RESUMEN
Background: Adaptive treatment strategies that can dynamically react to individual cancer progression can provide effective personalized care. Longitudinal multi-omics information, paired with an artificially intelligent clinical decision support system (AI-CDSS) can assist clinicians in determining optimal therapeutic options and treatment adaptations. However, AI-CDSS is not perfectly accurate, as such, clinicians' over/under reliance on AI may lead to unintended consequences, ultimately failing to develop optimal strategies. To investigate such collaborative decision-making process, we conducted a Human-AI interaction case study on response-adaptive radiotherapy (RT). Methods: We designed and conducted a two-phase study for two disease sites and two treatment modalities-adaptive RT for non-small cell lung cancer (NSCLC) and adaptive stereotactic body RT for hepatocellular carcinoma (HCC)-in which clinicians were asked to consider mid-treatment modification of the dose per fraction for a number of retrospective cancer patients without AI-support (Unassisted Phase) and with AI-assistance (AI-assisted Phase). The AI-CDSS graphically presented trade-offs in tumor control and the likelihood of toxicity to organs at risk, provided an optimal recommendation, and associated model uncertainties. In addition, we asked for clinicians' decision confidence level and trust level in individual AI recommendations and encouraged them to provide written remarks. We enrolled 13 evaluators (radiation oncology physicians and residents) from two medical institutions located in two different states, out of which, 4 evaluators volunteered in both NSCLC and HCC studies, resulting in a total of 17 completed evaluations (9 NSCLC, and 8 HCC). To limit the evaluation time to under an hour, we selected 8 treated patients for NSCLC and 9 for HCC, resulting in a total of 144 sets of evaluations (72 from NSCLC and 72 from HCC). Evaluation for each patient consisted of 8 required inputs and 2 optional remarks, resulting in up to a total of 1440 data points. Results: AI-assistance did not homogeneously influence all experts and clinical decisions. From NSCLC cohort, 41 (57%) decisions and from HCC cohort, 34 (47%) decisions were adjusted after AI assistance. Two evaluations (12%) from the NSCLC cohort had zero decision adjustments, while the remaining 15 (88%) evaluations resulted in at least two decision adjustments. Decision adjustment level positively correlated with dissimilarity in decision-making with AI [NSCLC: ρ = 0.53 ( p < 0.001); HCC: ρ = 0.60 ( p < 0.001)] indicating that evaluators adjusted their decision closer towards AI recommendation. Agreement with AI-recommendation positively correlated with AI Trust Level [NSCLC: ρ = 0.59 ( p < 0.001); HCC: ρ = 0.7 ( p < 0.001)] indicating that evaluators followed AI's recommendation if they agreed with that recommendation. The correlation between decision confidence changes and decision adjustment level showed an opposite trend [NSCLC: ρ = -0.24 ( p = 0.045), HCC: ρ = 0.28 ( p = 0.017)] reflecting the difference in behavior due to underlying differences in disease type and treatment modality. Decision confidence positively correlated with the closeness of decisions to the standard of care (NSCLC: 2 Gy/fx; HCC: 10 Gy/fx) indicating that evaluators were generally more confident in prescribing dose fractionations more similar to those used in standard clinical practice. Inter-evaluator agreement increased with AI-assistance indicating that AI-assistance can decrease inter-physician variability. The majority of decisions were adjusted to achieve higher tumor control in NSCLC and lower normal tissue complications in HCC. Analysis of evaluators' remarks indicated concerns for organs at risk and RT outcome estimates as important decision-making factors. Conclusions: Human-AI interaction depends on the complex interrelationship between expert's prior knowledge and preferences, patient's state, disease site, treatment modality, model transparency, and AI's learned behavior and biases. The collaborative decision-making process can be summarized as follows: (i) some clinicians may not believe in an AI system, completely disregarding its recommendation, (ii) some clinicians may believe in the AI system but will critically analyze its recommendations on a case-by-case basis; (iii) when a clinician finds that the AI recommendation indicates the possibility for better outcomes they will adjust their decisions accordingly; and (iv) When a clinician finds that the AI recommendation indicate a worse possible outcome they will disregard it and seek their own alternative approach.
RESUMEN
Leading successful change efforts first requires assessment of the "before change" environment and culture. At our institution, the radiation oncology (RO) residents follow a longitudinal didactic learning program consisting of weekly 1-h lectures, case conferences, and journal clubs. The resident didactic education series format has not changed since its inception over 10 years ago. We evaluated the perceptions of current residents and faculty about the effectiveness of the curriculum in its present form. Two parallel surveys were designed, one each for residents and attendings, to assess current attitudes regarding the effectiveness and need for change in the RO residency curriculum, specifically the traditional didactic lectures, the journal club sessions, and the case conferences. We also investigated perceived levels of engagement among residents and faculty, whether self-assessments would be useful to increase material retention, and how often the content of didactic lectures is updated. Surveys were distributed individually to each resident (N = 10) and attending (N = 24) either in-person or via Zoom. Following completion of the survey, respondents were informally interviewed about their perspectives on the curriculum's strengths and weaknesses. Compared to 46% of attendings, 80% of RO residents believed that the curriculum should be changed. Twenty percent of residents felt that the traditional didactic lectures were effective in preparing them to manage patients in the clinic, compared to 74% of attendings. Similarly, 10% of residents felt that the journal club sessions were effective vs. 42% of attendings. Finally, 40% of residents felt that the case conferences were effective vs. 67% of attendings. Overall, most respondents (56%) favored change in the curriculum. Our results suggest that the perceptions of the residents did not align with those of the attending physicians with respect to the effectiveness of the curriculum and the need for change. The discrepancies between resident and faculty views highlight the importance of a dedicated change management effort to mitigate this gap. Based on this project, we plan to propose recommended changes in structure to the residency program directors. Main changes would be to increase the interactive nature of the course material, incorporate more ways to increase faculty engagement, and consider self-assessment questions to promote retention. Once we get approval from the residency program leadership, we will follow Kotter's "Eight steps to transforming your organization" to ensure the highest potential for faculty to accept the expectations of a new curriculum.
RESUMEN
Purpose: Patients receiving respiratory gated magnetic resonance imaging-guided radiation therapy (MRIgRT) for abdominal targets must hold their breath for ≥25 seconds at a time. Virtual reality (VR) has shown promise for improving patient education and experience for diagnostic MRI scan acquisition. We aimed to develop and pilot-test the first VR app to educate, train, and reduce anxiety and discomfort in patients preparing to receive MRIgRT. Methods and Materials: A multidisciplinary team iteratively developed a new VR app with patient input. The app begins with minigames to help orient patients to using the VR device and to train patients on breath-holding. Next, app users are introduced to the MRI linear accelerator vault and practice breath-holding during MRIgRT. In this quality improvement project, clinic personnel and MRIgRT-eligible patients with pancreatic cancer tested the VR app for feasibility, acceptability, and potential efficacy for training patients on using breath-holding during MRIgRT. Results: The new VR app experience was tested by 19 patients and 67 clinic personnel. The experience was completed on average in 18.6 minutes (SD = 5.4) by patients and in 14.9 (SD = 3.5) minutes by clinic personnel. Patients reported the app was "extremely helpful" (58%) or "very helpful" (32%) for learning breath-holding used in MRIgRT and "extremely helpful" (28%) or "very helpful (50%) for reducing anxiety. Patients and clinic personnel also provided qualitative feedback on improving future versions of the VR app. Conclusion: The VR app was feasible and acceptable for training patients on breath-holding for MRIgRT. Patients eligible for MRIgRT for pancreatic cancer and clinic personnel reported on future improvements to the app to enhance its usability and efficacy.