The Impact of Expectation Management and Model Transparency on Radiologists' Trust and Utilization of AI Recommendations for Lung Nodule Assessment on Computed Tomography: Simulated Use Study.

BACKGROUND: Many promising artificial intelligence (AI) and computer-aided detection and diagnosis systems have been developed, but few have been successfully integrated into clinical practice. This is partially owing to a lack of user-centered design of AI-based computer-aided detection or diagnosis (AI-CAD) systems. OBJECTIVE: We aimed to assess the impact of different onboarding tutorials and levels of AI model explainability on radiologists' trust in AI and the use of AI recommendations in lung nodule assessment on computed tomography (CT) scans. METHODS: In total, 20 radiologists from 7 Dutch medical centers performed lung nodule assessment on CT scans under different conditions in a simulated use study as part of a 2×2 repeated-measures quasi-experimental design. Two types of AI onboarding tutorials (reflective vs informative) and 2 levels of AI output (black box vs explainable) were designed. The radiologists first received an onboarding tutorial that was either informative or reflective. Subsequently, each radiologist assessed 7 CT scans, first without AI recommendations. AI recommendations were shown to the radiologist, and they could adjust their initial assessment. Half of the participants received the recommendations via black box AI output and half received explainable AI output. Mental model and psychological trust were measured before onboarding, after onboarding, and after assessing the 7 CT scans. We recorded whether radiologists changed their assessment on found nodules, malignancy prediction, and follow-up advice for each CT assessment. In addition, we analyzed whether radiologists' trust in their assessments had changed based on the AI recommendations. RESULTS: Both variations of onboarding tutorials resulted in a significantly improved mental model of the AI-CAD system (informative P=.01 and reflective P=.01). After using AI-CAD, psychological trust significantly decreased for the group with explainable AI output (P=.02). On the basis of the AI recommendations, radiologists changed the number of reported nodules in 27 of 140 assessments, malignancy prediction in 32 of 140 assessments, and follow-up advice in 12 of 140 assessments. The changes were mostly an increased number of reported nodules, a higher estimated probability of malignancy, and earlier follow-up. The radiologists' confidence in their found nodules changed in 82 of 140 assessments, in their estimated probability of malignancy in 50 of 140 assessments, and in their follow-up advice in 28 of 140 assessments. These changes were predominantly increases in confidence. The number of changed assessments and radiologists' confidence did not significantly differ between the groups that received different onboarding tutorials and AI outputs. CONCLUSIONS: Onboarding tutorials help radiologists gain a better understanding of AI-CAD and facilitate the formation of a correct mental model. If AI explanations do not consistently substantiate the probability of malignancy across patient cases, radiologists' trust in the AI-CAD system can be impaired. Radiologists' confidence in their assessments was improved by using the AI recommendations.

Computer-Aided Detection for Pancreatic Cancer Diagnosis: Radiological Challenges and Future Directions.

Radiological imaging plays a crucial role in the detection and treatment of pancreatic ductal adenocarcinoma (PDAC). However, there are several challenges associated with the use of these techniques in daily clinical practice. Determination of the presence or absence of cancer using radiological imaging is difficult and requires specific expertise, especially after neoadjuvant therapy. Early detection and characterization of tumors would potentially increase the number of patients who are eligible for curative treatment. Over the last decades, artificial intelligence (AI)-based computer-aided detection (CAD) has rapidly evolved as a means for improving the radiological detection of cancer and the assessment of the extent of disease. Although the results of AI applications seem promising, widespread adoption in clinical practice has not taken place. This narrative review provides an overview of current radiological CAD systems in pancreatic cancer, highlights challenges that are pertinent to clinical practice, and discusses potential solutions for these challenges.

The Effects of Artificial Intelligence Assistance on the Radiologists' Assessment of Lung Nodules on CT Scans: A Systematic Review.

To reduce the number of missed or misdiagnosed lung nodules on CT scans by radiologists, many Artificial Intelligence (AI) algorithms have been developed. Some algorithms are currently being implemented in clinical practice, but the question is whether radiologists and patients really benefit from the use of these novel tools. This study aimed to review how AI assistance for lung nodule assessment on CT scans affects the performances of radiologists. We searched for studies that evaluated radiologists' performances in the detection or malignancy prediction of lung nodules with and without AI assistance. Concerning detection, radiologists achieved with AI assistance a higher sensitivity and AUC, while the specificity was slightly lower. Concerning malignancy prediction, radiologists achieved with AI assistance generally a higher sensitivity, specificity and AUC. The radiologists' workflows of using the AI assistance were often only described in limited detail in the papers. As recent studies showed improved performances of radiologists with AI assistance, AI assistance for lung nodule assessment holds great promise. To achieve added value of AI tools for lung nodule assessment in clinical practice, more research is required on the clinical validation of AI tools, impact on follow-up recommendations and ways of using AI tools.

Looking through the eyes of the multidisciplinary team: the design and clinical evaluation of a decision support system for lung cancer care.

BACKGROUND: Decision-making in lung cancer is complex due to a rapidly increasing amount of diagnostic data and treatment options. The need for timely and accurate diagnosis and delivery of care demands high-quality multidisciplinary team (MDT) collaboration and coordination. Clinical decision support systems (CDSSs) can potentially support MDTs in constructing a shared mental model of a patient case. This enables the team to assess the strength and completeness of collected diagnostic data, stratification for the right personalized therapy driven by clinical stage and other treatment-influencing factors, and adapt care management strategies when needed. Current CDSSs often have a suboptimal fit into the decision-making workflow, which hampers their impact in clinical practice. METHODS: A CDSS for multidisciplinary decision-making in lung cancer was designed to support the abovementioned goals through presentation of relevant clinical data in line with existing mental model structures of the MDT members. The CDSS was tested in a simulated multidisciplinary tumor board meeting for primary diagnosis and treatment selection, based on de-identified primary lung cancer cases (n=8). Decision course analysis, eye-tracking data and questionnaires were used to assess the impact of the CDSS on constructing shared mental models to improve the decision-making process and outcome. RESULTS: The CDSS supported the team in their self-correcting capacity for accurate diagnosis and TNM classification. It enabled cross-validation of diagnostic findings, surfaced discordance between diagnostic tests and facilitated cancer staging according the diagnostic evidence, as well as spotting contra-indications for personalized treatment selection. CONCLUSIONS: This study shows the potential of CDSS on clinical decision making, when these systems are properly designed in line with clinical thinking. The presented setup enables assessment of the impact of CDSS design on clinical decision making and optimization of CDSSs to maximize their effect on decision quality and confidence.

Immersive training: breaking the bubble and measuring the heat.

BACKGROUND: Minimal access surgery and, lately, single-incision laparoscopic procedures are challenging and demanding with regard to the skills of the surgeon performing the procedures. This article presents the results of an investigation of the performance and attention focus of 21 medical interns and surgical residents training in an immersive context. That is, training 'in situation', representing more realistically the demands imposed on the surgeons during minimal access surgery. METHODS: Twenty-one medical interns and surgical residents participated in simulation trainings in an integrated operating room for laparoscopic surgery. Various physiological measures of body heat expenditure were gathered as indicators of mental strain and attention focus. RESULTS: The results of the Mann-Whitney test indicated that participants with a poor performance in the two laparoscopic cholecystectomy cases had a significantly (U = 3, p = 0.038) higher heat flux at the start of the procedure (mean 107.08, standard deviation [SD] 24.34) than those who excelled in the two cases (mean 62.64, SD 23.41). Also, the average frontal head temperature of the participants who failed at the task was significantly lower (mean 33.27, SD 0.52) than those who performed well (mean 33.92, SD 0.27). CONCLUSIONS: Surgeons cannot operate in a bubble; thus, they should not be trained in one. Combining heat flux and frontal head temperature could be a good measure of deep involvement and attentional focus during performance of simulated surgical tasks.

Do absorption and realistic distraction influence performance of component task surgical procedure?

BACKGROUND: Surgeons perform complex tasks while exposed to multiple distracting sources that may increase stress in the operating room (e.g., music, conversation, and unadapted use of sophisticated technologies). This study aimed to examine whether such realistic social and technological distracting conditions may influence surgical performance. METHODS: Twelve medical interns performed a laparoscopic cholecystectomy task with the Xitact LC 3.0 virtual reality simulator under distracting conditions (exposure to music, conversation, and nonoptimal handling of the laparoscope) versus nondistracting conditions (control condition) as part of a 2 x 2 within-subject experimental design. RESULTS: Under distracting conditions, the medical interns showed a significant decline in task performance (overall task score, task errors, and operating time) and significantly increased levels of irritation toward both the assistant handling the laparoscope in a nonoptimal way and the sources of social distraction. Furthermore, individual differences in cognitive style (i.e., cognitive absorption and need for cognition) significantly influenced the levels of irritation experienced by the medical interns. CONCLUSION: The results suggest careful evaluation of the social and technological sources of distraction in the operation room to reduce irritation for the surgeon and provision of proper preclinical laparoscope navigation training to increase security for the patient.