Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction.

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Development and validation of AI-based automatic measurement of coronal Cobb angles in degenerative scoliosis using sagittal lumbar MRI.

OBJECTIVES: Severity of degenerative scoliosis (DS) is assessed by measuring the Cobb angle on anteroposterior radiographs. However, MRI images are often available to study the degenerative spine. This retrospective study aims to develop and evaluate the reliability of a novel automatic method that measures coronal Cobb angles on lumbar MRI in DS patients. MATERIALS AND METHODS: Vertebrae and intervertebral discs were automatically segmented using a 3D AI algorithm, trained on 447 lumbar MRI series. The segmentations were used to calculate all possible angles between the vertebral endplates, with the largest being the Cobb angle. The results were validated with 50 high-resolution sagittal lumbar MRI scans of DS patients, in which three experienced readers measured the Cobb angle. Reliability was determined using the intraclass correlation coefficient (ICC). RESULTS: The ICCs between the readers ranged from 0.90 (95% CI 0.83-0.94) to 0.93 (95% CI 0.88-0.96). The ICC between the maximum angle found by the algorithm and the average manually measured Cobb angles was 0.83 (95% CI 0.71-0.90). In 9 out of the 50 cases (18%), all readers agreed on both vertebral levels for Cobb angle measurement. When using the algorithm to extract the angles at the vertebral levels chosen by the readers, the ICCs ranged from 0.92 (95% CI 0.87-0.96) to 0.97 (95% CI 0.94-0.98). CONCLUSION: The Cobb angle can be accurately measured on MRI using the newly developed algorithm in patients with DS. The readers failed to consistently choose the same vertebral level for Cobb angle measurement, whereas the automatic approach ensures the maximum angle is consistently measured. CLINICAL RELEVANCE STATEMENT: Our AI-based algorithm offers reliable Cobb angle measurement on routine MRI for degenerative scoliosis patients, potentially reducing the reliance on conventional radiographs, ensuring consistent assessments, and therefore improving patient care. KEY POINTS: • While often available, MRI images are rarely utilized to determine the severity of degenerative scoliosis. • The presented MRI Cobb angle algorithm is more reliable than humans in patients with degenerative scoliosis. • Radiographic imaging for Cobb angle measurements is mitigated when lumbar MRI images are available.

Model-based Prediction of Critical Illness in Hospitalized Patients with COVID-19.

Background The prognosis of hospitalized patients with severe coronavirus disease 2019 (COVID-19) is difficult to predict, and the capacity of intensive care units was a limiting factor during the peak of the pandemic and is generally dependent on a country's clinical resources. Purpose To determine the value of chest radiographic findings together with patient history and laboratory markers at admission to predict critical illness in hospitalized patients with COVID-19. Materials and Methods In this retrospective study, which included patients from March 7, 2020, to April 24, 2020, a consecutive cohort of hospitalized patients with real-time reverse transcription polymerase chain reaction-confirmed COVID-19 from two large Dutch community hospitals was identified. After univariable analysis, a risk model to predict critical illness (ie, death and/or intensive care unit admission with invasive ventilation) was developed, using multivariable logistic regression including clinical, chest radiographic, and laboratory findings. Distribution and severity of lung involvement were visually assessed by using an eight-point scale (chest radiography score). Internal validation was performed by using bootstrapping. Performance is presented as an area under the receiver operating characteristic curve. Decision curve analysis was performed, and a risk calculator was derived. Results The cohort included 356 hospitalized patients (mean age, 69 years ± 12 [standard deviation]; 237 men) of whom 168 (47%) developed critical illness. The final risk model's variables included sex, chronic obstructive lung disease, symptom duration, neutrophil count, C-reactive protein level, lactate dehydrogenase level, distribution of lung disease, and chest radiography score at hospital presentation. The area under the receiver operating characteristic curve of the model was 0.77 (95% CI: 0.72, 0.81; P < .001). A risk calculator was derived for individual risk assessment: Dutch COVID-19 risk model. At an example threshold of 0.70, 71 of 356 patients would be predicted to develop critical illness, of which 59 (83%) would be true-positive results. Conclusion A risk model based on chest radiographic and laboratory findings obtained at admission was predictive of critical illness in hospitalized patients with coronavirus disease 2019. This risk calculator might be useful for triage of patients to the limited number of intensive care unit beds or facilities. © RSNA, 2020 Online supplemental material is available for this article.

An international survey on AI in radiology in 1,041 radiologists and radiology residents part 1: fear of replacement, knowledge, and attitude.

OBJECTIVES: Radiologists' perception is likely to influence the adoption of artificial intelligence (AI) into clinical practice. We investigated knowledge and attitude towards AI by radiologists and residents in Europe and beyond. METHODS: Between April and July 2019, a survey on fear of replacement, knowledge, and attitude towards AI was accessible to radiologists and residents. The survey was distributed through several radiological societies, author networks, and social media. Independent predictors of fear of replacement and a positive attitude towards AI were assessed using multivariable logistic regression. RESULTS: The survey was completed by 1,041 respondents from 54 mostly European countries. Most respondents were male (n = 670, 65%), median age was 38 (24-74) years, n = 142 (35%) residents, and n = 471 (45%) worked in an academic center. Basic AI-specific knowledge was associated with fear (adjusted OR 1.56, 95% CI 1.10-2.21, p = 0.01), while intermediate AI-specific knowledge (adjusted OR 0.40, 95% CI 0.20-0.80, p = 0.01) or advanced AI-specific knowledge (adjusted OR 0.43, 95% CI 0.21-0.90, p = 0.03) was inversely associated with fear. A positive attitude towards AI was observed in 48% (n = 501) and was associated with only having heard of AI, intermediate (adjusted OR 11.65, 95% CI 4.25-31.92, p < 0.001), or advanced AI-specific knowledge (adjusted OR 17.65, 95% CI 6.16-50.54, p < 0.001). CONCLUSIONS: Limited AI-specific knowledge levels among radiology residents and radiologists are associated with fear, while intermediate to advanced AI-specific knowledge levels are associated with a positive attitude towards AI. Additional training may therefore improve clinical adoption. KEY POINTS: • Forty-eight percent of radiologists and residents have an open and proactive attitude towards artificial intelligence (AI), while 38% fear of replacement by AI. • Intermediate and advanced AI-specific knowledge levels may enhance adoption of AI in clinical practice, while rudimentary knowledge levels appear to be inhibitive. • AI should be incorporated in radiology training curricula to help facilitate its clinical adoption.

An international survey on AI in radiology in 1041 radiologists and radiology residents part 2: expectations, hurdles to implementation, and education.

OBJECTIVES: Currently, hurdles to implementation of artificial intelligence (AI) in radiology are a much-debated topic but have not been investigated in the community at large. Also, controversy exists if and to what extent AI should be incorporated into radiology residency programs. METHODS: Between April and July 2019, an international survey took place on AI regarding its impact on the profession and training. The survey was accessible for radiologists and residents and distributed through several radiological societies. Relationships of independent variables with opinions, hurdles, and education were assessed using multivariable logistic regression. RESULTS: The survey was completed by 1041 respondents from 54 countries. A majority (n = 855, 82%) expects that AI will cause a change to the radiology field within 10 years. Most frequently, expected roles of AI in clinical practice were second reader (n = 829, 78%) and work-flow optimization (n = 802, 77%). Ethical and legal issues (n = 630, 62%) and lack of knowledge (n = 584, 57%) were mentioned most often as hurdles to implementation. Expert respondents added lack of labelled images and generalizability issues. A majority (n = 819, 79%) indicated that AI should be incorporated in residency programs, while less support for imaging informatics and AI as a subspecialty was found (n = 241, 23%). CONCLUSIONS: Broad community demand exists for incorporation of AI into residency programs. Based on the results of the current study, integration of AI education seems advisable for radiology residents, including issues related to data management, ethics, and legislation. KEY POINTS: • There is broad demand from the radiological community to incorporate AI into residency programs, but there is less support to recognize imaging informatics as a radiological subspecialty. • Ethical and legal issues and lack of knowledge are recognized as major bottlenecks for AI implementation by the radiological community, while the shortage in labeled data and IT-infrastructure issues are less often recognized as hurdles. • Integrating AI education in radiology curricula including technical aspects of data management, risk of bias, and ethical and legal issues may aid successful integration of AI into diagnostic radiology.

Comparison of Imaging Changes and Pain Responses in Patients with Intra- or Extraosseous Bone Metastases Treated Palliatively with Magnetic Resonance-Guided High-Intensity-Focused Ultrasound.

PURPOSE: This study compared changes in imaging and in pain relief between patients with intraosseous, as opposed to extraosseous bone metastases. Both groups were treated palliatively with magnetic resonance-guided high-intensity-focused ultrasound (MRgHIFU). MATERIALS AND METHODS: A total of 21 patients were treated prospectively with MRgHIFU at 3 centers. Intraprocedural thermal changes measured using proton resonance frequency shift (PRFS) thermometry and gadolinium-enhanced T1-weighted (Gd-T1W) image appearances after treatment were compared for intra- and extraosseous metastases. Pain scores and use of analgesic therapy documented before and up to 90 days after treatment were used to classify responses and were compared between the intra- and extraosseous groups. Gd-T1W changes were compared between responders and nonresponders in each group. RESULTS: Thermal dose volumes were significantly larger in the extraosseous group (P = 0.039). Tumor diameter did not change after treatment in either group. At day 30, Gd-T1W images showed focal nonenhancement in 7 of 9 patients with intraosseous tumors; in patients with extraosseous tumors, changes were heterogeneous. Cohort reductions in worst-pain scores were seen for both groups, but differences from baseline at days 14, 30, 60, and 90 were only significant for the intraosseous group (P = 0.027, P = 0.013, P = 0.012, and P = 0.027, respectively). By day 30, 67% of patients (6 of 9) with intraosseous tumors were classified as responders, and the rate was 33% (4 of 12) for patients with extraosseous tumors. In neither group was pain response indicated by nonenhancement on Gd-T1W. CONCLUSIONS: Intraosseous tumors showed focal nonenhancement by day 30, and patients had better pain response to MRgHIFU than those with extraosseous tumors. In this small cohort, post-treatment imaging was not informative of treatment efficacy.

International consensus on use of focused ultrasound for painful bone metastases: Current status and future directions.

Focused ultrasound surgery (FUS), in particular magnetic resonance guided FUS (MRgFUS), is an emerging non-invasive thermal treatment modality in oncology that has recently proven to be effective for the palliation of metastatic bone pain. A consensus panel of internationally recognised experts in focused ultrasound critically reviewed all available data and developed consensus statements to increase awareness, accelerate the development, acceptance and adoption of FUS as a treatment for painful bone metastases and provide guidance towards broader application in oncology. In this review, evidence-based consensus statements are provided for (1) current treatment goals, (2) current indications, (3) technical considerations, (4) future directions including research priorities, and (5) economic and logistical considerations.

CT and MRI radiomics of bone and soft-tissue sarcomas: an updated systematic review of reproducibility and validation strategies.

OBJECTIVE: To systematically review radiomic feature reproducibility and model validation strategies in recent studies dealing with CT and MRI radiomics of bone and soft-tissue sarcomas, thus updating a previous version of this review which included studies published up to 2020. METHODS: A literature search was conducted on EMBASE and PubMed databases for papers published between January 2021 and March 2023. Data regarding radiomic feature reproducibility and model validation strategies were extracted and analyzed. RESULTS: Out of 201 identified papers, 55 were included. They dealt with radiomics of bone (n = 23) or soft-tissue (n = 32) tumors. Thirty-two (out of 54 employing manual or semiautomatic segmentation, 59%) studies included a feature reproducibility analysis. Reproducibility was assessed based on intra/interobserver segmentation variability in 30 (55%) and geometrical transformations of the region of interest in 2 (4%) studies. At least one machine learning validation technique was used for model development in 34 (62%) papers, and K-fold cross-validation was employed most frequently. A clinical validation of the model was reported in 38 (69%) papers. It was performed using a separate dataset from the primary institution (internal test) in 22 (40%), an independent dataset from another institution (external test) in 14 (25%) and both in 2 (4%) studies. CONCLUSIONS: Compared to papers published up to 2020, a clear improvement was noted with almost double publications reporting methodological aspects related to reproducibility and validation. Larger multicenter investigations including external clinical validation and the publication of databases in open-access repositories could further improve methodology and bring radiomics from a research area to the clinical stage. CRITICAL RELEVANCE STATEMENT: An improvement in feature reproducibility and model validation strategies has been shown in this updated systematic review on radiomics of bone and soft-tissue sarcomas, highlighting efforts to enhance methodology and bring radiomics from a research area to the clinical stage. KEY POINTS: • 2021-2023 radiomic studies on CT and MRI of musculoskeletal sarcomas were reviewed. • Feature reproducibility was assessed in more than half (59%) of the studies. • Model clinical validation was performed in 69% of the studies. • Internal (44%) and/or external (29%) test datasets were employed for clinical validation.

Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects.

Although artificial intelligence (AI) methods hold promise for medical imaging-based prediction tasks, their integration into medical practice may present a double-edged sword due to bias (i.e., systematic errors). AI algorithms have the potential to mitigate cognitive biases in human interpretation, but extensive research has highlighted the tendency of AI systems to internalize biases within their model. This fact, whether intentional or not, may ultimately lead to unintentional consequences in the clinical setting, potentially compromising patient outcomes. This concern is particularly important in medical imaging, where AI has been more progressively and widely embraced than any other medical field. A comprehensive understanding of bias at each stage of the AI pipeline is therefore essential to contribute to developing AI solutions that are not only less biased but also widely applicable. This international collaborative review effort aims to increase awareness within the medical imaging community about the importance of proactively identifying and addressing AI bias to prevent its negative consequences from being realized later. The authors began with the fundamentals of bias by explaining its different definitions and delineating various potential sources. Strategies for detecting and identifying bias were then outlined, followed by a review of techniques for its avoidance and mitigation. Moreover, ethical dimensions, challenges encountered, and prospects were discussed.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

A deep learning-based application for COVID-19 diagnosis on CT: The Imaging COVID-19 AI initiative.

BACKGROUND: Recently, artificial intelligence (AI)-based applications for chest imaging have emerged as potential tools to assist clinicians in the diagnosis and management of patients with coronavirus disease 2019 (COVID-19). OBJECTIVES: To develop a deep learning-based clinical decision support system for automatic diagnosis of COVID-19 on chest CT scans. Secondarily, to develop a complementary segmentation tool to assess the extent of lung involvement and measure disease severity. METHODS: The Imaging COVID-19 AI initiative was formed to conduct a retrospective multicentre cohort study including 20 institutions from seven different European countries. Patients with suspected or known COVID-19 who underwent a chest CT were included. The dataset was split on the institution-level to allow external evaluation. Data annotation was performed by 34 radiologists/radiology residents and included quality control measures. A multi-class classification model was created using a custom 3D convolutional neural network. For the segmentation task, a UNET-like architecture with a backbone Residual Network (ResNet-34) was selected. RESULTS: A total of 2,802 CT scans were included (2,667 unique patients, mean [standard deviation] age = 64.6 [16.2] years, male/female ratio 1.3:1). The distribution of classes (COVID-19/Other type of pulmonary infection/No imaging signs of infection) was 1,490 (53.2%), 402 (14.3%), and 910 (32.5%), respectively. On the external test dataset, the diagnostic multiclassification model yielded high micro-average and macro-average AUC values (0.93 and 0.91, respectively). The model provided the likelihood of COVID-19 vs other cases with a sensitivity of 87% and a specificity of 94%. The segmentation performance was moderate with Dice similarity coefficient (DSC) of 0.59. An imaging analysis pipeline was developed that returned a quantitative report to the user. CONCLUSION: We developed a deep learning-based clinical decision support system that could become an efficient concurrent reading tool to assist clinicians, utilising a newly created European dataset including more than 2,800 CT scans.

Responsible AI practice and AI education are central to AI implementation: a rapid review for all medical imaging professionals in Europe.

Artificial intelligence (AI) has transitioned from the lab to the bedside, and it is increasingly being used in healthcare. Radiology and Radiography are on the frontline of AI implementation, because of the use of big data for medical imaging and diagnosis for different patient groups. Safe and effective AI implementation requires that responsible and ethical practices are upheld by all key stakeholders, that there is harmonious collaboration between different professional groups, and customised educational provisions for all involved. This paper outlines key principles of ethical and responsible AI, highlights recent educational initiatives for clinical practitioners and discusses the synergies between all medical imaging professionals as they prepare for the digital future in Europe. Responsible and ethical AI is vital to enhance a culture of safety and trust for healthcare professionals and patients alike. Educational and training provisions for medical imaging professionals on AI is central to the understanding of basic AI principles and applications and there are many offerings currently in Europe. Education can facilitate the transparency of AI tools, but more formalised, university-led training is needed to ensure the academic scrutiny, appropriate pedagogy, multidisciplinarity and customisation to the learners' unique needs are being adhered to. As radiographers and radiologists work together and with other professionals to understand and harness the benefits of AI in medical imaging, it becomes clear that they are faced with the same challenges and that they have the same needs. The digital future belongs to multidisciplinary teams that work seamlessly together, learn together, manage risk collectively and collaborate for the benefit of the patients they serve.

Endovascular treatment of a patient with an aneurysm of the proper hepatic artery and a duodenal fistula.

Aneurysms of the proper hepatic artery comprise a rare but potentially dangerous entity for which treatment is performed both surgically and endovascularly. Covered stents are generally used for endovascular treatment of such aneurysms. When the aneurysm is contaminated due to an enteric fistula, however, use of a covered stent is considered inappropriate. This case report describes the endovascular repair of a proper hepatic artery aneurysm using overlapping bare metal stents after the patient was surgically treated for duodenal hemorrhage.