Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

CT and MRI radiomics of bone and soft-tissue sarcomas: an updated systematic review of reproducibility and validation strategies.

OBJECTIVE: To systematically review radiomic feature reproducibility and model validation strategies in recent studies dealing with CT and MRI radiomics of bone and soft-tissue sarcomas, thus updating a previous version of this review which included studies published up to 2020. METHODS: A literature search was conducted on EMBASE and PubMed databases for papers published between January 2021 and March 2023. Data regarding radiomic feature reproducibility and model validation strategies were extracted and analyzed. RESULTS: Out of 201 identified papers, 55 were included. They dealt with radiomics of bone (n = 23) or soft-tissue (n = 32) tumors. Thirty-two (out of 54 employing manual or semiautomatic segmentation, 59%) studies included a feature reproducibility analysis. Reproducibility was assessed based on intra/interobserver segmentation variability in 30 (55%) and geometrical transformations of the region of interest in 2 (4%) studies. At least one machine learning validation technique was used for model development in 34 (62%) papers, and K-fold cross-validation was employed most frequently. A clinical validation of the model was reported in 38 (69%) papers. It was performed using a separate dataset from the primary institution (internal test) in 22 (40%), an independent dataset from another institution (external test) in 14 (25%) and both in 2 (4%) studies. CONCLUSIONS: Compared to papers published up to 2020, a clear improvement was noted with almost double publications reporting methodological aspects related to reproducibility and validation. Larger multicenter investigations including external clinical validation and the publication of databases in open-access repositories could further improve methodology and bring radiomics from a research area to the clinical stage. CRITICAL RELEVANCE STATEMENT: An improvement in feature reproducibility and model validation strategies has been shown in this updated systematic review on radiomics of bone and soft-tissue sarcomas, highlighting efforts to enhance methodology and bring radiomics from a research area to the clinical stage. KEY POINTS: • 2021-2023 radiomic studies on CT and MRI of musculoskeletal sarcomas were reviewed. • Feature reproducibility was assessed in more than half (59%) of the studies. • Model clinical validation was performed in 69% of the studies. • Internal (44%) and/or external (29%) test datasets were employed for clinical validation.

Development and validation of AI-based automatic measurement of coronal Cobb angles in degenerative scoliosis using sagittal lumbar MRI.

OBJECTIVES: Severity of degenerative scoliosis (DS) is assessed by measuring the Cobb angle on anteroposterior radiographs. However, MRI images are often available to study the degenerative spine. This retrospective study aims to develop and evaluate the reliability of a novel automatic method that measures coronal Cobb angles on lumbar MRI in DS patients. MATERIALS AND METHODS: Vertebrae and intervertebral discs were automatically segmented using a 3D AI algorithm, trained on 447 lumbar MRI series. The segmentations were used to calculate all possible angles between the vertebral endplates, with the largest being the Cobb angle. The results were validated with 50 high-resolution sagittal lumbar MRI scans of DS patients, in which three experienced readers measured the Cobb angle. Reliability was determined using the intraclass correlation coefficient (ICC). RESULTS: The ICCs between the readers ranged from 0.90 (95% CI 0.83-0.94) to 0.93 (95% CI 0.88-0.96). The ICC between the maximum angle found by the algorithm and the average manually measured Cobb angles was 0.83 (95% CI 0.71-0.90). In 9 out of the 50 cases (18%), all readers agreed on both vertebral levels for Cobb angle measurement. When using the algorithm to extract the angles at the vertebral levels chosen by the readers, the ICCs ranged from 0.92 (95% CI 0.87-0.96) to 0.97 (95% CI 0.94-0.98). CONCLUSION: The Cobb angle can be accurately measured on MRI using the newly developed algorithm in patients with DS. The readers failed to consistently choose the same vertebral level for Cobb angle measurement, whereas the automatic approach ensures the maximum angle is consistently measured. CLINICAL RELEVANCE STATEMENT: Our AI-based algorithm offers reliable Cobb angle measurement on routine MRI for degenerative scoliosis patients, potentially reducing the reliance on conventional radiographs, ensuring consistent assessments, and therefore improving patient care. KEY POINTS: • While often available, MRI images are rarely utilized to determine the severity of degenerative scoliosis. • The presented MRI Cobb angle algorithm is more reliable than humans in patients with degenerative scoliosis. • Radiographic imaging for Cobb angle measurements is mitigated when lumbar MRI images are available.

Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction.

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Responsible AI practice and AI education are central to AI implementation: a rapid review for all medical imaging professionals in Europe.

Artificial intelligence (AI) has transitioned from the lab to the bedside, and it is increasingly being used in healthcare. Radiology and Radiography are on the frontline of AI implementation, because of the use of big data for medical imaging and diagnosis for different patient groups. Safe and effective AI implementation requires that responsible and ethical practices are upheld by all key stakeholders, that there is harmonious collaboration between different professional groups, and customised educational provisions for all involved. This paper outlines key principles of ethical and responsible AI, highlights recent educational initiatives for clinical practitioners and discusses the synergies between all medical imaging professionals as they prepare for the digital future in Europe. Responsible and ethical AI is vital to enhance a culture of safety and trust for healthcare professionals and patients alike. Educational and training provisions for medical imaging professionals on AI is central to the understanding of basic AI principles and applications and there are many offerings currently in Europe. Education can facilitate the transparency of AI tools, but more formalised, university-led training is needed to ensure the academic scrutiny, appropriate pedagogy, multidisciplinarity and customisation to the learners' unique needs are being adhered to. As radiographers and radiologists work together and with other professionals to understand and harness the benefits of AI in medical imaging, it becomes clear that they are faced with the same challenges and that they have the same needs. The digital future belongs to multidisciplinary teams that work seamlessly together, learn together, manage risk collectively and collaborate for the benefit of the patients they serve.

A deep learning-based application for COVID-19 diagnosis on CT: The Imaging COVID-19 AI initiative.

BACKGROUND: Recently, artificial intelligence (AI)-based applications for chest imaging have emerged as potential tools to assist clinicians in the diagnosis and management of patients with coronavirus disease 2019 (COVID-19). OBJECTIVES: To develop a deep learning-based clinical decision support system for automatic diagnosis of COVID-19 on chest CT scans. Secondarily, to develop a complementary segmentation tool to assess the extent of lung involvement and measure disease severity. METHODS: The Imaging COVID-19 AI initiative was formed to conduct a retrospective multicentre cohort study including 20 institutions from seven different European countries. Patients with suspected or known COVID-19 who underwent a chest CT were included. The dataset was split on the institution-level to allow external evaluation. Data annotation was performed by 34 radiologists/radiology residents and included quality control measures. A multi-class classification model was created using a custom 3D convolutional neural network. For the segmentation task, a UNET-like architecture with a backbone Residual Network (ResNet-34) was selected. RESULTS: A total of 2,802 CT scans were included (2,667 unique patients, mean [standard deviation] age = 64.6 [16.2] years, male/female ratio 1.3:1). The distribution of classes (COVID-19/Other type of pulmonary infection/No imaging signs of infection) was 1,490 (53.2%), 402 (14.3%), and 910 (32.5%), respectively. On the external test dataset, the diagnostic multiclassification model yielded high micro-average and macro-average AUC values (0.93 and 0.91, respectively). The model provided the likelihood of COVID-19 vs other cases with a sensitivity of 87% and a specificity of 94%. The segmentation performance was moderate with Dice similarity coefficient (DSC) of 0.59. An imaging analysis pipeline was developed that returned a quantitative report to the user. CONCLUSION: We developed a deep learning-based clinical decision support system that could become an efficient concurrent reading tool to assist clinicians, utilising a newly created European dataset including more than 2,800 CT scans.

How COVID-19 kick-started online learning in medical education-The DigiMed study.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic led to far-reaching restrictions of social and professional life, affecting societies all over the world. To contain the virus, medical schools had to restructure their curriculum by switching to online learning. However, only few medical schools had implemented such novel learning concepts. We aimed to evaluate students' attitudes to online learning to provide a broad scientific basis to guide future development of medical education. METHODS: Overall, 3286 medical students from 12 different countries participated in this cross-sectional, web-based study investigating various aspects of online learning in medical education. On a 7-point Likert scale, participants rated the online learning situation during the pandemic at their medical schools, technical and social aspects, and the current and future role of online learning in medical education. RESULTS: The majority of medical schools managed the rapid switch to online learning (78%) and most students were satisfied with the quantity (67%) and quality (62%) of the courses. Online learning provided greater flexibility (84%) and led to unchanged or even higher attendance of courses (70%). Possible downsides included motivational problems (42%), insufficient possibilities for interaction with fellow students (67%) and thus the risk of social isolation (64%). The vast majority felt comfortable using the software solutions (80%). Most were convinced that medical education lags behind current capabilities regarding online learning (78%) and estimated the proportion of online learning before the pandemic at only 14%. In order to improve the current curriculum, they wish for a more balanced ratio with at least 40% of online teaching compared to on-site teaching. CONCLUSION: This study demonstrates the positive attitude of medical students towards online learning. Furthermore, it reveals a considerable discrepancy between what students demand and what the curriculum offers. Thus, the COVID-19 pandemic might be the long-awaited catalyst for a new "online era" in medical education.

'Improving Health through Reducing Stress': Parents' Priorities in the Participatory Development of a Multilevel Family Health Programme in a Low-Income Neighbourhood in The Netherlands.

In order to reduce health inequities, a socio-ecological approach and community engagement are needed to develop sustained interventions with a positive effect on the health of disadvantaged groups. This qualitative study was part of the development phase of a community health promotion programme. The study aimed to provide insight into the perceptions of parents in a disadvantaged neighbourhood about health, and their priorities for the community health programme. It also described the process of integrating these perceptions in the development of a multilevel plan for this programme. Participatory methods were applied to enable the engagement of all groups involved. Ten parents from a low-income neighbourhood in the Netherlands participated in five panel sessions. Parents' priorities for improving family health were reducing chronic stress and not so much healthy eating and physical activity. They prioritised solutions to reduce their financial stress, to provide a safe place for their children to meet and play and to establish good quality communication with authorities. The programme development process resulted in objectives in which both parents and professionals were willing to invest, such as a safe playground for children. This study shows that target population engagement in health programme development is possible and valuable.

An international survey on AI in radiology in 1041 radiologists and radiology residents part 2: expectations, hurdles to implementation, and education.

OBJECTIVES: Currently, hurdles to implementation of artificial intelligence (AI) in radiology are a much-debated topic but have not been investigated in the community at large. Also, controversy exists if and to what extent AI should be incorporated into radiology residency programs. METHODS: Between April and July 2019, an international survey took place on AI regarding its impact on the profession and training. The survey was accessible for radiologists and residents and distributed through several radiological societies. Relationships of independent variables with opinions, hurdles, and education were assessed using multivariable logistic regression. RESULTS: The survey was completed by 1041 respondents from 54 countries. A majority (n = 855, 82%) expects that AI will cause a change to the radiology field within 10 years. Most frequently, expected roles of AI in clinical practice were second reader (n = 829, 78%) and work-flow optimization (n = 802, 77%). Ethical and legal issues (n = 630, 62%) and lack of knowledge (n = 584, 57%) were mentioned most often as hurdles to implementation. Expert respondents added lack of labelled images and generalizability issues. A majority (n = 819, 79%) indicated that AI should be incorporated in residency programs, while less support for imaging informatics and AI as a subspecialty was found (n = 241, 23%). CONCLUSIONS: Broad community demand exists for incorporation of AI into residency programs. Based on the results of the current study, integration of AI education seems advisable for radiology residents, including issues related to data management, ethics, and legislation. KEY POINTS: • There is broad demand from the radiological community to incorporate AI into residency programs, but there is less support to recognize imaging informatics as a radiological subspecialty. • Ethical and legal issues and lack of knowledge are recognized as major bottlenecks for AI implementation by the radiological community, while the shortage in labeled data and IT-infrastructure issues are less often recognized as hurdles. • Integrating AI education in radiology curricula including technical aspects of data management, risk of bias, and ethical and legal issues may aid successful integration of AI into diagnostic radiology.

An international survey on AI in radiology in 1,041 radiologists and radiology residents part 1: fear of replacement, knowledge, and attitude.

OBJECTIVES: Radiologists' perception is likely to influence the adoption of artificial intelligence (AI) into clinical practice. We investigated knowledge and attitude towards AI by radiologists and residents in Europe and beyond. METHODS: Between April and July 2019, a survey on fear of replacement, knowledge, and attitude towards AI was accessible to radiologists and residents. The survey was distributed through several radiological societies, author networks, and social media. Independent predictors of fear of replacement and a positive attitude towards AI were assessed using multivariable logistic regression. RESULTS: The survey was completed by 1,041 respondents from 54 mostly European countries. Most respondents were male (n = 670, 65%), median age was 38 (24-74) years, n = 142 (35%) residents, and n = 471 (45%) worked in an academic center. Basic AI-specific knowledge was associated with fear (adjusted OR 1.56, 95% CI 1.10-2.21, p = 0.01), while intermediate AI-specific knowledge (adjusted OR 0.40, 95% CI 0.20-0.80, p = 0.01) or advanced AI-specific knowledge (adjusted OR 0.43, 95% CI 0.21-0.90, p = 0.03) was inversely associated with fear. A positive attitude towards AI was observed in 48% (n = 501) and was associated with only having heard of AI, intermediate (adjusted OR 11.65, 95% CI 4.25-31.92, p < 0.001), or advanced AI-specific knowledge (adjusted OR 17.65, 95% CI 6.16-50.54, p < 0.001). CONCLUSIONS: Limited AI-specific knowledge levels among radiology residents and radiologists are associated with fear, while intermediate to advanced AI-specific knowledge levels are associated with a positive attitude towards AI. Additional training may therefore improve clinical adoption. KEY POINTS: • Forty-eight percent of radiologists and residents have an open and proactive attitude towards artificial intelligence (AI), while 38% fear of replacement by AI. • Intermediate and advanced AI-specific knowledge levels may enhance adoption of AI in clinical practice, while rudimentary knowledge levels appear to be inhibitive. • AI should be incorporated in radiology training curricula to help facilitate its clinical adoption.

Model-based Prediction of Critical Illness in Hospitalized Patients with COVID-19.

Background The prognosis of hospitalized patients with severe coronavirus disease 2019 (COVID-19) is difficult to predict, and the capacity of intensive care units was a limiting factor during the peak of the pandemic and is generally dependent on a country's clinical resources. Purpose To determine the value of chest radiographic findings together with patient history and laboratory markers at admission to predict critical illness in hospitalized patients with COVID-19. Materials and Methods In this retrospective study, which included patients from March 7, 2020, to April 24, 2020, a consecutive cohort of hospitalized patients with real-time reverse transcription polymerase chain reaction-confirmed COVID-19 from two large Dutch community hospitals was identified. After univariable analysis, a risk model to predict critical illness (ie, death and/or intensive care unit admission with invasive ventilation) was developed, using multivariable logistic regression including clinical, chest radiographic, and laboratory findings. Distribution and severity of lung involvement were visually assessed by using an eight-point scale (chest radiography score). Internal validation was performed by using bootstrapping. Performance is presented as an area under the receiver operating characteristic curve. Decision curve analysis was performed, and a risk calculator was derived. Results The cohort included 356 hospitalized patients (mean age, 69 years ± 12 [standard deviation]; 237 men) of whom 168 (47%) developed critical illness. The final risk model's variables included sex, chronic obstructive lung disease, symptom duration, neutrophil count, C-reactive protein level, lactate dehydrogenase level, distribution of lung disease, and chest radiography score at hospital presentation. The area under the receiver operating characteristic curve of the model was 0.77 (95% CI: 0.72, 0.81; P < .001). A risk calculator was derived for individual risk assessment: Dutch COVID-19 risk model. At an example threshold of 0.70, 71 of 356 patients would be predicted to develop critical illness, of which 59 (83%) would be true-positive results. Conclusion A risk model based on chest radiographic and laboratory findings obtained at admission was predictive of critical illness in hospitalized patients with coronavirus disease 2019. This risk calculator might be useful for triage of patients to the limited number of intensive care unit beds or facilities. © RSNA, 2020 Online supplemental material is available for this article.

Comparison of Imaging Changes and Pain Responses in Patients with Intra- or Extraosseous Bone Metastases Treated Palliatively with Magnetic Resonance-Guided High-Intensity-Focused Ultrasound.

PURPOSE: This study compared changes in imaging and in pain relief between patients with intraosseous, as opposed to extraosseous bone metastases. Both groups were treated palliatively with magnetic resonance-guided high-intensity-focused ultrasound (MRgHIFU). MATERIALS AND METHODS: A total of 21 patients were treated prospectively with MRgHIFU at 3 centers. Intraprocedural thermal changes measured using proton resonance frequency shift (PRFS) thermometry and gadolinium-enhanced T1-weighted (Gd-T1W) image appearances after treatment were compared for intra- and extraosseous metastases. Pain scores and use of analgesic therapy documented before and up to 90 days after treatment were used to classify responses and were compared between the intra- and extraosseous groups. Gd-T1W changes were compared between responders and nonresponders in each group. RESULTS: Thermal dose volumes were significantly larger in the extraosseous group (P = 0.039). Tumor diameter did not change after treatment in either group. At day 30, Gd-T1W images showed focal nonenhancement in 7 of 9 patients with intraosseous tumors; in patients with extraosseous tumors, changes were heterogeneous. Cohort reductions in worst-pain scores were seen for both groups, but differences from baseline at days 14, 30, 60, and 90 were only significant for the intraosseous group (P = 0.027, P = 0.013, P = 0.012, and P = 0.027, respectively). By day 30, 67% of patients (6 of 9) with intraosseous tumors were classified as responders, and the rate was 33% (4 of 12) for patients with extraosseous tumors. In neither group was pain response indicated by nonenhancement on Gd-T1W. CONCLUSIONS: Intraosseous tumors showed focal nonenhancement by day 30, and patients had better pain response to MRgHIFU than those with extraosseous tumors. In this small cohort, post-treatment imaging was not informative of treatment efficacy.