Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC). RESULTS: 1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth. CONCLUSIONS: To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS: AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.

Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI.

Despite the surge in artificial intelligence (AI) development for health care applications, particularly for medical imaging applications, there has been limited adoption of such AI tools into clinical practice. During a 1-day workshop in November 2022, co-organized by the ACR and the RSNA, participants outlined experiences and problems with implementing AI in clinical practice, defined the needs of various stakeholders in the AI ecosystem, and elicited potential solutions and strategies related to the safety, effectiveness, reliability, and transparency of AI algorithms. Participants included radiologists from academic and community radiology practices, informatics leaders responsible for AI implementation, regulatory agency employees, and specialty society representatives. The major themes that emerged fell into two categories: (1) AI product development and (2) implementation of AI-based applications in clinical practice. In particular, participants highlighted key aspects of AI product development to include clear clinical task definitions; well-curated data from diverse geographic, economic, and health care settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings. For implementation, participants emphasized the need for strong institutional governance; systematic evaluation, selection, and validation methods conducted by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement. Participants predicted that clinical implementation of AI in radiology will continue to be limited until the safety, effectiveness, reliability, and transparency of such tools are more fully addressed.

Data Liberation and Crowdsourcing in Medical Research: The Intersection of Collective and Artificial Intelligence.

In spite of an exponential increase in the volume of medical data produced globally, much of these data are inaccessible to those who might best use them to develop improved health care solutions through the application of advanced analytics such as artificial intelligence. Data liberation and crowdsourcing represent two distinct but interrelated approaches to bridging existing data silos and accelerating the pace of innovation internationally. In this article, we examine these concepts in the context of medical artificial intelligence research, summarizing their potential benefits, identifying potential pitfalls, and ultimately making a case for their expanded use going forward. A practical example of a crowdsourced competition using an international medical imaging dataset is provided. Keywords: Artificial Intelligence, Data Liberation, Crowdsourcing © RSNA, 2023.

Performance of the Winning Algorithms of the RSNA 2022 Cervical Spine Fracture Detection Challenge.

Purpose To evaluate and report the performance of the winning algorithms of the Radiological Society of North America Cervical Spine Fracture AI Challenge. Materials and Methods The competition was open to the public on Kaggle from July 28 to October 27, 2022. A sample of 3112 CT scans with and without cervical spine fractures (CSFx) were assembled from multiple sites (12 institutions across six continents) and prepared for the competition. The test set had 1093 scans (private test set: n = 789; mean age, 53.40 years ± 22.86 [SD]; 509 males; public test set: n = 304; mean age, 52.51 years ± 20.73; 189 males) and 847 fractures. The eight top-performing artificial intelligence (AI) algorithms were retrospectively evaluated, and the area under the receiver operating characteristic curve (AUC) value, F1 score, sensitivity, and specificity were calculated. Results A total of 1108 contestants composing 883 teams worldwide participated in the competition. The top eight AI models showed high performance, with a mean AUC value of 0.96 (95% CI: 0.95, 0.96), mean F1 score of 90% (95% CI: 90%, 91%), mean sensitivity of 88% (95% Cl: 86%, 90%), and mean specificity of 94% (95% CI: 93%, 96%). The highest values reported for previous models were an AUC of 0.85, F1 score of 81%, sensitivity of 76%, and specificity of 97%. Conclusion The competition successfully facilitated the development of AI models that could detect and localize CSFx on CT scans with high performance outcomes, which appear to exceed known values of previously reported models. Further study is needed to evaluate the generalizability of these models in a clinical environment. Keywords: Cervical Spine, Fracture Detection, Machine Learning, Artificial Intelligence Algorithms, CT, Head/Neck Supplemental material is available for this article. © RSNA, 2024.

Harmonization of multi-site diffusion tensor imaging data for cervical and thoracic spinal cord at 1.5 T and 3 T using longitudinal ComBat.

MRI scanner hardware, field strengths, and sequence parameters are major variables in diffusion studies of the spinal cord. Reliability between scanners is not well known, particularly for the thoracic cord. DTI data was collected for the entire cervical and thoracic spinal cord in thirty healthy adult subjects with different MR vendors and field strengths. DTI metrics were extracted and averaged for all slices within each vertebral level. Metrics were examined for variability and then harmonized using longitudinal ComBat (longComBat). Four scanners were used: Siemens 3 T Prisma, Siemens 1.5 T Avanto, Philips 3 T Ingenia, Philips 1.5 T Achieva. Average full cord diffusion values/standard deviation for all subjects and scanners were FA: 0.63, σ = 0.10, MD: 1.11, σ = 0.12 × 10-3 mm2/s, AD: 1.98, σ = 0.55 × 10-3 mm2/s, RD: 0.67, σ = 0.31 × 10-3 mm2/s. FA metrics averaged for all subjects by level were relatively consistent across scanners, but large variability was found in diffusivity measures. Coefficients of variation were lowest in the cervical region, and relatively lower for FA than diffusivity measures. Harmonized metrics showed greatly improved agreement between scanners. Variability in DTI of the spinal cord arises from scanner hardware differences, pulse sequence differences, physiological motion, and subject compliance. The use of longComBat resulted in large improvement in agreement of all DTI metrics between scanners. This study shows the importance of harmonization of diffusion data in the spinal cord and potential for longitudinal and multisite clinical research and clinical trials.

The RSNA Cervical Spine Fracture CT Dataset.

This dataset is composed of cervical spine CT images with annotations related to fractures; it is available at https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/.

Radiomic signatures of meningiomas using the Ki-67 proliferation index as a prognostic marker of clinical outcomes.

OBJECTIVE: The clinical behavior of meningiomas is not entirely captured by its designated WHO grade, therefore other factors must be elucidated that portend increased tumor aggressiveness and associated risk of recurrence. In this study, the authors identify multiparametric MRI radiomic signatures of meningiomas using Ki-67 as a prognostic marker of clinical outcomes independent of WHO grade. METHODS: A retrospective analysis was conducted of all resected meningiomas between 2012 and 2018. Preoperative MR images were used for high-throughput radiomic feature extraction and subsequently used to develop a machine learning algorithm to stratify meningiomas based on Ki-67 indices < 5% and ≥ 5%, independent of WHO grade. Progression-free survival (PFS) was assessed based on machine learning prediction of Ki-67 strata and compared with outcomes based on histopathological Ki-67. RESULTS: Three hundred forty-three meningiomas were included: 291 with WHO grade I, 43 with grade II, and 9 with grade III. The overall rate of recurrence was 19.8% (15.1% in grade I, 44.2% in grade II, and 77.8% in grade III) over a median follow-up of 28.5 months. Grade II and III tumors had higher Ki-67 indices than grade I tumors, albeit tumor and peritumoral edema volumes had considerable variation independent of meningioma WHO grade. Forty-six high-performing radiomic features (1 morphological, 7 intensity-based, and 38 textural) were identified and used to build a support vector machine model to stratify tumors based on a Ki-67 cutoff of 5%, with resultant areas under the curve of 0.83 (95% CI 0.78-0.89) and 0.84 (95% CI 0.75-0.94) achieved for the discovery (n = 257) and validation (n = 86) data sets, respectively. Comparison of histopathological Ki-67 versus machine learning-predicted Ki-67 showed excellent performance (overall accuracy > 80%), with classification of grade I meningiomas exhibiting the greatest accuracy. Prediction of Ki-67 by machine learning classifier revealed shorter PFS for meningiomas with Ki-67 indices ≥ 5% compared with tumors with Ki-67 < 5% (p < 0.0001, log-rank test), which corroborates divergent patient outcomes observed using histopathological Ki-67. CONCLUSIONS: The Ki-67 proliferation index may serve as a surrogate marker of increased meningioma aggressiveness independent of WHO grade. Machine learning using radiomic feature analysis may be used for the preoperative prediction of meningioma Ki-67, which provides enhanced analytical insights to help improve diagnostic classification and guide patient-specific treatment strategies.

Association of partial T2-FLAIR mismatch sign and isocitrate dehydrogenase mutation in WHO grade 4 gliomas: results from the ReSPOND consortium.

PURPOSE: While the T2-FLAIR mismatch sign is highly specific for isocitrate dehydrogenase (IDH)-mutant, 1p/19q-noncodeleted astrocytomas among lower-grade gliomas, its utility in WHO grade 4 gliomas is not well-studied. We derived the partial T2-FLAIR mismatch sign as an imaging biomarker for IDH mutation in WHO grade 4 gliomas. METHODS: Preoperative MRI scans of adult WHO grade 4 glioma patients (n = 2165) from the multi-institutional ReSPOND (Radiomics Signatures for PrecisiON Diagnostics) consortium were analyzed. Diagnostic performance of the partial T2-FLAIR mismatch sign was evaluated. Subset analyses were performed to assess associations of imaging markers with overall survival (OS). RESULTS: One hundred twenty-one (5.6%) of 2165 grade 4 gliomas were IDH-mutant. Partial T2-FLAIR mismatch was present in 40 (1.8%) cases, 32 of which were IDH-mutant, yielding 26.4% sensitivity, 99.6% specificity, 80.0% positive predictive value, and 95.8% negative predictive value. Multivariate logistic regression demonstrated IDH mutation was significantly associated with partial T2-FLAIR mismatch (odds ratio [OR] 5.715, 95% CI [1.896, 17.221], p = 0.002), younger age (OR 0.911 [0.895, 0.927], p < 0.001), tumor centered in frontal lobe (OR 3.842, [2.361, 6.251], p < 0.001), absence of multicentricity (OR 0.173, [0.049, 0.612], p = 0.007), and presence of cystic (OR 6.596, [3.023, 14.391], p < 0.001) or non-enhancing solid components (OR 6.069, [3.371, 10.928], p < 0.001). Multivariate Cox analysis demonstrated cystic components (p = 0.024) and non-enhancing solid components (p = 0.003) were associated with longer OS, while older age (p < 0.001), frontal lobe center (p = 0.008), multifocality (p < 0.001), and multicentricity (p < 0.001) were associated with shorter OS. CONCLUSION: Partial T2-FLAIR mismatch sign is highly specific for IDH mutation in WHO grade 4 gliomas.

Opportunistic detection of type 2 diabetes using deep learning from frontal chest radiographs.

Deep learning (DL) models can harness electronic health records (EHRs) to predict diseases and extract radiologic findings for diagnosis. With ambulatory chest radiographs (CXRs) frequently ordered, we investigated detecting type 2 diabetes (T2D) by combining radiographic and EHR data using a DL model. Our model, developed from 271,065 CXRs and 160,244 patients, was tested on a prospective dataset of 9,943 CXRs. Here we show the model effectively detected T2D with a ROC AUC of 0.84 and a 16% prevalence. The algorithm flagged 1,381 cases (14%) as suspicious for T2D. External validation at a distinct institution yielded a ROC AUC of 0.77, with 5% of patients subsequently diagnosed with T2D. Explainable AI techniques revealed correlations between specific adiposity measures and high predictivity, suggesting CXRs' potential for enhanced T2D screening.

DTI of chronic spinal cord injury in children without MRI abnormalities (SCIWOMR) and with pathology on MRI and comparison to severity of motor impairment.

STUDY DESIGN: This investigation was a cohort study that included: 36 typically developing (TD) children and 19 children with spinal cord lesions who underwent spinal cord MRI. OBJECTIVES: To investigate diffusion tensor imaging (DTI) cervical and thoracic spinal cord changes in pediatric patients that have clinically traumatic and non-traumatic spinal cord injury (SCI) without MR (SCIWOMR) abnormalities. SETTING: Thomas Jefferson University, Temple University, Shriners Hospitals for Children all in Philadelphia, USA. METHODS: 36 TD children and 19 children with spinal cord lesions that represent either a chronic traumatic acquired SCI or chronic non-traumatic SCI (≥6 months post injury), age range, 6-16 years who underwent cervical and thoracic spinal cord MRI in 2014-2017. Additionally DTI was correlated to clinical American Spinal Injury Association Impairment Scale (AIS). RESULTS: Both SCIWOMR and MRI positive (+) groups showed abnormal FA and RD DTI values in the adjacent MRI-normal appearing segments of cephalad and caudal spinal cord compared to TD. The FA values demonstrated perilesional abnormal DTI findings in the middle and proximal segments of the cephalad and caudal cord in the SCIWOMR AIS A/B group compared to SCIWOMR AIS C/D group. CONCLUSIONS: We found DTI changes in children with SCIWOMR with different causes of spinal lesions. We also investigated the relationship between DTI and clinical AIS scores. This study further examined the potential diagnostic value of DTI and should be translatable to adults with spinal cord lesions.