Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Data Liberation and Crowdsourcing in Medical Research: The Intersection of Collective and Artificial Intelligence.

In spite of an exponential increase in the volume of medical data produced globally, much of these data are inaccessible to those who might best use them to develop improved health care solutions through the application of advanced analytics such as artificial intelligence. Data liberation and crowdsourcing represent two distinct but interrelated approaches to bridging existing data silos and accelerating the pace of innovation internationally. In this article, we examine these concepts in the context of medical artificial intelligence research, summarizing their potential benefits, identifying potential pitfalls, and ultimately making a case for their expanded use going forward. A practical example of a crowdsourced competition using an international medical imaging dataset is provided. Keywords: Artificial Intelligence, Data Liberation, Crowdsourcing © RSNA, 2023.

Machine Learning Detection and Characterization of Splenic Injuries on Abdominal Computed Tomography.

BACKGROUND: Multi-detector contrast-enhanced abdominal computed tomography (CT) allows for the accurate detection and classification of traumatic splenic injuries, leading to improved patient management. Their effective use requires rapid study interpretation, which can be a challenge on busy emergency radiology services. A machine learning system has the potential to automate the process, potentially leading to a faster clinical response. This study aimed to create such a system. METHOD: Using the American Association for the Surgery of Trauma (AAST), spleen injuries were classified into 3 classes: normal, low-grade (AAST grade I-III) injuries, and high-grade (AAST grade IV and V) injuries. Employing a 2-stage machine learning strategy, spleens were initially segmented from input CT images and subsequently underwent classification via a 3D dense convolutional neural network (DenseNet). RESULTS: This single-centre retrospective study involved trauma protocol CT scans performed between January 1, 2005, and July 31, 2021, totaling 608 scans with splenic injuries and 608 without. Five board-certified fellowship-trained abdominal radiologists utilizing the AAST injury scoring scale established ground truth labels. The model achieved AUC values of 0.84, 0.69, and 0.90 for normal, low-grade injuries, and high-grade splenic injuries, respectively. CONCLUSIONS: Our findings demonstrate the feasibility of automating spleen injury detection using our method with potential applications in improving patient care through radiologist worklist prioritization and injury stratification. Future endeavours should concentrate on further enhancing and optimizing our approach and testing its use in a real-world clinical environment.

Performance of the Winning Algorithms of the RSNA 2022 Cervical Spine Fracture Detection Challenge.

Purpose To evaluate and report the performance of the winning algorithms of the Radiological Society of North America Cervical Spine Fracture AI Challenge. Materials and Methods The competition was open to the public on Kaggle from July 28 to October 27, 2022. A sample of 3112 CT scans with and without cervical spine fractures (CSFx) were assembled from multiple sites (12 institutions across six continents) and prepared for the competition. The test set had 1093 scans (private test set: n = 789; mean age, 53.40 years ± 22.86 [SD]; 509 males; public test set: n = 304; mean age, 52.51 years ± 20.73; 189 males) and 847 fractures. The eight top-performing artificial intelligence (AI) algorithms were retrospectively evaluated, and the area under the receiver operating characteristic curve (AUC) value, F1 score, sensitivity, and specificity were calculated. Results A total of 1108 contestants composing 883 teams worldwide participated in the competition. The top eight AI models showed high performance, with a mean AUC value of 0.96 (95% CI: 0.95, 0.96), mean F1 score of 90% (95% CI: 90%, 91%), mean sensitivity of 88% (95% Cl: 86%, 90%), and mean specificity of 94% (95% CI: 93%, 96%). The highest values reported for previous models were an AUC of 0.85, F1 score of 81%, sensitivity of 76%, and specificity of 97%. Conclusion The competition successfully facilitated the development of AI models that could detect and localize CSFx on CT scans with high performance outcomes, which appear to exceed known values of previously reported models. Further study is needed to evaluate the generalizability of these models in a clinical environment. Keywords: Cervical Spine, Fracture Detection, Machine Learning, Artificial Intelligence Algorithms, CT, Head/Neck Supplemental material is available for this article. © RSNA, 2024.

Vision Transformer-based Decision Support for Neurosurgical Intervention in Acute Traumatic Brain Injury: Automated Surgical Intervention Support Tool.

Purpose To develop an automated triage tool to predict neurosurgical intervention for patients with traumatic brain injury (TBI). Materials and Methods A provincial trauma registry was reviewed to retrospectively identify patients with TBI from 2005 to 2022 treated at a specialized Canadian trauma center. Model training, validation, and testing were performed using head CT scans with binary reference standard patient-level labels corresponding to whether the patient received neurosurgical intervention. Performance and accuracy of the model, the Automated Surgical Intervention Support Tool for TBI (ASIST-TBI), were also assessed using a held-out consecutive test set of all patients with TBI presenting to the center between March 2021 and September 2022. Results Head CT scans from 2806 patients with TBI (mean age, 57 years ± 22 [SD]; 1955 [70%] men) were acquired between 2005 and 2021 and used for training, validation, and testing. Consecutive scans from an additional 612 patients (mean age, 61 years ± 22; 443 [72%] men) were used to assess the performance of ASIST-TBI. There was accurate prediction of neurosurgical intervention with an area under the receiver operating characteristic curve (AUC) of 0.92 (95% CI: 0.88, 0.94), accuracy of 87% (491 of 562), sensitivity of 87% (196 of 225), and specificity of 88% (295 of 337) on the test dataset. Performance on the held-out test dataset remained robust with an AUC of 0.89 (95% CI: 0.85, 0.91), accuracy of 84% (517 of 612), sensitivity of 85% (199 of 235), and specificity of 84% (318 of 377). Conclusion A novel deep learning model was developed that could accurately predict the requirement for neurosurgical intervention using acute TBI CT scans. Keywords: CT, Brain/Brain Stem, Surgery, Trauma, Prognosis, Classification, Application Domain, Traumatic Brain Injury, Triage, Machine Learning, Decision Support Supplemental material is available for this article. © RSNA, 2024 See also commentary by Haller in this issue.

Machine Learning Classification of Body Part, Imaging Axis, and Intravenous Contrast Enhancement on CT Imaging.

Purpose: The development and evaluation of machine learning models that automatically identify the body part(s) imaged, axis of imaging, and the presence of intravenous contrast material of a CT series of images. Methods: This retrospective study included 6955 series from 1198 studies (501 female, 697 males, mean age 56.5 years) obtained between January 2010 and September 2021. Each series was annotated by a trained board-certified radiologist with labels consisting of 16 body parts, 3 imaging axes, and whether an intravenous contrast agent was used. The studies were randomly assigned to the training, validation and testing sets with a proportion of 70%, 20% and 10%, respectively, to develop a 3D deep neural network for each classification task. External validation was conducted with a total of 35,272 series from 7 publicly available datasets. The classification accuracy for each series was independently assessed for each task to evaluate model performance. Results: The accuracies for identifying the body parts, imaging axes, and the presence of intravenous contrast were 96.0% (95% CI: 94.6%, 97.2%), 99.2% (95% CI: 98.5%, 99.7%), and 97.5% (95% CI: 96.4%, 98.5%) respectively. The generalizability of the models was demonstrated through external validation with accuracies of 89.7 - 97.8%, 98.6 - 100%, and 87.8 - 98.6% for the same tasks. Conclusions: The developed models demonstrated high performance on both internal and external testing in identifying key aspects of a CT series.

Interrater Agreement of CT Grading of Blunt Splenic Injuries: Does the AAST Grading Need to Be Reimagined?

Introduction: The Revised Organ Injury Scale (OIS) of the American Association for Surgery of Trauma (AAST) is the most widely accepted classification of splenic trauma. The objective of this study was to evaluate inter-rater agreement for CT grading of blunt splenic injuries. Methods: CT scans in adult patients with splenic injuries at a level 1 trauma centre were independently graded by 5 fellowship trained abdominal radiologists using the AAST OIS for splenic injuries - 2018 revision. The inter-rater agreement for AAST CT injury score, as well as low-grade (IIII) versus high-grade (IV-V) splenic injury was assessed. Disagreement in two key clinical scenarios (no injury versus injury, and high versus low grade) were qualitatively reviewed to identify possible sources of disagreement. Results: A total of 610 examinations were included. The inter-rater absolute agreement was low (Fleiss kappa statistic 0.38, P < 0.001), but improved when comparing agreement between low and high grade injuries (Fleiss kappa statistic of 0.77, P < .001). There were 34 cases (5.6%) of minimum two-rater disagreement about no injury vs injury (AAST grade ≥ I). There were 46 cases (7.5%) of minimum two-rater disagreement of low grade (AAST grade I-III) versus high grade (AAST grade IV-V) injuries. Likely sources of disagreement were interpretation of clefts versus lacerations, peri-splenic fluid versus subcapsular hematoma, application of adding multiple low grade injuries to higher grade injuries, and identification of subtle vascular injuries. Conclusion: There is low absolute agreement in grading of splenic injuries using the existing AAST OIS for splenic injuries.

The RSNA Cervical Spine Fracture CT Dataset.

This dataset is composed of cervical spine CT images with annotations related to fractures; it is available at https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/.

Augmentation of the RSNA Pulmonary Embolism CT Dataset with Bounding Box Annotations and Anatomic Localization of Pulmonary Emboli.

Supplemental material is available for this article. Keywords: CT, Pulmonary Arteries, Embolism/Thrombosis, Feature Detection © RSNA, 2023.

Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays.

Artificial intelligence (AI)-generated clinical advice is becoming more prevalent in healthcare. However, the impact of AI-generated advice on physicians' decision-making is underexplored. In this study, physicians received X-rays with correct diagnostic advice and were asked to make a diagnosis, rate the advice's quality, and judge their own confidence. We manipulated whether the advice came with or without a visual annotation on the X-rays, and whether it was labeled as coming from an AI or a human radiologist. Overall, receiving annotated advice from an AI resulted in the highest diagnostic accuracy. Physicians rated the quality of AI advice higher than human advice. We did not find a strong effect of either manipulation on participants' confidence. The magnitude of the effects varied between task experts and non-task experts, with the latter benefiting considerably from correct explainable AI advice. These findings raise important considerations for the deployment of diagnostic advice in healthcare.

Standardized Reporting on the Preoperative CT Assessment of Potential Living Renal Transplant Donors: Can We Create a Universal Report Standard to Meet the Needs of Transplant Urologists?

Purpose: Determine whether standardized template reporting for the preoperative assessment of potential living renal transplant donors improves the comprehensiveness of radiology reports to meet the needs of urologists performing renal transplants. Methods: Urologist and radiologist stakeholders from renal transplant centres in our province ratified a standardized reporting template for evaluation of potential renal donors. Three centres (A, B, and C) were designated "intervention" groups. Centre D was the control group, given employment of a site-specific standardized template prior to study commencement. Up to 100 consecutive CT scan reports per centre, pre- and post-implementation of standardized reporting, were evaluated for reporting specific outcome measures. Results: At baseline, all intervention groups demonstrated poor reporting of urologist-desired outcome measures. Centre A discussed 5/13 variables (38%), Centre B discussed 6/13 variables (46%), and Centre C only discussed 1/13 variables (8%) with ≥90% reliability. The control group exhibited consistent reporting, with 11/13 variables (85%) reported at ≥90% reliability. All institutions in the intervention group exhibited excellent compliance to structured reporting post-template implementation (Centres A = 95%, B = 100%, and C = 77%, respectively). Additionally, all intervention centres demonstrated a significant improvement in the comprehensiveness of reports post-template implementation, with statistically significant increases in the reporting of all variables under-reported at baseline (P > .01). Conclusion: Standardized templates across our province for CT scans of potential renal donors promote completeness of reports. Radiologists can reliably provide our surgical colleagues with needed preoperative anatomy and incidental findings, helping to determine suitable transplant donors and reduce potential complications associated with organ retrieval.

Deep learning characterization of brain tumours with diffusion weighted imaging.

Glioblastoma multiforme (GBM) is one of the most deadly forms of cancer. Methods of characterizing these tumours are valuable for improving predictions of their progression and response to treatment. A mathematical model called the proliferation-invasion (PI) model has been used extensively in the literature to model the growth of these tumours, though it relies on known values of two key parameters: the tumour cell diffusivity and proliferation rate. Unfortunately, these parameters are difficult to estimate in a patient-specific manner, making personalized tumour forecasting challenging. In this paper, we develop and apply a deep learning model capable of making accurate estimates of these key GBM-characterizing parameters while simultaneously producing a full prediction of the tumour progression curve. Our method uses two sets of multi sequence MRI in order to produce estimations and relies on a preprocessing pipeline which includes brain tumour segmentation and conversion to tumour cellularity. We first apply our deep learning model to synthetic tumours to showcase the model's capabilities and identify situations where prediction errors are likely to occur. We then apply our model to a clinical dataset consisting of five patients diagnosed with GBM. For all patients, we derive evidence-based estimates for each of the PI model parameters and predictions for the future progression of the tumour, along with estimates of the parameter uncertainties. Our work provides a new, easily generalizable method for the estimation of patient-specific tumour parameters, which can be built upon to aid physicians in designing personalized treatments.

Infection complications after transrectal ultrasound-guided prostate biopsy: A radiology department's experience and strategy for improvement.

INTRODUCTION: Transrectal ultrasound (TRUS)-guided prostate biopsy is a common procedure performed to diagnose prostate cancer. The risk of infection complications is well-described in the literature, and strategies to avoid such complications continue to evolve over time. We performed a retrospective review of our infection complications and propose a strategy for improvement. METHODS: We reviewed clinical outcomes from patients undergoing TRUS-guided prostate biopsy at our institution from November 2018 to November 2020. We reported the antimicrobial prophylaxis received, whether the biopsy was systematic or targeted, and we examined the rate of clinically significant infection complications and hospitalization. RESULTS: Among 312 men who underwent TRUS-guided prostate biopsy during the study period, seven (2.2%) had an infection. Four patient groups with distinct antimicrobial regimen were identified; the largest of these patient groups received a three-day course of cefixime and a single dose of fosfomycin (59%). The proportion of patients with infection complications across these groups did not demonstrate a statistically significant difference (p=0.803). There was no significant difference in proportion of infection between systematic and targeted biopsy groups (3.0% vs. 0%, p=0.204). The proportion of patients hospitalized was 1.3%, with a mean length of stay of four days. CONCLUSIONS: We report a rate of clinically significant infection following TRUS-guided prostate biopsy of 2.2%. Due to our referral pathway, we have an inconsistent approach to antimicrobial prophylaxis, although there was no statistically significant difference in infection rate between the groups. We propose a standardized approach that may lead to improved patient outcomes.

A real-world demonstration of machine learning generalizability in the detection of intracranial hemorrhage on head computerized tomography.

Machine learning (ML) holds great promise in transforming healthcare. While published studies have shown the utility of ML models in interpreting medical imaging examinations, these are often evaluated under laboratory settings. The importance of real world evaluation is best illustrated by case studies that have documented successes and failures in the translation of these models into clinical environments. A key prerequisite for the clinical adoption of these technologies is demonstrating generalizable ML model performance under real world circumstances. The purpose of this study was to demonstrate that ML model generalizability is achievable in medical imaging with the detection of intracranial hemorrhage (ICH) on non-contrast computed tomography (CT) scans serving as the use case. An ML model was trained using 21,784 scans from the RSNA Intracranial Hemorrhage CT dataset while generalizability was evaluated using an external validation dataset obtained from our busy trauma and neurosurgical center. This real world external validation dataset consisted of every unenhanced head CT scan (n = 5965) performed in our emergency department in 2019 without exclusion. The model demonstrated an AUC of 98.4%, sensitivity of 98.8%, and specificity of 98.0%, on the test dataset. On external validation, the model demonstrated an AUC of 95.4%, sensitivity of 91.3%, and specificity of 94.1%. Evaluating the ML model using a real world external validation dataset that is temporally and geographically distinct from the training dataset indicates that ML generalizability is achievable in medical imaging applications.

The RSNA Pulmonary Embolism CT Dataset.

Supplemental material is available for this article.

Applications of Machine Learning to Imaging of Spinal Disorders: Current Status and Future Directions.

STUDY DESIGN: Narrative review. OBJECTIVES: We aim to describe current progress in the application of artificial intelligence and machine learning technology to provide automated analysis of imaging in patients with spinal disorders. METHODS: A literature search utilizing the PubMed database was performed. Relevant studies from all the evidence levels have been included. RESULTS: Within spine surgery, artificial intelligence and machine learning technologies have achieved near-human performance in narrow image classification tasks on specific datasets in spinal degenerative disease, spinal deformity, spine trauma, and spine oncology. CONCLUSION: Although substantial challenges remain to be overcome it is clear that artificial intelligence and machine learning technology will influence the practice of spine surgery in the future.

Do as AI say: susceptibility in deployment of clinical decision-aids.

Artificial intelligence (AI) models for decision support have been developed for clinical settings such as radiology, but little work evaluates the potential impact of such systems. In this study, physicians received chest X-rays and diagnostic advice, some of which was inaccurate, and were asked to evaluate advice quality and make diagnoses. All advice was generated by human experts, but some was labeled as coming from an AI system. As a group, radiologists rated advice as lower quality when it appeared to come from an AI system; physicians with less task-expertise did not. Diagnostic accuracy was significantly worse when participants received inaccurate advice, regardless of the purported source. This work raises important considerations for how advice, AI and non-AI, should be deployed in clinical environments.

The RSNA International COVID-19 Open Radiology Database (RICORD).

The coronavirus disease 2019 (COVID-19) pandemic is a global health care emergency. Although reverse-transcription polymerase chain reaction testing is the reference standard method to identify patients with COVID-19 infection, chest radiography and CT play a vital role in the detection and management of these patients. Prediction models for COVID-19 imaging are rapidly being developed to support medical decision making. However, inadequate availability of a diverse annotated data set has limited the performance and generalizability of existing models. To address this unmet need, the RSNA and Society of Thoracic Radiology collaborated to develop the RSNA International COVID-19 Open Radiology Database (RICORD). This database is the first multi-institutional, multinational, expert-annotated COVID-19 imaging data set. It is made freely available to the machine learning community as a research and educational resource for COVID-19 chest imaging. Pixel-level volumetric segmentation with clinical annotations was performed by thoracic radiology subspecialists for all COVID-19-positive thoracic CT scans. The labeling schema was coordinated with other international consensus panels and COVID-19 data annotation efforts, the European Society of Medical Imaging Informatics, the American College of Radiology, and the American Association of Physicists in Medicine. Study-level COVID-19 classification labels for chest radiographs were annotated by three radiologists, with majority vote adjudication by board-certified radiologists. RICORD consists of 240 thoracic CT scans and 1000 chest radiographs contributed from four international sites. It is anticipated that RICORD will ideally lead to prediction models that can demonstrate sustained performance across populations and health care systems.