Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Checklist for Reproducibility of Deep Learning in Medical Imaging.

The application of deep learning (DL) in medicine introduces transformative tools with the potential to enhance prognosis, diagnosis, and treatment planning. However, ensuring transparent documentation is essential for researchers to enhance reproducibility and refine techniques. Our study addresses the unique challenges presented by DL in medical imaging by developing a comprehensive checklist using the Delphi method to enhance reproducibility and reliability in this dynamic field. We compiled a preliminary checklist based on a comprehensive review of existing checklists and relevant literature. A panel of 11 experts in medical imaging and DL assessed these items using Likert scales, with two survey rounds to refine responses and gauge consensus. We also employed the content validity ratio with a cutoff of 0.59 to determine item face and content validity. Round 1 included a 27-item questionnaire, with 12 items demonstrating high consensus for face and content validity that were then left out of round 2. Round 2 involved refining the checklist, resulting in an additional 17 items. In the last round, 3 items were deemed non-essential or infeasible, while 2 newly suggested items received unanimous agreement for inclusion, resulting in a final 26-item DL model reporting checklist derived from the Delphi process. The 26-item checklist facilitates the reproducible reporting of DL tools and enables scientists to replicate the study's results.

Use of Artificial Intelligence in Radiology: Impact on Pediatric Patients, a White Paper From the ACR Pediatric AI Workgroup.

In this white paper, the ACR Pediatric AI Workgroup of the Commission on Informatics educates the radiology community about the health equity issue of the lack of pediatric artificial intelligence (AI), improves the understanding of relevant pediatric AI issues, and offers solutions to address the inadequacies in pediatric AI development. In short, the design, training, validation, and safe implementation of AI in children require careful and specific approaches that can be distinct from those used for adults. On the eve of widespread use of AI in imaging practice, the group invites the radiology community to align and join Image IntelliGently (www.imageintelligently.org) to ensure that the use of AI is safe, reliable, and effective for children.

Changes in Radiology Due to Artificial Intelligence That Can Attract Medical Students to the Specialty.

The role of artificial intelligence (AI) in radiology has grown exponentially in the recent years. One of the primary worries by medical students is that AI will cause the roles of a radiologist to become automated and thus obsolete. Therefore, there is a greater hesitancy by medical students to choose radiology as a specialty. However, it is in this time of change that the specialty needs new thinkers and leaders. In this succinct viewpoint, 2 medical students involved in AI and 2 radiologists specializing in AI or clinical informatics posit that not only are these fears false, but the field of radiology will be transformed in such a way due to AI that there will be novel reasons to choose radiology. These new factors include greater impact on patient care, new space for innovation, interdisciplinary collaboration, increased patient contact, becoming master diagnosticians, and greater opportunity for global health initiatives, among others. Finally, since medical students view mentorship as a critical resource when deciding their career path, medical educators must also be cognizant of these changes and not give much credence to the prevalent fearmongering. As the field and practice of radiology continue to undergo significant change due to AI, it is urgent and necessary for the conversation to expand from expert to expert to expert to student. Medical students should be encouraged to choose radiology specifically because of the changes brought on by AI rather than being deterred by it.

The requirements for performing artificial-intelligence-related research and model development.

Artificial intelligence research in health care has undergone tremendous growth in the last several years thanks to the explosion of digital health care data and systems that can leverage large amounts of data to learn patterns that can be applied to clinical tasks. In addition, given broad acceleration in machine learning across industries like transportation, media and commerce, there has been a significant growth in demand for machine-learning practitioners such as engineers and data scientists, who have skill sets that can be applied to health care use cases but who simultaneously lack important health care domain expertise. The purpose of this paper is to discuss the requirements of building an artificial-intelligence research enterprise including the research team, technical software/hardware, and procurement and curation of health care data.

Attention-guided deep learning for gestational age prediction using fetal brain MRI.

Magnetic resonance imaging offers unrivaled visualization of the fetal brain, forming the basis for establishing age-specific morphologic milestones. However, gauging age-appropriate neural development remains a difficult task due to the constantly changing appearance of the fetal brain, variable image quality, and frequent motion artifacts. Here we present an end-to-end, attention-guided deep learning model that predicts gestational age with R2 score of 0.945, mean absolute error of 6.7 days, and concordance correlation coefficient of 0.970. The convolutional neural network was trained on a heterogeneous dataset of 741 developmentally normal fetal brain images ranging from 19 to 39 weeks in gestational age. We also demonstrate model performance and generalizability using independent datasets from four academic institutions across the U.S. and Turkey with R2 scores of 0.81-0.90 after minimal fine-tuning. The proposed regression algorithm provides an automated machine-enabled tool with the potential to better characterize in utero neurodevelopment and guide real-time gestational age estimation after the first trimester.

Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.

The RSNA Pulmonary Embolism CT Dataset.

Supplemental material is available for this article.

The Effect of Including Benchmark Prevalence Data of Common Imaging Findings in Spine Image Reports on Health Care Utilization Among Adults Undergoing Spine Imaging: A Stepped-Wedge Randomized Clinical Trial.

Importance: Lumbar spine imaging frequently reveals findings that may seem alarming but are likely unrelated to pain. Prior work has suggested that inserting data on the prevalence of imaging findings among asymptomatic individuals into spine imaging reports may reduce unnecessary subsequent interventions. Objective: To evaluate the impact of including benchmark prevalence data in routine spinal imaging reports on subsequent spine-related health care utilization and opioid prescriptions. Design, Setting, and Participants: This stepped-wedge, pragmatic randomized clinical trial included 250 401 adult participants receiving care from 98 primary care clinics at 4 large health systems in the United States. Participants had imaging of their backs between October 2013 and September 2016 without having had spine imaging in the prior year. Data analysis was conducted from November 2018 to October 2019. Interventions: Either standard lumbar spine imaging reports (control group) or reports containing age-appropriate prevalence data for common imaging findings in individuals without back pain (intervention group). Main Outcomes and Measures: Health care utilization was measured in spine-related relative value units (RVUs) within 365 days of index imaging. The number of subsequent opioid prescriptions written by a primary care clinician was a secondary outcome, and prespecified subgroup analyses examined results by imaging modality. Results: We enrolled 250 401 participants (of whom 238 886 [95.4%] met eligibility for this analysis, with 137 373 [57.5%] women and 105 497 [44.2%] aged >60 years) from 3278 primary care clinicians. A total of 117 455 patients (49.2%) were randomized to the control group, and 121 431 patients (50.8%) were randomized to the intervention group. There was no significant difference in cumulative spine-related RVUs comparing intervention and control conditions through 365 days. The adjusted median (interquartile range) RVU for the control group was 3.56 (2.71-5.12) compared with 3.53 (2.68-5.08) for the intervention group (difference, -0.7%; 95% CI, -2.9% to 1.5%; P = .54). Rates of subsequent RVUs did not differ between groups by specific clinical findings in the report but did differ by type of index imaging (eg, computed tomography: difference, -29.3%; 95% CI, -42.1% to -13.5%; magnetic resonance imaging: difference, -3.4%; 95% CI, -8.3% to 1.8%). We observed a small but significant decrease in the likelihood of opioid prescribing from a study clinician within 1 year of the intervention (odds ratio, 0.95; 95% CI, 0.91 to 1.00; P = .04). Conclusions and Relevance: In this study, inserting benchmark prevalence information in lumbar spine imaging reports did not decrease subsequent spine-related RVUs but did reduce subsequent opioid prescriptions. The intervention text is simple, inexpensive, and easily implemented. Trial Registration: ClinicalTrials.gov Identifier: NCT02015455.

Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge.

This dataset is composed of annotations of the five hemorrhage subtypes (subarachnoid, intraventricular, subdural, epidural, and intraparenchymal hemorrhage) typically encountered at brain CT.

Erratum: Construction of a Machine Learning Dataset through Collaboration: The RSNA 2019 Brain CT Hemorrhage Challenge.

[This corrects the article DOI: 10.1148/ryai.2020190211.].

Deep learning to automate Brasfield chest radiographic scoring for cystic fibrosis.

BACKGROUND: The aim of this study was to evaluate the hypothesis that a deep convolutional neural network (DCNN) model could facilitate automated Brasfield scoring of chest radiographs (CXRs) for patients with cystic fibrosis (CF), performing similarly to a pediatric radiologist. METHODS: All frontal/lateral chest radiographs (2058 exams) performed in CF patients at a single institution from January 2008-2018 were retrospectively identified, and ground-truth Brasfield scoring performed by a board-certified pediatric radiologist. 1858 exams (90.3%) were used to train and validate the DCNN model, while 200 exams (9.7%) were reserved for a test set. Five board-certified pediatric radiologists independently scored the test set according to the Brasfield method. DCNN model vs. radiologist performance was compared using Spearman correlation (ρ) as well as mean difference (MD), mean absolute difference (MAD), and root mean squared error (RMSE) estimation. RESULTS: For the total Brasfield score, ρ for the model-derived results computed pairwise with each radiologist's scores ranged from 0.79-0.83, compared to 0.85-0.90 for radiologist vs. radiologist scores. The MD between model estimates of the total Brasfield score and the average score of radiologists was -0.09. Based on MD, MAD, and RMSE, the model matched or exceeded radiologist performance for all subfeatures except air-trapping and large lesions. CONCLUSIONS: A DCNN model is promising for predicting CF Brasfield scores with accuracy similar to that of a pediatric radiologist.

Sonographic Diagnosis of Velamentous and Marginal Placental Cord Insertion.

Routine second trimester ultrasound (US) examinations include an assessment of the umbilical cord given its vital role as a vascular conduit between the maternal placenta and fetus during fetal development. Placental cord insertion abnormalities can be identified during prenatal US screening and are increasingly recognized as independent risk factors for various complications during pregnancy and delivery. The purpose of this pictorial review is to illustrate examples of velamentous and marginal placental cord insertion with an emphasis on how to differentiate their morphology using color Doppler US.

Challenges Related to Artificial Intelligence Research in Medical Imaging and the Importance of Image Analysis Competitions.

In recent years, there has been enormous interest in applying artificial intelligence (AI) to radiology. Although some of this interest may have been driven by exaggerated expectations that the technology can outperform radiologists in some tasks, there is a growing body of evidence that illustrates its limitations in medical imaging. The true potential of the technique probably lies somewhere in the middle, and AI will ultimately play a key role in medical imaging in the future. The limitless power of computers makes AI an ideal candidate to provide the standardization, consistency, and dependability needed to support radiologists in their mission to provide excellent patient care. However, important roadblocks currently limit the expansion of this field in medical imaging. This article reviews some of the challenges and potential solutions to advance the field forward, with focus on the experience gained by hosting image-based competitions.