Checklist for Reproducibility of Deep Learning in Medical Imaging.

The application of deep learning (DL) in medicine introduces transformative tools with the potential to enhance prognosis, diagnosis, and treatment planning. However, ensuring transparent documentation is essential for researchers to enhance reproducibility and refine techniques. Our study addresses the unique challenges presented by DL in medical imaging by developing a comprehensive checklist using the Delphi method to enhance reproducibility and reliability in this dynamic field. We compiled a preliminary checklist based on a comprehensive review of existing checklists and relevant literature. A panel of 11 experts in medical imaging and DL assessed these items using Likert scales, with two survey rounds to refine responses and gauge consensus. We also employed the content validity ratio with a cutoff of 0.59 to determine item face and content validity. Round 1 included a 27-item questionnaire, with 12 items demonstrating high consensus for face and content validity that were then left out of round 2. Round 2 involved refining the checklist, resulting in an additional 17 items. In the last round, 3 items were deemed non-essential or infeasible, while 2 newly suggested items received unanimous agreement for inclusion, resulting in a final 26-item DL model reporting checklist derived from the Delphi process. The 26-item checklist facilitates the reproducible reporting of DL tools and enables scientists to replicate the study's results.

Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Use of Artificial Intelligence in Radiology: Impact on Pediatric Patients, a White Paper From the ACR Pediatric AI Workgroup.

In this white paper, the ACR Pediatric AI Workgroup of the Commission on Informatics educates the radiology community about the health equity issue of the lack of pediatric artificial intelligence (AI), improves the understanding of relevant pediatric AI issues, and offers solutions to address the inadequacies in pediatric AI development. In short, the design, training, validation, and safe implementation of AI in children require careful and specific approaches that can be distinct from those used for adults. On the eve of widespread use of AI in imaging practice, the group invites the radiology community to align and join Image IntelliGently (www.imageintelligently.org) to ensure that the use of AI is safe, reliable, and effective for children.

Changes in Radiology Due to Artificial Intelligence That Can Attract Medical Students to the Specialty.

The role of artificial intelligence (AI) in radiology has grown exponentially in the recent years. One of the primary worries by medical students is that AI will cause the roles of a radiologist to become automated and thus obsolete. Therefore, there is a greater hesitancy by medical students to choose radiology as a specialty. However, it is in this time of change that the specialty needs new thinkers and leaders. In this succinct viewpoint, 2 medical students involved in AI and 2 radiologists specializing in AI or clinical informatics posit that not only are these fears false, but the field of radiology will be transformed in such a way due to AI that there will be novel reasons to choose radiology. These new factors include greater impact on patient care, new space for innovation, interdisciplinary collaboration, increased patient contact, becoming master diagnosticians, and greater opportunity for global health initiatives, among others. Finally, since medical students view mentorship as a critical resource when deciding their career path, medical educators must also be cognizant of these changes and not give much credence to the prevalent fearmongering. As the field and practice of radiology continue to undergo significant change due to AI, it is urgent and necessary for the conversation to expand from expert to expert to expert to student. Medical students should be encouraged to choose radiology specifically because of the changes brought on by AI rather than being deterred by it.

Perceptions of US Medical Students on Artificial Intelligence in Medicine: Mixed Methods Survey Study.

BACKGROUND: Given the rapidity with which artificial intelligence is gaining momentum in clinical medicine, current physician leaders have called for more incorporation of artificial intelligence topics into undergraduate medical education. This is to prepare future physicians to better work together with artificial intelligence technology. However, the first step in curriculum development is to survey the needs of end users. There has not been a study to determine which media and which topics are most preferred by US medical students to learn about the topic of artificial intelligence in medicine. OBJECTIVE: We aimed to survey US medical students on the need to incorporate artificial intelligence in undergraduate medical education and their preferred means to do so to assist with future education initiatives. METHODS: A mixed methods survey comprising both specific questions and a write-in response section was sent through Qualtrics to US medical students in May 2021. Likert scale questions were used to first assess various perceptions of artificial intelligence in medicine. Specific questions were posed regarding learning format and topics in artificial intelligence. RESULTS: We surveyed 390 US medical students with an average age of 26 (SD 3) years from 17 different medical programs (the estimated response rate was 3.5%). A majority (355/388, 91.5%) of respondents agreed that training in artificial intelligence concepts during medical school would be useful for their future. While 79.4% (308/388) were excited to use artificial intelligence technologies, 91.2% (353/387) either reported that their medical schools did not offer resources or were unsure if they did so. Short lectures (264/378, 69.8%), formal electives (180/378, 47.6%), and Q and A panels (167/378, 44.2%) were identified as preferred formats, while fundamental concepts of artificial intelligence (247/379, 65.2%), when to use artificial intelligence in medicine (227/379, 59.9%), and pros and cons of using artificial intelligence (224/379, 59.1%) were the most preferred topics for enhancing their training. CONCLUSIONS: The results of this study indicate that current US medical students recognize the importance of artificial intelligence in medicine and acknowledge that current formal education and resources to study artificial intelligence-related topics are limited in most US medical schools. Respondents also indicated that a hybrid formal/flexible format would be most appropriate for incorporating artificial intelligence as a topic in US medical schools. Based on these data, we conclude that there is a definitive knowledge gap in artificial intelligence education within current medical education in the US. Further, the results suggest there is a disparity in opinions on the specific format and topics to be introduced.

The requirements for performing artificial-intelligence-related research and model development.

Artificial intelligence research in health care has undergone tremendous growth in the last several years thanks to the explosion of digital health care data and systems that can leverage large amounts of data to learn patterns that can be applied to clinical tasks. In addition, given broad acceleration in machine learning across industries like transportation, media and commerce, there has been a significant growth in demand for machine-learning practitioners such as engineers and data scientists, who have skill sets that can be applied to health care use cases but who simultaneously lack important health care domain expertise. The purpose of this paper is to discuss the requirements of building an artificial-intelligence research enterprise including the research team, technical software/hardware, and procurement and curation of health care data.

Attention-guided deep learning for gestational age prediction using fetal brain MRI.

Magnetic resonance imaging offers unrivaled visualization of the fetal brain, forming the basis for establishing age-specific morphologic milestones. However, gauging age-appropriate neural development remains a difficult task due to the constantly changing appearance of the fetal brain, variable image quality, and frequent motion artifacts. Here we present an end-to-end, attention-guided deep learning model that predicts gestational age with R2 score of 0.945, mean absolute error of 6.7 days, and concordance correlation coefficient of 0.970. The convolutional neural network was trained on a heterogeneous dataset of 741 developmentally normal fetal brain images ranging from 19 to 39 weeks in gestational age. We also demonstrate model performance and generalizability using independent datasets from four academic institutions across the U.S. and Turkey with R2 scores of 0.81-0.90 after minimal fine-tuning. The proposed regression algorithm provides an automated machine-enabled tool with the potential to better characterize in utero neurodevelopment and guide real-time gestational age estimation after the first trimester.

A Review of Core Concepts of Imaging Informatics.

There are myriad systems and standards used in imaging informatics. Digital Imaging and Communications in Medicine (DICOM) is the standard for displaying, transferring, and storing medical images. Health Level Seven International (HL7) develops and maintains standards for exchanging, integrating, and sharing medical data. Picture archiving and communication system (PACS) serves as the health provider's primary tool for viewing and interpreting medical images. Medical imaging depends on the interoperability of several of these systems. From entering the order into the electronic medical record (EMR), several systems receive and share medical data, including the radiology information system (RIS) and hospital information system (HIS). After acquiring an image, transformations may be performed to better focus on a specific area. The workflow from entering the order to receiving the report depends on many systems. Having disaster recovery and business continuity procedures is important should any issues arise. This article intends to review these essential concepts of imaging informatics.

Data Sharing of Imaging in an Evolving Health Care World: Report of the ACR Data Sharing Workgroup, Part 2: Annotation, Curation, and Contracting.

A core principle of ethical data sharing is maintaining the security and anonymity of the data, and care must be taken to ensure medical records and images cannot be reidentified to be traced back to patients or misconstrued as a breach in the trust between health care providers and patients. Once those principles have been observed, those seeking to share data must take the appropriate steps to curate the data in a way that organizes the clinically relevant information so as to be useful to the data sharing party, assesses the ensuing value of the data set and its annotations, and informs the data sharing contracts that will govern use of the data. Embarking on a data sharing partnership engenders a host of ethical, practical, technical, legal, and commercial challenges that require a thoughtful, considered approach. In 2019 the ACR convened a Data Sharing Workgroup to develop philosophies around best practices in the sharing of health information. This is Part 2 of a Report on the workgroup's efforts in exploring these issues.

Data Sharing of Imaging in an Evolving Health Care World: Report of the ACR Data Sharing Workgroup, Part 1: Data Ethics of Privacy, Consent, and Anonymization.

Radiology is at the forefront of the artificial intelligence transformation of health care across multiple areas, from patient selection to study acquisition to image interpretation. Needing large data sets to develop and train these algorithms, developers enter contractual data sharing agreements involving data derived from health records, usually with postacquisition curation and annotation. In 2019 the ACR convened a Data Sharing Workgroup to develop philosophies around best practices in the sharing of health information. The workgroup identified five broad domains of activity important to collaboration using patient data: privacy, informed consent, standardization of data elements, vendor contracts, and data valuation. This is Part 1 of a Report on the workgroup's efforts in exploring these issues.

Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.

The RSNA Pulmonary Embolism CT Dataset.

Supplemental material is available for this article.

Multi-classifier-based identification of COVID-19 from chest computed tomography using generalizable and interpretable radiomics features.

PURPOSE: To investigate the efficacy of radiomics in diagnosing patients with coronavirus disease (COVID-19) and other types of viral pneumonia with clinical symptoms and CT signs similar to those of COVID-19. METHODS: Between 18 January 2020 and 20 May 2020, 110 SARS-CoV-2 positive and 108 SARS-CoV-2 negative patients were retrospectively recruited from three hospitals based on the inclusion criteria. Manual segmentation of pneumonia lesions on CT scans was performed by four radiologists. The latest version of Pyradiomics was used for feature extraction. Four classifiers (linear classifier, k-nearest neighbour, least absolute shrinkage and selection operator [LASSO], and random forest) were used to differentiate SARS-CoV-2 positive and SARS-CoV-2 negative patients. Comparison of the performance of the classifiers and radiologists was evaluated by ROC curve and Kappa score. RESULTS: We manually segmented 16,053 CT slices, comprising 32,625 pneumonia lesions, from the CT scans of all patients. Using Pyradiomics, 120 radiomic features were extracted from each image. The key radiomic features screened by different classifiers varied and lead to significant differences in classification accuracy. The LASSO achieved the best performance (sensitivity: 72.2%, specificity: 75.1%, and AUC: 0.81) on the external validation dataset and attained excellent agreement (Kappa score: 0.89) with radiologists (average sensitivity: 75.6%, specificity: 78.2%, and AUC: 0.81). All classifiers indicated that "Original_Firstorder_RootMeanSquared" and "Original_Firstorder_Uniformity" were significant features for this task. CONCLUSIONS: We identified radiomic features that were significantly associated with the classification of COVID-19 pneumonia using multiple classifiers. The quantifiable interpretation of the differences in features between the two groups extends our understanding of CT imaging characteristics of COVID-19 pneumonia.

Obstetric and neonatal outcomes in pregnancies complicated by fetal lung masses: does final histology matter?

PURPOSE: Fetal lung masses complicate approximately 1 in 2000 live births. Our aim was to determine whether obstetric and neonatal outcomes differ by final fetal lung mass histology. MATERIALS AND METHODS: A review of all pregnancies complicated by a prenatally diagnosed fetal lung mass between 2009 and 2017 at a single academic center was conducted. All cases included in the final analysis underwent surgical resection and histology diagnosis was determined by a trained pathologist. Clinical data were obtained from review of stored electronic medical records which contained linked maternal and neonatal records. Imaging records included both prenatal ultrasound and magnetic resonance imaging. Fisher's exact test was used for categorical variables and the Kruskal-Wallis test was used for continuous variables. The level of significance was p<.05. RESULTS: Of 61 pregnancies complicated by fetal lung mass during the study period, 45 cases underwent both prenatal care and postnatal resection. Final histology revealed 10 cases of congenital pulmonary airway malformation (CPAM) type 1, nine cases of CPAM type 2, and 16 cases of bronchopulmonary sequestration. There was no difference in initial, maximal, or final CPAM volume ratio between groups, with median final CPAM volume ratio of 0.6 for CPAM type 1, 0.7 for CPAM type 2, and 0.3 for bronchopulmonary sequestration (p = .12). There were no differences in any of the maternal or obstetric outcomes including gestational age at delivery and mode of delivery between the groups. The primary outcome of neonatal respiratory distress was not statistically different between groups (p = .66). Median neonatal length of stay following delivery ranged from 3 to 4 days, and time to postnatal resection was similar as well, with a median of 126 days for CPAM type 1, 122 days for CPAM type 2, and 132 days for bronchopulmonary sequestration (p = .76). CONCLUSIONS: In our cohort, there was no significant association between histologic lung mass subtypes and any obstetric or neonatal morbidity including respiratory distress.