Clinical, Cultural, Computational, and Regulatory Considerations to Deploy AI in Radiology: Perspectives of RSNA and MICCAI Experts.

The Radiological Society of North of America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society have led a series of joint panels and seminars focused on the present impact and future directions of artificial intelligence (AI) in radiology. These conversations have collected viewpoints from multidisciplinary experts in radiology, medical imaging, and machine learning on the current clinical penetration of AI technology in radiology and how it is impacted by trust, reproducibility, explainability, and accountability. The collective points-both practical and philosophical-define the cultural changes for radiologists and AI scientists working together and describe the challenges ahead for AI technologies to meet broad approval. This article presents the perspectives of experts from MICCAI and RSNA on the clinical, cultural, computational, and regulatory considerations-coupled with recommended reading materials-essential to adopt AI technology successfully in radiology and, more generally, in clinical practice. The report emphasizes the importance of collaboration to improve clinical deployment, highlights the need to integrate clinical and medical imaging data, and introduces strategies to ensure smooth and incentivized integration. Keywords: Adults and Pediatrics, Computer Applications-General (Informatics), Diagnosis, Prognosis © RSNA, 2024.

Standardized reporting for Head CT Scans in patients suspected of traumatic brain injury (TBI): An international expert endeavor.

BACKGROUND AND PURPOSE: Traumatic brain injury (TBI) is a major source of health loss and disability worldwide. Accurate and timely diagnosis of TBI is critical for appropriate treatment and management of the condition. Neuroimaging plays a crucial role in the diagnosis and characterization of TBI. Computed tomography (CT) is the first-line diagnostic imaging modality typically utilized in patients with suspected acute mild, moderate and severe TBI. Radiology reports play a crucial role in the diagnostic process, providing critical information about the location and extent of brain injury, as well as factors that could prevent secondary injury. However, the complexity and variability of radiology reports can make it challenging for healthcare providers to extract the necessary information for diagnosis and treatment planning. METHODS/RESULTS/CONCLUSION: In this article, we report the efforts of an international group of TBI imaging experts to develop a clinical radiology report template for CT scans obtained in patients suspected of TBI and consisting of fourteen different subdivisions (CT technique, mechanism of injury or clinical history, presence of scalp injuries, fractures, potential vascular injuries, potential injuries involving the extra-axial spaces, brain parenchymal injuries, potential injuries involving the cerebrospinal fluid spaces and the ventricular system, mass effect, secondary injuries, prior or coexisting pathology).

Correlation of Computed Tomography Angiography to Digital Subtraction Angiography in Carotid Stenosis with Real-World Assessment of Overestimation of Carotid Stenosis on Computed Tomography Angiography.

OBJECTIVE: Computed tomography angiography (CTA) is a well-established diagnostic modality for carotid stenosis. However, false-positive CTA results may expose patients to unnecessary procedural complications in cases where surgical intervention is not warranted. We aim to assess the correlation of CTA to digital subtraction angiography (DSA) in carotid stenosis and characterize patients who were referred for intervention based on CTA and did not require it based on DSA. METHODS: We retrospectively reviewed 186 patients who underwent carotid angioplasty and stenting following preprocedural CTA at our institution from April 2017 to December 2022. RESULTS: Twenty-one of 186 patients (11.2%) were found to have <50% carotid stenosis on DSA (discordant group). Severe plaque calcification on CTA was associated with a discordant degree of stenosis on DSA (LR+=7.4). Among 186 patients, agreement between the percentage of stenosis from CTA and DSA was weak-moderate (r2=0.27, P<0.01). Among concordant pairs, we found moderate-strong agreement between CTA and DSA (adj r2=0.37) (P < 0.0001). Of 186 patients, 127 patients had CTA stenosis of ≥70%, and 59 had CTA of 50%-69%. Correlation between CTA and DSA in severe CTA stenosis was weak (r2=0.11, P<0.01). CONCLUSIONS: In patients with stenosis found on CTA, over 88% also had stenosis on DSA, with this positive predictive value in line with previous studies. The percent-stenosis value from CTA and DSA was weakly correlated but does not affect the overall clinical judgement of stenosis. Severe calcification found on CTA may potentially indicate nonstenosis on DSA.

The impact of mechanical thrombectomy on the blood-brain barrier in patients with acute ischemic stroke: A non-contrast MR imaging study using DP-pCASL and NODDI.

BACKGROUND AND PURPOSE: While mechanical thrombectomy (MT) achieves restoration of cerebral blood flow to the area at risk in patients with acute ischemic stroke (AIS), the influx of blood flow may exacerbate the blood-brain barrier (BBB) disruption and extravasation across the BBB, and it therefore remains unclear how reperfusion impacts the blood-brain barrier integrity. In this study, we use diffusion-prepared pseudocontinuous ASL (DP-pCASL) and Neurite Orientation Dispersion and Density Imaging (NODDI) sequence to measure the water exchange rate (kw) in patients who underwent either MT or medical management and determine its impact on the brain tissue microstructure in order to elucidate the impact of MT on BBB complex integrity. MATERIALS AND METHODS: We prospectively enrolled 21 patients with AIS treated at our institution from 10/2021 to 6/2023 who underwent MR imaging at a 3.0-Tesla scanner. Patients underwent DP-pCASl and NODDI imaging in addition to the standard stroke protocol which generated cerebral blood flow (CBF), arterial transit time (ATT), water exchange rate (kw), orientation dispersion index (ODI), intracellular volume fraction (ICVF), and free water fraction (FWF) parametric maps. RESULTS: Of the 21 patients, 11 underwent MT and 10 were treated non-operatively. The average age and NIHSS for the MT cohort and non-MT cohorts were 69.3 ± 16.6 years old and 15.0 (12.0-20.0), and 70.2 ± 10.7 (p = 0.882) and 6.0 (3.8-9.0, p = 0.003) respectively. The average CBF, ATT, and kw in the infarcted territory of the MT cohort were 38.2 (18.4-59.6), 1347.6 (1182.5-1842.3), and 107.8 (79.2-140.1) respectively. The average CBF, ATT, and kw in the stroke ROI were 16.0 (8.8-36.6, p = 0.036), 1090.8 (937.1-1258.9, p = 0.013), 89.7 (68.0-122.7, p = 0.314) respectively. Linear regression analysis showed increasing CBF (p = 0.008) and undergoing mechanical thrombectomy (p = 0.048) were significant predictors of increased kw. CONCLUSION: Using our multimodal non-contrast MRI protocol, we demonstrate that increased CBF and mechanical thrombectomy increased kw, suggesting a better functioning BBB complex. Higher kw suggests less disruption of the BBB complex in the MT cohort.

Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs.

Background Generating radiologic findings from chest radiographs is pivotal in medical image analysis. The emergence of OpenAI's generative pretrained transformer, GPT-4 with vision (GPT-4V), has opened new perspectives on the potential for automated image-text pair generation. However, the application of GPT-4V to real-world chest radiography is yet to be thoroughly examined. Purpose To investigate the capability of GPT-4V to generate radiologic findings from real-world chest radiographs. Materials and Methods In this retrospective study, 100 chest radiographs with free-text radiology reports were annotated by a cohort of radiologists, two attending physicians and three residents, to establish a reference standard. Of 100 chest radiographs, 50 were randomly selected from the National Institutes of Health (NIH) chest radiographic data set, and 50 were randomly selected from the Medical Imaging and Data Resource Center (MIDRC). The performance of GPT-4V at detecting imaging findings from each chest radiograph was assessed in the zero-shot setting (where it operates without prior examples) and few-shot setting (where it operates with two examples). Its outcomes were compared with the reference standard with regards to clinical conditions and their corresponding codes in the International Statistical Classification of Diseases, Tenth Revision (ICD-10), including the anatomic location (hereafter, laterality). Results In the zero-shot setting, in the task of detecting ICD-10 codes alone, GPT-4V attained an average positive predictive value (PPV) of 12.3%, average true-positive rate (TPR) of 5.8%, and average F1 score of 7.3% on the NIH data set, and an average PPV of 25.0%, average TPR of 16.8%, and average F1 score of 18.2% on the MIDRC data set. When both the ICD-10 codes and their corresponding laterality were considered, GPT-4V produced an average PPV of 7.8%, average TPR of 3.5%, and average F1 score of 4.5% on the NIH data set, and an average PPV of 10.9%, average TPR of 4.9%, and average F1 score of 6.4% on the MIDRC data set. With few-shot learning, GPT-4V showed improved performance on both data sets. When contrasting zero-shot and few-shot learning, there were improved average TPRs and F1 scores in the few-shot setting, but there was not a substantial increase in the average PPV. Conclusion Although GPT-4V has shown promise in understanding natural images, it had limited effectiveness in interpreting real-world chest radiographs. © RSNA, 2024 Supplemental material is available for this article.

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve. RESULTS: Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth. CONCLUSIONS: To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI.

Despite the surge in artificial intelligence (AI) development for health care applications, particularly for medical imaging applications, there has been limited adoption of such AI tools into clinical practice. During a 1-day workshop in November 2022, co-organized by the ACR and the RSNA, participants outlined experiences and problems with implementing AI in clinical practice, defined the needs of various stakeholders in the AI ecosystem, and elicited potential solutions and strategies related to the safety, effectiveness, reliability, and transparency of AI algorithms. Participants included radiologists from academic and community radiology practices, informatics leaders responsible for AI implementation, regulatory agency employees, and specialty society representatives. The major themes that emerged fell into two categories: (1) AI product development and (2) implementation of AI-based applications in clinical practice. In particular, participants highlighted key aspects of AI product development to include clear clinical task definitions; well-curated data from diverse geographic, economic, and health care settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings. For implementation, participants emphasized the need for strong institutional governance; systematic evaluation, selection, and validation methods conducted by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement. Participants predicted that clinical implementation of AI in radiology will continue to be limited until the safety, effectiveness, reliability, and transparency of such tools are more fully addressed.

Performance of the Winning Algorithms of the RSNA 2022 Cervical Spine Fracture Detection Challenge.

Purpose To evaluate and report the performance of the winning algorithms of the Radiological Society of North America Cervical Spine Fracture AI Challenge. Materials and Methods The competition was open to the public on Kaggle from July 28 to October 27, 2022. A sample of 3112 CT scans with and without cervical spine fractures (CSFx) were assembled from multiple sites (12 institutions across six continents) and prepared for the competition. The test set had 1093 scans (private test set: n = 789; mean age, 53.40 years ± 22.86 [SD]; 509 males; public test set: n = 304; mean age, 52.51 years ± 20.73; 189 males) and 847 fractures. The eight top-performing artificial intelligence (AI) algorithms were retrospectively evaluated, and the area under the receiver operating characteristic curve (AUC) value, F1 score, sensitivity, and specificity were calculated. Results A total of 1108 contestants composing 883 teams worldwide participated in the competition. The top eight AI models showed high performance, with a mean AUC value of 0.96 (95% CI: 0.95, 0.96), mean F1 score of 90% (95% CI: 90%, 91%), mean sensitivity of 88% (95% Cl: 86%, 90%), and mean specificity of 94% (95% CI: 93%, 96%). The highest values reported for previous models were an AUC of 0.85, F1 score of 81%, sensitivity of 76%, and specificity of 97%. Conclusion The competition successfully facilitated the development of AI models that could detect and localize CSFx on CT scans with high performance outcomes, which appear to exceed known values of previously reported models. Further study is needed to evaluate the generalizability of these models in a clinical environment. Keywords: Cervical Spine, Fracture Detection, Machine Learning, Artificial Intelligence Algorithms, CT, Head/Neck Supplemental material is available for this article. © RSNA, 2024.

Data Liberation and Crowdsourcing in Medical Research: The Intersection of Collective and Artificial Intelligence.

In spite of an exponential increase in the volume of medical data produced globally, much of these data are inaccessible to those who might best use them to develop improved health care solutions through the application of advanced analytics such as artificial intelligence. Data liberation and crowdsourcing represent two distinct but interrelated approaches to bridging existing data silos and accelerating the pace of innovation internationally. In this article, we examine these concepts in the context of medical artificial intelligence research, summarizing their potential benefits, identifying potential pitfalls, and ultimately making a case for their expanded use going forward. A practical example of a crowdsourced competition using an international medical imaging dataset is provided. Keywords: Artificial Intelligence, Data Liberation, Crowdsourcing © RSNA, 2023.

Harmonization of multi-site diffusion tensor imaging data for cervical and thoracic spinal cord at 1.5 T and 3 T using longitudinal ComBat.

MRI scanner hardware, field strengths, and sequence parameters are major variables in diffusion studies of the spinal cord. Reliability between scanners is not well known, particularly for the thoracic cord. DTI data was collected for the entire cervical and thoracic spinal cord in thirty healthy adult subjects with different MR vendors and field strengths. DTI metrics were extracted and averaged for all slices within each vertebral level. Metrics were examined for variability and then harmonized using longitudinal ComBat (longComBat). Four scanners were used: Siemens 3 T Prisma, Siemens 1.5 T Avanto, Philips 3 T Ingenia, Philips 1.5 T Achieva. Average full cord diffusion values/standard deviation for all subjects and scanners were FA: 0.63, σ = 0.10, MD: 1.11, σ = 0.12 × 10-3 mm2/s, AD: 1.98, σ = 0.55 × 10-3 mm2/s, RD: 0.67, σ = 0.31 × 10-3 mm2/s. FA metrics averaged for all subjects by level were relatively consistent across scanners, but large variability was found in diffusivity measures. Coefficients of variation were lowest in the cervical region, and relatively lower for FA than diffusivity measures. Harmonized metrics showed greatly improved agreement between scanners. Variability in DTI of the spinal cord arises from scanner hardware differences, pulse sequence differences, physiological motion, and subject compliance. The use of longComBat resulted in large improvement in agreement of all DTI metrics between scanners. This study shows the importance of harmonization of diffusion data in the spinal cord and potential for longitudinal and multisite clinical research and clinical trials.

The RSNA Cervical Spine Fracture CT Dataset.

This dataset is composed of cervical spine CT images with annotations related to fractures; it is available at https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/.

The ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2023: Intracranial Meningioma.

Meningiomas are the most common primary intracranial tumor in adults and can be associated with significant morbidity and mortality. Radiologists, neurosurgeons, neuro-oncologists, and radiation oncologists rely on multiparametric MRI (mpMRI) for diagnosis, treatment planning, and longitudinal treatment monitoring; yet automated, objective, and quantitative tools for non-invasive assessment of meningiomas on mpMRI are lacking. The BraTS meningioma 2023 challenge will provide a community standard and benchmark for state-of-the-art automated intracranial meningioma segmentation models based on the largest expert annotated multilabel meningioma mpMRI dataset to date. Challenge competitors will develop automated segmentation models to predict three distinct meningioma sub-regions on MRI including enhancing tumor, non-enhancing tumor core, and surrounding nonenhancing T2/FLAIR hyperintensity. Models will be evaluated on separate validation and held-out test datasets using standardized metrics utilized across the BraTS 2023 series of challenges including the Dice similarity coefficient and Hausdorff distance. The models developed during the course of this challenge will aid in incorporation of automated meningioma MRI segmentation into clinical practice, which will ultimately improve care of patients with meningioma.

Radiomic signatures of meningiomas using the Ki-67 proliferation index as a prognostic marker of clinical outcomes.

OBJECTIVE: The clinical behavior of meningiomas is not entirely captured by its designated WHO grade, therefore other factors must be elucidated that portend increased tumor aggressiveness and associated risk of recurrence. In this study, the authors identify multiparametric MRI radiomic signatures of meningiomas using Ki-67 as a prognostic marker of clinical outcomes independent of WHO grade. METHODS: A retrospective analysis was conducted of all resected meningiomas between 2012 and 2018. Preoperative MR images were used for high-throughput radiomic feature extraction and subsequently used to develop a machine learning algorithm to stratify meningiomas based on Ki-67 indices < 5% and ≥ 5%, independent of WHO grade. Progression-free survival (PFS) was assessed based on machine learning prediction of Ki-67 strata and compared with outcomes based on histopathological Ki-67. RESULTS: Three hundred forty-three meningiomas were included: 291 with WHO grade I, 43 with grade II, and 9 with grade III. The overall rate of recurrence was 19.8% (15.1% in grade I, 44.2% in grade II, and 77.8% in grade III) over a median follow-up of 28.5 months. Grade II and III tumors had higher Ki-67 indices than grade I tumors, albeit tumor and peritumoral edema volumes had considerable variation independent of meningioma WHO grade. Forty-six high-performing radiomic features (1 morphological, 7 intensity-based, and 38 textural) were identified and used to build a support vector machine model to stratify tumors based on a Ki-67 cutoff of 5%, with resultant areas under the curve of 0.83 (95% CI 0.78-0.89) and 0.84 (95% CI 0.75-0.94) achieved for the discovery (n = 257) and validation (n = 86) data sets, respectively. Comparison of histopathological Ki-67 versus machine learning-predicted Ki-67 showed excellent performance (overall accuracy > 80%), with classification of grade I meningiomas exhibiting the greatest accuracy. Prediction of Ki-67 by machine learning classifier revealed shorter PFS for meningiomas with Ki-67 indices ≥ 5% compared with tumors with Ki-67 < 5% (p < 0.0001, log-rank test), which corroborates divergent patient outcomes observed using histopathological Ki-67. CONCLUSIONS: The Ki-67 proliferation index may serve as a surrogate marker of increased meningioma aggressiveness independent of WHO grade. Machine learning using radiomic feature analysis may be used for the preoperative prediction of meningioma Ki-67, which provides enhanced analytical insights to help improve diagnostic classification and guide patient-specific treatment strategies.

Soluble Nogo-Receptor-Fc decoy (AXER-204) in patients with chronic cervical spinal cord injury in the USA: a first-in-human and randomised clinical trial.

BACKGROUND: Spinal cord injury (SCI) causes neural disconnection and persistent neurological deficits, so axon sprouting and plasticity might promote recovery. Soluble Nogo-Receptor-Fc decoy (AXER-204) blocks inhibitors of axon growth and promotes recovery of motor function after SCI in animals. This first-in-human and randomised trial sought to determine primarily the safety and pharmacokinetics of AXER-204 in individuals with chronic SCI, and secondarily its effect on recovery. METHODS: We conducted a two-part study in adults (aged 18-65 years) with chronic (>1 year) cervical traumatic SCI at six rehabilitation centres in the USA. In part 1, AXER-204 was delivered open label as single intrathecal doses of 3 mg, 30 mg, 90 mg, or 200 mg, with primary outcomes of safety and pharmacokinetics. Part 2 was a randomised, parallel, double-blind comparison of six intrathecal doses of 200 mg AXER-204 over 104 days versus placebo. Participants were randomly allocated (1:1) by investigators using a central electronic system, stratified in blocks of four by American Spinal Injury Association Impairment Scale grade and receipt of AXER-204 in part 1. All investigators and patients were masked to treatment allocation until at least day 169. The part 2 primary objectives were safety and pharmacokinetics, with a key secondary objective to assess change in International Standards for Neurological Classification of SCI (ISNCSCI) Upper Extremity Motor Score (UEMS) at day 169 for all enrolled participants. This trial is registered with ClinicalTrials.gov, NCT03989440, and is completed. FINDINGS: We treated 24 participants in part 1 (six per dose; 18 men, six women), and 27 participants in part 2 (13 placebo, 14 AXER-204; 23 men, four women), between June 20, 2019, and June 21, 2022. There were no deaths and no discontinuations from the study due to an adverse event in part 1 and 2. In part 2, treatment-related adverse events were of similar incidence in AXER-204 and placebo groups (ten [71%] vs nine [69%]). Headache was the most common treatment-related adverse event (five [21%] in part 1, 11 [41%] in part 2). In part 1, AXER-204 reached mean maximal CSF concentration 1 day after dosing with 200 mg of 412 000 ng/mL (SD 129 000), exceeding those concentrations that were efficacious in animal studies. In part 2, mean changes from baseline to day 169 in ISNCSCI UEMS were 1·5 (SD 3·3) for AXER-204 and 0·9 (2·3) for placebo (mean difference 0·54, 95% CI -1·48 to 2·55; p=0·59). INTERPRETATION: This study delivers the first, to our knowledge, clinical trial of a rationally designed pharmacological treatment intended to promote neural repair in chronic SCI. AXER-204 appeared safe and reached target CSF concentrations; exploratory biomarker results were consistent with target engagement and synaptic stabilisation. Post-hoc subgroup analyses suggest that future trials could investigate efficacy in patients with moderately severe SCI without prior AXER-204 exposure. FUNDING: Wings for Life Foundation, National Institute of Neurological Disorders and Stroke, National Center for Advancing Translational Sciences, National Institute on Drug Abuse, and ReNetX Bio.