Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA.

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever­growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi­society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Deep Learning to Simulate Contrast-enhanced Breast MRI of Invasive Breast Cancer.

Background There is increasing interest in noncontrast breast MRI alternatives for tumor visualization to increase the accessibility of breast MRI. Purpose To evaluate the feasibility and accuracy of generating simulated contrast-enhanced T1-weighted breast MRI scans from precontrast MRI sequences in biopsy-proven invasive breast cancer with use of deep learning. Materials and Methods Women with invasive breast cancer and a contrast-enhanced breast MRI examination that was performed for initial evaluation of the extent of disease between January 2015 and December 2019 at a single academic institution were retrospectively identified. A three-dimensional, fully convolutional deep neural network simulated contrast-enhanced T1-weighted breast MRI scans from five precontrast sequences (T1-weighted non-fat-suppressed [FS], T1-weighted FS, T2-weighted FS, apparent diffusion coefficient, and diffusion-weighted imaging). For qualitative assessment, four breast radiologists (with 3-15 years of experience) blinded to whether the method of contrast was real or simulated assessed image quality (excellent, acceptable, good, poor, or unacceptable), presence of tumor enhancement, and maximum index mass size by using 22 pairs of real and simulated contrast-enhanced MRI scans. Quantitative comparison was performed using whole-breast similarity and error metrics and Dice coefficient analysis of enhancing tumor overlap. Results Ninety-six MRI examinations in 96 women (mean age, 52 years ± 12 [SD]) were evaluated. The readers assessed all simulated MRI scans as having the appearance of a real MRI scan with tumor enhancement. Index mass sizes on real and simulated MRI scans demonstrated good to excellent agreement (intraclass correlation coefficient, 0.73-0.86; P < .001) without significant differences (mean differences, -0.8 to 0.8 mm; P = .36-.80). Almost all simulated MRI scans (84 of 88 [95%]) were considered of diagnostic quality (ratings of excellent, acceptable, or good). Quantitative analysis demonstrated strong similarity (structural similarity index, 0.88 ± 0.05), low voxel-wise error (symmetric mean absolute percent error, 3.26%), and Dice coefficient of enhancing tumor overlap of 0.75 ± 0.25. Conclusion It is feasible to generate simulated contrast-enhanced breast MRI scans with use of deep learning. Simulated and real contrast-enhanced MRI scans demonstrated comparable tumor sizes, areas of tumor enhancement, and image quality without significant qualitative or quantitative differences. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Slanetz in this issue. An earlier incorrect version appeared online. This article was corrected on January 17, 2023.

Effect of an ultrasound-first clinical decision tool in emergency department patients with suspected nephrolithiasis: A randomized trial.

INTRODUCTION: Previously, we found that the use of ultrasonography for patients with suspected nephrolithiasis resulted in similar outcomes and less radiation exposure vs. CT scan. In this study, we evaluated the implementation of an ultrasound-first clinical decision support (CDS) tool in patients with suspected nephrolithiasis. METHODS: This randomized trial was conducted at an academic emergency department (ED). We implemented the ultrasound-first CDS tool, deployed when an ED provider placed a CT order for suspected nephrolithiasis. Providers were randomized to receiving the CDS tool vs. usual care. The primary outcome was receipt of CT during the index ED visit. Secondary outcomes included radiation dose and ED revisit. RESULTS: 64 ED Providers and 254 patients with suspected nephrolithiasis were enrolled from January 2019 through Dec 2020. The US-First CDS tool was deployed for 128 patients and was not deployed for 126 patients. 86.7% of patients in the CDS arm received a CT vs. 94.4% in the usual care arm, resulting in an absolute risk difference of -7.7% (-14.8 to -0.6%). Mean radiation dose in the CDS arm was 6.8 mSv (95% CI 5.7-7.9 mSv) vs. 6.1 mSv (95% CI 5.1-7.1 mSv) in the usual care arm. The CDS arm did not result in increased ED revisits, CT scans, or hospitalizations at 7 or 30 days. CONCLUSIONS AND RELEVANCE: Implementation of the US-first CDS tool resulted in lower CT use for ED patients with suspected nephrolithiasis. The use of this decision support may improve the evaluation of a common problem in the ED. TRIAL REGISTRATION: ClinicalTrials.gov#NCT03461536.

The RSNA International COVID-19 Open Radiology Database (RICORD).

The coronavirus disease 2019 (COVID-19) pandemic is a global health care emergency. Although reverse-transcription polymerase chain reaction testing is the reference standard method to identify patients with COVID-19 infection, chest radiography and CT play a vital role in the detection and management of these patients. Prediction models for COVID-19 imaging are rapidly being developed to support medical decision making. However, inadequate availability of a diverse annotated data set has limited the performance and generalizability of existing models. To address this unmet need, the RSNA and Society of Thoracic Radiology collaborated to develop the RSNA International COVID-19 Open Radiology Database (RICORD). This database is the first multi-institutional, multinational, expert-annotated COVID-19 imaging data set. It is made freely available to the machine learning community as a research and educational resource for COVID-19 chest imaging. Pixel-level volumetric segmentation with clinical annotations was performed by thoracic radiology subspecialists for all COVID-19-positive thoracic CT scans. The labeling schema was coordinated with other international consensus panels and COVID-19 data annotation efforts, the European Society of Medical Imaging Informatics, the American College of Radiology, and the American Association of Physicists in Medicine. Study-level COVID-19 classification labels for chest radiographs were annotated by three radiologists, with majority vote adjudication by board-certified radiologists. RICORD consists of 240 thoracic CT scans and 1000 chest radiographs contributed from four international sites. It is anticipated that RICORD will ideally lead to prediction models that can demonstrate sustained performance across populations and health care systems.

What Physicians and Health Organizations Should Know About Mandated Imaging Appropriate Use Criteria.

The Appropriate Use Criteria Program, enacted by the Centers for Medicare & Medicaid Services in response to the Protecting Access to Medicare Act of 2014 (PAMA), aims to reduce inappropriate and unnecessary imaging by mandating use of clinical decision support (CDS) by all providers who order advanced imaging examinations (magnetic resonance imaging; computed tomography; and nuclear medicine studies, including positron emission tomography). Beginning 1 January 2020, documentation of an interaction with a certified CDS system using approved appropriate use criteria will be required on all Medicare claims for advanced imaging in all emergency department patients and outpatients as a prerequisite for payment. The Appropriate Use Criteria Program will initially cover 8 priority clinical areas, including several (such as headache and low back pain) commonly encountered by internal medicine providers. All providers and organizations that order and provide advanced imaging must understand program requirements and their options for compliance strategies. Substantial resources and planning will be needed to comply with PAMA regulations and avoid unintended negative consequences on workflow and payments. However, robust evidence supporting the desired outcome of reducing inappropriate use of advanced imaging is lacking.

Effect of shelter-in-place on emergency department radiology volumes during the COVID-19 pandemic.

PURPOSE: The coronavirus disease 2019 (COVID-19) pandemic has led to significant disruptions in the healthcare system including surges of infected patients exceeding local capacity, closures of primary care offices, and delays of non-emergent medical care. Government-initiated measures to decrease healthcare utilization (i.e., "flattening the curve") have included shelter-in-place mandates and social distancing, which have taken effect across most of the USA. We evaluate the immediate impact of the Public Health Messaging and shelter-in-place mandates on Emergency Department (ED) demand for radiology services. METHODS: We analyzed ED radiology volumes from the five University of California health systems during a 2-week time period following the shelter-in-place mandate and compared those volumes with March 2019 and early April 2019 volumes. RESULTS: ED radiology volumes declined from the 2019 baseline by 32 to 40% (p < 0.001) across the five health systems with a total decrease in volumes across all 5 systems by 35% (p < 0.001). Stratifying by subspecialty, the smallest declines were seen in non-trauma thoracic imaging, which decreased 18% (p value < 0.001), while all other non-trauma studies decreased by 48% (p < 0.001). CONCLUSION: Total ED radiology demand may be a marker for public adherence to shelter-in-place mandates, though ED chest radiology demand may increase with an increase in COVID-19 cases.

Potential Effect of the Protecting Access to Medicare Act on Use of Advanced Diagnostic Imaging in the Emergency Department: An Analysis of the National Hospital Ambulatory Care Survey.

Background Clinical decision support is increasingly used to enhance clinicians' exposure to established evidence and patient information during an episode of patient care. Pending legislation specifies clinical decision support before performing advanced imaging at emergency department (ED) visits. Purpose To estimate the volume of advanced imaging tests (CT and MRI) that would require use of clinical decision support to achieve Protecting Access to Medicare Act (PAMA) compliance in the ED. Materials and Methods A retrospective, cross-sectional analysis of ED visits was conducted by using data from the 2012-2015 National Hospital Ambulatory Care Survey. PAMA-related visits were identified by selecting the patient reasons for visit (RFVs) related to the eight clinical conditions. Results Among the adult ED visits, 26.7% (20 506 of 77 299, representing 113 000 000 visits across 4 years, or 28 000 000 visits annually) patients presented with a RFV consistent with a PAMA priority clinical area (PCA). Among visits in which a patient described an RFV code consistent with a PAMA PCA, up to 22.9% (4681 of 20 506; 95% confidence interval: 21.8%, 24.1%) patients underwent advanced imaging, translating to approximately 6 000 000 visits annually. Conclusion Protecting Access to Medicare Act legislation targets eight priority clinical areas, estimated to be prevalent among one in four adult emergency department visits. CT and/or MRI studies are performed during up to 23% of these visits. Depending on the particular clinical decision support systems selected within a health system, and how they are implemented, the potential volume of studies in which clinicians must interact with clinical decision support system may either exceed or fall short of these estimates. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Forman in this issue.

Automated detection of moderate and large pneumothorax on frontal chest X-rays using deep convolutional neural networks: A retrospective study.

BACKGROUND: Pneumothorax can precipitate a life-threatening emergency due to lung collapse and respiratory or circulatory distress. Pneumothorax is typically detected on chest X-ray; however, treatment is reliant on timely review of radiographs. Since current imaging volumes may result in long worklists of radiographs awaiting review, an automated method of prioritizing X-rays with pneumothorax may reduce time to treatment. Our objective was to create a large human-annotated dataset of chest X-rays containing pneumothorax and to train deep convolutional networks to screen for potentially emergent moderate or large pneumothorax at the time of image acquisition. METHODS AND FINDINGS: In all, 13,292 frontal chest X-rays (3,107 with pneumothorax) were visually annotated by radiologists. This dataset was used to train and evaluate multiple network architectures. Images showing large- or moderate-sized pneumothorax were considered positive, and those with trace or no pneumothorax were considered negative. Images showing small pneumothorax were excluded from training. Using an internal validation set (n = 1,993), we selected the 2 top-performing models; these models were then evaluated on a held-out internal test set based on area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and positive predictive value (PPV). The final internal test was performed initially on a subset with small pneumothorax excluded (as in training; n = 1,701), then on the full test set (n = 1,990), with small pneumothorax included as positive. External evaluation was performed using the National Institutes of Health (NIH) ChestX-ray14 set, a public dataset labeled for chest pathology based on text reports. All images labeled with pneumothorax were considered positive, because the NIH set does not classify pneumothorax by size. In internal testing, our "high sensitivity model" produced a sensitivity of 0.84 (95% CI 0.78-0.90), specificity of 0.90 (95% CI 0.89-0.92), and AUC of 0.94 for the test subset with small pneumothorax excluded. Our "high specificity model" showed sensitivity of 0.80 (95% CI 0.72-0.86), specificity of 0.97 (95% CI 0.96-0.98), and AUC of 0.96 for this set. PPVs were 0.45 (95% CI 0.39-0.51) and 0.71 (95% CI 0.63-0.77), respectively. Internal testing on the full set showed expected decreased performance (sensitivity 0.55, specificity 0.90, and AUC 0.82 for high sensitivity model and sensitivity 0.45, specificity 0.97, and AUC 0.86 for high specificity model). External testing using the NIH dataset showed some further performance decline (sensitivity 0.28-0.49, specificity 0.85-0.97, and AUC 0.75 for both). Due to labeling differences between internal and external datasets, these findings represent a preliminary step towards external validation. CONCLUSIONS: We trained automated classifiers to detect moderate and large pneumothorax in frontal chest X-rays at high levels of performance on held-out test data. These models may provide a high specificity screening solution to detect moderate or large pneumothorax on images collected when human review might be delayed, such as overnight. They are not intended for unsupervised diagnosis of all pneumothoraces, as many small pneumothoraces (and some larger ones) are not detected by the algorithm. Implementation studies are warranted to develop appropriate, effective clinician alerts for the potentially critical finding of pneumothorax, and to assess their impact on reducing time to treatment.

Impact of PACS-EMR Integration on Radiologist Usage of the EMR.

The purpose of this study was to objectively quantify the impact of implementing picture archiving and communication system-electronic medical record (PACS-EMR) integration on the time required to access data in the EMR and the frequency with which data are accessed by radiologists. Time to access a clinic note in the EMR was measured before and after integration with a stopwatch and compared by t test. An IRB-approved, HIPAA-compliant retrospective review of EMR access data from security audit logs was conducted for a 14-month period spanning the integration. Correlation of these data with report signatures identified the studies in which the radiologist accessed the EMR to obtain additional clinical data. Proportions of studies with EMR access were plotted and compared before and after integration using a chi-square test. Time to access the EMR decreased from 52 to 6 s (p < 0.001). Proportion of studies with EMR access increased from 36.7% (10,175/27,773) to 44.9% (10,843/24,153) after integration (p < 0.001). Integrating PACS and the EMR substantially decreases the time to access the EMR and is associated with a significant increase in the proportion of studies for which radiologists obtain additional clinical data.

Feasibility of Antegrade Contrast-enhanced US Nephrostograms to Evaluate Ureteral Patency.

Purpose To demonstrate the feasibility of contrast material-enhanced ulrasonographic (US) nephrostograms to assess ureteral patency after percutaneous nephrolithotomy (PCNL) in this proof-of-concept study. Materials and Methods For this HIPAA-compliant, institutional review board-approved prospective blinded pilot study, patients undergoing PCNL provided consent to undergo contrast-enhanced US and fluoroscopic nephrostograms on postoperative day 1. For contrast-enhanced US, 1.5 mL of Optison (GE Healthcare, Oslo, Norway) microbubble contrast agent solution (perflutren protein-type A microspheres) was injected via the nephrostomy tube. Unobstructed antegrade ureteral flow was defined by the presence of contrast material in the bladder. Contrast-enhanced US results were compared against those of fluoroscopic nephrostograms for concordance. Results Ten studies were performed in nine patients (four women, five men). Contrast-enhanced US demonstrated ureteral patency in eight studies and obstruction in two. One patient underwent two studies, one showing obstruction and the second showing patency. Concordance between US and fluoroscopic assessments of ureteral patency was evaluated by using a Clopper-Pearson exact binomial test. These results were perfectly concordant with fluoroscopic nephrostogram results, with a 95% confidence interval of 69.2% and 100%. No complications or adverse events related to contrast-enhanced US occurred. Conclusion Contrast-enhanced US nephrostograms are simple to perform and are capable of demonstrating both patency and obstruction of the ureter. The perfect concordance with fluoroscopic results across 10 studies demonstrated here is not sufficient to establish diagnostic accuracy of this technique, but motivates further, larger scale investigation. If subsequent larger studies confirm these preliminary results, contrast-enhanced US may provide a safer, more convenient way to evaluate ureteral patency than fluoroscopy. © RSNA, 2016 Online supplemental material is available for this article.

Contrast Enhanced Ultrasound as a Radiation-Free Alternative to Fluoroscopic Nephrostogram for Evaluating Ureteral Patency.

PURPOSE: We compared contrast enhanced ultrasound and fluoroscopic nephrostography in the evaluation of ureteral patency following percutaneous nephrolithotomy. MATERIALS AND METHODS: This prospective cohort, noninferiority study was performed after obtaining institutional review board approval. We enrolled eligible patients with kidney and proximal ureteral stones who underwent percutaneous nephrolithotomy at our center. On postoperative day 1 patients received contrast enhanced ultrasound and fluoroscopic nephrostogram within 2 hours of each other to evaluate ureteral patency, which was the primary outcome of this study. RESULTS: A total of 92 pairs of imaging studies were performed in 82 patients during the study period. Five study pairs were excluded due to technical errors that prevented imaging interpretation. Females slightly predominated over males with a mean ± SD age of 50.5 ± 15.9 years and a mean body mass index of 29.6 ± 8.6 kg/m2. Of the remaining 87 sets of studies 69 (79.3%) demonstrated concordant findings regarding ureteral patency for the 2 imaging techniques and 18 (20.7%) were discordant. The nephrostomy tube was removed on the same day in 15 of the 17 patients who demonstrated antegrade urine flow only on contrast enhanced ultrasound and they had no subsequent adverse events. No adverse events were noted related to ultrasound contrast injection. While contrast enhanced ultrasound used no ionizing radiation, fluoroscopic nephrostograms provided a mean radiation exposure dose of 2.8 ± 3.7 mGy. CONCLUSIONS: A contrast enhanced ultrasound nephrostogram can be safely performed to evaluate for ureteral patency following percutaneous nephrolithotomy. This imaging technique was mostly concordant with fluoroscopic findings. Most discordance was likely attributable to the higher sensitivity for patency of contrast enhanced ultrasound compared to fluoroscopy.

Correlate: A PACS- and EHR-integrated Tool Leveraging Natural Language Processing to Provide Automated Clinical Follow-up.

A major challenge for radiologists is obtaining meaningful clinical follow-up information for even a small percentage of cases encountered and dictated. Traditional methods, such as keeping medical record number follow-up lists, discussing cases with rounding clinical teams, and discussing cases at tumor boards, are effective at keeping radiologists informed of clinical outcomes but are time intensive and provide follow-up for a small subset of cases. To this end, the authors developed a picture archiving and communication system-accessible electronic health record (EHR)-integrated program called Correlate, which allows the user to easily enter free-text search queries regarding desired clinical follow-up information, with minimal interruption to the workflow. The program uses natural language processing (NLP) to process the query and parse relevant future clinical data from the EHR. Results are ordered in terms of clinical relevance, and the user is e-mailed a link to results when these are available for viewing. A customizable personal database of queries and results is also maintained for convenient future access. Correlate aids radiologists in efficiently obtaining useful clinical follow-up information that can improve patient care, help keep radiologists integrated with other specialties and referring physicians, and provide valuable experiential learning. The authors briefly review the history of automated clinical follow-up tools and discuss the design and function of the Correlate program, which uses NLP to perform intelligent prospective searches of the EHR. © RSNA, 2017.

High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks.

The study aimed to determine if computer vision techniques rooted in deep learning can use a small set of radiographs to perform clinically relevant image classification with high fidelity. One thousand eight hundred eighty-five chest radiographs on 909 patients obtained between January 2013 and July 2015 at our institution were retrieved and anonymized. The source images were manually annotated as frontal or lateral and randomly divided into training, validation, and test sets. Training and validation sets were augmented to over 150,000 images using standard image manipulations. We then pre-trained a series of deep convolutional networks based on the open-source GoogLeNet with various transformations of the open-source ImageNet (non-radiology) images. These trained networks were then fine-tuned using the original and augmented radiology images. The model with highest validation accuracy was applied to our institutional test set and a publicly available set. Accuracy was assessed by using the Youden Index to set a binary cutoff for frontal or lateral classification. This retrospective study was IRB approved prior to initiation. A network pre-trained on 1.2 million greyscale ImageNet images and fine-tuned on augmented radiographs was chosen. The binary classification method correctly classified 100 % (95 % CI 99.73-100 %) of both our test set and the publicly available images. Classification was rapid, at 38 images per second. A deep convolutional neural network created using non-radiological images, and an augmented set of radiographs is effective in highly accurate classification of chest radiograph view type and is a feasible, rapid method for high-throughput annotation.

Beyond Prostate Adenocarcinoma: Expanding the Differential Diagnosis in Prostate Pathologic Conditions.

Recent advances in magnetic resonance (MR) imaging of the prostate gland have dramatically improved the ability to detect and stage adenocarcinoma of the prostate, one of the most frequently diagnosed cancers in men and one of the most frequently diagnosed pathologic conditions of the prostate gland. A wide variety of nonadenocarcinoma diseases can also be seen with MR imaging, ranging from benign to malignant diseases, as well as infectious and inflammatory manifestations. Many of these diseases have distinctive imaging features that allow differentiation from prostate acinar adenocarcinoma. Early recognition of these entities produces a more accurate differential diagnosis and may enable more expeditious clinical workup. Benign neoplasms of the prostate include plexiform neurofibroma and cystadenoma, both of which demonstrate distinctive imaging features. Stromal neoplasms of uncertain malignant potential are rare tumors of uncertain malignant potential that are often difficult to distinguish at imaging from more-malignant prostate sarcomas. Other malignant neoplasms of the prostate include urothelial carcinoma, primary prostatic carcinoid, carcinosarcoma, endometrioid or ductal adenocarcinoma, and mucinous adenocarcinoma. Prostatic infections can lead to abscesses of pyogenic, tuberculous, or fungal origins. Finally, miscellaneous idiopathic disorders of the prostate include amyloidosis, exophytic benign prostatic hyperplasia, and various congenital cysts. Considerable overlap can exist in the clinical history and imaging findings associated with these prostate pathologic conditions, and biopsy is often required for ultimate confirmation of the diagnosis. However, many diagnoses, including cystadenoma, mucinous adenocarcinoma, sarcoma, and abscesses, have distinct imaging features, which can enable the informed radiologist to identify the diagnosis and recommend appropriate clinical workup and management. (©)RSNA, 2016.

Age and sex-dependent trends in pulmonary embolism testing and derivation of a clinical decision rule for young patients.

IMPORTANCE: Despite low prevalence of pulmonary embolism (PE) in young adults, they are frequently imaged for PE, which involves radiation exposure and substantial financial cost. OBJECTIVE: Determine the use and positive proportions for PE imaging by age, differences in clinical presentation of PE by age and the projected impact of an age-targeted decision rule. DESIGN: Analysis of two national population-based datasets: the 2009 Nationwide Emergency Department Sample, a 20% sample of US emergency departments (EDs) and the 2003-2006 Pulmonary Embolism Rule-out Criteria (PERC) dataset, a multisite cohort of ED patients with suspected PE from 12 US EDs. RESULTS: Prevalence of PE was 10 times lower in young patients (18-35 years) than in older patients (>65 years) (0.06% vs 0.60%, p<0.001), but young patients were imaged for PE almost as frequently as older patients (2.3% vs 3.2%). This resulted in a lower proportion of positive examinations in young adults than older adults (2.3% vs 17.4%, p<0.001 in women; 4.0% vs 21.4%, p<0.001 in men). Clinical predictors of PE varied by age. Tachycardia was a significant predictor of PE in older patients (OR: 1.2-1.9, p<0.001), but not young patients. Fever was a significant predictor only in young patients (OR: 1.4-7.2, p<0.01). A modification of the previously described PERC rule to include age-specific risk factors could reduce PE imaging by 51% in young patients, with a missed PE rate of 0.6% in those excluded from imaging. CONCLUSIONS AND RELEVANCE: Young patients are frequently imaged for PE and have lower positive imaging rates than older patients. After further validation, application of our proposed rule for excluding young patients from PE imaging could reduce imaging, increase the positive rate of imaging and result in a low rate of missed PE among those excluded from imaging.

Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI.

Despite the surge in artificial intelligence (AI) development for health care applications, particularly for medical imaging applications, there has been limited adoption of such AI tools into clinical practice. During a 1-day workshop in November 2022, co-organized by the ACR and the RSNA, participants outlined experiences and problems with implementing AI in clinical practice, defined the needs of various stakeholders in the AI ecosystem, and elicited potential solutions and strategies related to the safety, effectiveness, reliability, and transparency of AI algorithms. Participants included radiologists from academic and community radiology practices, informatics leaders responsible for AI implementation, regulatory agency employees, and specialty society representatives. The major themes that emerged fell into two categories: (1) AI product development and (2) implementation of AI-based applications in clinical practice. In particular, participants highlighted key aspects of AI product development to include clear clinical task definitions; well-curated data from diverse geographic, economic, and health care settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings. For implementation, participants emphasized the need for strong institutional governance; systematic evaluation, selection, and validation methods conducted by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement. Participants predicted that clinical implementation of AI in radiology will continue to be limited until the safety, effectiveness, reliability, and transparency of such tools are more fully addressed.

The University of California San Francisco Adult Longitudinal Post-Treatment Diffuse Glioma MRI Dataset.

Supplemental material is available for this article.