Redesign of the American Board of Radiology Diagnostic Radiology Certifying Examination.

On April 13, 2023, the American Board of Radiology (ABR) announced plans to replace the current computer-based diagnostic radiology (DR) certifying examination with a new oral examination to be administered remotely, beginning in 2028. This article describes the planned changes and the process that led to those changes. In keeping with its commitment to continuous improvement, the ABR gathered input regarding the DR initial certification process. Respondents generally agreed that the qualifying (core) examination was satisfactory but expressed concerns regarding the computer-based certifying examination's effectiveness and impact on training. Examination redesign was conducted using input from key groups with a goal of effectively evaluating competence and incentivizing study behaviors that best prepare candidates for radiology practice. Major design elements included examination structure, breadth and depth of content, and timing. The new oral examination will focus on critical findings as well as common and important diagnoses routinely encountered in all diagnostic specialties, including radiology procedures. Candidates will first be eligible for the examination in the calendar year after residency graduation. Additional details will be finalized and announced in coming years. The ABR will continue to engage with interested parties throughout the implementation process.

Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How?

As the role of artificial intelligence (AI) in clinical practice evolves, governance structures oversee the implementation, maintenance, and monitoring of clinical AI algorithms to enhance quality, manage resources, and ensure patient safety. In this article, a framework is established for the infrastructure required for clinical AI implementation and presents a road map for governance. The road map answers four key questions: Who decides which tools to implement? What factors should be considered when assessing an application for implementation? How should applications be implemented in clinical practice? Finally, how should tools be monitored and maintained after clinical implementation? Among the many challenges for the implementation of AI in clinical practice, devising flexible governance structures that can quickly adapt to a changing environment will be essential to ensure quality patient care and practice improvement objectives.

Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.

Prospective Deployment of Deep Learning in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices.

Artificial intelligence algorithms based on principles of deep learning (DL) have made a large impact on the acquisition, reconstruction, and interpretation of MRI data. Despite the large number of retrospective studies using DL, there are fewer applications of DL in the clinic on a routine basis. To address this large translational gap, we review the recent publications to determine three major use cases that DL can have in MRI, namely, that of model-free image synthesis, model-based image reconstruction, and image or pixel-level classification. For each of these three areas, we provide a framework for important considerations that consist of appropriate model training paradigms, evaluation of model robustness, downstream clinical utility, opportunities for future advances, as well recommendations for best current practices. We draw inspiration for this framework from advances in computer vision in natural imaging as well as additional healthcare fields. We further emphasize the need for reproducibility of research studies through the sharing of datasets and software. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY STAGE: 2.

Needs of Referring Providers by Practice Type: Results of a Survey at an Academic Medical Center.

OBJECTIVE. The purpose of this study was to test a published hypothetic framework of different referring provider needs for primary care, specialty care, and urgent or emergency care practitioners through questions asked in an annual survey at an academic medical center. MATERIALS AND METHODS. Seven questions regarding provider needs were included in an annual online anonymous survey of referring providers. Multiple-choice response options were provided. Differences in responses between provider types were assessed using the Mann-Whitney U test. RESULTS. The survey was sent to 3325 providers, and 514 responses were received (response rate, 15.5%). The analysis included 340 responses: 81 from primary care, 205 from specialty care, and 54 from urgent or emergency care. Results indicated that urgent or emergency care providers need examinations to be performed and interpreted more quickly, specialist providers prefer greater radiologist specialization, urgent or emergency care providers order imaging with greater frequency, primary care and urgent or emergency care providers order a greater breadth of imaging, primary care providers report greater reliance on radiologist interpretations, and all provider types highly value direct interactions with radiologists. All results were statistically significant and matched established hypotheses. CONCLUSION. Our results support the concept that referring providers tend to value different aspects of radiology services differently, according to predictable characteristics. The findings suggest that the concept of value in radiology is highly context-specific and can be evaluated, at least in part, using practice-specific referring provider assessments.

Program for Supporting Frontline Improvement Projects in an Academic Radiology Department.

OBJECTIVE. The purpose of this study was to describe the results of an ongoing program implemented in an academic radiology department to support the execution of small- to medium-size improvement projects led by frontline staff and leaders. MATERIALS AND METHODS. Staff members were assigned a coach, were instructed in improvement methods, were given time to work on the project, and presented progress to department leaders in weekly 30-minute reports. Estimated costs and outcomes were calculated for each project and aggregated. An anonymous survey was administered to participants at the end of the first year. RESULTS. A total of 73 participants completed 102 projects in the first 2 years of the program. The project type mix included 25 quality improvement projects, 22 patient satisfaction projects, 14 staff engagement projects, 27 efficiency improvement projects, and 14 regulatory compliance and readiness projects. Estimated annualized outcomes included approximately 4500 labor hours saved, $315K in supply cost savings, $42.2M in potential increased revenues, 8- and 2-point increase in top-box patient experience scores at two clinics, and a 60-incident reduction in near-miss safety events. Participant time equated to approximately 0.35 full-time equivalent positions per year. Approximately 0.4 full-time equivalent was required to support the program. Survey results indicated that the participants generally viewed the program favorably. CONCLUSION. The program was successful in providing a platform for simultaneously solving a large number of organizational problems while also providing a positive experience to frontline personnel.

Increasing the Utilization of Moderate Sedation Services for Pediatric Imaging.

Performing motion-free imaging is frequently challenging in children. To bridge the gap between examinations performed in children who are awake and in those under general anesthesia, a moderate sedation program was implemented at our institution but was seldom used despite substantial eligibility. In conjunction with a 5-month quality improvement (QI) course, a multidisciplinary team was assembled and, by using an A3 approach, sought to address the most important key drivers of low utilization, namely the need for clear moderate sedation eligibility criteria, reliable protocol routing order, consistent moderate sedation screening performed by registered nurses (RNs), and enhanced visibility of moderate sedation services to ordering providers. Initial steps focused on developing better-defined criteria and protocoling standard work for technologists and RNs, with coaching and audits. Modality-specific forecasting was then implemented to reroute profiles of patients who were awaiting scheduling or already scheduled for an examination with general anesthesia to the moderate sedation queue to identify more eligible patients. These manual efforts were coupled with higher reliability but more protracted electronic health record changes, facilitating automated protocol routing on the basis of moderate sedation eligibility and order entry constraints. As a result, scheduled imaging examinations requiring moderate sedation increased from a mean of 1.2 examinations per week to a sustained 6.1 examinations per week (range, 4-8) over the 5-month period, exceeding the team SMART (specific, measurable, achievable, relevant, and time bound) goal to achieve an average of five examinations per week by the QI course end. By targeting the most high-impact yet modifiable process deficiencies through a multifaceted team approach and initially investing in manual efforts to gain cultural buy-in while awaiting higher-reliability interventions, the project achieved success and may serve as a more general model for workflow change when there is organizational resistance. ©RSNA, 2021.

Ethics of Using and Sharing Clinical Imaging Data for Artificial Intelligence: A Proposed Framework.

In this article, the authors propose an ethical framework for using and sharing clinical data for the development of artificial intelligence (AI) applications. The philosophical premise is as follows: when clinical data are used to provide care, the primary purpose for acquiring the data is fulfilled. At that point, clinical data should be treated as a form of public good, to be used for the benefit of future patients. In their 2013 article, Faden et al argued that all who participate in the health care system, including patients, have a moral obligation to contribute to improving that system. The authors extend that framework to questions surrounding the secondary use of clinical data for AI applications. Specifically, the authors propose that all individuals and entities with access to clinical data become data stewards, with fiduciary (or trust) responsibilities to patients to carefully safeguard patient privacy, and to the public to ensure that the data are made widely available for the development of knowledge and tools to benefit future patients. According to this framework, the authors maintain that it is unethical for providers to "sell" clinical data to other parties by granting access to clinical data, especially under exclusive arrangements, in exchange for monetary or in-kind payments that exceed costs. The authors also propose that patient consent is not required before the data are used for secondary purposes when obtaining such consent is prohibitively costly or burdensome, as long as mechanisms are in place to ensure that ethical standards are strictly followed. Rather than debate whether patients or provider organizations "own" the data, the authors propose that clinical data are not owned at all in the traditional sense, but rather that all who interact with or control the data have an obligation to ensure that the data are used for the benefit of future patients and society.

Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet.

BACKGROUND: Magnetic resonance imaging (MRI) of the knee is the preferred method for diagnosing knee injuries. However, interpretation of knee MRI is time-intensive and subject to diagnostic error and variability. An automated system for interpreting knee MRI could prioritize high-risk patients and assist clinicians in making diagnoses. Deep learning methods, in being able to automatically learn layers of features, are well suited for modeling the complex relationships between medical images and their interpretations. In this study we developed a deep learning model for detecting general abnormalities and specific diagnoses (anterior cruciate ligament [ACL] tears and meniscal tears) on knee MRI exams. We then measured the effect of providing the model's predictions to clinical experts during interpretation. METHODS AND FINDINGS: Our dataset consisted of 1,370 knee MRI exams performed at Stanford University Medical Center between January 1, 2001, and December 31, 2012 (mean age 38.0 years; 569 [41.5%] female patients). The majority vote of 3 musculoskeletal radiologists established reference standard labels on an internal validation set of 120 exams. We developed MRNet, a convolutional neural network for classifying MRI series and combined predictions from 3 series per exam using logistic regression. In detecting abnormalities, ACL tears, and meniscal tears, this model achieved area under the receiver operating characteristic curve (AUC) values of 0.937 (95% CI 0.895, 0.980), 0.965 (95% CI 0.938, 0.993), and 0.847 (95% CI 0.780, 0.914), respectively, on the internal validation set. We also obtained a public dataset of 917 exams with sagittal T1-weighted series and labels for ACL injury from Clinical Hospital Centre Rijeka, Croatia. On the external validation set of 183 exams, the MRNet trained on Stanford sagittal T2-weighted series achieved an AUC of 0.824 (95% CI 0.757, 0.892) in the detection of ACL injuries with no additional training, while an MRNet trained on the rest of the external data achieved an AUC of 0.911 (95% CI 0.864, 0.958). We additionally measured the specificity, sensitivity, and accuracy of 9 clinical experts (7 board-certified general radiologists and 2 orthopedic surgeons) on the internal validation set both with and without model assistance. Using a 2-sided Pearson's chi-squared test with adjustment for multiple comparisons, we found no significant differences between the performance of the model and that of unassisted general radiologists in detecting abnormalities. General radiologists achieved significantly higher sensitivity in detecting ACL tears (p-value = 0.002; q-value = 0.019) and significantly higher specificity in detecting meniscal tears (p-value = 0.003; q-value = 0.019). Using a 1-tailed t test on the change in performance metrics, we found that providing model predictions significantly increased clinical experts' specificity in identifying ACL tears (p-value < 0.001; q-value = 0.006). The primary limitations of our study include lack of surgical ground truth and the small size of the panel of clinical experts. CONCLUSIONS: Our deep learning model can rapidly generate accurate clinical pathology classifications of knee MRI exams from both internal and external datasets. Moreover, our results support the assertion that deep learning models can improve the performance of clinical experts during medical imaging interpretation. Further research is needed to validate the model prospectively and to determine its utility in the clinical setting.

Performance of a Deep-Learning Neural Network Model in Assessing Skeletal Maturity on Pediatric Hand Radiographs.

Purpose To compare the performance of a deep-learning bone age assessment model based on hand radiographs with that of expert radiologists and that of existing automated models. Materials and Methods The institutional review board approved the study. A total of 14 036 clinical hand radiographs and corresponding reports were obtained from two children's hospitals to train and validate the model. For the first test set, composed of 200 examinations, the mean of bone age estimates from the clinical report and three additional human reviewers was used as the reference standard. Overall model performance was assessed by comparing the root mean square (RMS) and mean absolute difference (MAD) between the model estimates and the reference standard bone ages. Ninety-five percent limits of agreement were calculated in a pairwise fashion for all reviewers and the model. The RMS of a second test set composed of 913 examinations from the publicly available Digital Hand Atlas was compared with published reports of an existing automated model. Results The mean difference between bone age estimates of the model and of the reviewers was 0 years, with a mean RMS and MAD of 0.63 and 0.50 years, respectively. The estimates of the model, the clinical report, and the three reviewers were within the 95% limits of agreement. RMS for the Digital Hand Atlas data set was 0.73 years, compared with 0.61 years of a previously reported model. Conclusion A deep-learning convolutional neural network model can estimate skeletal maturity with accuracy similar to that of an expert radiologist and to that of existing automated models. © RSNA, 2017 An earlier incorrect version of this article appeared online. This article was corrected on January 19, 2018.

Deep Learning to Classify Radiology Free-Text Reports.

Purpose To evaluate the performance of a deep learning convolutional neural network (CNN) model compared with a traditional natural language processing (NLP) model in extracting pulmonary embolism (PE) findings from thoracic computed tomography (CT) reports from two institutions. Materials and Methods Contrast material-enhanced CT examinations of the chest performed between January 1, 1998, and January 1, 2016, were selected. Annotations by two human radiologists were made for three categories: the presence, chronicity, and location of PE. Classification of performance of a CNN model with an unsupervised learning algorithm for obtaining vector representations of words was compared with the open-source application PeFinder. Sensitivity, specificity, accuracy, and F1 scores for both the CNN model and PeFinder in the internal and external validation sets were determined. Results The CNN model demonstrated an accuracy of 99% and an area under the curve value of 0.97. For internal validation report data, the CNN model had a statistically significant larger F1 score (0.938) than did PeFinder (0.867) when classifying findings as either PE positive or PE negative, but no significant difference in sensitivity, specificity, or accuracy was found. For external validation report data, no statistical difference between the performance of the CNN model and PeFinder was found. Conclusion A deep learning CNN model can classify radiology free-text reports with accuracy equivalent to or beyond that of an existing traditional NLP model. © RSNA, 2017 Online supplemental material is available for this article.

Improving and Maintaining Radiologic Technologist Skill Using a Medical Director Partnership and Technologist Coaching Model.

OBJECTIVE: Consistent excellence in radiologic technologist performance, including ensuring high technical image quality, patient safety and comfort, and efficient workflow, largely depends on individual technologist skill. However, sustained growth in the size and complexity of health care organizations has increased the difficulty in developing and maintaining technologist expertise. In this article, we explore underlying organizational structures that contribute to this problem and propose organizational models to promote continued excellence in technologist skill. CONCLUSION: We have found that a relatively modest investment in medical directorship combined with a coaching model can bring about a significant level of improvement in skilled clinical performance. We believe that widespread implementation of similar programs could contribute to substantial improvements in quality in radiology and other health care settings.

Practical Suggestions on How to Move From Peer Review to Peer Learning.

OBJECTIVE: The purpose of this article is to outline practical steps that a department can take to transition to a peer learning model. CONCLUSION: The 2015 Institute of Medicine report on improving diagnosis emphasized that organizations and industries that embrace error as an opportunity to learn tend to outperform those that do not. To meet this charge, radiology must transition from a peer review to a peer learning approach.

Improving Performance of Mammographic Breast Positioning in an Academic Radiology Practice.

OBJECTIVE: The purpose of this project was to achieve sustained improvement in mammographic breast positioning in our department. MATERIALS AND METHODS: Between June 2013 and December 2016, we conducted a team-based performance improvement initiative with the goal of improving mammographic positioning. The team of technologists and radiologists established quantitative measures of positioning performance based on American College of Radiology (ACR) criteria, audited at least 35 mammograms per week for positioning quality, displayed performance in dashboards, provided technologists with positioning training, developed a supportive environment fostering technologist and radiologist communication surrounding mammographic positioning, and employed a mammography positioning coach to develop, improve, and maintain technologist positioning performance. Statistical significance in changes in the percentage of mammograms passing the ACR criteria were evaluated using a two-proportion z test. RESULTS: A baseline mammogram audit performed in June 2013 showed that 67% (82/122) met ACR passing criteria for positioning. Performance improved to 80% (588/739; p < 0.01) after positioning training and technologist and radiologist agreement on positioning criteria. With individual technologist feedback, positioning further improved, with 91% of mammograms passing ACR criteria (p < 0.01). Seven months later, performance temporarily decreased to 80% but improved to 89% with implementation of a positioning coach. The overall mean performance of 91% has been sustained for 23 months. The program cost approximately $30,000 to develop, $42,000 to launch, and $25,000 per year to maintain. Almost all costs were related to personnel time. CONCLUSION: Dedicated performance improvement methods may achieve significant and sustained improvement in mammographic breast positioning, which may better enable facilities to pass the recently instated Enhancing Quality Using the Inspection Program portion of a practice's annual Mammography Quality Standards Act inspections.

Strategies for Implementing a Standardized Structured Radiology Reporting Program.

Radiology practices are increasingly implementing standardized report templates to overcome the drawbacks of individual templates. However, implementing a standardized structured reporting program is not necessarily straightforward. This article provides practical guidance for radiologists who wish to implement standardized structured reporting in their practice. Challenges that radiology groups encounter tend to fall into two categories: technical and organizational. Defining and carrying out technical work can be tedious but tends to be relatively straightforward, whereas overcoming organizational challenges often requires changes in individuals' strongly held values, beliefs, roles, and relationships. Established organizational change models can help frame the organizational strategy to implement a standardized structured reporting program. Once leadership support is secured, a standardized structured reporting committee can be convened to establish report priorities, standards, design principles, and guidelines. Report standards help to establish the common framework upon which all report templates are constructed, helping to ensure report consistency. By using these standards, committee members can create reports relevant to their subspecialties, which can then be edited for formatting and content. Once report templates have been developed, edited, and published, an abbreviated form of the same process can be used to maintain the reports, which can be accomplished with much less effort than that initially required to create the templates. After standardized structured report templates are implemented and become embedded in practice, most radiologists eventually appreciate the merits of the program. ©RSNA, 2018.

Peer Feedback, Learning, and Improvement: Answering the Call of the Institute of Medicine Report on Diagnostic Error.

In September 2015, the Institute of Medicine (IOM) published a report titled "Improving Diagnosis in Health Care," in which it was recommended that "health care organizations should adopt policies and practices that promote a nonpunitive culture that values open discussion and feedback on diagnostic performance." It may seem counterintuitive that a report addressing a highly technical skill such as medical diagnosis would be focused on organizational culture. The wisdom becomes clearer, however, when examined in the light of recent advances in the understanding of human error and individual and organizational performance. The current dominant model for radiologist performance improvement is scoring-based peer review, which reflects a traditional quality assurance approach, derived from manufacturing in the mid-1900s. Far from achieving the goals of the IOM, which are celebrating success, recognizing mistakes as an opportunity to learn, and fostering openness and trust, we have found that scoring-based peer review tends to drive radiologists inward, against each other, and against practice leaders. Modern approaches to quality improvement focus on using and enhancing interpersonal professional relationships to achieve and maintain high levels of individual and organizational performance. In this article, the authors review the recommendations set forth by the recent IOM report, discuss the science and theory that underlie several of those recommendations, and assess how well they fit with the current dominant approach to radiology peer review. The authors also offer an alternative approach to peer review: peer feedback, learning, and improvement (or more succinctly, "peer learning"), which they believe is better aligned with the principles promoted by the IOM. © RSNA, 2016.

The Role of Radiology in the Diagnostic Process: Information, Communication, and Teamwork.

OBJECTIVE: The diagnostic radiology process represents a partnership between clinical and radiology teams. As such, breakdowns in interpersonal interactions and communication can result in patient harm. CONCLUSION: We explore the role of radiology in the diagnostic process, focusing on key concepts of information and communication, as well as key interpersonal interactions of teamwork, collaboration, and collegiality, all based on trust. We propose 10 principles to facilitate effective information flow in the diagnostic process.