Improving Prostate MR Image Quality in Practice - Initial results from the ACR Prostate MR Image Quality Improvement Collaborative.

OBJECTIVE: Variability in prostate MRI quality is an increasingly recognized problem that negatively affects patient care. This report aims to describe the results and key learnings of the first cohort of the ACR Learning Network Prostate MR Image Quality Improvement Collaborative. METHODS: Teams from five organizations in the U.S. were trained on a structured improvement method. After reaching a consensus on image quality and auditing their images using the Prostate Imaging Quality (PI-QUAL) system, teams conducted a current state analysis to identify barriers to obtaining high-quality images. Through plan-do-study-act cycles involving frontline staff, each site designed and tested interventions targeting image quality key drivers. The percentage of exams meeting quality criteria (i.e., PI-QUAL score ≥ 4) was plotted on a run chart, and project progress was reviewed in weekly meetings. At the collaborative level, the goal was to increase the percentage of exams with PI-QUAL ≥ 4 to at least 85%. RESULTS: Across 2380 exams audited, the mean weekly rates of prostate MR exams meeting image quality criteria increased from 67% (range: 60-74%) at baseline to 87% (range: 80-97%) upon program completion. The most commonly employed interventions were MR protocol adjustments, development and implementation of patient preparation instructions, personell training and development of an auditing process mechanism. CONCLUSION: A Learning Network model, where organizations share knowledge and work together toward a common goal, can improve prostate MR image quality at multiple sites simultaneously. The inaugural cohort's key learnings provide a roadmap for improvement on a broader scale.

Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI.

Despite the surge in artificial intelligence (AI) development for health care applications, particularly for medical imaging applications, there has been limited adoption of such AI tools into clinical practice. During a 1-day workshop in November 2022, co-organized by the ACR and the RSNA, participants outlined experiences and problems with implementing AI in clinical practice, defined the needs of various stakeholders in the AI ecosystem, and elicited potential solutions and strategies related to the safety, effectiveness, reliability, and transparency of AI algorithms. Participants included radiologists from academic and community radiology practices, informatics leaders responsible for AI implementation, regulatory agency employees, and specialty society representatives. The major themes that emerged fell into two categories: (1) AI product development and (2) implementation of AI-based applications in clinical practice. In particular, participants highlighted key aspects of AI product development to include clear clinical task definitions; well-curated data from diverse geographic, economic, and health care settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings. For implementation, participants emphasized the need for strong institutional governance; systematic evaluation, selection, and validation methods conducted by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement. Participants predicted that clinical implementation of AI in radiology will continue to be limited until the safety, effectiveness, reliability, and transparency of such tools are more fully addressed.

A Vision for Global CT Radiation Dose Optimization.

The topic of CT radiation dose management is receiving renewed attention since the recent approval by CMS for new CT dose measures. Widespread variation in CT dose persists in practices across the world, suggesting that current dose optimization techniques are lacking. The author outlines a proposed strategy for facilitating global CT radiation dose optimization. CT radiation dose optimization can be defined as the routine use of CT scan parameters that consistently produce images just above the minimum threshold of acceptable image quality for a given clinical indication, accounting for relevant patient characteristics, using the most dose-efficient techniques available on the scanner. To accomplish this, an image quality-based target dose must be established for every protocol; for nonhead CT applications, these target dose values must be expressed as a function of patient size. As variation in outcomes is reduced, the dose targets can be decreased to more closely approximate the minimum image quality threshold. Maintaining CT radiation dose optimization requires a process control program, including measurement, evaluation, feedback, and control. This is best accomplished by local teams made up of radiologists, medical physicists, and technologists, supported with protected time and needed tools, including analytics and protocol management applications. Other stakeholders critical to facilitating CT radiation dose management include researchers, funding agencies, industry, regulators, accreditors, payers, and the ACR. Analogous coordinated approaches have transformed quality in other industries and can be the mechanism for achieving the universal goal of CT radiation dose optimization.

Perceptions of Data Set Experts on Important Characteristics of Health Data Sets Ready for Machine Learning: A Qualitative Study.

Importance: The lack of data quality frameworks to guide the development of artificial intelligence (AI)-ready data sets limits their usefulness for machine learning (ML) research in health care and hinders the diagnostic excellence of developed clinical AI applications for patient care. Objective: To discern what constitutes high-quality and useful data sets for health and biomedical ML research purposes according to subject matter experts. Design, Setting, and Participants: This qualitative study interviewed data set experts, particularly those who are creators and ML researchers. Semistructured interviews were conducted in English and remotely through a secure video conferencing platform between August 23, 2022, and January 5, 2023. A total of 93 experts were invited to participate. Twenty experts were enrolled and interviewed. Using purposive sampling, experts were affiliated with a diverse representation of 16 health data sets/databases across organizational sectors. Content analysis was used to evaluate survey information and thematic analysis was used to analyze interview data. Main Outcomes and Measures: Data set experts' perceptions on what makes data sets AI ready. Results: Participants included 20 data set experts (11 [55%] men; mean [SD] age, 42 [11] years), of whom all were health data set creators, and 18 of the 20 were also ML researchers. Themes (3 main and 11 subthemes) were identified and integrated into an AI-readiness framework to show their association within the health data ecosystem. Participants partially determined the AI readiness of data sets using priority appraisal elements of accuracy, completeness, consistency, and fitness. Ethical acquisition and societal impact emerged as appraisal considerations in that participant samples have not been described to date in prior data quality frameworks. Factors that drive creation of high-quality health data sets and mitigate risks associated with data reuse in ML research were also relevant to AI readiness. The state of data availability, data quality standards, documentation, team science, and incentivization were associated with elements of AI readiness and the overall perception of data set usefulness. Conclusions and Relevance: In this qualitative study of data set experts, participants contributed to the development of a grounded framework for AI data set quality. Data set AI readiness required the concerted appraisal of many elements and the balancing of transparency and ethical reflection against pragmatic constraints. The movement toward more reliable, relevant, and ethical AI and ML applications for patient care will inevitably require strategic updates to data set creation practices.

Establishing a Point-of-Care Ultrasound Program: An Institutional Approach for Developing a Point-of-Care Ultrasound Program Infrastructure.

Point-of-care ultrasound (POCUS) is rapidly accelerating in adoption and applications outside the traditional realm of diagnostic radiology departments. Although the use of this imaging technology in a distributed fashion has great potential, there are many associated challenges. To address these challenges, the authors developed an enterprise-wide POCUS program at their institution (Stanford Health Care). Here, the authors share their experience, the governance organization, and their approaches to device and information security, training, and quality assurance. The authors also share the basic principles they use to guide their approach to manage these challenges. Through their work, the authors have learned that a foundational framework of defining POCUS and the different levels of POCUS use and delineating program management elements are critical. The authors hope that their experience will be helpful to others who are also interested in POCUS or in the process of creating POCUS programs at their institutions. With a clearly established framework, patient safety and quality of care are improved for everyone.

Redesign of the American Board of Radiology Diagnostic Radiology Certifying Examination.

On April 13, 2023, the American Board of Radiology (ABR) announced plans to replace the current computer-based diagnostic radiology (DR) certifying examination with a new oral examination to be administered remotely, beginning in 2028. This article describes the planned changes and the process that led to those changes. In keeping with its commitment to continuous improvement, the ABR gathered input regarding the DR initial certification process. Respondents generally agreed that the qualifying (core) examination was satisfactory but expressed concerns regarding the computer-based certifying examination's effectiveness and impact on training. Examination redesign was conducted using input from key groups with a goal of effectively evaluating competence and incentivizing study behaviors that best prepare candidates for radiology practice. Major design elements included examination structure, breadth and depth of content, and timing. The new oral examination will focus on critical findings as well as common and important diagnoses routinely encountered in all diagnostic specialties, including radiology procedures. Candidates will first be eligible for the examination in the calendar year after residency graduation. Additional details will be finalized and announced in coming years. The ABR will continue to engage with interested parties throughout the implementation process.

From Acceptable to Superlative: Scaling a Technologist Coaching Intervention to Improve Image Quality.

OBJECTIVE: To explore factors influencing the expansion of the peer-based technologist Coaching Model Program (CMP) from its origins in mammography and ultrasound to all imaging modalities at a single tertiary academic medical center. METHODS: After success in mammography and ultrasound, efforts to expand the CMP across all Stanford Radiology modalities commenced in September 2020. From February to April 2021 as lead coaches piloted the program in these novel modalities, an implementation science team designed and conducted semistructured stakeholder interviews and took observational notes at learning collaborative meetings. Data were analyzed using inductive-deductive approaches informed by two implementation science frameworks. RESULTS: Twenty-seven interviews were collected across modalities with radiologists (n = 5), managers (n = 6), coaches (n = 11), and technologists (n = 5) and analyzed with observational notes from six learning meetings with 25 to 40 recurrent participants. The number of technologists, the complexity of examinations, or the existence of standardized auditing criteria for each modality influenced CMP adaptations. Facilitators underlying program expansion included cross-modality learning collaborative, thoughtful pairing of coach and technologist, flexibility in feedback frequency and format, radiologist engagement, and staged rollout. Barriers included lack of protected coaching time, lack of pre-existing audit criteria for some modalities, and the need for privacy of auditing and feedback data. DISCUSSION: Adaptations to each radiology modality and communication of these learnings were key to disseminating the existing CMP to new modalities across the entire department. An intermodality learning collaborative can facilitate the dissemination of evidence-based practices across modalities.

American College of Radiology initiatives on prostate magnetic resonance imaging quality.

Magnetic resonance imaging (MRI) has become integral to diagnosing and managing patients with suspected or confirmed prostate cancer. However, the benefits of utilizing MRI can be hindered by quality issues during imaging acquisition, interpretation, and reporting. As the utilization of prostate MRI continues to increase in clinical practice, the variability in MRI quality and how it can negatively impact patient care have become apparent. The American College of Radiology (ACR) has recognized this challenge and developed several initiatives to address the issue of inconsistent MRI quality and ensure that imaging centers deliver high-quality patient care. These initiatives include the Prostate Imaging Reporting and Data System (PI-RADS), developed in collaboration with an international panel of experts and members of the European Society of Urogenital Radiology (ESUR), the Prostate MR Image Quality Improvement Collaborative, which is part of the ACR Learning Network, the ACR Prostate Cancer MRI Center Designation, and the ACR Appropriateness Criteria. In this article, we will discuss the importance of these initiatives in establishing quality assurance and quality control programs for prostate MRI and how they can improve patient outcomes.

The ACR Learning Network: Facilitating Local Performance Improvement Through Shared Learning.

PURPOSE: The ACR Learning Network was established to test the viability of the learning network model in radiology. In this report, the authors review the learning network concept, introduce the ACR Learning Network and its components, and report progress to date and plans for the future. METHODS: Patterned after institutional programs developed by the principal investigator, the ACR Learning Network was composed of four distinct improvement collaboratives. Initial participating sites were solicited through broad program advertisement. Candidate programs were selected on the basis of assessments of local leadership support, experience with quality improvement initiatives, intraorganizational relationships, and access to data and analytic support. Participation began with completing a 27-week formal quality improvement training and project support program, with local teams reporting weekly progress on a common performance measure. RESULTS: Four improvement collaborative topics were chosen for the initial cohort with the following numbers of participating sites: mammography positioning (6), prostate MR image quality (6), lung cancer screening (6), and follow-up on recommendations for management of incidental findings (4). To date, all sites have remained actively engaged and have progressed in an expected fashion. A detailed report of the results of the improvement phase will be provided in a future publication. CONCLUSIONS: To date, the ACR Learning Network has successfully achieved planned milestones outlined in the program's plan, with preparation under way for the second and third cohorts. By providing a shared platform for improvement training and knowledge sharing, the authors are optimistic that the network may facilitate widespread performance improvement in radiology on a number of topics for years to come.

Results of a Virtual Multi-Institutional Program for Quality Improvement Training and Project Facilitation.

OBJECTIVE: The purpose of this project was to describe the results of a multi-institutional quality improvement (QI) program conducted in a virtual format. METHODS: Developed at Stanford in 2016, the Realizing Improvement Through Team Empowerment program uses a team-based, project-based improvement approach to QI. The program was planned to be replicated at two other institutions through respective on-site programs but was converted to a multi-institutional virtual format in 2020 in response to the COVID-19 pandemic. The virtual program began in July 2020 and ended in December 2020. The two institutions participated jointly in the cohort, with 10 2-hour training sessions every 2 weeks for a total of 18 weeks. Project progress was monitored using a predetermined project progress scale by the program manager, who provided more direct project support as needed. RESULTS: The cohort consisted of six teams (37 participants) from two institutions. Each team completed a QI project in subjects including MRI, ultrasound, CT, diagnostic radiography, and ACR certification. All projects reached levels of between 3.0 (initial test cycles begun with evidence of modest improvement) and 4.0 (performance data meeting goal and statistical process control criteria for improvement) and met graduation criteria for program completion. DISCUSSION: We found the structured problem-solving method, along with timely focused QI education materials via a virtual platform, to be effective in simultaneously facilitating improvement projects from multiple institutions. The combination of two institutions fostered encouragement and shared learning across institutions.

Evaluating clinician-led quality improvement initiatives: A system-wide embedded research partnership at Stanford Medicine.

Introduction: Many healthcare delivery systems have developed clinician-led quality improvement (QI) initiatives but fewer have also developed in-house evaluation units. Engagement between the two entities creates unique opportunities. Stanford Medicine funded a collaboration between their Improvement Capability Development Program (ICDP), which coordinates and incentivizes clinician-led QI efforts, and the Evaluation Sciences Unit (ESU), a multidisciplinary group of embedded researchers with expertise in implementation and evaluation sciences. Aim: To describe the ICDP-ESU partnership and report key learnings from the first 2 y of operation September 2019 to August 2021. Methods: Department-level physician and operational QI leaders were offered an ESU consultation to workshop design, methods, and overall scope of their annual QI projects. A steering committee of high-level stakeholders from operational, clinical, and research perspectives subsequently selected three projects for in-depth partnered evaluation with the ESU based on evaluability, importance to the health system, and broader relevance. Selected project teams met regularly with the ESU to develop mixed methods evaluations informed by relevant implementation science frameworks, while aligning the evaluation approach with the clinical teams' QI goals. Results: Sixty and 62 ICDP projects were initiated during the 2 cycles, respectively, across 18 departments, of which ESU consulted with 15 (83%). Within each annual cycle, evaluators made actionable, summative findings rapidly available to partners to inform ongoing improvement. Other reported benefits of the partnership included rapid adaptation to COVID-19 needs, expanded clinician evaluation skills, external knowledge dissemination through scholarship, and health system-wide knowledge exchange. Ongoing considerations for improving the collaboration included the need for multi-year support to enable nimble response to dynamic health system needs and timely data access. Conclusion: Presence of embedded evaluation partners in the enterprise-wide QI program supported identification of analogous endeavors (eg, telemedicine adoption) and cross-cutting lessons across QI efforts, clinician capacity building, and knowledge dissemination through scholarship.

Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How?

As the role of artificial intelligence (AI) in clinical practice evolves, governance structures oversee the implementation, maintenance, and monitoring of clinical AI algorithms to enhance quality, manage resources, and ensure patient safety. In this article, a framework is established for the infrastructure required for clinical AI implementation and presents a road map for governance. The road map answers four key questions: Who decides which tools to implement? What factors should be considered when assessing an application for implementation? How should applications be implemented in clinical practice? Finally, how should tools be monitored and maintained after clinical implementation? Among the many challenges for the implementation of AI in clinical practice, devising flexible governance structures that can quickly adapt to a changing environment will be essential to ensure quality patient care and practice improvement objectives.

Moving Toward Seamless Interinstitutional Electronic Image Transfer.

The fact that medical images are still predominately exchanged between institutions via physical media is unacceptable in the era of value-driven health care. Although better solutions are technically possible, problems of coordination and market dynamics may be inhibiting progress more than technical factors. We provide a macrosystem analysis of the problem of interinstitutional medical image exchange and propose a strategy for nudging the market toward a patient-friendly solution. The system can be viewed as a network, with autonomous nodes interconnected by links through which information is exchanged. A variety of potential network configurations include those that depend on individual carriers, peer-to-peer links, one or multiple hubs, or a hybrid of models. We find the linked multihub model, in which individual institutions are connected to other institutions via image exchange companies, to be the configuration most likely to create a patient-friendly electronic image exchange system. To achieve this configuration, image exchange companies, which operate in a competitive marketplace, must exchange images with each other. We call on these vendors to immediately commit to coordinating in this manner. We call on all other stakeholders, including local care provider institutions, medical societies, payers, and regulators, to actively encourage and facilitate this behavior. Specifically, we call on institutions to create appropriate market incentives by only contracting with image exchange vendors who are committed to begin vendor-to-vendor image exchange by no later than 2024.

Increasing the Utilization of Moderate Sedation Services for Pediatric Imaging.

Performing motion-free imaging is frequently challenging in children. To bridge the gap between examinations performed in children who are awake and in those under general anesthesia, a moderate sedation program was implemented at our institution but was seldom used despite substantial eligibility. In conjunction with a 5-month quality improvement (QI) course, a multidisciplinary team was assembled and, by using an A3 approach, sought to address the most important key drivers of low utilization, namely the need for clear moderate sedation eligibility criteria, reliable protocol routing order, consistent moderate sedation screening performed by registered nurses (RNs), and enhanced visibility of moderate sedation services to ordering providers. Initial steps focused on developing better-defined criteria and protocoling standard work for technologists and RNs, with coaching and audits. Modality-specific forecasting was then implemented to reroute profiles of patients who were awaiting scheduling or already scheduled for an examination with general anesthesia to the moderate sedation queue to identify more eligible patients. These manual efforts were coupled with higher reliability but more protracted electronic health record changes, facilitating automated protocol routing on the basis of moderate sedation eligibility and order entry constraints. As a result, scheduled imaging examinations requiring moderate sedation increased from a mean of 1.2 examinations per week to a sustained 6.1 examinations per week (range, 4-8) over the 5-month period, exceeding the team SMART (specific, measurable, achievable, relevant, and time bound) goal to achieve an average of five examinations per week by the QI course end. By targeting the most high-impact yet modifiable process deficiencies through a multifaceted team approach and initially investing in manual efforts to gain cultural buy-in while awaiting higher-reliability interventions, the project achieved success and may serve as a more general model for workflow change when there is organizational resistance. ©RSNA, 2021.

Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.

Program for Supporting Frontline Improvement Projects in an Academic Radiology Department.

OBJECTIVE. The purpose of this study was to describe the results of an ongoing program implemented in an academic radiology department to support the execution of small- to medium-size improvement projects led by frontline staff and leaders. MATERIALS AND METHODS. Staff members were assigned a coach, were instructed in improvement methods, were given time to work on the project, and presented progress to department leaders in weekly 30-minute reports. Estimated costs and outcomes were calculated for each project and aggregated. An anonymous survey was administered to participants at the end of the first year. RESULTS. A total of 73 participants completed 102 projects in the first 2 years of the program. The project type mix included 25 quality improvement projects, 22 patient satisfaction projects, 14 staff engagement projects, 27 efficiency improvement projects, and 14 regulatory compliance and readiness projects. Estimated annualized outcomes included approximately 4500 labor hours saved, $315K in supply cost savings, $42.2M in potential increased revenues, 8- and 2-point increase in top-box patient experience scores at two clinics, and a 60-incident reduction in near-miss safety events. Participant time equated to approximately 0.35 full-time equivalent positions per year. Approximately 0.4 full-time equivalent was required to support the program. Survey results indicated that the participants generally viewed the program favorably. CONCLUSION. The program was successful in providing a platform for simultaneously solving a large number of organizational problems while also providing a positive experience to frontline personnel.