Checklist for Reproducibility of Deep Learning in Medical Imaging.

The application of deep learning (DL) in medicine introduces transformative tools with the potential to enhance prognosis, diagnosis, and treatment planning. However, ensuring transparent documentation is essential for researchers to enhance reproducibility and refine techniques. Our study addresses the unique challenges presented by DL in medical imaging by developing a comprehensive checklist using the Delphi method to enhance reproducibility and reliability in this dynamic field. We compiled a preliminary checklist based on a comprehensive review of existing checklists and relevant literature. A panel of 11 experts in medical imaging and DL assessed these items using Likert scales, with two survey rounds to refine responses and gauge consensus. We also employed the content validity ratio with a cutoff of 0.59 to determine item face and content validity. Round 1 included a 27-item questionnaire, with 12 items demonstrating high consensus for face and content validity that were then left out of round 2. Round 2 involved refining the checklist, resulting in an additional 17 items. In the last round, 3 items were deemed non-essential or infeasible, while 2 newly suggested items received unanimous agreement for inclusion, resulting in a final 26-item DL model reporting checklist derived from the Delphi process. The 26-item checklist facilitates the reproducible reporting of DL tools and enables scientists to replicate the study's results.

AI pitfalls and what not to do: mitigating bias in AI.

Various forms of artificial intelligence (AI) applications are being deployed and used in many healthcare systems. As the use of these applications increases, we are learning the failures of these models and how they can perpetuate bias. With these new lessons, we need to prioritize bias evaluation and mitigation for radiology applications; all the while not ignoring the impact of changes in the larger enterprise AI deployment which may have downstream impact on performance of AI models. In this paper, we provide an updated review of known pitfalls causing AI bias and discuss strategies for mitigating these biases within the context of AI deployment in the larger healthcare enterprise. We describe these pitfalls by framing them in the larger AI lifecycle from problem definition, data set selection and curation, model training and deployment emphasizing that bias exists across a spectrum and is a sequela of a combination of both human and machine factors.

Lessons Learned in Change Management in Deploying Novel Informatics Solutions: Experience Implementing a Point-of-Care Patient Photography System with Radiography.

We describe implementation of a point-of-care system for simultaneous acquisition of patient photographs along with portable radiographs at a large academic hospital. During the implementation process, we observed several technical challenges in the areas of (1) hardware-automatic triggering for photograph acquisition, camera hardware enclosure, networking, and system server hardware and (2) software-post-processing of photographs. Additionally, we also faced cultural challenges involving workflow issues, communication with technologists and users, and system maintenance. We describe our solutions to address these challenges. We anticipate that these experiences will provide useful insights into deploying and iterating new technologies in imaging informatics.

Sociodemographic Variables Reporting in Human Radiology Artificial Intelligence Research.

PURPOSE: Artificial intelligence (AI) is rapidly reshaping how radiology is practiced. Its susceptibility to biases, however, is a primary concern as more AI algorithms become available for widespread use. So far, there has been limited evaluation of how sociodemographic variables are reported in radiology AI research. This study aims to evaluate the presence and extent of sociodemographic reporting in human subjects radiology AI original research. METHODS: All human subjects original radiology AI articles published from January to December 2020 in the top six US radiology journals, as determined by impact factor, were reviewed. Reporting of any sociodemographic variables (age, gender, and race or ethnicity) as well as any sociodemographic-based results were extracted. RESULTS: Of the 160 included articles, 54% reported at least one sociodemographic variable, 53% reported age, 47% gender, and 4% race or ethnicity. Six percent reported any sociodemographic-based results. There was significant variation in reporting of at least one sociodemographic variable by journal, ranging from 33% to 100%. CONCLUSIONS: Reporting of sociodemographic variables in human subjects original radiology AI research remains poor, putting the results and subsequent algorithms at increased risk of biases.

Imaging Informatics Fellowship Curriculum: Building Consensus on the Most Critical Topics and the Future of the Informatics Fellowship.

The existing fellowship imaging informatics curriculum, established in 2004, has not undergone formal revision since its inception and inaccurately reflects present-day radiology infrastructure. It insufficiently equips trainees for today's informatics challenges as current practices require an understanding of advanced informatics processes and more complex system integration. We sought to address this issue by surveying imaging informatics fellowship program directors across the country to determine the components and cutline for essential topics in a standardized imaging informatics curriculum, the consensus on essential versus supplementary knowledge, and the factors individual programs may use to determine if a newly developed topic is an essential topic. We further identified typical program structural elements and sought fellowship director consensus on offering official graduate trainee certification to imaging informatics fellows. Here, we aim to provide an imaging informatics fellowship director consensus on topics considered essential while still providing a framework for informatics fellowship programs to customize their individual curricula.

Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice.

Artificial intelligence has become a ubiquitous term in radiology over the past several years, and much attention has been given to applications that aid radiologists in the detection of abnormalities and diagnosis of diseases. However, there are many potential applications related to radiologic image quality, safety, and workflow improvements that present equal, if not greater, value propositions to radiology practices, insurance companies, and hospital systems. This review focuses on six major categories for artificial intelligence applications: study selection and protocoling, image acquisition, worklist prioritization, study reporting, business applications, and resident education. All of these categories can substantially affect different aspects of radiology practices and workflows. Each of these categories has different value propositions in terms of whether they could be used to increase efficiency, improve patient safety, increase revenue, or save costs. Each application is covered in depth in the context of both current and future areas of work. Keywords: Use of AI in Education, Application Domain, Supervised Learning, Safety © RSNA, 2022.

Leveraging Social Media and Web Presence to Discuss and Promote Diversity, Equity, and Inclusion in Radiology.

PURPOSE: This article seeks to better understand how radiology residency programs leverage their social media presences during the 2020 National Residency Match Program (NRMP) application cycle to engage with students and promote diversity, equity, and inclusion to prospective residency applicants. METHODS: We used publicly available information to determine how broad a presence radiology programs have across specific platforms (Twitter [Twitter, Inc, San Francisco, California], Facebook [Facebook, Inc, Menlo Park, California], Instagram [Facebook, Inc], and website pages) as well as what strategies these programs use to promote diversity, equity, and inclusion. RESULTS: During the 2020 NRMP application cycle, radiology residency programs substantially increased their social media presence across the platforms we examined. We determined that 29.3% (39 of 133), 58.9% (43 of 73), and 29.55% (13 of 44) of programs used Twitter, Instagram, and Facebook, respectively; these accounts were established after an April 1, 2020, advisory statement from the NRMP. Program size and university affiliation were correlated with the degree of social media presence. Those programs using social media to promote diversity, equity, and inclusion used a broad but similar approach across programs and platforms. CONCLUSION: The events of 2020 expedited the growth of social media among radiology residency programs, which subsequently ushered in a new medium for conversations about representation in medicine. However, the effectiveness of this medium to promote meaningful expansion of diversity, equity, and inclusion in the field of radiology remains to be seen.

Development and Implementation of an Integrated Imaging Informatics Track for Radiology Residents: Our 3-Year Experience.

RATIONALE AND OBJECTIVES: Imaging Informatics is an emerging and fast-evolving field that encompasses the management of information during all steps of the imaging value chain. With many information technology tools being essential to the radiologists' day-to-day work, there is an increasing need for qualified professionals with clinical background, technology expertise, and leadership skills. To answer this, we describe our experience in the development and implementation of an Integrated Imaging Informatics Track (I3T) for radiology residents at our institution. MATERIALS AND METHODS: The I3T was created by a resident-driven initiative funded by an intradepartmental resident grant. Its curriculum is delivered through a combination of monthly small group discussions, operational meetings, recommended readings, lectures, and early exposure to the National Imaging Informatics Course. The track is steered and managed by the I3T Committee, including trainees and faculty advisors. Up to two first-year residents are selected annually based on their curriculum vitae and an interest application. Successful completion of the program requires submission of a capstone project and at least one academic deliverable (national meeting presentation, poster, exhibit, manuscript and/or grant). RESULTS: In our three-year experience, the seven I3T radiology residents have reported a total of 58 scholarly activities related to Imaging Informatics. I3T residents have assumed leadership roles within our organization and nationally. All residents have successfully carried out their clinical responsibilities. CONCLUSION: We have developed and implemented an I3T for radiology residents at our institution. These residents have been successful in their clinical, scholarship and leadership pursuits.

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images.

Real-time execution of machine learning (ML) pipelines on radiology images is difficult due to limited computing resources in clinical environments, whereas running them in research clusters requires efficient data transfer capabilities. We developed Niffler, an open-source Digital Imaging and Communications in Medicine (DICOM) framework that enables ML and processing pipelines in research clusters by efficiently retrieving images from the hospitals' PACS and extracting the metadata from the images. We deployed Niffler at our institution (Emory Healthcare, the largest healthcare network in the state of Georgia) and retrieved data from 715 scanners spanning 12 sites, up to 350 GB/day continuously in real-time as a DICOM data stream over the past 2 years. We also used Niffler to retrieve images bulk on-demand based on user-provided filters to facilitate several research projects. This paper presents the architecture and three such use cases of Niffler. First, we executed an IVC filter detection and segmentation pipeline on abdominal radiographs in real-time, which was able to classify 989 test images with an accuracy of 96.0%. Second, we applied the Niffler Metadata Extractor to understand the operational efficiency of individual MRI systems based on calculated metrics. We benchmarked the accuracy of the calculated exam time windows by comparing Niffler against the Clinical Data Warehouse (CDW). Niffler accurately identified the scanners' examination timeframes and idling times, whereas CDW falsely depicted several exam overlaps due to human errors. Third, with metadata extracted from the images by Niffler, we identified scanners with misconfigured time and reconfigured five scanners. Our evaluations highlight how Niffler enables real-time ML and processing pipelines in a research cluster.

Sharing and Selling Images: Ethical and Regulatory Considerations for Radiologists.

Opportunities to share or sell images are common in radiology. But because these images typically originate as protected health information, their use admits a host of ethical and regulatory considerations. This article discusses four scenarios that reflect data sharing or selling arrangements in radiology, especially as they might occur in "big data" systems or applications. The objective of this article is to acquaint radiologists with a variety of regulatory standards and ethical perspectives that pertain to certain data use agreements, such that the attitudes and practices of data holders and their sharers or purchasers can withstand ethical or regulatory scrutiny and not invite undesirable outcomes.

Teaching Physician Supervision: Medicare's Ongoing Imperative for Academic Radiologists.

OBJECTIVE. Medicare permits radiologists to bill for trainee work but only in narrowly defined circumstances and with considerable consequences for noncompliance. The purpose of this article is to introduce relevant policy rationale and definitions, review payment requirements, outline documentation and operational considerations for diagnostic and interventional radiology services, and offer practical suggestions for academic radiologists striving to optimize regulatory compliance. CONCLUSION. As academic radiology departments advance their missions of service, teaching, and scholarship, most rely on residents and fellows to support expanding clinical demands. Given the risks of technical noncompliance, institutional commitment and ongoing education regarding teaching supervision compliance are warranted.

Artificial Intelligence in Quality Improvement: Reviewing Uses of Artificial Intelligence in Noninterpretative Processes from Clinical Decision Support to Education and Feedback.

The radiology workflow can be segmented into three large groups: pre-interpretative processes, interpretation, and postinterpretative processes. Each stage of this workflow represents quality improvement opportunities for artificial intelligence and machine learning. Although the focus of recent research has been targeted toward optimization of image interpretation, this article describes significant use cases for artificial intelligence in both the pre-interpretative and postinterpretative aspects of radiology. We provide examples of how current applications of AI for quality improvement purposes across the radiology workflow have been implemented and how further integration of these technologies can significantly improve clinical efficiency, reduce radiologist work burden, and ultimately optimize patient care and outcomes.

Virtual and augmented reality: potential applications in radiology.

The modern-day radiologist must be adept at image interpretation, and the one who most successfully leverages new technologies may provide the highest value to patients, clinicians, and trainees. Applications of virtual reality (VR) and augmented reality (AR) have the potential to revolutionize how imaging information is applied in clinical practice and how radiologists practice. This review provides an overview of VR and AR, highlights current applications, future developments, and limitations hindering adoption.

Improving Billing Accuracy Through Enterprise-Wide Standardized Structured Reporting With Cross-Divisional Shared Templates.

OBJECTIVE: We describe our experience in implementing enterprise-wide standardized structured reporting for chest radiographs (CXRs) via change management strategies and assess the economic impact of structured template adoption. METHODS: Enterprise-wide standardized structured CXR reporting was implemented in a large urban health care enterprise in two phases from September 2016 to March 2019: initial implementation of division-specific structured templates followed by introduction of auto launching cross-divisional consensus structured templates. Usage was tracked over time, and potential radiologist time savings were estimated. Correct-to-bill (CTB) rates were collected between January 2018 and May 2019 for radiography. RESULTS: CXR structured template adoption increased from 46% to 92% in phase 1 and to 96.2% in phase 2, resulting in an estimated 8.5 hours per month of radiologist time saved. CTB rates for both radiographs and all radiology reports showed a linearly increasing trend postintervention with radiography CTB rate showing greater absolute values with an average difference of 20% throughout the sampling period. The CTB rate for all modalities increased by 12%, and the rate for radiography increased by 8%. DISCUSSION: Change management strategies prompted adoption of division-specific structured templates, and exposure via auto launching enforced widespread adoption of consensus templates. Standardized structured reporting resulted in both economic gains and projected radiologist time saved.

The State of Radiology AI: Considerations for Purchase Decisions and Current Market Offerings.

PURPOSE: To provide an overview of important factors to consider when purchasing radiology artificial intelligence (AI) software and current software offerings by type, subspecialty, and modality. MATERIALS AND METHODS: Important factors for consideration when purchasing AI software, including key decision makers, data ownership and privacy, cost structures, performance indicators, and potential return on investment are described. For the market overview, a list of radiology AI companies was aggregated from the Radiological Society of North America and the Society for Imaging Informatics in Medicine conferences (November 2016-June 2019), then narrowed to companies using deep learning for imaging analysis and diagnosis. Software created for image enhancement, reporting, or workflow management was excluded. Software was categorized by task (repetitive, quantitative, explorative, and diagnostic), modality, and subspecialty. RESULTS: A total of 119 software offerings from 55 companies were identified. There were 46 algorithms that currently have Food and Drug Administration and/or Conformité Européenne approval (as of November 2019). Of the 119 offerings, distribution of software targets was 34 of 70 (49%), 21 of 70 (30%), 14 of 70 (20%), and one of 70 (1%) for diagnostic, quantitative, repetitive, and explorative tasks, respectively. A plurality of companies are focused on nodule detection at chest CT and two-dimensional mammography. There is very little activity in certain subspecialties, including pediatrics and nuclear medicine. A comprehensive table is available on the website hitilab.org/pages/ai-companies. CONCLUSION: The radiology AI marketplace is rapidly maturing, with an increase in product offerings. Radiologists and practice administrators should educate themselves on current product offerings and important factors to consider before purchase and implementation.© RSNA, 2020See also the invited commentary by Sala and Ursprung in this issue.

Initial Experience With Patient Visible Light Images Obtained Simultaneously With Portable Radiographs.

OBJECTIVE. Visible light images in the form of point-of-care photographs obtained at the time of medical imaging can be useful for detecting wrong-patient errors and providing image-related clinical context. Our goal was to implement a system to automatically obtain point-of-care patient photographs along with portable radiographs. CONCLUSION. We discuss one academic medical center's initial experience in integrating the system into the clinical workflow and initial use cases ranging from cardiothoracic and abdominal imaging to musculoskeletal imaging, for which such point-of-care photographs were deemed clinically beneficial.

Ethical considerations in artificial intelligence.

With artificial intelligence (AI) precipitously perched at the apex of the hype curve, the promise of transforming the disparate fields of healthcare, finance, journalism, and security and law enforcement, among others, is enormous. For healthcare - particularly radiology - AI is anticipated to facilitate improved diagnostics, workflow, and therapeutic planning and monitoring. And, while it is also causing some trepidation among radiologists regarding its uncertain impact on the demand and training of our current and future workforce, most of us welcome the potential to harness AI for transformative improvements in our ability to diagnose disease more accurately and earlier in the populations we serve.