Standardizing imaging findings representation: harnessing Common Data Elements semantics and Fast Healthcare Interoperability Resources structures.

OBJECTIVES: Designing a framework representing radiology results in a standards-based data structure using joint Radiological Society of North America/American College of Radiology Common Data Elements (CDEs) as the semantic labels on standard structures. This allows radiologist-created report data to integrate with artificial intelligence-generated results for use throughout downstream systems. MATERIALS AND METHODS: We developed a framework modeling radiology findings as Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) observations using CDE set/element identifiers as standardized semantic labels. This framework deploys CDE identifiers to specify radiology findings and attributes, providing consistent labels for radiology report concepts-diagnoses, recommendations, tabular/quantitative data-with built-in integration with RadLex, SNOMED CT, LOINC, and other ontologies. Observation structures fit within larger HL7 FHIR DiagnosticReport resources, providing output including both nuanced text and structured data. RESULTS: Labeling radiology findings as discrete data for interchange between systems requires two components: structure and semantics. CDE definitions provide semantic identifiers for findings and their component values. The FHIR observation resource specifies a structure for associating identifiers with radiology findings in the context of reports, with CDE-encoded observations referring to definitions for CDE identifiers in a central repository. The discussion includes an example of encoding pulmonary nodules on a chest CT as CDE-labeled observations, demonstrating the application of this framework to exchange findings throughout the imaging workflow, making imaging data available to downstream clinical systems. DISCUSSION: CDE-labeled observations establish a lingua franca for encoding, exchanging, and consuming radiology data at the level of individual findings, facilitating use throughout healthcare systems. IMPORTANCE: CDE-labeled FHIR observation objects can increase the value of radiology results by facilitating their use throughout patient care.

Development of Medical Imaging Data Standardization for Imaging-Based Observational Research: OMOP Common Data Model Extension.

The rapid growth of artificial intelligence (AI) and deep learning techniques require access to large inter-institutional cohorts of data to enable the development of robust models, e.g., targeting the identification of disease biomarkers and quantifying disease progression and treatment efficacy. The Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) has been designed to accommodate a harmonized representation of observational healthcare data. This study proposes the Medical Imaging CDM (MI-CDM) extension, adding two new tables and two vocabularies to the OMOP CDM to address the structural and semantic requirements to support imaging research. The tables provide the capabilities of linking DICOM data sources as well as tracking the provenance of imaging features derived from those images. The implementation of the extension enables phenotype definitions using imaging features and expanding standardized computable imaging biomarkers. This proposal offers a comprehensive and unified approach for conducting imaging research and outcome studies utilizing imaging features.

Implementation of an Institution-Wide Rules-Based Automated CT Protocoling System.

BACKGROUND. Examination protocoling is a noninterpretive task that increases radiologists' workload and can cause workflow inefficiencies. OBJECTIVE. The purpose of this study was to evaluate effects of an automated CT protocoling system on examination process times and protocol error rates. METHODS. This retrospective study included 317,597 CT examinations (mean age, 61.8 ± 18.1 [SD] years; male, 161,125; female, 156,447; unspecified sex, 25) from July 2020 to June 2022. A rules-based automated protocoling system was implemented institution-wide; the system evaluated all CT orders in the EHR and assigned a protocol or directed the order for manual radiologist protocoling. The study period comprised pilot (July 2020 to December 2020), implementation (January 2021 to December 2021), and postimplementation (January 2022 to June 2022) phases. Proportions of automatically protocoled examinations were summarized. Process times were recorded. Protocol error rates were assessed by counts of quality improvement (QI) reports and examination recalls and comparison with retrospectively assigned protocols in 450 randomly selected examinations. RESULTS. Frequency of automatic protocoling was 19,366/70,780 (27.4%), 68,875/163,068 (42.2%), and 54,045/83,749 (64.5%) in pilot, implementation, and postimplementation phases, respectively (p < .001). Mean (± SD) times from order entry to protocol assignment for automatically and manually protocoled examinations for emergency department examinations were 0.2 ± 18.2 and 2.1 ± 69.7 hours, respectively; mean inpatient examination times were 0.5 ± 50.0 and 3.5 ± 105.5 hours; and mean outpatient examination times were 361.7 ± 1165.5 and 1289.9 ± 2050.9 hours (all p < .001). Mean (± SD) times from order entry to examination completion for automatically and manually protocoled examinations for emergency department examinations were 2.6 ± 38.6 and 4.2 ± 73.0 hours, respectively (p < .001); for inpatient examinations were 6.3 ± 74.6 and 8.7 ± 109.3 hours (p = .001); and for outpatient examinations were 1367.2 ± 1795.8 and 1471.8 ± 2118.3 hours (p < .001). In the three phases, there were three, 19, and 25 QI reports and zero, one, and three recalls, respectively, for automatically protocoled examinations, versus nine, 19, and five QI reports and one, seven, and zero recalls for manually protocoled examinations. Retrospectively assigned protocols were concordant with 212/214 (99.1%) of automatically protocoled versus 233/236 (98.7%) of manually protocoled examinations. CONCLUSION. The automated protocoling system substantially reduced radiologists' protocoling workload and decreased times from order entry to protocol assignment and examination completion; protocol errors and recalls were infrequent. CLINICAL IMPACT. The system represents a solution for reducing radiologists' time spent performing noninterpretive tasks and improving care efficiency.

Addressing the Challenges of Implementing Artificial Intelligence Tools in Clinical Practice: Principles From Experience.

The multitude of artificial intelligence (AI)-based solutions, vendors, and platforms poses a challenging proposition to an already complex clinical radiology practice. Apart from assessing and ensuring acceptable local performance and workflow fit to improve imaging services, AI tools require multiple stakeholders, including clinical, technical, and financial, who collaborate to move potential deployable applications to full clinical deployment in a structured and efficient manner. Postdeployment monitoring and surveillance of such tools require an infrastructure that ensures proper and safe use. Herein, the authors describe their experience and framework for implementing and supporting the use of AI applications in radiology workflow.

Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How?

As the role of artificial intelligence (AI) in clinical practice evolves, governance structures oversee the implementation, maintenance, and monitoring of clinical AI algorithms to enhance quality, manage resources, and ensure patient safety. In this article, a framework is established for the infrastructure required for clinical AI implementation and presents a road map for governance. The road map answers four key questions: Who decides which tools to implement? What factors should be considered when assessing an application for implementation? How should applications be implemented in clinical practice? Finally, how should tools be monitored and maintained after clinical implementation? Among the many challenges for the implementation of AI in clinical practice, devising flexible governance structures that can quickly adapt to a changing environment will be essential to ensure quality patient care and practice improvement objectives.

Validation pipeline for machine learning algorithm assessment for multiple vendors.

A standardized objective evaluation method is needed to compare machine learning (ML) algorithms as these tools become available for clinical use. Therefore, we designed, built, and tested an evaluation pipeline with the goal of normalizing performance measurement of independently developed algorithms, using a common test dataset of our clinical imaging. Three vendor applications for detecting solid, part-solid, and groundglass lung nodules in chest CT examinations were assessed in this retrospective study using our data-preprocessing and algorithm assessment chain. The pipeline included tools for image cohort creation and de-identification; report and image annotation for ground-truth labeling; server partitioning to receive vendor "black box" algorithms and to enable model testing on our internal clinical data (100 chest CTs with 243 nodules) from within our security firewall; model validation and result visualization; and performance assessment calculating algorithm recall, precision, and receiver operating characteristic curves (ROC). Algorithm true positives, false positives, false negatives, recall, and precision for detecting lung nodules were as follows: Vendor-1 (194, 23, 49, 0.80, 0.89); Vendor-2 (182, 270, 61, 0.75, 0.40); Vendor-3 (75, 120, 168, 0.32, 0.39). The AUCs for detection of solid (0.61-0.74), groundglass (0.66-0.86) and part-solid (0.52-0.86) nodules varied between the three vendors. Our ML model validation pipeline enabled testing of multi-vendor algorithms within the institutional firewall. Wide variations in algorithm performance for detection as well as classification of lung nodules justifies the premise for a standardized objective ML algorithm evaluation process.

Implementation of a Semistructured Clinical Event Documentation Tool for Acute Adverse Contrast Reactions.

PURPOSE: To improve the efficiency and accuracy of clinicians documenting acute clinical events related to contrast agent administration using a web browser-based semistructured documentation support tool. METHODS: A new tool called Contrast Incident Support and Reporting (CISaR) was developed to enable radiologists responding to contrast reactions to document inciting contrast class, type of event, severity of contrast reaction, and recommendation for future contrast use. Retrospective analysis was conducted of all CT and MRI examinations performed between February 2018 and December 2019 across our hospital system with associated contrast reaction documentation. Time periods were defined as before tool deployment, early adoption, and steady-state deployment. The primary outcome measure was the presence of event documentation by a radiologist. The secondary outcome measure was completeness of the documentation parameters. RESULTS: A total of 431 CT and MRI studies with reactions were included in the study, and 50% of studies had radiologist documentation during the pre-CISaR period. This increased to 66% during the early adoption period and 89% in the post-CISaR period. It took approximately 9 months from the introduction of CISaR to reach full adoption and become the main method for adverse contrast reaction documentation. The percentage of radiologist documentation that detailed provoking contrast agent class, severity of reaction, reaction type, and future contrast agent recommendation all significantly increased (P < .0001), with greater than 95% inclusion of each element. CONCLUSION: The implementation of a semistructured electronic application for adverse contrast reaction reporting significantly increased radiologist documentation rate and completeness of the documentation.

Computer-assisted Reporting and Decision Support Increases Compliance with Follow-up Imaging and Hormonal Screening of Adrenal Incidentalomas.

OBJECTIVE: To assess the impact of using a computer-assisted reporting and decision support (CAR/DS) tool at the radiologist point-of-care on ordering provider compliance with recommendations for adrenal incidentaloma workup. METHOD: Abdominal CT reports describing adrenal incidentalomas (2014 - 2016) were retrospectively extracted from the radiology database. Exclusion criteria were history of cancer, suspected functioning adrenal tumor, dominant nodule size < 1 cm or ≥ 4 cm, myelolipomas, cysts, and hematomas. Multivariable logistic regression models were employed to predict follow-up imaging (FUI) and hormonal screening orders as a function of patient age and sex, nodule size, and CAR/DS use. CAR/DS reports were compared to conventional reports regarding ordering provider compliance with, frequency, and completeness of, guideline-warranted recommendations for FUI and hormonal screening of adrenal incidentalomas using Chi-square test. RESULT: Of 174 patients (mean age 62.4; 51.1% women) with adrenal incidentalomas, 62% (108/174) received CAR/DS-based recommendations versus 38% (66/174) unassisted recommendations. CAR/DS use was an independent predictor of provider compliance both with FUI (Odds Ratio [OR]=2.47, p = 0.02) and hormonal screening (OR=2.38, p = 0.04). CAR/DS reports recommended FUI (97.2%,105/108) and hormonal screening (87.0%,94/108) more often than conventional reports (respectively, 69.7% [46/66], 3.0% [2/66], both p <0.0001). CAR/DS recommendations more frequently included instructions for FUI time, protocol, and modality than conventional reports (all p <0.001). CONCLUSION: Ordering providers were at least twice as likely to comply with report recommendations for FUI and hormonal evaluation of adrenal incidentalomas generated using CAR/DS versus unassisted reporting. CAR/DS-directed recommendations were more adherent to guidelines than those generated without.

Artificial Intelligence Enabling Radiology Reporting.

The radiology reporting process is beginning to incorporate structured, semantically labeled data. Tools based on artificial intelligence technologies using a structured reporting context can assist with internal report consistency and longitudinal tracking. To-do lists of relevant issues could be assembled by artificial intelligence tools, incorporating components of the patient's history. Radiologists will review and select artificial intelligence-generated and other data to be transmitted to the electronic health record and generate feedback for ongoing improvement of artificial intelligence tools. These technologies should make reports more valuable by making reports more accessible and better able to integrate into care pathways.

Automated tracking of emergency department abdominal CT findings during the COVID-19 pandemic using natural language processing.

PURPOSE: During the COVID-19 pandemic, emergency department (ED) volumes have fluctuated. We hypothesized that natural language processing (NLP) models could quantify changes in detection of acute abdominal pathology (acute appendicitis (AA), acute diverticulitis (AD), or bowel obstruction (BO)) on CT reports. METHODS: This retrospective study included 22,182 radiology reports from CT abdomen/pelvis studies performed at an urban ED between January 1, 2018 to August 14, 2020. Using a subset of 2448 manually annotated reports, we trained random forest NLP models to classify the presence of AA, AD, and BO in report impressions. Performance was assessed using 5-fold cross validation. The NLP classifiers were then applied to all reports. RESULTS: The NLP classifiers for AA, AD, and BO demonstrated cross-validation classification accuracies between 0.97 and 0.99 and F1-scores between 0.86 and 0.91. When applied to all CT reports, the estimated numbers of AA, AD, and BO cases decreased 43-57% in April 2020 (first regional peak of COVID-19 cases) compared to 2018-2019. However, the number of abdominal pathologies detected rebounded in May-July 2020, with increases above historical averages for AD. The proportions of CT studies with these pathologies did not significantly increase during the pandemic period. CONCLUSION: Dramatic decreases in numbers of acute abdominal pathologies detected by ED CT studies were observed early on during the COVID-19 pandemic, though these numbers rapidly rebounded. The proportions of CT cases with these pathologies did not increase, which suggests patients deferred care during the first pandemic peak. NLP can help automatically track findings in ED radiology reporting.

Computer-Assisted Reporting and Decision Support in Standardized Radiology Reporting for Cancer Imaging.

PURPOSE: Recent advances in structured reporting are providing an opportunity to enhance cancer imaging assessment to drive value-based care and improve patient safety. METHODS: The computer-assisted reporting and decision support (CAR/DS) framework has been developed to enable systematic ingestion of guidelines as clinical decision structured reporting tools embedded within the radiologist's workflow. RESULTS: CAR/DS tools can reduce the radiology reporting variability and increase compliance with clinical guidelines. The lung cancer use-case is used to describe various scenarios of a cancer imaging structured reporting pathway, including incidental findings, screening, staging, and restaging or continued care. Various aspects of these tools are also described using cancer-related examples for different imaging modalities and applications such as calculators. Such systems can leverage artificial intelligence (AI) algorithms to assist with the generation of structured reports and there are opportunities for new AI applications to be created using the structured data associated with CAR/DS tools. CONCLUSION: These AI-enabled systems are starting to allow information from multiple sources to be integrated and inserted into structured reports to drive improvements in clinical decision support and patient care.

Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs.

RATIONALE AND OBJECTIVES: Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement. MATERIALS AND METHODS: We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend. RESULTS: Use of the AI system improved radiologist interrater agreement (Fleiss κ = 0.40 to 0.66, before and after use of the system). The Fleiss κ for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days. CONCLUSION: An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists.

Artificial Intelligence and Clinical Decision Support for Radiologists and Referring Providers.

Recent advances in artificial intelligence (AI) are providing an opportunity to enhance existing clinical decision support (CDS) tools to improve patient safety and drive value-based imaging. We discuss the advantages and potential applications that may be realized with the synergy between AI and CDS systems. From the perspective of both radiologist and ordering provider, CDS could be significantly empowered using AI. CDS enhanced by AI could reduce friction in radiology workflows and can aid AI developers to identify relevant imaging features their tools should be seeking to extract from images. Furthermore, these systems can generate structured data to be used as input to develop machine learning algorithms, which can drive downstream care pathways. For referring providers, an AI-enabled CDS solution could enable an evolution from existing imaging-centric CDS toward decision support that takes into account a holistic patient perspective. More intelligent CDS could suggest imaging examinations in highly complex clinical scenarios, assist on the identification of appropriate imaging opportunities at the health system level, suggest appropriate individualized screening, or aid health care providers to ensure continuity of care. AI has the potential to enable the next generation of CDS, improving patient care and enhancing providers' and radiologists' experience.

Importance of clinical decision support system response time monitoring: a case report.

Clinical decision support (CDS) systems are prevalent in electronic health records and drive many safety advantages. However, CDS systems can also cause unintended consequences. Monitoring programs focused on alert firing rates are important to detect anomalies and ensure systems are working as intended. Monitoring efforts do not generally include system load and time to generate decision support, which is becoming increasingly important as more CDS systems rely on external, web-based content and algorithms. We report a case in which a web-based service caused significant increase in the time to generate decision support, in turn leading to marked delays in electronic health record system responsiveness, which could have led to patient safety events. Given this, it is critical to consider adding decision support-time generation to ongoing CDS system monitoring programs.

Bending the Artificial Intelligence Curve for Radiology: Informatics Tools From ACR and RSNA.

Artificial intelligence (AI) will reshape radiology over the coming years. The radiology community has a strong history of embracing new technology for positive change, and AI is no exception. As with any new technology, rapid, successful implementation faces several challenges that will require creation and adoption of new integration technology. Use cases important to real-world application of AI are described, including clinical registries, AI research, AI product validation, and computer assistance for radiology reporting. Furthermore, the informatics technologies required for successful implementation of the use cases are described, including open Computer-Assisted Radiologist Decision Support, ACR Assist, ACR Data Science Institute use cases, common data elements (radelement.org), RadLex (radlex.org), LOINC/RSNA RadLex Playbook (loinc.org), and Radiology Report Templates (radreport.org).