Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

How AI May Transform Musculoskeletal Imaging.

While musculoskeletal imaging volumes are increasing, there is a relative shortage of subspecialized musculoskeletal radiologists to interpret the studies. Will artificial intelligence (AI) be the solution? For AI to be the solution, the wide implementation of AI-supported data acquisition methods in clinical practice requires establishing trusted and reliable results. This implementation will demand close collaboration between core AI researchers and clinical radiologists. Upon successful clinical implementation, a wide variety of AI-based tools can improve the musculoskeletal radiologist's workflow by triaging imaging examinations, helping with image interpretation, and decreasing the reporting time. Additional AI applications may also be helpful for business, education, and research purposes if successfully integrated into the daily practice of musculoskeletal radiology. The question is not whether AI will replace radiologists, but rather how musculoskeletal radiologists can take advantage of AI to enhance their expert capabilities.

Strategies for Implementing Machine Learning Algorithms in the Clinical Practice of Radiology.

Despite recent advancements in machine learning (ML) applications in health care, there have been few benefits and improvements to clinical medicine in the hospital setting. To facilitate clinical adaptation of methods in ML, this review proposes a standardized framework for the step-by-step implementation of artificial intelligence into the clinical practice of radiology that focuses on three key components: problem identification, stakeholder alignment, and pipeline integration. A review of the recent literature and empirical evidence in radiologic imaging applications justifies this approach and offers a discussion on structuring implementation efforts to help other hospital practices leverage ML to improve patient care. Clinical trial registration no. 04242667 © RSNA, 2024 Supplemental material is available for this article.

Consensus statements on the current landscape of artificial intelligence applications in endoscopy, addressing roadblocks, and advancing artificial intelligence in gastroenterology.

BACKGROUND AND AIMS: The American Society for Gastrointestinal Endoscopy (ASGE) AI Task Force along with experts in endoscopy, technology space, regulatory authorities, and other medical subspecialties initiated a consensus process that analyzed the current literature, highlighted potential areas, and outlined the necessary research in artificial intelligence (AI) to allow a clearer understanding of AI as it pertains to endoscopy currently. METHODS: A modified Delphi process was used to develop these consensus statements. RESULTS: Statement 1: Current advances in AI allow for the development of AI-based algorithms that can be applied to endoscopy to augment endoscopist performance in detection and characterization of endoscopic lesions. Statement 2: Computer vision-based algorithms provide opportunities to redefine quality metrics in endoscopy using AI, which can be standardized and can reduce subjectivity in reporting quality metrics. Natural language processing-based algorithms can help with the data abstraction needed for reporting current quality metrics in GI endoscopy effortlessly. Statement 3: AI technologies can support smart endoscopy suites, which may help optimize workflows in the endoscopy suite, including automated documentation. Statement 4: Using AI and machine learning helps in predictive modeling, diagnosis, and prognostication. High-quality data with multidimensionality are needed for risk prediction, prognostication of specific clinical conditions, and their outcomes when using machine learning methods. Statement 5: Big data and cloud-based tools can help advance clinical research in gastroenterology. Multimodal data are key to understanding the maximal extent of the disease state and unlocking treatment options. Statement 6: Understanding how to evaluate AI algorithms in the gastroenterology literature and clinical trials is important for gastroenterologists, trainees, and researchers, and hence education efforts by GI societies are needed. Statement 7: Several challenges regarding integrating AI solutions into the clinical practice of endoscopy exist, including understanding the role of human-AI interaction. Transparency, interpretability, and explainability of AI algorithms play a key role in their clinical adoption in GI endoscopy. Developing appropriate AI governance, data procurement, and tools needed for the AI lifecycle are critical for the successful implementation of AI into clinical practice. Statement 8: For payment of AI in endoscopy, a thorough evaluation of the potential value proposition for AI systems may help guide purchasing decisions in endoscopy. Reliable cost-effectiveness studies to guide reimbursement are needed. Statement 9: Relevant clinical outcomes and performance metrics for AI in gastroenterology are currently not well defined. To improve the quality and interpretability of research in the field, steps need to be taken to define these evidence standards. Statement 10: A balanced view of AI technologies and active collaboration between the medical technology industry, computer scientists, gastroenterologists, and researchers are critical for the meaningful advancement of AI in gastroenterology. CONCLUSIONS: The consensus process led by the ASGE AI Task Force and experts from various disciplines has shed light on the potential of AI in endoscopy and gastroenterology. AI-based algorithms have shown promise in augmenting endoscopist performance, redefining quality metrics, optimizing workflows, and aiding in predictive modeling and diagnosis. However, challenges remain in evaluating AI algorithms, ensuring transparency and interpretability, addressing governance and data procurement, determining payment models, defining relevant clinical outcomes, and fostering collaboration between stakeholders. Addressing these challenges while maintaining a balanced perspective is crucial for the meaningful advancement of AI in gastroenterology.

Ontologies in the New Computational Age of Radiology: RadLex for Semantics and Interoperability in Imaging Workflows.

From basic research to the bedside, precise terminology is key to advancing medicine and ensuring optimal and appropriate patient care. However, the wide spectrum of diseases and their manifestations superimposed on medical team-specific and discipline-specific communication patterns often impairs shared understanding and the shared use of common medical terminology. Common terms are currently used in medicine to ensure interoperability and facilitate integration of biomedical information for clinical practice and emerging scientific and educational applications alike, from database integration to supporting basic clinical operations such as billing. Such common terminologies can be provided in ontologies, which are formalized representations of knowledge in a particular domain. Ontologies unambiguously specify common concepts and describe the relationships between those concepts by using a form that is mathematically precise and accessible to humans and machines alike. RadLex® is a key RSNA initiative that provides a shared domain model, or ontology, of radiology to facilitate integration of information in radiology education, clinical care, and research. As the contributions of the computational components of common radiologic workflows continue to increase with the ongoing development of big data, artificial intelligence, and novel image analysis and visualization tools, the use of common terminologies is becoming increasingly important for supporting seamless computational resource integration across medicine. This article introduces ontologies, outlines the fundamental semantic web technologies used to create and apply RadLex, and presents examples of RadLex applications in everyday radiology and research. It concludes with a discussion of emerging applications of RadLex, including artificial intelligence applications. © RSNA, 2023 Quiz questions for this article are available in the supplemental material.

Quality Improvement Report: Design and Implementation of a Radiology E-Consult Service.

Electronic consultations (e-consults) mediated through an electronic health record system or web-based platform allow synchronous or asynchronous physician-to-physician communication. E-consults have been explored in various clinical specialties, but relatively few instances in the literature describe e-consults to connect health care providers directly with radiologists.The authors outline how a radiology department can implement an e-consult service and review the development of such a service in a large academic health system. They describe the logistics, workflow, turnaround time expectations, stakeholder management, and pilot implementation and highlight challenges and lessons learned.

Ensemble Approaches to Recognize Protected Health Information in Radiology Reports.

Natural language processing (NLP) techniques for electronic health records have shown great potential to improve the quality of medical care. The text of radiology reports frequently constitutes a large fraction of EHR data, and can provide valuable information about patients' diagnoses, medical history, and imaging findings. The lack of a major public repository for radiological reports severely limits the development, testing, and application of new NLP tools. De-identification of protected health information (PHI) presents a major challenge to building such repositories, as many automated tools for de-identification were trained or designed for clinical notes and do not perform sufficiently well to build a public database of radiology reports. We developed and evaluated six ensemble models based on three publically available de-identification tools: MIT de-id, NeuroNER, and Philter. A set of 1023 reports was set aside as the testing partition. Two individuals with medical training annotated the test set for PHI; differences were resolved by consensus. Ensemble methods included simple voting schemes (1-Vote, 2-Votes, and 3-Votes), a decision tree, a naïve Bayesian classifier, and Adaboost boosting. The 1-Vote ensemble achieved recall of 998 / 1043 (95.7%); the 3-Votes ensemble had precision of 1035 / 1043 (99.2%). F1 scores were: 93.4% for the decision tree, 71.2% for the naïve Bayesian classifier, and 87.5% for the boosting method. Basic voting algorithms and machine learning classifiers incorporating the predictions of multiple tools can outperform each tool acting alone in de-identifying radiology reports. Ensemble methods hold substantial potential to improve automated de-identification tools for radiology reports to make such reports more available for research use to improve patient care and outcomes.

To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines).

Artificial intelligence (AI) has made impressive progress over the past few years, including many applications in medical imaging. Numerous commercial solutions based on AI techniques are now available for sale, forcing radiology practices to learn how to properly assess these tools. While several guidelines describing good practices for conducting and reporting AI-based research in medicine and radiology have been published, fewer efforts have focused on recommendations addressing the key questions to consider when critically assessing AI solutions before purchase. Commercial AI solutions are typically complicated software products, for the evaluation of which many factors are to be considered. In this work, authors from academia and industry have joined efforts to propose a practical framework that will help stakeholders evaluate commercial AI solutions in radiology (the ECLAIR guidelines) and reach an informed decision. Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services. KEY POINTS: • Numerous commercial solutions based on artificial intelligence techniques are now available for sale, and radiology practices have to learn how to properly assess these tools. • We propose a framework focusing on practical points to consider when assessing an AI solution in medical imaging, allowing all stakeholders to conduct relevant discussions with manufacturers and reach an informed decision as to whether to purchase an AI commercial solution for imaging applications. • Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services.

Biomedical Ontologies to Guide AI Development in Radiology.

The advent of deep learning has engendered renewed and rapidly growing interest in artificial intelligence (AI) in radiology to analyze images, manipulate textual reports, and plan interventions. Applications of deep learning and other AI approaches must be guided by sound medical knowledge to assure that they are developed successfully and that they address important problems in biomedical research or patient care. To date, AI has been applied to a limited number of real-world radiology applications. As AI systems become more pervasive and are applied more broadly, they will benefit from medical knowledge on a larger scale, such as that available through computer-based approaches. A key approach to represent computer-based knowledge in a particular domain is an ontology. As defined in informatics, an ontology defines a domain's terms through their relationships with other terms in the ontology. Those relationships, then, define the terms' semantics, or "meaning." Biomedical ontologies commonly define the relationships between terms and more general terms, and can express causal, part-whole, and anatomic relationships. Ontologies express knowledge in a form that is both human-readable and machine-computable. Some ontologies, such as RSNA's RadLex radiology lexicon, have been applied to applications in clinical practice and research, and may be familiar to many radiologists. This article describes how ontologies can support research and guide emerging applications of AI in radiology, including natural language processing, image-based machine learning, radiomics, and planning.

Automating Import and Reconciliation of Outside Examinations Submitted to an Academic Radiology Department.

Although advances in electronic image sharing have made continuity of patient care easier, currently, the majority of outside studies are received on CD. At our institution, there were 9 full-time employees (FTE) at three locations using three workflows to manually upload, schedule, and process studies to PACS. As the demand to view and store outside studies has grown, so has the processing turnaround time. To reduce turnaround time and the need for human intervention, we developed an automated workflow to import outside studies from a CD to our PACS and reconcile them with an internal accession number and exam code.

Evaluating Completeness of a Radiology Glossary Using Iterative Refinement.

A lay-language glossary of radiology, built to help patients better understand the content of their radiology reports, has been analyzed for its coverage and readability, but not for its completeness. We present an iterative method to sample radiology reports, identify "missing" terms, and measure the glossary's completeness. We hypothesized that the refinement process would reduce the number of missing terms to fewer than 1 per report. A random sample of 1000 radiology reports from a large US academic health system was divided into 10 cohorts of 100 reports each. Each cohort was reviewed in sequence by two investigators to identify terms (single words and multi-word phrases) absent from the glossary. Terms marked as new were added to the glossary and hence was shown as matched in subsequent cohorts. This HIPAA-compliant study was IRB-approved; informed consent was waived. The refinement process added a mean of 288.0 new terms per 100 reports in the first 5 cohorts vs. a mean of 66.0 new terms per 100 reports in the last 5 cohorts; the difference was statistically significant (p < .01). After reviewing 500 reports, the review process found fewer than 1 new term per report in each of 500 subsequent reports. The findings suggest that 500 to 1000 reports is adequate to test the completeness of a glossary, and that the glossary after iterative refinement achieved a high level of completeness to cover the vocabulary of radiology reports.

Integrating Wikipedia Articles and Images into an Information Resource for Radiology Patients.

Wikipedia-an open-access online encyclopedia-contains a large number of medically relevant articles and images that may help supplement glossaries of radiology terms. We sought to determine the extent to which concepts from a large online radiology glossary developed as part of the Patient-Oriented Radiology Reporter (PORTER) initiative could be mapped to relevant Wikipedia web pages and images using automated or semi-automated approaches. The glossary included 4090 concepts with their definitions; the concept's preferred name and lexical variants, such as plurals, adjectival forms, synonyms, and abbreviations, yielded a total of 13,030 terms. Of the 4090 concepts, 3063 (74.9%) had a corresponding English-language Wikipedia page identified by automated search with subsequent manual review. We applied the MediaWiki application programming interface (API) to generate web-service calls to identify the images from each concept's corresponding Wikipedia page; three reviewers selected relevant images to associate with the glossary's concepts. Licensing terms for the images were reviewed. For 800 randomly sampled concepts that had associated Wikipedia pages, 362 distinct images were identified from the MediaWiki library and matched to 404 concepts (51%). Three images (1%) had unspecified licensing terms; the rest were in the public domain or available via a Creative Commons license. Wikipedia and the MediaWiki library offer a large collection of medical articles and images that can be incorporated into an online lay-language glossary of radiology terms though a semi-automated approach.

Integrating an Ontology of Radiology Differential Diagnosis with ICD-10-CM, RadLex, and SNOMED CT.

An ontology offers a human-readable and machine-computable representation of the concepts in a domain and the relationships among them. Mappings between ontologies enable the reuse and interoperability of biomedical knowledge. We sought to map concepts of the Radiology Gamuts Ontology (RGO), an ontology that links diseases and imaging findings to support differential diagnosis in radiology, to terms in three key vocabularies for clinical radiology: the International Classification of Diseases, version 10, Clinical Modification (ICD-10-CM), the Radiological Society of North America's radiology lexicon (RadLex), and the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT). RGO (version 0.7; Jan 2018) incorporated 16,918 terms (classes) for diseases, interventions, and imaging observations linked by 1782 subsumption (class-subclass) relations and 55,569 causal ("may cause") relations. RGO classes were mapped to RadLex (46,656 classes, version 3.15), SNOMED CT (347,358 classes, version 2018AA), and ICD-10-CM (94,645 classes, version 2018AA) using the National Center for Biomedical Ontology (NCBO) Annotator web service. We identified 1275 exact mappings from RGO to RadLex, 5302 to SNOMED CT, and 941 to ICD-10-CM. RGO terms mapped to one ontology (n = 3401), two ontologies (n = 1515), or all three ontologies (n = 198). The mapped ontologies provide additional terms to support data mining from textual information in the electronic health record. The current work builds on efforts to map RGO to ontologies of diseases and phenotypes. Mappings between ontologies can support automated knowledge discovery, diagnostic reasoning, and data mining.

DICOMweb™: Background and Application of the Web Standard for Medical Imaging.

This paper describes why and how DICOM, the standard that has been the basis for medical imaging interoperability around the world for several decades, has been extended into a full web technology-based standard, DICOMweb. At the turn of the century, healthcare embraced information technology, which created new problems and new opportunities for the medical imaging industry; at the same time, web technologies matured and began serving other domains well. This paper describes DICOMweb, how it extended the DICOM standard, and how DICOMweb can be applied to problems facing healthcare applications to address workflow and the changing healthcare climate.

Common Data Elements in Radiology.

Diagnostic radiologists generally produce unstructured information in the form of images and narrative text reports. Although designed for human consumption, radiologic reports contain a wealth of information that could be valuable for clinical care, research, and quality improvement if that information could be extracted by automated systems. Unfortunately, the lack of structure in radiologic reports limits the ability of information systems to share information easily with other systems. A common data element (CDE)-a unit of information used in a shared, predefined fashion-can improve the ability to exchange information seamlessly among information systems. In this article, a model and a repository of radiologic CDEs is described, and three important applications are highlighted. CDEs can help advance radiologic practice, research, and performance improvement, and thus, it is crucial that CDEs be adopted widely in radiologic information systems. © RSNA, 2016.

Enabling the Next-Generation Radiology Report: Description of Two New System Standards.

Currently, most radiology reports are highly variable and consist of unconstrained narrative text. This variability limits the ability to extract information from the report to guide clinical care, populate a data registry, or support quality improvement. This article introduces two newly available standards that pertain to radiology reports. Management of Radiology Reporting Templates (MRRT) is an integration profile that defines the format and exchange mechanisms for radiology report templates. Digital Imaging and Communications in Medicine Part 20 defines how reports built using MRRT-based templates can be transmitted into an electronic health record (EHR). Together, these two standards enable new ways to improve report consistency and completeness, ensure proper clinical action, and improve the quality of patient care. Commercial and open-source developers are beginning to incorporate these standards into clinical systems. The authors use an example of a patient with an incidentally detected lung nodule to illustrate how these standards improve the exchange of information. The clinical scenario follows the use of the appropriate template through the completion of the radiology report, with the incidental finding structured and coded to enable automated follow-up in the EHR. ©RSNA, 2017.