Ontologies in the New Computational Age of Radiology: RadLex for Semantics and Interoperability in Imaging Workflows.

From basic research to the bedside, precise terminology is key to advancing medicine and ensuring optimal and appropriate patient care. However, the wide spectrum of diseases and their manifestations superimposed on medical team-specific and discipline-specific communication patterns often impairs shared understanding and the shared use of common medical terminology. Common terms are currently used in medicine to ensure interoperability and facilitate integration of biomedical information for clinical practice and emerging scientific and educational applications alike, from database integration to supporting basic clinical operations such as billing. Such common terminologies can be provided in ontologies, which are formalized representations of knowledge in a particular domain. Ontologies unambiguously specify common concepts and describe the relationships between those concepts by using a form that is mathematically precise and accessible to humans and machines alike. RadLex® is a key RSNA initiative that provides a shared domain model, or ontology, of radiology to facilitate integration of information in radiology education, clinical care, and research. As the contributions of the computational components of common radiologic workflows continue to increase with the ongoing development of big data, artificial intelligence, and novel image analysis and visualization tools, the use of common terminologies is becoming increasingly important for supporting seamless computational resource integration across medicine. This article introduces ontologies, outlines the fundamental semantic web technologies used to create and apply RadLex, and presents examples of RadLex applications in everyday radiology and research. It concludes with a discussion of emerging applications of RadLex, including artificial intelligence applications. © RSNA, 2023 Quiz questions for this article are available in the supplemental material.

Improving the Efficacy of ACR TI-RADS Through Deep Learning-Based Descriptor Augmentation.

Thyroid nodules occur in up to 68% of people, 95% of which are benign. Of the 5% of malignant nodules, many would not result in symptoms or death, yet 600,000 FNAs are still performed annually, with a PPV of 5-7% (up to 30%). Artificial intelligence (AI) systems have the capacity to improve diagnostic accuracy and workflow efficiency when integrated into clinical decision pathways. Previous studies have evaluated AI systems against physicians, whereas we aim to compare the benefits of incorporating AI into their final diagnostic decision. This work analyzed the potential for artificial intelligence (AI)-based decision support systems to improve physician accuracy, variability, and efficiency. The decision support system (DSS) assessed was Koios DS, which provides automated sonographic nodule descriptor predictions and a direct cancer risk assessment aligned to ACR TI-RADS. The study was conducted retrospectively between (08/2020) and (10/2020). The set of cases used included 650 patients (21% male, 79% female) of age 53 ± 15. Fifteen physicians assessed each of the cases in the set, both unassisted and aided by the DSS. The order of the reading condition was randomized, and reading blocks were separated by a period of 4 weeks. The system's impact on reader accuracy was measured by comparing the area under the ROC curve (AUC), sensitivity, and specificity of readers with and without the DSS with FNA as ground truth. The impact on reader variability was evaluated using Pearson's correlation coefficient. The impact on efficiency was determined by comparing the average time per read. There was a statistically significant increase in average AUC of 0.083 [0.066, 0.099] and an increase in sensitivity and specificity of 8.4% [5.4%, 11.3%] and 14% [12.5%, 15.5%], respectively, when aided by Koios DS. The average time per case decreased by 23.6% (p = 0.00017), and the observed Pearson's correlation coefficient increased from r = 0.622 to r = 0.876 when aided by Koios DS. These results indicate that providing physicians with automated clinical decision support significantly improved diagnostic accuracy, as measured by AUC, sensitivity, and specificity, and reduced inter-reader variability and interpretation times.

Imaging Informatics Fellowship Curriculum: Building Consensus on the Most Critical Topics and the Future of the Informatics Fellowship.

The existing fellowship imaging informatics curriculum, established in 2004, has not undergone formal revision since its inception and inaccurately reflects present-day radiology infrastructure. It insufficiently equips trainees for today's informatics challenges as current practices require an understanding of advanced informatics processes and more complex system integration. We sought to address this issue by surveying imaging informatics fellowship program directors across the country to determine the components and cutline for essential topics in a standardized imaging informatics curriculum, the consensus on essential versus supplementary knowledge, and the factors individual programs may use to determine if a newly developed topic is an essential topic. We further identified typical program structural elements and sought fellowship director consensus on offering official graduate trainee certification to imaging informatics fellows. Here, we aim to provide an imaging informatics fellowship director consensus on topics considered essential while still providing a framework for informatics fellowship programs to customize their individual curricula.

Deep Learning Algorithms for Interpretation of Upper Extremity Radiographs: Laterality and Technologist Initial Labels as Confounding Factors.

Convolutional neural networks (CNNs) trained to identify abnormalities on upper extremity radiographs achieved an AUC of 0.844 with a frequent emphasis on radiograph laterality and/or technologist labels for decision-making. Covering the labels increased the AUC to 0.857 (p = .02) and redirected CNN attention from the labels to the bones. Using images of radiograph labels alone, the AUC was 0.638, indicating that radiograph labels are associated with abnormal examinations. Potential radiographic confounding features should be considered when curating data for radiology CNN development.

Pancreaticoduodenal Groove: Spectrum of Disease and Imaging Features.

The pancreaticoduodenal groove (PDG) is a small space between the pancreatic head and duodenum where vital interactions between multiple organs and physiologic processes take place. Muscles, nerves, and hormones perform a coordinated dance, allowing bile and pancreatic enzymes to aid in digestion and absorption of critical nutrition. Given the multitude of organs and cells working together, a variety of benign and malignant entities can arise in or adjacent to this space. Management of lesions in this region is also complex and can involve observation, endoscopic resection, or challenging surgeries such as the Whipple procedure. The radiologist plays an important role in evaluation of abnormalities involving the PDG. While CT is usually the first-line examination for evaluation of this complex region, MRI offers complementary information. Although features of abnormalities involving the PDG can often overlap, understanding the characteristic imaging and pathologic features generally allows categorization of disease entities based on the suspected organ of origin and the presence of ancillary features. The goal of the authors is to provide radiologists with a conceptual approach to entities implicating the PDG to increase the accuracy of diagnosis and assist in appropriate management or presurgical planning. They briefly discuss the anatomy of the PDG, followed by a more in-depth presentation of the features of disease categories. A table summarizing the entities that occur in this region by underlying cause and anatomic location is provided. ©RSNA, 2022.

Federated Deep Learning to More Reliably Detect Body Part for Hanging Protocols, Relevant Priors, and Workflow Optimization.

Preparing radiology examinations for interpretation requires prefetching relevant prior examinations and implementing hanging protocols to optimally display the examination along with comparisons. Body part is a critical piece of information to facilitate both prefetching and hanging protocols, but body part information encoded using the Digital Imaging and Communications in Medicine (DICOM) standard is widely variable, error-prone, not granular enough, or missing altogether. This results in inappropriate examinations being prefetched or relevant examinations left behind; hanging protocol optimization suffers as well. Modern artificial intelligence (AI) techniques, particularly when harnessing federated deep learning techniques, allow for highly accurate automatic detection of body part based on the image data within a radiological examination; this allows for much more reliable implementation of this categorization and workflow. Additionally, new avenues to further optimize examination viewing such as dynamic hanging protocol and image display can be implemented using these techniques.

Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.

Biomedical Ontologies to Guide AI Development in Radiology.

The advent of deep learning has engendered renewed and rapidly growing interest in artificial intelligence (AI) in radiology to analyze images, manipulate textual reports, and plan interventions. Applications of deep learning and other AI approaches must be guided by sound medical knowledge to assure that they are developed successfully and that they address important problems in biomedical research or patient care. To date, AI has been applied to a limited number of real-world radiology applications. As AI systems become more pervasive and are applied more broadly, they will benefit from medical knowledge on a larger scale, such as that available through computer-based approaches. A key approach to represent computer-based knowledge in a particular domain is an ontology. As defined in informatics, an ontology defines a domain's terms through their relationships with other terms in the ontology. Those relationships, then, define the terms' semantics, or "meaning." Biomedical ontologies commonly define the relationships between terms and more general terms, and can express causal, part-whole, and anatomic relationships. Ontologies express knowledge in a form that is both human-readable and machine-computable. Some ontologies, such as RSNA's RadLex radiology lexicon, have been applied to applications in clinical practice and research, and may be familiar to many radiologists. This article describes how ontologies can support research and guide emerging applications of AI in radiology, including natural language processing, image-based machine learning, radiomics, and planning.

Radiology Reporting in the Era of Patient-Centered Care: How Can We Improve Readability?

Radiology reports are consumed not only by referring physicians and healthcare providers, but also by patients. We assessed report readability in our enterprise and implemented a two-part quality improvement intervention with the goal of improving report accessibility. A total of 491,813 radiology reports from ten hospitals within the enterprise from May to October, 2018 were collected. We excluded echocardiograms, rehabilitation reports, administrator reports, and reports with negative scores leaving 461,219 reports and report impressions for analysis. A grade level (GL) was calculated for each report and impression by averaging four readability metrics. Next, we conducted a readability workshop and distributed weekly emails with readability GLs over a period of 6 months to each attending radiologist at our primary institution. Following this intervention, we utilized the same exclusion criteria and analyzed 473,612 reports from May to October, 2019. The mean GL for all reports and report impressions was above 13 at every hospital in the enterprise. Following our intervention, a statistically significant drop in GL for reports and impressions was demonstrated at all locations, but a larger and significant improvement was observed in impressions at our primary site. Radiology reports across the enterprise are written at an advanced reading level making them difficult for patients and their families to understand. We observed a significantly larger drop in GL for impressions at our primary site than at all other sites following our intervention. Radiologists at our home institution improved their report readability after becoming more aware of their writing practices.

Streamlining Radiologist Workflow for Multidisciplinary Conferences: A Web-Based System to Represent Radiology.

Radiologists are an integral component in patient care and provide valuable information at multidisciplinary tumor boards. However, the radiologists' role at such meetings can be compromised by technical and workflow limitations, typically including the need for complex software such as picture archiving and communication system (PACS) applications which are difficult to install and manage in disparate locations with increasing security and network limitations. Our purpose was to develop a web-based system for easy retrieval of images and notes for presentation during multidisciplinary conferences and tumor boards. Our system allows images to be viewed from any computer with a web browser and does not require a stand-alone PACS software installation. The tool is launched by the radiologist marking the exam in PACS. It stores relevant text-based information in a MySQL server and is indexed to the conference for which it is to be used. The exams are then viewed through a web browser, via the hospital intranet or virtual private network (VPN). A web-based viewing platform, provided by our PACS vendor, is used for image display. In the 28 months following implementation, our web-based conference system was well-received by our radiologists and is now fully integrated into daily practice. Our method streamlines radiologist workflow in preparing and presenting medical imaging at multidisciplinary conferences and overcomes many previous technical obstacles. In addition to its primary role for interdepartmental conferences, our system also functions as a teaching file, fostering radiologist education within our department.

Cinebot: Creation of Movies and Animated GIFs Directly from PACS-Efficiency in Presentation and Education.

The presentation of radiology exams can be enhanced through the use of dynamic images. Dynamic images differ from static images by the use of animation and are especially useful for depicting real-time activity such as the scrolling or the flow of contrast to enhance pathology. This is generally superior to a collection of static images as a representation of clinical workflow and provides a more robust appreciation of the case in question. Dynamic images can be shared electronically to facilitate teaching, case review, presentation, and sharing of interesting cases to be viewed in detail on a computer or mobile devices for education. The creation of movies or animated images from radiology data has traditionally been challenging based on technological limitations inherent in converting the Digital Imaging and Communications in Medicine (DICOM) standard to other formats or concerns related to the presence of protected health information (PHI). The solution presented here, named Cinebot, allows a simple "one-click" generation of anonymized dynamic movies or animated images within the picture archiving and communication system (PACS) workflow. This approach works across all imaging modalities, including stacked cross-sectional and multi-frame cine formats. Usage statistics over 2 years have shown this method to be well-received and useful throughout our enterprise.

Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset.

Pneumothorax is a potentially life-threatening condition that requires prompt recognition and often urgent intervention. In the ICU setting, large numbers of chest radiographs are performed and must be interpreted on a daily basis which may delay diagnosis of this entity. Development of artificial intelligence (AI) techniques to detect pneumothorax could help expedite detection as well as localize and potentially quantify pneumothorax. Open image analysis competitions are useful in advancing state-of-the art AI algorithms but generally require large expert annotated datasets. We have annotated and adjudicated a large dataset of chest radiographs to be made public with the goal of sparking innovation in this space. Because of the cumbersome and time-consuming nature of image labeling, we explored the value of using AI models to generate annotations for review. Utilization of this machine learning annotation (MLA) technique appeared to expedite our annotation process with relatively high sensitivity at the expense of specificity. Further research is required to confirm and better characterize the value of MLAs. Our adjudicated dataset is now available for public consumption in the form of a challenge.

Integrating an Ontology of Radiology Differential Diagnosis with ICD-10-CM, RadLex, and SNOMED CT.

An ontology offers a human-readable and machine-computable representation of the concepts in a domain and the relationships among them. Mappings between ontologies enable the reuse and interoperability of biomedical knowledge. We sought to map concepts of the Radiology Gamuts Ontology (RGO), an ontology that links diseases and imaging findings to support differential diagnosis in radiology, to terms in three key vocabularies for clinical radiology: the International Classification of Diseases, version 10, Clinical Modification (ICD-10-CM), the Radiological Society of North America's radiology lexicon (RadLex), and the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT). RGO (version 0.7; Jan 2018) incorporated 16,918 terms (classes) for diseases, interventions, and imaging observations linked by 1782 subsumption (class-subclass) relations and 55,569 causal ("may cause") relations. RGO classes were mapped to RadLex (46,656 classes, version 3.15), SNOMED CT (347,358 classes, version 2018AA), and ICD-10-CM (94,645 classes, version 2018AA) using the National Center for Biomedical Ontology (NCBO) Annotator web service. We identified 1275 exact mappings from RGO to RadLex, 5302 to SNOMED CT, and 941 to ICD-10-CM. RGO terms mapped to one ontology (n = 3401), two ontologies (n = 1515), or all three ontologies (n = 198). The mapped ontologies provide additional terms to support data mining from textual information in the electronic health record. The current work builds on efforts to map RGO to ontologies of diseases and phenotypes. Mappings between ontologies can support automated knowledge discovery, diagnostic reasoning, and data mining.

Effectiveness of Deep Learning Algorithms to Determine Laterality in Radiographs.

Develop a highly accurate deep learning model to reliably classify radiographs by laterality. Digital Imaging and Communications in Medicine (DICOM) data for nine body parts was extracted retrospectively. Laterality was determined directly if encoded properly or inferred using other elements. Curation confirmed categorization and identified inaccurate labels due to human error. Augmentation enriched training data to semi-equilibrate classes. Classification and object detection models were developed on a dedicated workstation and tested on novel images. Receiver operating characteristic (ROC) curves, sensitivity, specificity, and accuracy were calculated. Study-level accuracy was determined and both were compared to human performance. An ensemble model was tested for the rigorous use-case of automatically classifying exams retrospectively. The final classification model identified novel images with an ROC area under the curve (AUC) of 0.999, improving on previous work and comparable to human performance. A similar ROC curve was observed for per-study analysis with AUC of 0.999. The object detection model classified images with accuracy of 99% or greater at both image and study level. Confidence scores allow adjustment of sensitivity and specificity as needed; the ensemble model designed for the highly specific use-case of automatically classifying exams was comparable and arguably better than human performance demonstrating 99% accuracy with 1% of exams unchanged and no incorrect classification. Deep learning models can classify radiographs by laterality with high accuracy and may be applied in a variety of settings that could improve patient safety and radiologist satisfaction. Rigorous use-cases requiring high specificity are achievable.

Developing Deeper Radiology Exam Insight to Optimize MRI Workflow and Patient Experience.

Process variability during the acquisition of magnetic resonance imaging (MRI) can lengthen examination times and introduce unexpected exam differences which can negatively impact the cost and quality of care provided to patients. Digital Imaging and Communications in Medicine (DICOM) metadata can provide more accurate study data and granular series-level information that can be used to increase operational efficiency, optimize patient care, and reduce costs associated with MRI examinations. Systematic use of such data analysis could be used as a continuous operational optimization and quality control mechanism.

The Radiologist's Gaze: Mapping Three-Dimensional Visual Search in Computed Tomography of the Abdomen and Pelvis.

A radiologist's search pattern can directly influence patient management. A missed finding is a missed opportunity for intervention. Multiple studies have attempted to describe and quantify search patterns but have mainly focused on chest radiographs and chest CTs. Here, we describe and quantify the visual search patterns of 17 radiologists as they scroll through 6 CTs of the abdomen and pelvis. Search pattern tracings varied among individuals and remained relatively consistent per individual between cases. Attendings and trainees had similar eye metric statistics with respect to time to first fixation (TTFF), number of fixations in the region of interest (ROI), fixation duration in ROI, mean saccadic amplitude, or total number of fixations. Attendings had fewer numbers of fixations per second versus trainees (p < 0.001), suggesting efficiency due to expertise. In those cases that were accurately interpreted, TTFF was shorter (p = 0.04), the number of fixations per second and number of fixations in ROI were higher (p = 0.04, p = 0.02, respectively), and fixation duration in ROI was increased (p = 0.02). We subsequently categorized radiologists as "scanners" or "drillers" by both qualitative and quantitative methods and found no differences in accuracy with most radiologists being categorized as "drillers." This study describes visual search patterns of radiologists in interpretation of CTs of the abdomen and pelvis to better approach future endeavors in determining the effects of manipulations such as fatigue, interruptions, and computer-aided detection.

Minimizing Barriers in Learning for On-Call Radiology Residents-End-to-End Web-Based Resident Feedback System.

Feedback is an essential part of medical training, where trainees are provided with information regarding their performance and further directions for improvement. In diagnostic radiology, feedback entails a detailed review of the differences between the residents' preliminary interpretation and the attendings' final interpretation of imaging studies. While the on-call experience of independently interpreting complex cases is important to resident education, the more traditional synchronous "read-out" or joint review is impossible due to multiple constraints. Without an efficient method to compare reports, grade discrepancies, convey salient teaching points, and view images, valuable lessons in image interpretation and report construction are lost. We developed a streamlined web-based system, including report comparison and image viewing, to minimize barriers in asynchronous communication between attending radiologists and on-call residents. Our system provides real-time, end-to-end delivery of case-specific and user-specific feedback in a streamlined, easy-to-view format. We assessed quality improvement subjectively through surveys and objectively through participation metrics. Our web-based feedback system improved user satisfaction for both attending and resident radiologists, and increased attending participation, particularly with regards to cases where substantive discrepancies were identified.

Implementing Machine Learning in Radiology Practice and Research.

OBJECTIVE: The purposes of this article are to describe concepts that radiologists should understand to evaluate machine learning projects, including common algorithms, supervised as opposed to unsupervised techniques, statistical pitfalls, and data considerations for training and evaluation, and to briefly describe ethical dilemmas and legal risk. CONCLUSION: Machine learning includes a broad class of computer programs that improve with experience. The complexity of creating, training, and monitoring machine learning indicates that the success of the algorithms will require radiologist involvement for years to come, leading to engagement rather than replacement.

PathBot: A Radiology-Pathology Correlation Dashboard.

Pathology is considered the "gold standard" of diagnostic medicine. The importance of radiology-pathology correlation is seen in interdepartmental patient conferences such as "tumor boards" and by the tradition of radiology resident immersion in a radiologic-pathology course at the American Institute of Radiologic Pathology. In practice, consistent pathology follow-up can be difficult due to time constraints and cumbersome electronic medical records. We present a radiology-pathology correlation dashboard that presents radiologists with pathology reports matched to their dictations, for both diagnostic imaging and image-guided procedures. In creating our dashboard, we utilized the RadLex ontology and National Center for Biomedical Ontology (NCBO) Annotator to identify anatomic concepts in pathology reports that could subsequently be mapped to relevant radiology reports, providing an automated method to match related radiology and pathology reports. Radiology-pathology matches are presented to the radiologist on a web-based dashboard. We found that our algorithm was highly specific in detecting matches. Our sensitivity was slightly lower than expected and could be attributed to missing anatomy concepts in the RadLex ontology, as well as limitations in our parent term hierarchical mapping and synonym recognition algorithms. By automating radiology-pathology correlation and presenting matches in a user-friendly dashboard format, we hope to encourage pathology follow-up in clinical radiology practice for purposes of self-education and to augment peer review. We also hope to provide a tool to facilitate the production of quality teaching files, lectures, and publications. Diagnostic images have a richer educational value when they are backed up by the gold standard of pathology.