Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework.

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Upgrade Rate of Atypical Ductal Hyperplasia: Ten Years Experience and Predictive Factors.

BACKGROUND: Atypical ductal hyperplasia (ADH) is a benign epithelial proliferative lesion with histologic features resembling those seen in low grade ductal carcinoma in situ (DCIS). Surgical excision of the biopsy site is the standard management approach. The objective of this study was to determine the upgrade rate from ADH on stereotactic breast biopsies to DCIS or invasive carcinoma (IC) in our institution. We also sought to identify clinical, pathologic and radiologic predictive factors associated with risk of upgrade. MATERIALS AND METHODS: Clinical charts, mammograms and pathology reports were reviewed for all women with a stereotactic breast biopsy showing ADH and subsequent surgery at our institution between 2008 and 2018. When available, mammograms were re-reviewed by a radiologist for this study. RESULTS: 295 biopsies were analyzed in 290 patients. Mean age was 56 y old. Upgrade rate was 10.5% of which 7.5% were DCIS and 3.1% IC. Mammograms were reviewed by a radiologist in 161 patients from 2013 to 2018. In this subset of patients, the rate of upgrade was 8.7% (4.35% DCIS and 4.35% IC). A statistically significant difference he largest size of the microcalcification clusters on mammogram was observed between the upgraded and the non-upgraded subgroups (14.2 mm versus 8.9 mm, P = 0.03) CONCLUSIONS: The evaluation of the largest size of microcalcification clusters on mammogram as a cut-off feature could be considered to choose between an observational versus a surgical approach. This large series provides contemporary data to assist informed decision making regarding the treatment of our patients.

Deep Learning: An Update for Radiologists.

Deep learning is a class of machine learning methods that has been successful in computer vision. Unlike traditional machine learning methods that require hand-engineered feature extraction from input images, deep learning methods learn the image features by which to classify data. Convolutional neural networks (CNNs), the core of deep learning methods for imaging, are multilayered artificial neural networks with weighted connections between neurons that are iteratively adjusted through repeated exposure to training data. These networks have numerous applications in radiology, particularly in image classification, object detection, semantic segmentation, and instance segmentation. The authors provide an update on a recent primer on deep learning for radiologists, and they review terminology, data requirements, and recent trends in the design of CNNs; illustrate building blocks and architectures adapted to computer vision tasks, including generative architectures; and discuss training and validation, performance metrics, visualization, and future directions. Familiarity with the key concepts described will help radiologists understand advances of deep learning in medical imaging and facilitate clinical adoption of these techniques. Online supplemental material is available for this article. ©RSNA, 2021.

A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images.

Real-time execution of machine learning (ML) pipelines on radiology images is difficult due to limited computing resources in clinical environments, whereas running them in research clusters requires efficient data transfer capabilities. We developed Niffler, an open-source Digital Imaging and Communications in Medicine (DICOM) framework that enables ML and processing pipelines in research clusters by efficiently retrieving images from the hospitals' PACS and extracting the metadata from the images. We deployed Niffler at our institution (Emory Healthcare, the largest healthcare network in the state of Georgia) and retrieved data from 715 scanners spanning 12 sites, up to 350 GB/day continuously in real-time as a DICOM data stream over the past 2 years. We also used Niffler to retrieve images bulk on-demand based on user-provided filters to facilitate several research projects. This paper presents the architecture and three such use cases of Niffler. First, we executed an IVC filter detection and segmentation pipeline on abdominal radiographs in real-time, which was able to classify 989 test images with an accuracy of 96.0%. Second, we applied the Niffler Metadata Extractor to understand the operational efficiency of individual MRI systems based on calculated metrics. We benchmarked the accuracy of the calculated exam time windows by comparing Niffler against the Clinical Data Warehouse (CDW). Niffler accurately identified the scanners' examination timeframes and idling times, whereas CDW falsely depicted several exam overlaps due to human errors. Third, with metadata extracted from the images by Niffler, we identified scanners with misconfigured time and reconfigured five scanners. Our evaluations highlight how Niffler enables real-time ML and processing pipelines in a research cluster.

Tackling the Radiological Society of North America Pneumonia Detection Challenge.

OBJECTIVE. We provide overviews of deep learning approaches used by two top-placing teams for the 2018 Radiological Society of North America (RSNA) Pneumonia Detection Challenge. CONCLUSION. Practical applications of deep learning techniques, as well as insights into the annotation of the data, were keys to success in accurately detecting pneumonia on chest radiographs for the competition.

Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology.

Artificial intelligence (AI) is rapidly moving from an experimental phase to an implementation phase in many fields, including medicine. The combination of improved availability of large datasets, increasing computing power, and advances in learning algorithms has created major performance breakthroughs in the development of AI applications. In the last 5 years, AI techniques known as deep learning have delivered rapidly improving performance in image recognition, caption generation, and speech recognition. Radiology, in particular, is a prime candidate for early adoption of these techniques. It is anticipated that the implementation of AI in radiology over the next decade will significantly improve the quality, value, and depth of radiology's contribution to patient care and population health, and will revolutionize radiologists' workflows. The Canadian Association of Radiologists (CAR) is the national voice of radiology committed to promoting the highest standards in patient-centered imaging, lifelong learning, and research. The CAR has created an AI working group with the mandate to discuss and deliberate on practice, policy, and patient care issues related to the introduction and implementation of AI in imaging. This white paper provides recommendations for the CAR derived from deliberations between members of the AI working group. This white paper on AI in radiology will inform CAR members and policymakers on key terminology, educational needs of members, research and development, partnerships, potential clinical applications, implementation, structure and governance, role of radiologists, and potential impact of AI on radiology in Canada.

Simulating federated learning for steatosis detection using ultrasound images.

We aimed to implement four data partitioning strategies evaluated with four federated learning (FL) algorithms and investigate the impact of data distribution on FL model performance in detecting steatosis using B-mode US images. A private dataset (153 patients; 1530 images) and a public dataset (55 patient; 550 images) were included in this retrospective study. The datasets contained patients with metabolic dysfunction-associated fatty liver disease (MAFLD) with biopsy-proven steatosis grades and control individuals without steatosis. We employed four data partitioning strategies to simulate FL scenarios and we assessed four FL algorithms. We investigated the impact of class imbalance and the mismatch between the global and local data distributions on the learning outcome. Classification performance was assessed with area under the receiver operating characteristic curve (AUC) on a separate test set. AUCs were 0.93 (95% CI 0.92, 0.94) for source-based partitioning scenario with FedAvg, 0.90 (95% CI 0.89, 0.91) for a centralized model, and 0.83 (95% CI 0.81, 0.85) for a model trained in a single-center scenario. When data was perfectly balanced on the global level and each site had an identical data distribution, the model yielded an AUC of 0.90 (95% CI 0.88, 0.92). When each site contained data exclusively from one single class, irrespective of the global data distribution, the AUC fell in the range of 0.34-0.70. FL applied to B-mode US images provide performance comparable to a centralized model and higher than single-center scenario. Global data imbalance and local data heterogeneity influenced the learning outcome.

Interpretable clinical phenotypes among patients hospitalized with COVID-19 using cluster analysis.

Background: Multiple clinical phenotypes have been proposed for coronavirus disease (COVID-19), but few have used multimodal data. Using clinical and imaging data, we aimed to identify distinct clinical phenotypes in patients admitted with COVID-19 and to assess their clinical outcomes. Our secondary objective was to demonstrate the clinical applicability of this method by developing an interpretable model for phenotype assignment. Methods: We analyzed data from 547 patients hospitalized with COVID-19 at a Canadian academic hospital. We processed the data by applying a factor analysis of mixed data (FAMD) and compared four clustering algorithms: k-means, partitioning around medoids (PAM), and divisive and agglomerative hierarchical clustering. We used imaging data and 34 clinical variables collected within the first 24 h of admission to train our algorithm. We conducted a survival analysis to compare the clinical outcomes across phenotypes. With the data split into training and validation sets (75/25 ratio), we developed a decision-tree-based model to facilitate the interpretation and assignment of the observed phenotypes. Results: Agglomerative hierarchical clustering was the most robust algorithm. We identified three clinical phenotypes: 79 patients (14%) in Cluster 1, 275 patients (50%) in Cluster 2, and 203 (37%) in Cluster 3. Cluster 2 and Cluster 3 were both characterized by a low-risk respiratory and inflammatory profile but differed in terms of demographics. Compared with Cluster 3, Cluster 2 comprised older patients with more comorbidities. Cluster 1 represented the group with the most severe clinical presentation, as inferred by the highest rate of hypoxemia and the highest radiological burden. Intensive care unit (ICU) admission and mechanical ventilation risks were the highest in Cluster 1. Using only two to four decision rules, the classification and regression tree (CART) phenotype assignment model achieved an AUC of 84% (81.5-86.5%, 95 CI) on the validation set. Conclusions: We conducted a multidimensional phenotypic analysis of adult inpatients with COVID-19 and identified three distinct phenotypes associated with different clinical outcomes. We also demonstrated the clinical usability of this approach, as phenotypes can be accurately assigned using a simple decision tree. Further research is still needed to properly incorporate these phenotypes in the management of patients with COVID-19.

Impact of Fasting Status on the Use of Klotho as a Biomarker.

BACKGROUND: Klotho is a protein secreted physiologically in humans. It acts like a hormone that regulates many biological processes. It is also a novel serological biomarker that is increasingly used as a predictive factor for several physiological and psychological conditions. Surprisingly, there is no consensus about the fasting state of the patient who is tested for klotho. Most studies are done on fasting patients, although others are done without concern about fasting status. There is a lack of evidence about this variable in klotho serological testing. Performing fasting tests on patients can be deleterious and can affect compliance. We investigated the effect of fasting status on klotho serological value. METHODS: We conducted an observational study in which klotho serology was evaluated in a fasting state and 2 h after a meal. In total, 35 participants came to the laboratory without having eaten for 10 h. Blood samples were taken on arrival at our laboratory and 2 h after eating a standardized meal. RESULTS: The mean age of our participants was 32.7 years old. There were 13 men and 22 women. In the fasting state, the klotho value was 1060.5 pg/mL (SD: 557.5 pg/mL). At 2 h after the meal, the klotho value was 1077.5 pg/mL (SD: 576.9 pg/mL). Statistical tests showed no difference before and after a meal in our study (P = 0.2425). CONCLUSIONS: Our results suggest that it is not necessary to perform klotho serology in a fasting state.

Deep learning workflow in radiology: a primer.

Interest for deep learning in radiology has increased tremendously in the past decade due to the high achievable performance for various computer vision tasks such as detection, segmentation, classification, monitoring, and prediction. This article provides step-by-step practical guidance for conducting a project that involves deep learning in radiology, from defining specifications, to deployment and scaling. Specifically, the objectives of this article are to provide an overview of clinical use cases of deep learning, describe the composition of multi-disciplinary team, and summarize current approaches to patient, data, model, and hardware selection. Key ideas will be illustrated by examples from a prototypical project on imaging of colorectal liver metastasis. This article illustrates the workflow for liver lesion detection, segmentation, classification, monitoring, and prediction of tumor recurrence and patient survival. Challenges are discussed, including ethical considerations, cohorting, data collection, anonymization, and availability of expert annotations. The practical guidance may be adapted to any project that requires automated medical image analysis.

A comparison of cervical spine motion during orotracheal intubation with the trachlight(r) or the flexible fiberoptic bronchoscope.

BACKGROUND: Tracheal intubation of an unstable cervical spine (c-spine) patient with the flexible fiberoptic bronchoscope (FOB) is thought to minimize c-spine movement but may be technically difficult in certain patients. Intubation using a luminous stylet, such as the Trachlight(R) (TL), also produces minimal motion of the c-spine and may be an interesting alternative technique for patients with an unstable c-spine. In this study, we compared the cervical motion caused by the TL and the FOB during intubation. METHODS: Twenty patients with a normal c-spine undergoing general anesthesia, including neuromuscular blockade, for a neuroradiologic intervention were included in a prospective, randomized, controlled, nonblinded, crossover trial. Each patient was tracheally intubated sequentially with the TL and the FOB in a randomized order. Manual in-line stabilization was applied by an assistant during intubation. The motions produced by intubation from the occiput (C0) to C5 were recorded in the sagittal plane using continuous cinefluoroscopy. For movement analysis, the recordings were divided into four stages: "baseline" before intubation began; "introduction" of the intubation device; "intubation" (passage of the tube through the vocal cords); and "removal" of the device. For each intubating device, the average maximal segmental motion observed in every patient at any stage or cervical segment was calculated and compared using Student's t-test. The time required to intubate with each device was also compared. RESULTS: There was no significant difference in the mean maximum segmental motion produced during intubation with the TL versus the FOB (12 degrees +/- 6 degrees vs 11 degrees +/- 5 degrees ; P = 0.5). Segmental movements occurred predominantly at the C0-1 and C1-2 levels, and maximal movements were observed during the introduction stage in 18/20 patients for both devices. Intubation took less time with the TL (34 +/- 17 vs 60 +/- 15 s, P < 0.001). CONCLUSION: In patients under general anesthesia with neuromuscular blockade and manual in-line stabilization, we found no difference in the segmental c-spine motion produced during endotracheal intubation using the FOB and the TL.

Coronary computed tomography angiography: overview of technical aspects, current concepts, and perspectives.

Multidetector-row electrocardiogram (ECG)-gated cardiac computed tomography (CT) will probably be a major noninvasive imaging option in the near future. Recent developments indicate that this new technology is improving rapidly. This article presents an overview of the current concepts, perspectives, and technical capabilities in coronary CT angiography (CTA). We have reviewed the recent literature on the different applications of this technology; of particular note are the many studies that have demonstrated the high negative predictive value (NPV) of coronary CTA, when performed under optimal conditions, for significant stenoses in native coronary arteries. This new technology's level of performance allows it to be used to evaluate the presence of calcified plaques, coronary bypass graft patency, and the origin and course of congenital coronary anomalies. Despite a high NPV, the robustness of the technology is limited by arrhythmias, the requirement of low heart rates, and calcium-related artifacts. Some improvements are needed in the imaging of coronary stents, especially the smaller stents, and in the detection and characterization of noncalcified plaques. Further studies are needed to more precisely determine the role of CTA in various symptomatic and asymptomatic patient groups. Clinical testing of 64-slice scanners has recently begun. As the technology improves, so does the spatial and temporal resolution. To date, this is being achieved through the development of systems with an increased number of detectors and shorter gantry rotation time, as well as the development of systems equipped with 2 X-ray tubes and the eventual development of flat-panel technology. Thus further improvement of image quality is expected.