Real-World Performance of Large Vessel Occlusion Artificial Intelligence-Based Computer-Aided Triage and Notification Algorithms-What the Stroke Team Needs to Know.

PURPOSE: To evaluate the real-world performance of two FDA-approved artificial intelligence (AI)-based computer-aided triage and notification (CADt) detection devices and compare them with the manufacturer-reported performance testing in the instructions for use. MATERIALS AND METHODS: Clinical performance of two FDA-cleared CADt large-vessel occlusion (LVO) devices was retrospectively evaluated at two separate stroke centers. Consecutive "code stroke" CT angiography examinations were included and assessed for patient demographics, scanner manufacturer, presence or absence of CADt result, CADt result, and LVO in the internal carotid artery (ICA), horizontal middle cerebral artery (MCA) segment (M1), Sylvian MCA segments after the bifurcation (M2), precommunicating part of cerebral artery, postcommunicating part of the cerebral artery, vertebral artery, basilar artery vessel segments. The original radiology report served as the reference standard, and a study radiologist extracted the above data elements from the imaging examination and radiology report. RESULTS: At hospital A, the CADt algorithm manufacturer reports assessment of intracranial ICA and MCA with sensitivity of 97% and specificity of 95.6%. Real-world performance of 704 cases included 79 in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 85.3% and 91.9%. Sensitivity decreased to 68.5% when M2 segments were included and to 59.9% when all proximal vessel segments were included. At hospital B the CADt algorithm manufacturer reports sensitivity of 87.8% and specificity of 89.6%, without specifying the vessel segments. Real-world performance of 642 cases included 20 cases in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 90.7% and 97.9%. Sensitivity decreased to 76.4% when M2 segments were included and to 59.4% when all proximal vessel segments are included. DISCUSSION: Real-world testing of two CADt LVO detection algorithms identified gaps in the detection and communication of potentially treatable LVOs when considering vessels beyond the intracranial ICA and M1 segments and in cases with absent and uninterpretable data.

An FDA Guide on Indications for Use and Device Reporting of Artificial Intelligence-Enabled Devices: Significance for Pediatric Use.

Radiology has been a pioneer in adopting artificial intelligence (AI)-enabled devices into the clinic. However, initial clinical experience has identified concerns of inconsistent device performance across different patient populations. Medical devices, including those using AI, are cleared by the FDA for their specific indications for use (IFUs). IFU describes the disease or condition the device will diagnose or treat, including a description of the intended patient population. Performance data evaluated during the premarket submission support the IFU and include the intended patient population. Understanding the IFUs of a given device is thus critical to ensuring that the device is used properly and performs as expected. When devices do not perform as expected or malfunction, medical device reporting is an important way to provide feedback about the device to the manufacturer, the FDA, and other users. This article describes the ways to retrieve the IFU and performance data information as well as the FDA medical device reporting systems for unexpected performance discrepancy. It is crucial that imaging professionals, including radiologists, know how to access and use these tools to improve the informed use of medical devices for patients of all ages.

Regulatory considerations for medical imaging AI/ML devices in the United States: concepts and challenges.

Purpose: To introduce developers to medical device regulatory processes and data considerations in artificial intelligence and machine learning (AI/ML) device submissions and to discuss ongoing AI/ML-related regulatory challenges and activities. Approach: AI/ML technologies are being used in an increasing number of medical imaging devices, and the fast evolution of these technologies presents novel regulatory challenges. We provide AI/ML developers with an introduction to U.S. Food and Drug Administration (FDA) regulatory concepts, processes, and fundamental assessments for a wide range of medical imaging AI/ML device types. Results: The device type for an AI/ML device and appropriate premarket regulatory pathway is based on the level of risk associated with the device and informed by both its technological characteristics and intended use. AI/ML device submissions contain a wide array of information and testing to facilitate the review process with the model description, data, nonclinical testing, and multi-reader multi-case testing being critical aspects of the AI/ML device review process for many AI/ML device submissions. The agency is also involved in AI/ML-related activities that support guidance document development, good machine learning practice development, AI/ML transparency, AI/ML regulatory research, and real-world performance assessment. Conclusion: FDA's AI/ML regulatory and scientific efforts support the joint goals of ensuring patients have access to safe and effective AI/ML devices over the entire device lifecycle and stimulating medical AI/ML innovation.

Multiparametric Data-driven Imaging Markers: Guidelines for Development, Application and Reporting of Model Outputs in Radiomics.

This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of "Radiomic ANalysis and DAta Modeling"), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies.

Multiparametric Quantitative Imaging in Risk Prediction: Recommendations for Data Acquisition, Technical Performance Assessment, and Model Development and Validation.

Combinations of multiple quantitative imaging biomarkers (QIBs) are often able to predict the likelihood of an event of interest such as death or disease recurrence more effectively than single imaging measurements can alone. The development of such multiparametric quantitative imaging and evaluation of its fitness of use differs from the analogous processes for individual QIBs in several key aspects. A computational procedure to combine the QIB values into a model output must be specified. The output must also be reproducible and be shown to have reasonably strong ability to predict the risk of an event of interest. Attention must be paid to statistical issues not often encountered in the single QIB scenario, including overfitting and bias in the estimates of model performance. This is the fourth in a five-part series on statistical methodology for assessing the technical performance of multiparametric quantitative imaging. Considerations for data acquisition are discussed and recommendations from the literature on methodology to construct and evaluate QIB-based models for risk prediction are summarized. The findings in the literature upon which these recommendations are based are demonstrated through simulation studies. The concepts in this manuscript are applied to a real-life example involving prediction of major adverse cardiac events using automated plaque analysis.

Multiparametric Quantitative Imaging Biomarker as a Multivariate Descriptor of Health: A Roadmap.

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

Multiparametric Quantitative Imaging Biomarkers for Phenotype Classification: A Framework for Development and Validation.

This manuscript is the third in a five-part series related to statistical assessment methodology for technical performance of multi-parametric quantitative imaging biomarkers (mp-QIBs). We outline approaches and statistical methodologies for developing and evaluating a phenotype classification model from a set of multiparametric QIBs. We then describe validation studies of the classifier for precision, diagnostic accuracy, and interchangeability with a comparator classifier. We follow with an end-to-end real-world example of development and validation of a classifier for atherosclerotic plaque phenotypes. We consider diagnostic accuracy and interchangeability to be clinically meaningful claims for a phenotype classification model informed by mp-QIB inputs, aiming to provide tools to demonstrate agreement between imaging-derived characteristics and clinically established phenotypes. Understanding that we are working in an evolving field, we close our manuscript with an acknowledgement of existing challenges and a discussion of where additional work is needed. In particular, we discuss the challenges involved with technical performance and analytical validation of mp-QIBs. We intend for this manuscript to further advance the robust and promising science of multiparametric biomarker development.

Challenges in ensuring the generalizability of image quantitation methods for MRI.

Image quantitation methods including quantitative MRI, multiparametric MRI, and radiomics offer great promise for clinical use. However, many of these methods have limited clinical adoption, in part due to issues of generalizability, that is, the ability to translate methods and models across institutions. Researchers can assess generalizability through measurement of repeatability and reproducibility, thus quantifying different aspects of measurement variance. In this article, we review the challenges to ensuring repeatability and reproducibility of image quantitation methods as well as present strategies to minimize their variance to enable wider clinical implementation. We present possible solutions for achieving clinically acceptable performance of image quantitation methods and briefly discuss the impact of minimizing variance and achieving generalizability towards clinical implementation and adoption.

MRI-related FDA adverse event reports: A 10-yr review.

PURPOSE: To provide an overview of the types of adverse events reported to the US Food and Drug Administration (US FDA) for magnetic resonance (MR) systems over a 10-yr period. METHODS: Two reviewers independently reviewed adverse events reported to FDA for MR systems from 1 January 2008 to 31 December 2017 and manually categorized events into eight event types. Thermal events were further subcategorized by probable cause. Objects that became projectiles were also categorized. RESULTS: FDA received 1568 adverse event reports for MR systems between 1 January 2008 and 31 December 2017. This analysis included 1548 reports. Thermal events were the most commonly reported serious injury (59% of analyzed reports). Mechanical events - defined as slips, falls, crush injuries, broken bones, and cuts; musculoskeletal injuries from lifting or movement of the device - (11%), projectile events (9%), and acoustic events (6%) were also observed. CONCLUSIONS: Adverse events related to MR systems consistent with the known hazards of the MR environment continue to be reported to FDA. Increased awareness of the types of adverse events occurring for MR imaging systems is important for prevention.

Recommendations towards standards for quantitative MRI (qMRI) and outstanding needs.

LEVEL OF EVIDENCE: 5 Technical Efficacy: Stage 5 J. Magn. Reson. Imaging 2019.

Response to Standardized MR Terminology and Reporting of Implants and Devices as Recommended by the American College of Radiology Subcommittee on MR Safety.

The U.S. Food and Drug Administration (FDA) continually works toward the goal of safety. For patients with magnetic resonance (MR) Conditional devices, safety is achieved when MR Conditional labeling is clear and accessible and can be unambiguously interpreted and applied. The FDA supports the three facets of standardization listed by the American College of Radiology (ACR) Subcommittee on MR Safety in their special report: (a) standardization in terminology and reporting of spatial gradient magnetic fields associated with MR systems; (b) standardization in reporting of ferromagnetic testing results for implants and devices; and (c) standardization, consistency, and clarity in radiofrequency power deposition guidelines and terminology. While the FDA is in agreement with the ACR Subcommittee on MR Safety that patient safety is of primary concern, the authors disagree with the Subcommittee on several important points and offer a point-by-point response to the Subcommittee's four recommendations. (©) RSNA, 2015.

Method to create regional mechanical dyssynchrony maps from short-axis cine steady-state free-precession images.

PURPOSE: To develop a robust method to assess regional mechanical dyssynchrony from cine short-axis MR images. Cardiac resynchronization therapy (CRT) is an effective treatment for patients with heart failure and evidence of left-ventricular (LV) dyssynchrony. Patient response to CRT is greatest when the LV pacing lead is placed in the most dyssynchronous segment. Existing techniques for assessing regional dyssynchrony require difficult acquisition and/or postprocessing. Our goal was to develop a widely applicable and robust method to assess regional mechanical dyssynchrony. MATERIALS AND METHODS: Using the endocardial boundary, radial displacement curves (RDCs) were generated throughout the LV. Cross-correlation was used to determine the delay time between each RDC and a patient-specific reference. Delay times were projected onto the American Heart Association 17-segment model creating a regional dyssynchrony map. Our method was tested in 10 normal individuals and 10 patients enrolled for CRT (QRS > 120 ms, NYHA III-IV, EF < 35%). RESULTS: Delay times over the LV were 23.9 ± 33.8 ms and 93.1 ± 99.9 ms (P < 0.001) in normal subjects and patients, respectively. Interobserver reproducibility for segment averages was 6.8 ± 39.3 ms and there was 70% agreement in identifying the latest contracting segment. CONCLUSION: We have developed a method that can reliably calculate regional delay times from cine steady-state free-precession (SSFP) images. Maps of regional dyssynchrony could be used to identify the latest-contracting segment to assist in CRT lead implantation.

Cardiovascular magnetic resonance at 3.0 T: current state of the art.

There are advantages to conducting cardiovascular magnetic resonance (CMR) studies at a field strength of 3.0 Telsa, including the increase in bulk magnetization, the increase in frequency separation of off-resonance spins, and the increase in T1 of many tissues. However, there are significant challenges to routinely performing CMR at 3.0 T, including the reduction in main magnetic field homogeneity, the increase in RF power deposition, and the increase in susceptibility-based artifacts.In this review, we outline the underlying physical effects that occur when imaging at higher fields, examine the practical results these effects have on the CMR applications, and examine methods used to compensate for these effects. Specifically, we will review cine imaging, MR coronary angiography, myocardial perfusion imaging, late gadolinium enhancement, and vascular wall imaging.

Role of MRI in patient selection for CRT.

Magnetic resonance imaging has great potential for aiding in the selection of patients who will respond to CRT. MRI is the only imaging tool that can simultaneously assess mechanical dyssynchrony, determine the amount and location of myocardial scar tissue, and map the location of cardiac venous anatomy-three important factors in predicting a patient's response to CRT. The goal of this manuscript is to review the MRI methods that can be used in the selection of patients for CRT.

Cross-correlation delay to quantify myocardial dyssynchrony from phase contrast magnetic resonance (PCMR) velocity data.

PURPOSE: To apply cross-correlation delay (XCD) analysis to myocardial phase contrast magnetic resonance (PCMR) tissue velocity data and to compare XCD to three established "time-to-peak" dyssynchrony parameters. MATERIALS AND METHODS: Myocardial tissue velocity was acquired using PCMR in 10 healthy volunteers (negative controls) and 10 heart failure patients who met criteria for cardiac resynchronization therapy (positive controls). All dyssynchrony parameters were computed from PCMR velocity curves. Sensitivity, specificity, and receiver operator curve (ROC) analysis for separating positive and negative controls were computed for each dyssynchrony parameter. RESULTS: XCD had higher sensitivity (90%) and specificity (100%) for discriminating between normal and patient groups than any of the time-to-peak dyssynchrony parameters. ROC analysis showed that XCD was the best parameter for separating the positive and negative control groups. CONCLUSION: XCD is superior to time-to-peak dyssynchrony parameters for discriminating between subjects with and without dyssynchrony and may aid in the selection of patients for cardiac resynchronization therapy.

Quantification of left ventricular internal flow from cardiac magnetic resonance images in patients with dyssynchronous heart failure.

PURPOSE: To develop a method for quantifying left ventricular (LV) internal flow as a measure of dyssynchrony using standard cine cardiac magnetic resonance (CMR) images. MATERIALS AND METHODS: CMR images were obtained from 10 healthy controls and 10 patients with dyssynchronous heart failure (class III/IV, LV ejection fraction <35%, pattern seen in an electrocardiogram QRS duration > 150 msec). The LV volume was reconstructed and divided into 16 regions. Internal flow was defined as the sum of the regional volume changes minus the global volume change during each time step in the cardiac cycle. Internal flow fraction (IFF) was defined as the total internal flow as a percentage of stroke volume during systole (IFF(systole)), diastole (IFF(diastole)), or the whole cycle (IFF(whole)). RESULTS: IFF(whole) was significantly increased in the patients (9.9 +/- 5.0% vs. 1.5 +/- 0.5% in the controls, P < 0.001). An IFF(whole) threshold of 4% discriminated between patients and controls with 90% sensitivity and 100% specificity. IFF(diastole) (2.3 +/- 0.8%) was greater than IFF(systole) (0.8 +/- 0.5%) in the normal controls (P < 0.001) while the patients had similar IFF(diastole) (7.8 +/- 4.2%) and IFF(systole) (12.0 +/- 7.8%). CONCLUSION: Left ventricular internal flow fraction can be quantified from standard CMR images. In this preliminary study, Left ventricular internal flow fraction discriminated patients with dyssynchronous heart failure from normal controls with 95% accuracy.