Fair evaluation of federated learning algorithms for automated breast density classification: The results of the 2022 ACR-NCI-NVIDIA federated learning challenge.

The correct interpretation of breast density is important in the assessment of breast cancer risk. AI has been shown capable of accurately predicting breast density, however, due to the differences in imaging characteristics across mammography systems, models built using data from one system do not generalize well to other systems. Though federated learning (FL) has emerged as a way to improve the generalizability of AI without the need to share data, the best way to preserve features from all training data during FL is an active area of research. To explore FL methodology, the breast density classification FL challenge was hosted in partnership with the American College of Radiology, Harvard Medical Schools' Mass General Brigham, University of Colorado, NVIDIA, and the National Institutes of Health National Cancer Institute. Challenge participants were able to submit docker containers capable of implementing FL on three simulated medical facilities, each containing a unique large mammography dataset. The breast density FL challenge ran from June 15 to September 5, 2022, attracting seven finalists from around the world. The winning FL submission reached a linear kappa score of 0.653 on the challenge test data and 0.413 on an external testing dataset, scoring comparably to a model trained on the same data in a central location.

Real-World Performance of Large Vessel Occlusion Artificial Intelligence-Based Computer-Aided Triage and Notification Algorithms-What the Stroke Team Needs to Know.

PURPOSE: To evaluate the real-world performance of two FDA-approved artificial intelligence (AI)-based computer-aided triage and notification (CADt) detection devices and compare them with the manufacturer-reported performance testing in the instructions for use. MATERIALS AND METHODS: Clinical performance of two FDA-cleared CADt large-vessel occlusion (LVO) devices was retrospectively evaluated at two separate stroke centers. Consecutive "code stroke" CT angiography examinations were included and assessed for patient demographics, scanner manufacturer, presence or absence of CADt result, CADt result, and LVO in the internal carotid artery (ICA), horizontal middle cerebral artery (MCA) segment (M1), Sylvian MCA segments after the bifurcation (M2), precommunicating part of cerebral artery, postcommunicating part of the cerebral artery, vertebral artery, basilar artery vessel segments. The original radiology report served as the reference standard, and a study radiologist extracted the above data elements from the imaging examination and radiology report. RESULTS: At hospital A, the CADt algorithm manufacturer reports assessment of intracranial ICA and MCA with sensitivity of 97% and specificity of 95.6%. Real-world performance of 704 cases included 79 in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 85.3% and 91.9%. Sensitivity decreased to 68.5% when M2 segments were included and to 59.9% when all proximal vessel segments were included. At hospital B the CADt algorithm manufacturer reports sensitivity of 87.8% and specificity of 89.6%, without specifying the vessel segments. Real-world performance of 642 cases included 20 cases in which no CADt result was available. Sensitivity and specificity in ICA and M1 segments were 90.7% and 97.9%. Sensitivity decreased to 76.4% when M2 segments were included and to 59.4% when all proximal vessel segments are included. DISCUSSION: Real-world testing of two CADt LVO detection algorithms identified gaps in the detection and communication of potentially treatable LVOs when considering vessels beyond the intracranial ICA and M1 segments and in cases with absent and uninterpretable data.

Projected Growth in FDA-Approved Artificial Intelligence Products Given Venture Capital Funding.

PURPOSE: Medical imaging accounts for 85% of digital health's venture capital funding. As funding grows, it is expected that artificial intelligence (AI) products will increase commensurately. The study's objective is to project the number of new AI products given the statistical association between historical funding and FDA-approved AI products. METHODS: The study used data from the ACR Data Science Institute and for the number of FDA-approved AI products (2008-2022) and data from Rock Health for AI funding (2013-2022). Employing a 6-year lag between funding and product approved, we used linear regression to estimate the association between new products approved in a certain year, based on the lagged funding (ie, product-year funding). Using this statistical relationship, we forecasted the number of new FDA-approved products. RESULTS: The results show that there are 11.33 (95% confidence interval: 7.03-15.64) new AI products for every $1 billion in funding assuming a 6-year lag between funding and product approval. In 2022 there were 69 new FDA-approved products associated with $4.8 billion in funding. In 2035, product-year funding is projected to reach $30.8 billion, resulting in 350 new products that year. CONCLUSIONS: FDA-approved AI products are expected to grow from 69 in 2022 to 350 in 2035 given the expected funding growth in the coming years. AI is likely to change the practice of diagnostic radiology as new products are developed and integrated into practice. As more AI products are integrated, it may incentivize increased investment for future AI products.

ACR's Connect and AI-LAB technical framework.

Objective: To develop a free, vendor-neutral software suite, the American College of Radiology (ACR) Connect, which serves as a platform for democratizing artificial intelligence (AI) for all individuals and institutions. Materials and Methods: Among its core capabilities, ACR Connect provides educational resources; tools for dataset annotation; model building and evaluation; and an interface for collaboration and federated learning across institutions without the need to move data off hospital premises. Results: The AI-LAB application within ACR Connect allows users to investigate AI models using their own local data while maintaining data security. The software enables non-technical users to participate in the evaluation and training of AI models as part of a larger, collaborative network. Discussion: Advancements in AI have transformed automated quantitative analysis for medical imaging. Despite the significant progress in research, AI is currently underutilized in current clinical workflows. The success of AI model development depends critically on the synergy between physicians who can drive clinical direction, data scientists who can design effective algorithms, and the availability of high-quality datasets. ACR Connect and AI-LAB provide a way to perform external validation as well as collaborative, distributed training. Conclusion: In order to create a collaborative AI ecosystem across clinical and technical domains, the ACR developed a platform that enables non-technical users to participate in education and model development.

Envisioning healthy futures: Youth perceptions of justice-oriented environments and communities in Northern British Columbia Canada.

Through an anti-colonial and critical race theoretical framework as well as arts-based methods (photovoice) that engage Indigenous and non-Indigenous youth, we explore the question: what do youth perceive as healthy and just environments and communities? Youth identified two overarching, strength-based messages: Firstly, youth demonstrate the need for a structural-level analysis of the conditions that influence individual-level outcomes of environmental health. Secondly, youth perspectives on healthy and justice-oriented environments and communities challenge environmental health scholars to consider youth as powerful actors. Youth perspectives of healthy and justice-oriented communities present a necessarily structural perspective to consider not only the impacts of environmental decision-making on health, but the conditions that have allowed for harmful impacts. In doing so, youth demonstrate the need for intersectional and complex understandings of health and wellbeing when discussing the environment. And, as we argue here, challenge us as scholars of environmental health to do the same.

Data Sharing of Imaging in an Evolving Health Care World: Report of the ACR Data Sharing Workgroup, Part 2: Annotation, Curation, and Contracting.

A core principle of ethical data sharing is maintaining the security and anonymity of the data, and care must be taken to ensure medical records and images cannot be reidentified to be traced back to patients or misconstrued as a breach in the trust between health care providers and patients. Once those principles have been observed, those seeking to share data must take the appropriate steps to curate the data in a way that organizes the clinically relevant information so as to be useful to the data sharing party, assesses the ensuing value of the data set and its annotations, and informs the data sharing contracts that will govern use of the data. Embarking on a data sharing partnership engenders a host of ethical, practical, technical, legal, and commercial challenges that require a thoughtful, considered approach. In 2019 the ACR convened a Data Sharing Workgroup to develop philosophies around best practices in the sharing of health information. This is Part 2 of a Report on the workgroup's efforts in exploring these issues.

Data Sharing of Imaging in an Evolving Health Care World: Report of the ACR Data Sharing Workgroup, Part 1: Data Ethics of Privacy, Consent, and Anonymization.

Radiology is at the forefront of the artificial intelligence transformation of health care across multiple areas, from patient selection to study acquisition to image interpretation. Needing large data sets to develop and train these algorithms, developers enter contractual data sharing agreements involving data derived from health records, usually with postacquisition curation and annotation. In 2019 the ACR convened a Data Sharing Workgroup to develop philosophies around best practices in the sharing of health information. The workgroup identified five broad domains of activity important to collaboration using patient data: privacy, informed consent, standardization of data elements, vendor contracts, and data valuation. This is Part 1 of a Report on the workgroup's efforts in exploring these issues.

Evaluation and Real-World Performance Monitoring of Artificial Intelligence Models in Clinical Practice: Try It, Buy It, Check It.

The pace of regulatory clearance of artificial intelligence (AI) algorithms for radiology continues to accelerate, and numerous algorithms are becoming available for use in clinical practice. End users of AI in radiology should be aware that AI algorithms may not work as expected when used beyond the institutions in which they were trained, and model performance may degrade over time. In this article, we discuss why regulatory clearance alone may not be enough to ensure AI will be safe and effective in all radiological practices and review strategies available resources for evaluating before clinical use and monitoring performance of AI models to ensure efficacy and patient safety.

2020 ACR Data Science Institute Artificial Intelligence Survey.

PURPOSE: The ACR Data Science Institute conducted its first annual survey of ACR members to understand how radiologists are using artificial intelligence (AI) in clinical practice and to provide a baseline for monitoring trends in AI use over time. METHODS: The ACR Data Science Institute sent a brief electronic survey to all ACR members via email. Invitees were asked for demographic information about their practice and if and how they were currently using AI as part of their clinical work. They were also asked to evaluate the performance of AI models in their practices and to assess future needs. RESULTS: Approximately 30% of radiologists are currently using AI as part of their practice. Large practices were more likely to use AI than smaller ones, and of those using AI in clinical practice, most were using AI to enhance interpretation, most commonly detection of intracranial hemorrhage, pulmonary emboli, and mammographic abnormalities. Of practices not currently using AI, 20% plan to purchase AI tools in the next 1 to 5 years. CONCLUSION: The survey results indicate a modest penetrance of AI in clinical practice. Information from the survey will help researchers and industry develop AI tools that will enhance radiological practice and improve quality and efficiency in patient care.

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density.

OBJECTIVE: We developed deep learning algorithms to automatically assess BI-RADS breast density. METHODS: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting. RESULTS: Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists. CONCLUSION: We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.

Interreader Reliability of LI-RADS Version 2014 Algorithm and Imaging Features for Diagnosis of Hepatocellular Carcinoma: A Large International Multireader Study.

Purpose To determine in a large multicenter multireader setting the interreader reliability of Liver Imaging Reporting and Data System (LI-RADS) version 2014 categories, the major imaging features seen with computed tomography (CT) and magnetic resonance (MR) imaging, and the potential effect of reader demographics on agreement with a preselected nonconsecutive image set. Materials and Methods Institutional review board approval was obtained, and patient consent was waived for this retrospective study. Ten image sets, comprising 38-40 unique studies (equal number of CT and MR imaging studies, uniformly distributed LI-RADS categories), were randomly allocated to readers. Images were acquired in unenhanced and standard contrast material-enhanced phases, with observation diameter and growth data provided. Readers completed a demographic survey, assigned LI-RADS version 2014 categories, and assessed major features. Intraclass correlation coefficient (ICC) assessed with mixed-model regression analyses was the metric for interreader reliability of assigning categories and major features. Results A total of 113 readers evaluated 380 image sets. ICC of final LI-RADS category assignment was 0.67 (95% confidence interval [CI]: 0.61, 0.71) for CT and 0.73 (95% CI: 0.68, 0.77) for MR imaging. ICC was 0.87 (95% CI: 0.84, 0.90) for arterial phase hyperenhancement, 0.85 (95% CI: 0.81, 0.88) for washout appearance, and 0.84 (95% CI: 0.80, 0.87) for capsule appearance. ICC was not significantly affected by liver expertise, LI-RADS familiarity, or years of postresidency practice (ICC range, 0.69-0.70; ICC difference, 0.003-0.01 [95% CI: -0.003 to -0.01, 0.004-0.02]. ICC was borderline higher for private practice readers than for academic readers (ICC difference, 0.009; 95% CI: 0.000, 0.021). Conclusion ICC is good for final LI-RADS categorization and high for major feature characterization, with minimal reader demographic effect. Of note, our results using selected image sets from nonconsecutive examinations are not necessarily comparable with those of prior studies that used consecutive examination series. © RSNA, 2017.

U.S. Diagnostic Reference Levels and Achievable Doses for 10 Adult CT Examinations.

Purpose To develop diagnostic reference levels (DRLs) and achievable doses (ADs) for the 10 most common adult computed tomographic (CT) examinations in the United States as a function of patient size by using the CT Dose Index Registry. Materials and Methods Data from the 10 most commonly performed adult CT head, neck, and body examinations from 583 facilities were analyzed. For head examinations, the lateral thickness was used as an indicator of patient size; for neck and body examinations, water-equivalent diameter was used. Data from 1 310 727 examinations (analyzed by using SAS 9.3) provided median values, as well as means and 25th and 75th (DRL) percentiles for volume CT dose index (CTDIvol), dose-length product (DLP), and size-specific dose estimate (SSDE). Applicable results were compared with DRLs from eight countries. Results More than 46% of the facilities were community hospitals; 13% were academic facilities. More than 48% were in metropolitan areas, 39% were suburban, and 13% were rural. More than 50% of the facilities performed fewer than 500 examinations per month. The abdomen and pelvis was the most frequently performed examination in the study (45%). For body examinations, DRLs (75th percentile) and ADs (median) for CTDIvol, SSDE, and DLP increased consistently with the patient's size (water-equivalent diameter). The relationships between patient size and DRLs and ADs were not as strong for head and neck examinations. These results agree well with the data from other countries. Conclusion DRLs and ADs as a function of patient size were developed for the 10 most common adult CT examinations performed in the United States. © RSNA, 2017.

Radiation dose index of renal colic protocol CT studies in the United States: a report from the American College of Radiology National Radiology Data Registry.

PURPOSE: To determine radiation dose indexes for computed tomography (CT) performed with renal colic protocols in the United States, including frequency of reduced-dose technique usage and any institutional-level factors associated with high or low dose indexes. MATERIALS AND METHODS: The Dose Imaging Registry (DIR) collects deidentified CT data, including examination type and dose indexes, for CT performed at participating institutions; thus, the DIR portion of the study was exempt from institutional review board approval and was HIPAA compliant. CT dose indexes were examined at the institutional level for CT performed with a renal colic protocol at institutions that contributed at least 10 studies to the registry as of January 2013. Additionally, patients undergoing CT for renal colic at a single institution (with institutional review board approval and informed consent from prospective subjects and waiver of consent from retrospective subjects) were studied to examine individual renal colic CT dose index patterns and explore relationships between patient habitus, demographics, and dose indexes. Descriptive statistics were used to analyze dose indexes, and linear regression and Spearman correlations were used to examine relationships between dose indexes and institutional factors. RESULTS: There were 49 903 renal colic protocol CT examinations conducted at 93 institutions between May 2011 and January 2013. Mean age ± standard deviation was 49 years ± 18, and 53.9% of patients were female. Institutions contributed a median of 268 (interquartile range, 77-699) CT studies. Overall mean institutional dose-length product (DLP) was 746 mGy ⋅ cm (effective dose, 11.2 mSv), with a range of 307-1497 mGy ⋅ cm (effective dose, 4.6-22.5 mSv) for mean DLPs. Only 2% of studies were conducted with a DLP of 200 mGy ⋅ cm or lower (a "reduced dose") (effective dose, 3 mSv), and only 10% of institutions kept DLP at 400 mGy ⋅ cm (effective dose, 6 mSv) or less in at least 50% of patients. CONCLUSION: Reduced-dose renal protocol CT is used infrequently in the United States. Mean dose index is higher than reported previously, and institutional variation is substantial.

Diagnostic reference ranges for pediatric abdominal CT.

PURPOSE: To develop diagnostic reference ranges (DRRs) and a method for an individual practice to calculate site-specific reference doses for computed tomographic (CT) scans of the abdomen or abdomen and pelvis in children on the basis of body width (BW). MATERIALS AND METHODS: This HIPAA-compliant multicenter retrospective study was approved by institutional review boards of participating institutions; informed consent was waived. In 939 pediatric patients, CT doses were reviewed in 499 (53%) male and 440 (47%) female patients (mean age, 10 years). Doses were from 954 scans obtained from September 1 to December 1, 2009, through Quality Improvement Registry for CT Scans in Children within the National Radiology Data Registry, American College of Radiology. Size-specific dose estimate (SSDE), a dose estimate based on BW, CT dose index, dose-length product, and effective dose were analyzed. BW measurement was obtained with electronic calipers from the axial image at the splenic vein level after completion of the CT scan. An adult-sized patient was defined as a patient with BW of 34 cm. An appropriate dose range for each DRR was developed by reviewing image quality on a subset of CT scans through comparison with a five-point visual reference scale with increments of added simulated quantum mottle and by determining DRR to establish lower and upper bounds for each range. RESULTS: For 954 scans, DRRs (SSDEs) were 5.8-12.0, 7.3-12.2, 7.6-13.4, 9.8-16.4, and 13.1-19.0 mGy for BWs less than 15, 15-19, 20-24, 25-29, and 30 cm or greater, respectively. The fractions of adult doses, adult SSDEs, used within the consortium for patients with BWs of 10, 14, 18, 22, 26, and 30 cm were 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9, respectively. CONCLUSION: The concept of DRRs addresses the balance between the patient's risk (radiation dose) and benefit (diagnostic image quality). Calculation of reference doses as a function of BW for an individual practice provides a tool to help develop site-specific CT protocols that help manage pediatric patient radiation doses.

Variation in institutional review board responses to a standard protocol for a multicenter randomized, controlled surgical trial.

PURPOSE: The primary responsibility of institutional review boards is to protect human research subjects and, therefore, ensure that studies are performed in accordance with a standard set of ethical principles. A number of groups have compared the responses of institutional review boards in multicenter clinical trials involving medical therapies. To our knowledge no such studies have been performed to date of trials investigating surgical intervention. We investigated the consistency of the recommendations issued by various institutional review boards in the Minimally Invasive Surgical Therapies study for benign prostatic hyperplasia, a multicenter trial with a uniform consent and study protocol. MATERIALS AND METHODS: We obtained the institutional review board response from 6 of the 7 participating institutions after initial submission of the Minimally Invasive Surgical Therapies study protocol and classified the responses. We then redistributed the approved protocols to an institutional review board at another participating institution and analyzed that review of these protocols. RESULTS: We found that the number and type of responses required for institutional review board approval of an identical study protocol varied significantly among participating institutions. We also found that institutional review board responses were inconsistent in the second review, although all protocols were ultimately approved. CONCLUSIONS: The current system of local institutional review board review in the context of a multicenter surgical trial is inefficient in the review process and may not provide expertise for overseeing surgical trials. Based on these results a central surgical institutional review board may be needed to improve the ethical review process in multicenter trials.

Examining the economic costs related to lifestyle and pharmacological interventions in youth with Type 2 diabetes.

The best treatment option for children with Type 2 diabetes has not yet been established. The Treatment Options for Type 2 Diabetes in Adolescents and Youth (TODAY) study is currently testing the efficacy of three therapies: metformin, metformin plus rosiglitazone and metformin plus an intensive lifestyle intervention. The relative cost-effectiveness of these therapies is also being examined. This review discusses the rationale for the design and methods applied in the economic analysis. The design of the economic analysis in the TODAY study was influenced by the existing literature and two primary study parameters: the nature of the interventions and the participants' age. The lifestyle intervention is an intensive behavioral intervention comprising diet and physical activity. Since economic factors influence both diet and physical activity, the analytical plan includes measurement of food and exercise-related purchases. Due to the young age of the participants, the impact of the intervention on adult caregivers is also included in the analysis. This analysis focuses on the time spent by the caregivers in both medical treatment and nutrition- and activity-related activities, and the value of this time relative to usual activities. Important methodological questions include how and when to collect information, not only on medical costs, but also on the impact of caregiver time, travel, food and equipment purchases. In the TODAY study, these latter resources are being measured by regularly administered surveys completed by the caregivers. The approach to the cost-effectiveness assessment undertaken by the TODAY study is one of the first in diabetes research to focus on youth and to include a societal perspective, regular and prospective assessment of clinician and caregiver time, and a comprehensive assessment of the costs associated with lifestyle behaviors. It can serve as a model for future studies of diabetes treatments.

The evaluation of treatment when center-specific selection criteria vary with respect to patient risk.

Many standards of medical care are based on the demonstrated effects of various treatment strategies or processes. Unlike pharmacological treatments, these strategies or processes are not necessarily subjected to rigorous clinical trials and their benefit is frequently assessed from observational data. For evaluating the influence of such medical processes on patient outcomes, not only is risk adjustment an issue, but also the "center effect" represents an important, often overlooked consideration. Both the quality of care and the tendency to use certain treatments or processes vary from one center to another. The induced similarity in outcomes within center, as well as the potential for confounding by center, needs to be addressed within the context of risk adjustment. In addition, center-specific selection criteria for a treatment strategy can vary with respect to patient risk. Because of these considerations, it is important to adequately separate the within-center effects of the treatment or strategy from the across-center effects, which relate more to center performance. The primary objective of this article is to explore and extend current methods of dealing with center confounding for dichotomous outcomes, primarily for the situation where selection on the basis of patient risk can vary from center to center. A simulation study compares results from several different analytic methods and provides evidence for the importance of considering confounding due to both risk and center when evaluating the effectiveness of a process. An example that examines the effect of early extubation after bypass surgery is also presented.

Selecting patients with mitral regurgitation and left ventricular dysfunction for isolated mitral valve surgery.

BACKGROUND: American College of Cardiology/American Heart Association (ACC/AHA) Guidelines state that patients with an ejection fraction (EF) of 30% or less should not undergo mitral valve replacement for mitral regurgitation (MR). We sought to establish, using a national cardiac surgery database, whether patients with left ventricular dysfunction may safely undergo mitral valve surgery for MR, and if so, which ones. METHODS: We queried the Society of Thoracic Surgeons (STS) National Database to identify patients who had isolated mitral valve replacement or repair for MR between 1998 and 2001. Mortality and morbidity outcomes were compared by EF category (< or = 30% vs > 30%), and observed mortality compared by EF group, stratified by predicted risk for mortality. A classification and regression tree (CART) model was then used to determine which patient characteristics contributed most to designate the high-risk patient. RESULTS: Of the 14,582 patients who had mitral valve surgery, 727 had an EF of 30% or less and 13,855 had an EF of more than 30%. Observed mortality rates were higher for patients with an EF of 30% or less (5.4% vs 3.1%). However, for low-risk to medium-risk patients, mortality rates remained fairly constant across levels of EF. Mortality is notably increased in the high-risk patients (predicted risk > 10%). A classification tree identifies three key characteristics for high risk: age more than 75 years, renal failure, and emergent or salvage procedure. CONCLUSIONS: When the predicted mortality risk is less than 10%, EF has minimal impact on operative mortality for mitral regurgitation. In contrast to the ACC/AHA Guidelines, our data show that operative risk for mitral valve surgery is not prohibitive for most patients with ventricular dysfunction.