Projected Growth in FDA-Approved Artificial Intelligence Products Given Venture Capital Funding.

PURPOSE: Medical imaging accounts for 85% of digital health's venture capital funding. As funding grows, it is expected that artificial intelligence (AI) products will increase commensurately. The study's objective is to project the number of new AI products given the statistical association between historical funding and FDA-approved AI products. METHODS: The study used data from the ACR Data Science Institute and for the number of FDA-approved AI products (2008-2022) and data from Rock Health for AI funding (2013-2022). Employing a 6-year lag between funding and product approved, we used linear regression to estimate the association between new products approved in a certain year, based on the lagged funding (ie, product-year funding). Using this statistical relationship, we forecasted the number of new FDA-approved products. RESULTS: The results show that there are 11.33 (95% confidence interval: 7.03-15.64) new AI products for every $1 billion in funding assuming a 6-year lag between funding and product approval. In 2022 there were 69 new FDA-approved products associated with $4.8 billion in funding. In 2035, product-year funding is projected to reach $30.8 billion, resulting in 350 new products that year. CONCLUSIONS: FDA-approved AI products are expected to grow from 69 in 2022 to 350 in 2035 given the expected funding growth in the coming years. AI is likely to change the practice of diagnostic radiology as new products are developed and integrated into practice. As more AI products are integrated, it may incentivize increased investment for future AI products.

Empathy and Equity: Key Considerations for Large Language Model Adoption in Health Care.

The growing presence of large language models (LLMs) in health care applications holds significant promise for innovative advancements in patient care. However, concerns about ethical implications and potential biases have been raised by various stakeholders. Here, we evaluate the ethics of LLMs in medicine along 2 key axes: empathy and equity. We outline the importance of these factors in novel models of care and develop frameworks for addressing these alongside LLM deployment.

Validation pipeline for machine learning algorithm assessment for multiple vendors.

A standardized objective evaluation method is needed to compare machine learning (ML) algorithms as these tools become available for clinical use. Therefore, we designed, built, and tested an evaluation pipeline with the goal of normalizing performance measurement of independently developed algorithms, using a common test dataset of our clinical imaging. Three vendor applications for detecting solid, part-solid, and groundglass lung nodules in chest CT examinations were assessed in this retrospective study using our data-preprocessing and algorithm assessment chain. The pipeline included tools for image cohort creation and de-identification; report and image annotation for ground-truth labeling; server partitioning to receive vendor "black box" algorithms and to enable model testing on our internal clinical data (100 chest CTs with 243 nodules) from within our security firewall; model validation and result visualization; and performance assessment calculating algorithm recall, precision, and receiver operating characteristic curves (ROC). Algorithm true positives, false positives, false negatives, recall, and precision for detecting lung nodules were as follows: Vendor-1 (194, 23, 49, 0.80, 0.89); Vendor-2 (182, 270, 61, 0.75, 0.40); Vendor-3 (75, 120, 168, 0.32, 0.39). The AUCs for detection of solid (0.61-0.74), groundglass (0.66-0.86) and part-solid (0.52-0.86) nodules varied between the three vendors. Our ML model validation pipeline enabled testing of multi-vendor algorithms within the institutional firewall. Wide variations in algorithm performance for detection as well as classification of lung nodules justifies the premise for a standardized objective ML algorithm evaluation process.

Artificial Intelligence has Similar Performance to Subjective Assessment of Emphysema Severity on Chest CT.

RATIONALE AND OBJECTIVES: To compare an artificial intelligence (AI)-based prototype and subjective grading for predicting disease severity in patients with emphysema. METHODS: Our IRB approved HIPAA-compliant study included 113 adults (71±8 years; 47 females, 66 males) who had both non-contrast chest CT and pulmonary function tests performed within a span of 2 months. The disease severity was classified based on the forced expiratory volume in 1 second (FEV1 as % of predicted) into mild, moderate, and severe. 2 thoracic radiologists (RA), blinded to the clinical and AI results, graded severity of emphysema on a 5-point scale suggested by the Fleischner Society for each lobe. The whole lung scores were derived from the summation of lobar scores. Thin-section CT images were processed with the AI-Rad Companion Chest prototype (Siemens Healthineers) to quantify low attenuation areas (LAA < - 950 HU) in whole lung and each lobe separately. Bronchial abnormality was assessed by both radiologists and a fully automated software (Philips Healthcare). RESULTS: Both AI (AUC of 0.77; 95% CI: 0.68 - 0.85) and RA (AUC: 0.76, 95% CI: 0.65 - 0.84) emphysema quantification could differentiate mild, moderate, and severe disease based on FEV1. There was a strong positive correlation between AI and RA (r = 0.72 - 0.80; p <0.001). The combination of emphysema and bronchial abnormality quantification from radiologists' and AI assessment could differentiate between different severities with AUC of 0.80 - 0.82 and 0.87, respectively. CONCLUSION: The assessed AI-prototypes can predict the disease severity in patients with emphysema with the same predictive value as the radiologists.

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density.

OBJECTIVE: We developed deep learning algorithms to automatically assess BI-RADS breast density. METHODS: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting. RESULTS: Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists. CONCLUSION: We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.

The Role of the ACR Data Science Institute in Advancing Health Equity in Radiology.

Commercially available artificial intelligence (AI) algorithms outside of health care have been shown to be susceptible to ethnic, gender, and social bias, which has important implications in the development of AI algorithms in health care and the radiologic sciences. To prevent the introduction bias in health care AI, the physician community should work with developers and regulators to develop pathways to ensure that algorithms marketed for widespread clinical practice are safe, effective, and free of unintended bias. The ACR Data Science Institute has developed structured AI use cases with data elements that allow the development of standardized data sets for AI testing and training across multiple institutions to promote the availability of diverse data for algorithm development. Additionally, the ACR Data Science Institute validation and monitoring services, ACR Certify-AI and ACR Assess-AI, incorporate standards to mitigate algorithm bias and promote health equity. In addition to promoting diversity, the ACR should promote and advocate for payment models for AI that afford access to AI tools for all of our patients regardless of socioeconomic status or the inherent resources of their health systems.

Expanding Role of Certified Electronic Health Records Technology in Radiology: The MACRA Mandate.

Radiology has historically been at the forefront of innovation and the advancement of technology for the benefit of patient care. However, challenges to early implementation prevented most radiologists from adopting and integrating certified electronic health record technology (CEHRT) into their daily workflow despite the early and potential advantages it offered. This circumstance places radiology at a disadvantage in the two payment pathways of the Medicare Access and CHIP Reauthorization Act of 2015: the Merit-Based Incentive Payment System (MIPS) and advanced alternative payment models (APMs). Specifically, not integrating CEHRT hampers radiology's ability to receive bonus points in the quality performance category of the MIPS and in parallel threatens certain threshold requirements for advanced APMs under the new Quality Payment Program. Radiology must expand the availability and use of CEHRT to satisfy existing performance measures while creating new performance measures that create value for the health care system. In addition, radiology IT vendors will need to ensure their products (eg, radiology information systems, PACS, and radiology reporting systems) are CEHRT compliant and approved. Such collective efforts will increase radiologists' quality of patient care, contribution to value driven activities, and overall health care relevance.

Utilization Management of High-Cost Imaging in an Outpatient Setting in a Large Stable Patient and Provider Cohort over 7 Years.

Purpose To quantify the effect of a comprehensive, long-term, provider-led utilization management (UM) program on high-cost imaging (computed tomography, magnetic resonance imaging, nuclear imaging, and positron emission tomography) performed on an outpatient basis. Materials and Methods This retrospective, 7-year cohort study included all patients regularly seen by primary care physicians (PCPs) at an urban academic medical center. The main outcome was the number of outpatient high-cost imaging examinations per patient per year ordered by the patient's PCP or by any specialist. The authors determined the probability of a patient undergoing any high-cost imaging procedure during a study year and the number of examinations per patient per year (intensity) in patients who underwent high-cost imaging. Risk-adjusted hierarchical models were used to directly quantify the physician component of variation in probability and intensity of high-cost imaging use, and clinicians were provided with regular comparative feedback on the basis of the results. Observed trends in high-cost imaging use and provider variation were compared with the same measures for outpatient laboratory studies because laboratory use was not subject to UM during this period. Finally, per-member per-year high-cost imaging use data were compared with statewide high-cost imaging use data from a major private payer on the basis of the same claim set. Results The patient cohort steadily increased in size from 88 959 in 2007 to 109 823 in 2013. Overall high-cost imaging utilization went from 0.43 examinations per year in 2007 to 0.34 examinations per year in 2013, a decrease of 21.33% (P < .0001). At the same time, similarly adjusted routine laboratory study utilization decreased by less than half that rate (9.4%, P < .0001). On the basis of unadjusted data, outpatient high-cost imaging utilization in this cohort decreased 28%, compared with a 20% decrease in statewide utilization (P = .0023). Conclusion Analysis of high-cost imaging utilization in a stable cohort of patients cared for by PCPs during a 7-year period showed that comprehensive UM can produce a significant and sustained reduction in risk-adjusted per-patient year outpatient high-cost imaging volume. © RSNA, 2017.

Images of imaging: how to process and display imaging utilization for large populations.

OBJECTIVE: We propose a method of processing and displaying imaging utilization data for large populations. CONCLUSION: The comprehensive and finely grained picture of imaging utilization yielded by our methods is a first step toward population-based imaging utilization management. We believe that our methods for the categorization and display of imaging utilization will prove to be widely useful.

Changes to stage 1 meaningful use in 2014: impact on radiologists.

The goal of this work is to provide radiologists an update regarding changes to stage 1 of meaningful use in 2014. These changes were promulgated in the final rulemaking released by the Centers for Medicare and Medicaid Services and the Office of the National Coordinator for Health Information Technology in September 2012. Under the new rules, radiologists are exempt from meaningful use penalties provided that they are listed as radiologists under the Provider Enrollment, Chain and Ownership System (PECOS). A major caveat is that this exemption can be removed at any time. Additional concerns are discussed in the main text. Additional changes discussed include software editions independent of meaningful use stage (i.e., 2011 edition versus 2014 edition), changes to the definition of certified electronic health record technology (CEHRT), and changes to specific measures and exemptions to those measures. The new changes regarding stage 1 add complexity to an already complex program, but overall make achieving meaningful use a win-win situation for radiologists. There are no penalties for failure and incentive payments for success. The cost of upgrading to CEHRT may be much less than the incentive payments, adding a potential new source of revenue. Additional benefits may be realized if the radiology department can build upon a modern electronic health record to improve their practice and billing patterns. Meaningful use and electronic health records represent an important evolutionary step in US healthcare, and it is imperative that radiologists are active participants in the process.

Meaningful use a call to arms.

The benefits of an interactive online world have affected the way we purchase products and plan our vacations. It is only a matter of time before consumers start demanding health care with the same convenience that comes with booking an airline flight or managing a bank account. The health care industry itself requires periodic and mandatory data analysis for outcome analysis, clinical benchmarking, quality improvement, forming guidelines, and making decisions. The federal government and health care community have been working together to come up with more robust and cost-effective health care informatics solutions. Meaningful use (MU) intends to establish a new standard for health care informatics in the United States. The term "meaningful use" implies that health care information and technology systems not just exist, but also serve as an integral part of physician and hospital workflow; leading to cost savings as well as improved outcomes. Under this concept, the federal government is offering maximum incentive payments of up to $44,000 per physician (including radiologists) if they can meet all the requirements as laid down in the MU measures. Unfortunately, penalties will kick in if physicians are not compliant with MU by 2015. This will be done in at least three stages, with Stage 1 already in effect (as of January 3, 2011). This will be the first in a series of articles outlining MU and what is in store for radiology. We will go in depth about who is eligible, and how the payment schedule is set up. We will break down the core and menu set measures to suggest what can be excluded by most radiologists. We will also go through some case studies and examine what lies in store for radiology.

When does a radiologist's recommendation for follow-up result in high-cost imaging?

PURPOSE: To measure the proportion of high-cost imaging generated by a radiologist's recommendation and to identify the imaging findings resulting in follow-up. MATERIALS AND METHODS: This retrospective HIPAA-compliant study had institutional review board approval, with waiver of informed consent. A recommended examination was defined as one performed within a single episode of care (defined as fewer than 60 days after the initial imaging) following a radiologist's recommendation in a prior examination report. Chest and abdominal computed tomography (CT), brain and lumbar spine magnetic resonance (MR) imaging, and body positron emission tomography were included for analysis. From a database of all radiology examinations (approximately 200,000) at one institution over a 6-month period, a computerized search identified all high-cost examinations that were preceded by an examination containing a radiologist recommendation. Medical records were reviewed to verify accuracy of the recommending-recommended examination pairs and to determine the reason for the radiologist's recommendation. For proportions, 95% confidence intervals were calculated. RESULTS: Overall, 1558 of 29,232 (5.3%) high-cost examinations followed a radiologist's recommendation. Chest CT was the high-cost examination most often resulting from a radiologist's recommendation (878 of 9331, 9.4%), followed by abdominal CT (390 of 10,258, 3.8%) and brain MR imaging (222 of 6436, 3.4%). The examination types with the highest numbers of follow-up examinations were chest radiography (n=431), chest CT (n=410), abdominal CT (n=214), and abdominal ultrasonography (n=120). The most common findings resulting in follow-up were pulmonary nodules or masses (559 of 1558, 35.9%), other pulmonary abnormalities (150 of 1558, 9.6%), adenopathy (103 of 1558, 6.6%), renal lesions (101 of 1558, 6.5%), and negative examination findings (101 of 1558, 6.5%). CONCLUSION: Radiologists' recommendations account for only a small proportion of outpatient high-cost imaging examinations. Pulmonary nodule follow-up is the most common cause for radiologist-generated examinations.

Does radiologist recommendation for follow-up with the same imaging modality contribute substantially to high-cost imaging volume?

PURPOSE: To retrospectively measure repeat rates for high-cost imaging studies, determining their causes and trends, and the impact of radiologist recommendations for a repeat examination on imaging volume. MATERIALS AND METHODS: This HIPAA-compliant study had institutional review board approval, with waiver of informed consent. Repeat examination was defined as a same-modality examination performed in the same patient within 0 days to 7 months of a first examination. From a database of all radiology examinations (>2.9 million) at one institution from May 1996 to June 2003, a computerized search identified head, spine, chest, and abdominal computed tomographic (CT), brain and spine magnetic resonance (MR) imaging, pelvic ultrasonography (US), and nuclear cardiology examinations with a prior examination of the same type within 7 months. Examination pairs were subdivided into studies repeated at less than 2 weeks, between 2 weeks and 2 months, or between 2 and 7 months. Automated classification of radiology reports revealed whether a repeat examination from June 2002 to June 2003 had been preceded by a radiologist recommendation on the prior report. Trends over time were analyzed with linear regression, and 95% confidence intervals were calculated. RESULTS: Between July 2002 and June 2003, 31 111 of 100 335 examinations (31%) were repeat examinations. Body CT (9057 of 20 177 [45%] chest and 8319 of 22 438 [37%] abdomen) and brain imaging (6823 of 18 378 [37%] CT and 3427 of 11 455 [30%] MR imaging) represented the highest repeat categories. Among five high-cost, high-volume imaging examinations, 6426 of 85 014 (8%) followed a report with a radiologist recommendation. Most common indications for examination repetition were neurologic surveillance within 2 weeks and cancer follow-up at 2-7 months. From 1997 to mid-2003, MR imaging and CT repeat rates increased (0.71% per year [P < .01] and 1.87% per year [P < .01], respectively). CONCLUSION: Repeat examinations account for nearly one-third of high-cost radiology examinations and represent an increasing proportion of such examinations. Most repeat examinations are initiated clinically without a recommendation by a radiologist.