Development and preliminary testing of Health Equity Across the AI Lifecycle (HEAAL): A framework for healthcare delivery organizations to mitigate the risk of AI solutions worsening health inequities.

The use of data-driven technologies such as Artificial Intelligence (AI) and Machine Learning (ML) is growing in healthcare. However, the proliferation of healthcare AI tools has outpaced regulatory frameworks, accountability measures, and governance standards to ensure safe, effective, and equitable use. To address these gaps and tackle a common challenge faced by healthcare delivery organizations, a case-based workshop was organized, and a framework was developed to evaluate the potential impact of implementing an AI solution on health equity. The Health Equity Across the AI Lifecycle (HEAAL) is co-designed with extensive engagement of clinical, operational, technical, and regulatory leaders across healthcare delivery organizations and ecosystem partners in the US. It assesses 5 equity assessment domains-accountability, fairness, fitness for purpose, reliability and validity, and transparency-across the span of eight key decision points in the AI adoption lifecycle. It is a process-oriented framework containing 37 step-by-step procedures for evaluating an existing AI solution and 34 procedures for evaluating a new AI solution in total. Within each procedure, it identifies relevant key stakeholders and data sources used to conduct the procedure. HEAAL guides how healthcare delivery organizations may mitigate the potential risk of AI solutions worsening health inequities. It also informs how much resources and support are required to assess the potential impact of AI solutions on health inequities.

Strengthening the use of artificial intelligence within healthcare delivery organizations: balancing regulatory compliance and patient safety.

OBJECTIVES: Surface the urgent dilemma that healthcare delivery organizations (HDOs) face navigating the US Food and Drug Administration (FDA) final guidance on the use of clinical decision support (CDS) software. MATERIALS AND METHODS: We use sepsis as a case study to highlight the patient safety and regulatory compliance tradeoffs that 6129 hospitals in the United States must navigate. RESULTS: Sepsis CDS remains in broad, routine use. There is no commercially available sepsis CDS system that is FDA cleared as a medical device. There is no public disclosure of an HDO turning off sepsis CDS due to regulatory compliance concerns. And there is no public disclosure of FDA enforcement action against an HDO for using sepsis CDS that is not cleared as a medical device. DISCUSSION AND CONCLUSION: We present multiple policy interventions that would relieve the current tension to enable HDOs to utilize artificial intelligence to improve patient care while also addressing FDA concerns about product safety, efficacy, and equity.

A framework for understanding label leakage in machine learning for health care.

INTRODUCTION: The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule." FRAMEWORK: We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice. RECOMMENDATIONS: Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.

Accelerating health system innovation: principles and practices from the Duke Institute for Health Innovation.

The Duke Institute for Health Innovation (DIHI) was launched in 2013. Frontline staff members submit proposals for innovation projects that align with strategic priorities set by organizational leadership. Funded projects receive operational and technical support from institute staff members and a transdisciplinary network of collaborators to develop and implement solutions as part of routine clinical care, ranging from machine learning algorithms to mobile applications. DIHI's operations are shaped by four guiding principles: build to show value, build to integrate, build to scale, and build responsibly. Between 2013 and 2021, more than 600 project proposals have been submitted to DIHI. More than 85 innovation projects, both through the application process and other strategic partnerships, have been supported and implemented. DIHI's funding has incubated 12 companies, engaged more than 300 faculty members, staff members, and students, and contributed to more than 50 peer-reviewed publications. DIHI's practices can serve as a model for other health systems to systematically source, develop, implement, and scale innovations.

Respiratory Syncytial Virus During the COVID-19 Pandemic Compared to Historic Levels: A Retrospective Cohort Study of a Health System.

BACKGROUND: Surveillance in 2020-2021 showed that seasonal respiratory illnesses were below levels seen during prior seasons, with the exception of interseasonal respiratory syncytial virus (RSV). METHODS: Electronic health record data of infants aged <1 year visiting the Duke University Health System from 4 October 2015 to 28 March 2020 (pre-COVID-19) and 29 March 2020 to 30 October 2021 (COVID-19) were assessed. International Classification of Diseases-Tenth Revision (ICD-10) codes for RSV (B97.4, J12.1, J20.5, J21.0) and bronchiolitis (RSV codes plus J21.8, J21.9) were used to detail encounters in the inpatient (IP), emergency department (ED), outpatient (OP), urgent care (UC), and telemedicine (TM) settings. RESULTS: Pre-COVID-19, 88% of RSV and 92% of bronchiolitis encounters were seen in ambulatory settings. During COVID-19, 94% and 93%, respectively, occurred in ambulatory settings. Pre-COVID-19, the highest RSV proportion was observed in December-January (up to 38% in ED), while the peaks during COVID-19 were seen in July-September (up to 41% in ED) across all settings. RSV laboratory testing among RSV encounters was low during pre-COVID-19 (IP, 51%; ED, 51%; OP, 41%; UC, 84%) and COVID-19 outside of UC (IP, 33%; ED, 47%; OP, 47%; UC, 87%). Full-term, otherwise healthy infants comprised most RSV encounters (pre-COVID-19, up to 57% in OP; COVID-19, up to 82% in TM). CONCLUSIONS: With the interruption of historical RSV epidemiologic trends and the emergence of interseasonal disease during COVID-19, continued monitoring of RSV is warranted across all settings as the changing RSV epidemiology could affect the distribution of health care resources and public health policy.

Social determinants of health data in solid organ transplantation: National data sources and future directions.

Health equity research in transplantation has largely relied on national data sources, yet the availability of social determinants of health (SDOH) data varies widely among these sources. We sought to characterize the extent to which national data sources contain SDOH data applicable to end-stage organ disease (ESOD) and transplant patients. We reviewed 10 active national data sources based in the United States. For each data source, we examined patient inclusion criteria and explored strengths and limitations regarding SDOH data, using the National Institutes of Health PhenX toolkit of SDOH as a data collection instrument. Of the 28 SDOH variables reviewed, eight-core demographic variables were included in ≥80% of the data sources, and seven variables that described elements of social status ranged between 30 and 60% inclusion. Variables regarding identity, healthcare access, and social need were poorly represented (≤20%) across the data sources, and five of these variables were included in none of the data sources. The results of our review highlight the need for improved SDOH data collection systems in ESOD and transplant patients via: enhanced inter-registry collaboration, incorporation of standardized SDOH variables into existing data sources, and transplant center and consortium-based investigation and innovation.

Preliminary results of a clinical research and innovation scholarship to prepare medical students to lead innovations in health care.

There is consensus amongst national organizations to integrate health innovation and augmented intelligence (AI) into medical education. However, there is scant evidence to guide policymakers and medical educators working to revise curricula. This study presents academic, operational, and domain understanding outcomes for the first three cohorts of participants in a clinical research and innovation scholarship program.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study.

BACKGROUND: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. OBJECTIVE: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. METHODS: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. RESULTS: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. CONCLUSIONS: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study.

BACKGROUND: Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature. OBJECTIVE: This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care. METHODS: In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch. RESULTS: Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch. CONCLUSIONS: Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.

Presenting machine learning model information to clinical end users with model facts labels.

There is tremendous enthusiasm surrounding the potential for machine learning to improve medical prognosis and diagnosis. However, there are risks to translating a machine learning model into clinical care and clinical end users are often unaware of the potential harm to patients. This perspective presents the "Model Facts" label, a systematic effort to ensure that front-line clinicians actually know how, when, how not, and when not to incorporate model output into clinical decisions. The "Model Facts" label was designed for clinicians who make decisions supported by a machine learning model and its purpose is to collate relevant, actionable information in 1-page. Practitioners and regulators must work together to standardize presentation of machine learning model information to clinical end users in order to prevent harm to patients. Efforts to integrate a model into clinical practice should be accompanied by an effort to clearly communicate information about a machine learning model with a "Model Facts" label.

Barriers to Achieving Economies of Scale in Analysis of EHR Data. A Cautionary Tale.

Signed in 2009, the Health Information Technology for Economic and Clinical Health Act infused $28 billion of federal funds to accelerate adoption of electronic health records (EHRs). Yet, EHRs have produced mixed results and have even raised concern that the current technology ecosystem stifles innovation. We describe the development process and report initial outcomes of a chronic kidney disease analytics application that identifies high-risk patients for nephrology referral. The cost to validate and integrate the analytics application into clinical workflow was $217,138. Despite the success of the program, redundant development and validation efforts will require $38.8 million to scale the application across all multihospital systems in the nation. We address the shortcomings of current technology investments and distill insights from the technology industry. To yield a return on technology investments, we propose policy changes that address the underlying issues now being imposed on the system by an ineffective technology business model.