Addressing cost barriers to healthy eating with eat Well, a prescription produce subsidy, for patients with diabetes and at risk for food insecurity: Study protocol for a type 1 hybrid effectiveness-implementation pragmatic randomized controlled trial.

BACKGROUND: Patients with diabetes at risk of food insecurity face cost barriers to healthy eating and, as a result, poor health outcomes. Population health management strategies are needed to improve food security in real-world health system settings. We seek to test the effect of a prescription produce program, 'Eat Well' on cardiometabolic health and healthcare utilization. We will also assess the implementation of an automated, affirmative outreach strategy. METHODS: We will recruit approximately 2400 patients from an integrated academic health system in the southeastern United States as part of a two-arm parallel hybrid type 1 pragmatic randomized controlled trial. Patients with diabetes, at risk for food insecurity, and a recent hemoglobin A1c reading will be eligible to participate. The intervention arm receives, 'Eat Well', which provides a debit card with $80 (added monthly) for 12 months valid for fresh, frozen, or canned fruits and vegetables across grocery retailers. The control arm does not. Both arms receive educational resources with diabetes nutrition and self-management materials, and information on existing care management resources. Using an intent-to-treat analysis, primary outcomes include hemoglobin A1C levels and emergency department visits in the 12 months following enrollment. Reach and fidelity data will be collected to assess implementation. DISCUSSION: Addressing food insecurity, particularly among those at heightened cardiometabolic risk, is critical to equitable and effective population health management. Pragmatic trials provide important insights into the effectiveness and implementation of 'Eat Well' and approaches like it in real-world settings. REGISTRATION: ClinicalTrials.gov Identifier: NCT05896644; Clinical Trial Registration Date: 2023-06-09.

Incorporating informatively collected laboratory data from EHR in clinical prediction models.

BACKGROUND: Electronic Health Records (EHR) are widely used to develop clinical prediction models (CPMs). However, one of the challenges is that there is often a degree of informative missing data. For example, laboratory measures are typically taken when a clinician is concerned that there is a need. When data are the so-called Not Missing at Random (NMAR), analytic strategies based on other missingness mechanisms are inappropriate. In this work, we seek to compare the impact of different strategies for handling missing data on CPMs performance. METHODS: We considered a predictive model for rapid inpatient deterioration as an exemplar implementation. This model incorporated twelve laboratory measures with varying levels of missingness. Five labs had missingness rate levels around 50%, and the other seven had missingness levels around 90%. We included them based on the belief that their missingness status can be highly informational for the prediction. In our study, we explicitly compared the various missing data strategies: mean imputation, normal-value imputation, conditional imputation, categorical encoding, and missingness embeddings. Some of these were also combined with the last observation carried forward (LOCF). We implemented logistic LASSO regression, multilayer perceptron (MLP), and long short-term memory (LSTM) models as the downstream classifiers. We compared the AUROC of testing data and used bootstrapping to construct 95% confidence intervals. RESULTS: We had 105,198 inpatient encounters, with 4.7% having experienced the deterioration outcome of interest. LSTM models generally outperformed other cross-sectional models, where embedding approaches and categorical encoding yielded the best results. For the cross-sectional models, normal-value imputation with LOCF generated the best results. CONCLUSION: Strategies that accounted for the possibility of NMAR missing data yielded better model performance than those did not. The embedding method had an advantage as it did not require prior clinical knowledge. Using LOCF could enhance the performance of cross-sectional models but have countereffects in LSTM models.

Enhancing Clinical Decision Support in Nephrology: Addressing Algorithmic Bias Through Artificial Intelligence Governance.

There has been a steady rise in the use of clinical decision support (CDS) tools to guide nephrology as well as general clinical care. Through guidance set by federal agencies and concerns raised by clinical investigators, there has been an equal rise in understanding whether such tools exhibit algorithmic bias leading to unfairness. This has spurred the more fundamental question of whether sensitive variables such as race should be included in CDS tools. In order to properly answer this question, it is necessary to understand how algorithmic bias arises. We break down 3 sources of bias encountered when using electronic health record data to develop CDS tools: (1) use of proxy variables, (2) observability concerns and (3) underlying heterogeneity. We discuss how answering the question of whether to include sensitive variables like race often hinges more on qualitative considerations than on quantitative analysis, dependent on the function that the sensitive variable serves. Based on our experience with our own institution's CDS governance group, we show how health system-based governance committees play a central role in guiding these difficult and important considerations. Ultimately, our goal is to foster a community practice of model development and governance teams that emphasizes consciousness about sensitive variables and prioritizes equity.

Designing an Implementable Clinical Prediction Model for Near-Term Mortality and Long-Term Survival in Patients on Maintenance Hemodialysis.

RATIONALE & OBJECTIVE: The life expectancy of patients treated with maintenance hemodialysis (MHD) is heterogeneous. Knowledge of life-expectancy may focus care decisions on near-term versus long-term goals. The current tools are limited and focus on near-term mortality. Here, we develop and assess potential utility for predicting near-term mortality and long-term survival on MHD. STUDY DESIGN: Predictive modeling study. SETTING & PARTICIPANTS: 42,351 patients contributing 997,381 patient months over 11 years, abstracted from the electronic health record (EHR) system of midsize, nonprofit dialysis providers. NEW PREDICTORS & ESTABLISHED PREDICTORS: Demographics, laboratory results, vital signs, and service utilization data available within dialysis EHR. OUTCOME: For each patient month, we ascertained death within the next 6 months (ie, near-term mortality) and survival over more than 5 years during receipt of MHD or after kidney transplantation (ie, long-term survival). ANALYTICAL APPROACH: We used least absolute shrinkage and selection operator logistic regression and gradient-boosting machines to predict each outcome. We compared these to time-to-event models spanning both time horizons. We explored the performance of decision rules at different cut points. RESULTS: All models achieved an area under the receiver operator characteristic curve of≥0.80 and optimal calibration metrics in the test set. The long-term survival models had significantly better performance than the near-term mortality models. The time-to-event models performed similarly to binary models. Applying different cut points spanning from the 1st to 90th percentile of the predictions, a positive predictive value (PPV) of 54% could be achieved for near-term mortality, but with poor sensitivity of 6%. A PPV of 71% could be achieved for long-term survival with a sensitivity of 67%. LIMITATIONS: The retrospective models would need to be prospectively validated before they could be appropriately used as clinical decision aids. CONCLUSIONS: A model built with readily available clinical variables to support easy implementation can predict clinically important life expectancy thresholds and shows promise as a clinical decision support tool for patients on MHD. Predicting long-term survival has better decision rule performance than predicting near-term mortality. PLAIN-LANGUAGE SUMMARY: Clinical prediction models (CPMs) are not widely used for patients undergoing maintenance hemodialysis (MHD). Although a variety of CPMs have been reported in the literature, many of these were not well-designed to be easily implementable. We consider the performance of an implementable CPM for both near-term mortality and long-term survival for patients undergoing MHD. Both near-term and long-term models have similar predictive performance, but the long-term models have greater clinical utility. We further consider how the differential performance of predicting over different time horizons may be used to impact clinical decision making. Although predictive modeling is not regularly used for MHD patients, such tools may help promote individualized care planning and foster shared decision making.

Impact of preeclampsia on infant and maternal health among women with rheumatic diseases.

OBJECTIVES: We sought to identify the impact of preeclampsia on infant and maternal health among women with rheumatic diseases. METHODS: A retrospective single-center cohort study was conducted to describe pregnancy and infant outcomes among women with systemic lupus erythematosus (SLE) with and without preeclampsia as compared to women with other rheumatic diseases with and without preeclampsia. RESULTS: We identified 263 singleton deliveries born to 226 individual mothers (mean age 31 years, 35% non-Hispanic Black). Overall, 14% of women had preeclampsia; preeclampsia was more common among women with SLE than other rheumatic diseases (27% vs 8%). Women with preeclampsia had a longer hospital stay post-delivery. Infants born to mothers with preeclampsia were delivered an average of 3.3 weeks earlier than those without preeclampsia, were 4 times more likely to be born preterm, and twice as likely to be admitted to the neonatal intensive care unit. The large majority of women with SLE in this cohort were prescribed hydroxychloroquine and aspirin, with no clear association of these medications with preeclampsia. CONCLUSIONS: We found preeclampsia was an important driver of adverse infant and maternal outcomes. While preeclampsia was particularly common among women with SLE in this cohort, the impact of preeclampsia on the infants of all women with rheumatic diseases was similarly severe. In order to improve infant outcomes for women with rheumatic diseases, attention must be paid to preventing, identifying, and managing preeclampsia.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Comparing Natural Language Processing and Structured Medical Data to Develop a Computable Phenotype for Patients Hospitalized Due to COVID-19: Retrospective Analysis.

Background: Throughout the COVID-19 pandemic, many hospitals conducted routine testing of hospitalized patients for SARS-CoV-2 infection upon admission. Some of these patients are admitted for reasons unrelated to COVID-19 and incidentally test positive for the virus. Because COVID-19-related hospitalizations have become a critical public health indicator, it is important to identify patients who are hospitalized because of COVID-19 as opposed to those who are admitted for other indications. Objective: We compared the performance of different computable phenotype definitions for COVID-19 hospitalizations that use different types of data from electronic health records (EHRs), including structured EHR data elements, clinical notes, or a combination of both data types. Methods: We conducted a retrospective data analysis, using clinician chart review-based validation at a large academic medical center. We reviewed and analyzed the charts of 586 hospitalized individuals who tested positive for SARS-CoV-2 in January 2022. We used LASSO (least absolute shrinkage and selection operator) regression and random forests to fit classification algorithms that incorporated structured EHR data elements, clinical notes, or a combination of structured data and clinical notes. We used natural language processing to incorporate data from clinical notes. The performance of each model was evaluated based on the area under the receiver operator characteristic curve (AUROC) and an associated decision rule based on sensitivity and positive predictive value. We also identified top words and clinical indicators of COVID-19-specific hospitalization and assessed the impact of different phenotyping strategies on estimated hospital outcome metrics. Results: Based on a chart review, 38.2% (224/586) of patients were determined to have been hospitalized for reasons other than COVID-19, despite having tested positive for SARS-CoV-2. A computable phenotype that used clinical notes had significantly better discrimination than one that used structured EHR data elements (AUROC: 0.894 vs 0.841; P<.001) and performed similarly to a model that combined clinical notes with structured data elements (AUROC: 0.894 vs 0.893; P=.91). Assessments of hospital outcome metrics significantly differed based on whether the population included all hospitalized patients who tested positive for SARS-CoV-2 or those who were determined to have been hospitalized due to COVID-19. Conclusions: These findings highlight the importance of cause-specific phenotyping for COVID-19 hospitalizations. More generally, this work demonstrates the utility of natural language processing approaches for deriving information related to patient hospitalizations in cases where there may be multiple conditions that could serve as the primary indication for hospitalization.

Impact of state legislation and institutional protocols on opioid prescribing practices following pediatric tonsillectomy.

Objectives: Tonsillectomy is a common pediatric surgery, and pain is an important consideration in recovery. Due to the opioid epidemic, individual states, medical societies, and institutions have all taken steps to limit postoperative opioids, yet few studies have examined the effect of these interventions on pediatric otolaryngology practices. The primary aim of this study was to characterize opioid prescribing practices following North Carolina state opioid legislation and targeted institutional changes. Methods: This single center retrospective cohort study included 1552 pediatric tonsillectomy patient records from 2014 to 2021. The primary outcome was number of oxycodone doses per prescription. This outcome was assessed over three time periods: (1) Before 2018 North Carolina opioid legislation. (2) Following legislation, before institutional changes. (3) After institutional opioid-specific protocols. Results: The mean (± standard deviation) number of doses per prescription in Periods 1, 2, and 3 was: 58 ± 53, range 4-493; 28 ± 36, range 3-488; and 23 ± 17, range 1-139, respectively. In the adjusted model, Periods 2 and 3 had lower doses by -41% (95% CI -49%, -32%) and -40% (95% CI -55%, -19%) compared to Period 1. After 2018 North Carolina legislation, dosage decreased by -9% (95% CI -13%, -5%) per year. Despite interventions, ongoing variability in prescription regimens remained in all periods. Conclusion: Legislative and institution specific opioid interventions was associated with a 40% decrease in oxycodone doses per prescription following pediatric tonsillectomy. While variability in opioid practices decreased post-interventions, it was not eliminated. Level of evidence: 3.

Predictive Modeling to Identify Children With Complex Health Needs At Risk for Hospitalization.

BACKGROUND: Identifying children at high risk with complex health needs (CCHN) who have intersecting medical and social needs is challenging. This study's objectives were to (1) develop and evaluate an electronic health record (EHR)-based clinical predictive model ("model") for identifying high-risk CCHN and (2) compare the model's performance as a clinical decision support (CDS) to other CDS tools available for identifying high-risk CCHN. METHODS: This retrospective cohort study included children aged 0 to 20 years with established care within a single health system. The model development/validation cohort included 33 months (January 1, 2016-September 30, 2018) and the testing cohort included 18 months (October 1, 2018-March 31, 2020) of EHR data. Machine learning methods generated a model that predicted probability (0%-100%) for hospitalization within 6 months. Model performance measures included sensitivity, positive predictive value, area under receiver-operator curve, and area under precision-recall curve. Three CDS rules for identifying high-risk CCHN were compared: (1) hospitalization probability ≥10% (model-predicted); (2) complex chronic disease classification (using Pediatric Medical Complexity Algorithm [PMCA]); and (3) previous high hospital utilization. RESULTS: Model development and testing cohorts included 116 799 and 27 087 patients, respectively. The model demonstrated area under receiver-operator curve = 0.79 and area under precision-recall curve = 0.13. PMCA had the highest sensitivity (52.4%) and classified the most children as high risk (17.3%). Positive predictive value of the model-based CDS rule (19%) was higher than CDS based on the PMCA (1.9%) and previous hospital utilization (15%). CONCLUSIONS: A novel EHR-based predictive model was developed and validated as a population-level CDS tool for identifying CCHN at high risk for future hospitalization.

Use of Structured Electronic Health Records Data Elements for the Development of Computable Phenotypes to Identify Potential Adverse Events Associated with Intravenous Immunoglobulin Infusion.

INTRODUCTION: Detection of adverse reactions to drugs and biologic agents is an important component of regulatory approval and post-market safety evaluation. Real-world data, including insurance claims and electronic health records data, are increasingly used for the evaluation of potential safety outcomes; however, there are different types of data elements available within these data resources, impacting the development and performance of computable phenotypes for the identification of adverse events (AEs) associated with a given therapy. OBJECTIVE: To evaluate the utility of different types of data elements to the performance of computable phenotypes for AEs. METHODS: We used intravenous immunoglobulin (IVIG) as a model therapeutic agent and conducted a single-center, retrospective study of 3897 individuals who had at least one IVIG administration between 1 January 2014 and 31 December 2019. We identified the potential occurrence of four different AEs, including two proximal AEs (anaphylaxis and heart rate alterations) and two distal AEs (thrombosis and hemolysis). We considered three different computable phenotypes: (1) an International Classification of Disease (ICD)-based phenotype; (2) a phenotype-based on EHR-derived contextual information based on structured data elements, including laboratory values, medication administrations, or vital signs; and (3) a compound phenotype that required both an ICD code for the AE in combination with additional EHR-derived structured data elements. We evaluated the performance of each of these computable phenotypes compared with chart review-based identification of AEs, assessing the positive predictive value (PPV), specificity, and estimated sensitivity of each computable phenotype method. RESULTS: Compound computable phenotypes had a high positive predictive value for acute AEs such as anaphylaxis and bradycardia or tachycardia; however, few patients had both ICD codes and the relevant contextual data, which decreased the sensitivity of these computable phenotypes. In contrast, computable phenotypes for distal AEs (i.e., thrombotic events or hemolysis) frequently had ICD codes for these conditions in the absence of an AE due to a prior history of such events, suggesting that patient medical history of AEs negatively impacted the PPV of computable phenotypes based on ICD codes. CONCLUSIONS: These data provide evidence for the utility of different structured data elements in computable phenotypes for AEs. Such computable phenotypes can be used across different data sources for the detection of infusion-related adverse events.

Utilization and Outcomes of Extracorporeal Membrane Oxygenation Following Traumatic Brain Injury in the United States.

Objectives: Describe contemporary ECMO utilization patterns among patients with traumatic brain injury (TBI) and examine clinical outcomes among TBI patients requiring ECMO. Design: Retrospective cohort study. Setting: Premier Healthcare Database (PHD) between January 2016 to June 2020. Subjects: Adult patients with TBI who were mechanically ventilated and stratified by exposure to ECMO. Results: Among patients exposed to ECMO, we examined the following clinical outcomes: hospital LOS, ICU LOS, duration of mechanical ventilation, and hospital mortality. Of our initial cohort (n = 59,612), 118 patients (0.2%) were placed on ECMO during hospitalization. Most patients were placed on ECMO within the first 2 days of admission (54.3%). Factors associated with ECMO utilization included younger age (OR 0.96, 95% CI (0.95-0.97)), higher injury severity score (ISS) (OR 1.03, 95% CI (1.01-1.04)), vasopressor utilization (2.92, 95% CI (1.90-4.48)), tranexamic acid utilization (OR 1.84, 95% CI (1.12-3.04)), baseline comorbidities (OR 1.06, 95% CI (1.03-1.09)), and care in a teaching hospital (OR 3.04, 95% CI 1.31-7.05). A moderate degree (ICC = 19.5%) of variation in ECMO use was explained at the individual hospital level. Patients exposed to ECMO had longer median (IQR) hospital and ICU length of stay (LOS) [26 days (11-36) versus 9 days (4-8) and 19.5 days (8-32) versus 5 days (2-11), respectively] and a longer median (IQR) duration of mechanical ventilation [18 days (8-31) versus 3 days (2-8)]. Patients exposed to ECMO experienced a hospital mortality rate of 33.9%, compared to 21.2% of TBI patients unexposed to ECMO. Conclusions: ECMO utilization in mechanically ventilated patients with TBI is rare, with significant variation across hospitals. The impact of ECMO on healthcare utilization and hospital mortality following TBI is comparable to non-TBI conditions requiring ECMO. Further research is necessary to better understand the role of ECMO following TBI and identify patients who may benefit from this therapy.

Transition from pediatric to adult care in type 1 diabetes mellitus: a longitudinal analysis of age at transfer and gap in care.

INTRODUCTION: Adolescents and young adults (AYAs) with type 1 diabetes (T1D) are at risk of suboptimal glycemic control and high acute care utilization. Little is known about the optimal age to transfer people with T1D to adult care, or time gap between completing pediatric care and beginning adult endocrinology care. RESEARCH DESIGN AND METHODS: This retrospective, longitudinal study examined the transition of AYAs with T1D who received endocrinology care within Duke University Health System. We used linear multivariable or Poisson regression modeling to assess the association of (1) sociodemographic and clinical factors associated with gap in care and age at transfer among AYAs and (2) the impact of gap in care and age at transfer on subsequent glycemic control and acute care utilization. RESULTS: There were 214 subjects included in the analysis (54.2% female, 72.8% white). The median time to transition and age at transition were 8.0 months and 21.5 years old, respectively. The median gap in care was extended by a factor of 3.39 (95% CI=1.25 to 9.22, p=0.02) for those who did not see a mental health provider pre-transfer. Individuals who did not see a diabetes educator in pediatrics had an increase in mean age at transition of 2.62 years (95% CI=0.93 to 4.32, p<0.01). The post-transfer emergency department visit rate was increased for every month increase in gap in care by a relative factor of 1.07 (95% CI=1.03 to 1.11, p<0.01). For every year increase in age at transition, post-transfer hospitalization rate was associated with a reduction of a relative factor of 0.62 (95% CI=0.45 to 0.85, p<0.01) and emergency department visit rate by 0.58 (95% CI=0.45 to 0.76, p<0.01). CONCLUSIONS: Most AYAs with T1D have a prolonged gap in care. When designing interventions to improve health outcomes for AYAs transitioning from pediatric to adult-based care, we should aim to minimize gaps in care.

Association between Gentrification and Health and Healthcare Utilization.

There is tremendous interest in understanding how neighborhoods impact health by linking extant social and environmental drivers of health (SDOH) data with electronic health record (EHR) data. Studies quantifying such associations often use static neighborhood measures. Little research examines the impact of gentrification-a measure of neighborhood change-on the health of long-term neighborhood residents using EHR data, which may have a more generalizable population than traditional approaches. We quantified associations between gentrification and health and healthcare utilization by linking longitudinal socioeconomic data from the American Community Survey with EHR data across two health systems accessed by long-term residents of Durham County, NC, from 2007 to 2017. Census block group-level neighborhoods were eligible to be gentrified if they had low socioeconomic status relative to the county average. Gentrification was defined using socioeconomic data from 2006 to 2010 and 2011-2015, with the Steinmetz-Wood definition. Multivariable logistic and Poisson regression models estimated associations between gentrification and development of health indicators (cardiovascular disease, hypertension, diabetes, obesity, asthma, depression) or healthcare encounters (emergency department [ED], inpatient, or outpatient). Sensitivity analyses examined two alternative gentrification measures. Of the 99 block groups within the city of Durham, 28 were eligible (N = 10,807; median age = 42; 83% Black; 55% female) and 5 gentrified. Individuals in gentrifying neighborhoods had lower odds of obesity (odds ratio [OR] = 0.89; 95% confidence interval [CI]: 0.81-0.99), higher odds of an ED encounter (OR = 1.10; 95% CI: 1.01-1.20), and lower risk for outpatient encounters (incidence rate ratio = 0.93; 95% CI: 0.87-1.00) compared with non-gentrifying neighborhoods. The association between gentrification and health and healthcare utilization was sensitive to gentrification definition.

Risk Factors and Neurological Outcomes Associated With Circulatory Shock After Moderate-Severe Traumatic Brain Injury: A TRACK-TBI Study.

BACKGROUND: Extracranial multisystem organ failure is a common sequela of severe traumatic brain injury (TBI). Risk factors for developing circulatory shock and long-term functional outcomes of this patient subset are poorly understood. OBJECTIVE: To identify emergency department predictors of circulatory shock after moderate-severe TBI and examine long-term functional outcomes in patients with moderate-severe TBI who developed circulatory shock. METHODS: We conducted a retrospective cohort study using the Transforming Clinical Research and Knowledge in TBI database for adult patients with moderate-severe TBI, defined as a Glasgow Coma Scale (GCS) score of <13 and stratified by the development of circulatory shock within 72 hours of hospital admission (Sequential Organ Failure Assessment score ≥2). Demographic and clinical data were assessed with descriptive statistics. A forward selection regression model examined risk factors for the development of circulatory shock. Functional outcomes were examined using multivariable regression models. RESULTS: Of our moderate-severe TBI population (n = 407), 168 (41.2%) developed circulatory shock. Our predictive model suggested that race, computed tomography Rotterdam scores <3, GCS in the emergency department, and development of hypotension in the emergency department were associated with developing circulatory shock. Those who developed shock had less favorable 6-month functional outcomes measured by the 6-month GCS-Extended (odds ratio 0.36, P = .002) and 6-month Disability Rating Scale score (Diff. in means 3.86, P = .002) and a longer length of hospital stay (Diff. in means 11.0 days, P < .001). CONCLUSION: We report potential risk factors for circulatory shock after moderate-severe TBI. Our study suggests that developing circulatory shock after moderate-severe TBI is associated with poor long-term functional outcomes.

A framework for the oversight and local deployment of safe and high-quality prediction models.

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.