A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning-Based Risk Prediction Models.

OBJECTIVES: We propose a framework of health outcomes modeling with dynamic decision making and real-world data (RWD) to evaluate the potential utility of novel risk prediction models in clinical practice. Lung transplant (LTx) referral decisions in cystic fibrosis offer a complex case study. METHODS: We used longitudinal RWD for a cohort of adults (n = 4247) from the Cystic Fibrosis Foundation Patient Registry to compare outcomes of an LTx referral policy based on machine learning (ML) mortality risk predictions to referral based on (1) forced expiratory volume in 1 second (FEV1) alone and (2) heterogenous usual care (UC). We then developed a patient-level simulation model to project number of patients referred for LTx and 5-year survival, accounting for transplant availability, organ allocation policy, and heterogenous treatment effects. RESULTS: Only 12% of patients (95% confidence interval 11%-13%) were referred for LTx over 5 years under UC, compared with 19% (18%-20%) under FEV1 and 20% (19%-22%) under ML. Of 309 patients who died before LTx referral under UC, 31% (27%-36%) would have been referred under FEV1 and 40% (35%-45%) would have been referred under ML. Given a fixed supply of organs, differences in referral time did not lead to significant differences in transplants, pretransplant or post-transplant deaths, or overall survival in 5 years. CONCLUSIONS: Health outcomes modeling with RWD may help to identify novel ML risk prediction models with high potential real-world clinical utility and rule out further investment in models that are unlikely to offer meaningful real-world benefits.

Perioperative Opioid Use and Chronic Post-Surgical Pain after Liver Transplantation: A Single Center Observational Study.

OBJECTIVES: To investigate the opioid requirements and prevalence of chronic postsurgical pain (CPSP) in liver transplant (LT) recipients and to evaluate the association of opioid use with postoperative survival. DESIGN: Retrospective analysis. SETTING: A large academic medical center. PATIENTS: Cadaveric liver transplants recipients from 2008 to 2016. INTERVENTIONS: Analysis of demographic, perioperative, and outcome data. MEASUREMENTS AND MAIN RESULTS: This study measured the incidence and quantity of preoperative opioid use, postoperative opioid requirements, the incidence of CPSP, and survival in patients with and without CPSP. Opioid requirements were calculated in morphine milligram equivalents. In total, 322 LT recipients satisfied the inclusion criteria. The cohort of interest included 61 patients (18.9%) who were prescribed opioids before LT, compared to the control group of 261. Postoperative opioid requirements were significantly higher in the cohort of interest in the first 24 hours (205.9 ± 318.5 v 60.4 ± 33.6 mg, p < 0.0001) and at 7 days after transplant (57.0 ± 70.6 mg v 19.2 ± 15.4 mg, p < 0.0001). Incidence of CPSP was significantly higher in the cohort of interest at 3 months (70.5% v 45.5%, p < 0.0001), at 2 years (38% v 12%), and at 5 years (29.8% v 6.9%) postoperatively. CPSP was a significant risk factor for patient mortality after transplantation (p = 0.038, HR 1.26). CONCLUSIONS: Opioid use is relatively frequent in patients waiting for LT. It significantly affects the postoperative opioid requirements and the incidence of CSPS. CPSP may significantly affect survival after LT.

Semaglutide vs Tirzepatide for Weight Loss in Adults With Overweight or Obesity.

Importance: Although tirzepatide and semaglutide were shown to reduce weight in randomized clinical trials, data from head-to-head comparisons in populations with overweight or obesity are not yet available. Objective: To compare on-treatment weight loss and rates of gastrointestinal adverse events (AEs) among adults with overweight or obesity receiving tirzepatide or semaglutide labeled for type 2 diabetes (T2D) in a clinical setting. Design, Setting, and Participants: In this cohort study, adults with overweight or obesity receiving semaglutide or tirzepatide between May 2022 and September 2023 were identified using electronic health record (EHR) data linked to dispensing information from a collective of US health care systems. On-treatment weight outcomes through November 3, 2023, were assessed. Adults with overweight or obesity and regular care in the year before initiation, no prior glucagon-like peptide 1 receptor agonist receptor agonist use, a prescription within 60 days prior to initiation, and an available baseline weight were identified. The analysis was completed on April 3, 2024. Exposures: Tirzepatide or semaglutide in formulations labeled for T2D, on or off label. Main Outcomes and Measures: On-treatment weight change in a propensity score-matched population, assessed as hazard of achieving 5% or greater, 10% or greater, and 15% or greater weight loss, and percentage change in weight at 3, 6, and 12 months. Hazards of gastrointestinal AEs were compared. Results: Among 41 222 adults meeting the study criteria (semaglutide, 32 029; tirzepatide, 9193), 18 386 remained after propensity score matching. The mean (SD) age was 52.0 (12.9) years, 12 970 were female (70.5%), 14 182 were white (77.1%), 2171 Black (11.8%), 354 Asian (1.9%), 1679 were of other or unknown race, and 9563 (52.0%) had T2D. The mean (SD) baseline weight was 110 (25.8) kg. Follow-up was ended by discontinuation for 5140 patients (55.9%) receiving tirzepatide and 4823 (52.5%) receiving semaglutide. Patients receiving tirzepatide were significantly more likely to achieve weight loss (≥5%; hazard ratio [HR], 1.76, 95% CI, 1.68, 1.84; ≥10%; HR, 2.54; 95% CI, 2.37, 2.73; and ≥15%; HR, 3.24; 95% CI, 2.91, 3.61). On-treatment changes in weight were larger for patients receiving tirzepatide at 3 months (difference, -2.4%; 95% CI -2.5% to -2.2%), 6 months (difference, -4.3%; 95% CI, -4.7% to -4.0%), and 12 months (difference, -6.9%; 95% CI, -7.9% to -5.8%). Rates of gastrointestinal AEs were similar between groups. Conclusions and Relevance: In this population of adults with overweight or obesity, use of tirzepatide was associated with significantly greater weight loss than semaglutide. Future study is needed to understand differences in other important outcomes.

Using Machine Learning to Leverage Biomarker Change and Predict Colorectal Cancer Recurrence.

PURPOSE: The risk of colorectal cancer (CRC) recurrence after primary treatment varies across individuals and over time. Using patients' most up-to-date information, including carcinoembryonic antigen (CEA) biomarker profiles, to predict risk could improve personalized decision making. METHODS: We used electronic health record data from an integrated health system on a cohort of patients diagnosed with American Joint Committee on Cancer stage I-III CRC between 2008 and 2013 (N = 3,970) and monitored until recurrence or end of follow-up. We addressed missingness in recurrence outcomes and longitudinal CEA measures, and engineered CEA features using current and past biomarker values for inclusion in a risk prediction model. We used a discrete time Superlearner model to evaluate various algorithms for predicting recurrence. We evaluated the time-varying discrimination and calibration of the algorithms and assessed the role of individual predictors. RESULTS: Recurrence was documented in 448 (11.3%) patients. XGBoost with depth = 1 (XGB-D1) predicted recurrence substantially better than all other algorithms at all time points, with AUC ranging from 0.87 (95% CI, 0.86 to 0.88) at 6 months to 0.94 (95% CI, 0.92 to 0.96) at 54 months. The only variable used by XGB-D1 was 6-month change in log CEA. Predicted 1-year risk of recurrence was nearly zero for patients whose log CEA did not increase in the last 6 months, between 12.2% and 34.1% for patients whose log CEA increased between 0.10 and 0.40, and 43.6% for those with a log CEA increase >0.40. Compared with XGB, penalized regression approaches (lasso, ridge, and elastic net) performed poorly, with AUCs ranging from 0.58 to 0.69. CONCLUSION: A flexible, machine learning approach that incorporated longitudinal CEA information yielded a simple and high-performing model for predicting recurrence on the basis of 6-month change in log CEA.

Development and Internal Validation of a Prognostic Model of the Probability of Death or Lung Transplantation Within 2 Years for Patients With Cystic Fibrosis and FEV1 ≤ 50% Predicted.

BACKGROUND: Improved methods are needed to risk-stratify patients with cystic fibrosis (CF) and reduced FEV1. RESEARCH QUESTIONS: What are the predictors of death or lung transplantation (LTx) within 2 years among patients with CF whose FEV1 ≤ 50% predicted? Do these markers similarly predict outcomes among G551D patients taking ivacaftor since 2012? STUDY DESIGN AND METHODS: Patients with CF, age ≥ 6 years with FEV1 ≤ 50% predicted as of December 31, 2014, were identified in a data set that merged Cystic Fibrosis Foundation and United Network for Organ Sharing (UNOS) registries. The least absolute shrinkage and selection operator (LASSO) method was applied to a randomly selected training set to select important prognostic variables. Accuracy and association of the model with death or LTx with 2 years (2-year death or LTx) were validated via logistic regression on an independent test set. Sensitivity analyses explored predictors for patients with UNOS data. RESULTS: FEV1 percent predicted (OR, 1.51 for 5% decrease; 95% CI, 1.27-1.81), number of pulmonary exacerbations treated with IV antibiotics (OR, 1.35; 95% CI, 1.11-1.65), and continuous or nocturnal oxygen (OR, 3.71; 95% CI, 1.81-7.59) were significantly associated with 2-year death or LTx. Our model predicted outcomes with greater sensitivity (ratio of sensitivity, 1.26; 95% CI, 1.02-1.54), ratio of positive predictive value (1.25; 95% CI, 1.05-1.51), and ratio of negative predictive value (1.04; 95% CI, 1.01-1.07) than FEV1 < 30% predicted. Among those taking ivacaftor in 2014, only FEV1 remained associated with 2-year death or LTx. For patients with UNOS data, LASSO identified additional covariates of interest, including noninvasive ventilation use, low hemoglobin, pulmonary arterial systolic pressure, supplemental oxygen, mechanical ventilation, FEV1 percent predicted, and cardiac index. INTERPRETATION: Among individuals with CF and FEV1 ≤ 50% predicted, FEV1 percent predicted, oxygen therapy, and number of pulmonary exacerbations predicted 2-year death or LTx. Although limited by small sample size, only FEV1 remained predictive in patients receiving highly effective modulator therapy. Additional physiologic variables could improve prognostication in CF.

Applied Methods for Estimating Transition Probabilities from Electronic Health Record Data.

BACKGROUND: Electronic health record (EHR) data contain longitudinal patient information and standardized diagnostic codes. EHR data may be useful for estimating transition probabilities for state-transition models, but no guidelines exist on appropriate methods. We applied 3 potential methods to estimate transition probabilities from EHR data, using pediatric eating disorders (EDs) as a case study. METHODS: We obtained EHR data from PEDsnet, which includes 8 US children's hospitals. Data included inpatient, outpatient, and emergency department visits for all patients with an ED. We mapped diagnoses to 3 ED health states: anorexia nervosa, bulimia nervosa, and other specified feeding or eating disorder. We estimated 1-y transition probabilities for males and females using 3 approaches: simple first-last proportions, a multistate Markov (MSM) model, and independent survival models. RESULTS: Transition probability estimates varied widely between approaches. The first-last proportion approach estimated higher probabilities of remaining in the same health state, while the MSM and independent survival approaches estimated higher probabilities of transitioning to a different health state. All estimates differed substantially from published literature. LIMITATIONS: As a source of health state information, EHR data are incomplete and sometimes inaccurate. EHR data were especially challenging for EDs, limiting the estimation and interpretation of transition probabilities. CONCLUSIONS: The 3 approaches produced very different transition probability estimates. Estimates varied considerably from published literature and were rescaled and calibrated for use in a microsimulation model. Estimation of transition probabilities from EHR data may be more promising for diseases that are well documented in the EHR. Furthermore, clinicians and health systems should work to improve documentation of ED in the EHR. Further research is needed on methods for using EHR data to inform transition probabilities.

Cost-effectiveness of dual maternal HIV and syphilis testing strategies in high and low HIV prevalence countries: a modelling study.

BACKGROUND: Dual HIV and syphilis testing might help to prevent mother-to-child transmission (MTCT) of HIV and syphilis through increased case detection and treatment. We aimed to model and assess the cost-effectiveness of dual testing during antenatal care in four countries with varying HIV and syphilis prevalence. METHODS: In this modelling study, we developed Markov models of HIV and syphilis in pregnant women to estimate costs and infant health outcomes of maternal testing at the first antenatal care visit with individual HIV and syphilis tests (base case) and at the first antenatal care visit with a dual rapid diagnostic test (scenario one). We additionally evaluated retesting during late antenatal care and at delivery with either individual tests (scenario two) or a dual rapid diagnosis test (scenario three). We modelled four countries: South Africa, Kenya, Colombia, and Ukraine. Strategies with an incremental cost-effectiveness ratio (ICER) less than the country-specific cost-effectiveness threshold (US$500 in Kenya, $750 in South Africa, $3000 in Colombia, and $1000 in Ukraine) per disability-adjusted life-year averted were considered cost-effective. FINDINGS: Routinely offering testing at the first antenatal care visit with a dual rapid diagnosis test was cost-saving compared with the base case in all four countries (ICER: -$26 in Kenya,-$559 in South Africa, -$844 in Colombia, and -$454 in Ukraine). Retesting during late antenatal care with a dual rapid diagnostic test (scenario three) was cost-effective compared with scenario one in all four countries (ICER: $270 in Kenya, $260 in South Africa, $2207 in Colombia, and $205 in Ukraine). INTERPRETATION: Incorporating dual rapid diagnostic tests in antenatal care can be cost-saving across countries with varying HIV prevalence. Countries should consider incorporating dual HIV and syphilis rapid diagnostic tests as the first test in antenatal care to support efforts to eliminate MTCT of HIV and syphilis. FUNDING: WHO, US Agency for International Development, and the Bill & Melinda Gates Foundation.

Time-Series Analysis of First-Time Pediatric Speech Delays From 2018 to 2022.

This cohort study evaluates changes in pediatric speech delay diagnoses before and after the COVID-19 pandemic.