Developing and validating a machine-learning algorithm to predict opioid overdose in Medicaid beneficiaries in two US states: a prognostic modelling study.

BACKGROUND: Little is known about whether machine-learning algorithms developed to predict opioid overdose using earlier years and from a single state will perform as well when applied to other populations. We aimed to develop a machine-learning algorithm to predict 3-month risk of opioid overdose using Pennsylvania Medicaid data and externally validated it in two data sources (ie, later years of Pennsylvania Medicaid data and data from a different state). METHODS: This prognostic modelling study developed and validated a machine-learning algorithm to predict overdose in Medicaid beneficiaries with one or more opioid prescription in Pennsylvania and Arizona, USA. To predict risk of hospital or emergency department visits for overdose in the subsequent 3 months, we measured 284 potential predictors from pharmaceutical and health-care encounter claims data in 3-month periods, starting 3 months before the first opioid prescription and continuing until loss to follow-up or study end. We developed and internally validated a gradient-boosting machine algorithm to predict overdose using 2013-16 Pennsylvania Medicaid data (n=639 693). We externally validated the model using (1) 2017-18 Pennsylvania Medicaid data (n=318 585) and (2) 2015-17 Arizona Medicaid data (n=391 959). We reported several prediction performance metrics (eg, C-statistic, positive predictive value). Beneficiaries were stratified into risk-score subgroups to support clinical use. FINDINGS: A total of 8641 (1·35%) 2013-16 Pennsylvania Medicaid beneficiaries, 2705 (0·85%) 2017-18 Pennsylvania Medicaid beneficiaries, and 2410 (0·61%) 2015-17 Arizona beneficiaries had one or more overdose during the study period. C-statistics for the algorithm predicting 3-month overdoses developed from the 2013-16 Pennsylvania training dataset and validated on the 2013-16 Pennsylvania internal validation dataset, 2017-18 Pennsylvania external validation dataset, and 2015-17 Arizona external validation dataset were 0·841 (95% CI 0·835-0·847), 0·828 (0·822-0·834), and 0·817 (0·807-0·826), respectively. In external validation datasets, 71 361 (22·4%) of 318 585 2017-18 Pennsylvania beneficiaries were in high-risk subgroups (positive predictive value of 0·38-4·08%; capturing 73% of overdoses in the subsequent 3 months) and 40 041 (10%) of 391 959 2015-17 Arizona beneficiaries were in high-risk subgroups (positive predictive value of 0·19-1·97%; capturing 55% of overdoses). Lower risk subgroups in both validation datasets had few individuals (≤0·2%) with an overdose. INTERPRETATION: A machine-learning algorithm predicting opioid overdose derived from Pennsylvania Medicaid data performed well in external validation with more recent Pennsylvania data and with Arizona Medicaid data. The algorithm might be valuable for overdose risk prediction and stratification in Medicaid beneficiaries. FUNDING: National Institute of Health, National Institute on Drug Abuse, National Institute on Aging.

Changes in predicted opioid overdose risk over time in a state Medicaid program: a group-based trajectory modeling analysis.

BACKGROUND AND AIMS: The time lag encountered when accessing health-care data is one major barrier to implementing opioid overdose prediction measures in practice. Little is known regarding how one's opioid overdose risk changes over time. We aimed to identify longitudinal patterns of individual predicted overdose risks among Medicaid beneficiaries after initiation of opioid prescriptions. DESIGN, SETTING AND PARTICIPANTS: A retrospective cohort study in Pennsylvania, USA among Pennsylvania Medicaid beneficiaries aged 18-64 years who initiated opioid prescriptions between July 2017 and September 2018 (318 585 eligible beneficiaries (mean age = 39 ± 12 years, female = 65.7%, White = 62.2% and Black = 24.9%). MEASUREMENTS: We first applied a previously developed and validated machine-learning algorithm to obtain risk scores for opioid overdose emergency room or hospital visits in 3-month intervals for each beneficiary who initiated opioid therapy, until disenrollment from Medicaid, death or the end of observation (December 2018). We performed group-based trajectory modeling to identify trajectories of these predicted overdose risk scores over time. FINDINGS: Among eligible beneficiaries, 0.61% had one or more occurrences of opioid overdose in a median follow-up of 15 months. We identified five unique opioid overdose risk trajectories: three trajectories (accounting for 92% of the cohort) had consistent overdose risk over time, including consistent low-risk (63%), consistent medium-risk (25%) and consistent high-risk (4%) groups; another two trajectories (accounting for 8%) had overdose risks that substantially changed over time, including a group that transitioned from high- to medium-risk (3%) and another group that increased from medium- to high-risk over time (5%). CONCLUSIONS: More than 90% of Medicaid beneficiaries in Pennsylvania USA with one or more opioid prescriptions had consistent, predicted opioid overdose risks over 15 months. Applying opioid prediction algorithms developed from historical data may not be a major barrier to implementation in practice for the large majority of individuals.

Integrating human services and criminal justice data with claims data to predict risk of opioid overdose among Medicaid beneficiaries: A machine-learning approach.

Health system data incompletely capture the social risk factors for drug overdose. This study aimed to improve the accuracy of a machine-learning algorithm to predict opioid overdose risk by integrating human services and criminal justice data with health claims data to capture the social determinants of overdose risk. This prognostic study included Medicaid beneficiaries (n = 237,259) in Allegheny County, Pennsylvania enrolled between 2015 and 2018, randomly divided into training, testing, and validation samples. We measured 290 potential predictors (239 derived from Medicaid claims data) in 30-day periods, beginning with the first observed Medicaid enrollment date during the study period. Using a gradient boosting machine, we predicted a composite outcome (i.e., fatal or nonfatal opioid overdose constructed using medical examiner and claims data) in the subsequent month. We compared prediction performance between a Medicaid claims only model to one integrating human services and criminal justice data with Medicaid claims (i.e., integrated model) using several metrics (e.g., C-statistic, number needed to evaluate [NNE] to identify one overdose). Beneficiaries were stratified into risk-score decile subgroups. The samples (training = 79,087, testing = 79,086, validation = 79,086) had similar characteristics (age = 38±18 years, female = 56%, white = 48%, having at least one overdose = 1.7% during study period). Using the validation sample, the integrated model slightly improved on the Medicaid claims only model (C-statistic = 0.885; 95%CI = 0.877-0.892 vs. C-statistic = 0.871; 95%CI = 0.863-0.878), with small corresponding improvements in the NNE and positive predictive value. Nine of the top 30 most important predictors in the integrated model were human services and criminal justice variables. Using the integrated model, approximately 70% of individuals with overdoses were members of the top risk decile (overdose rates in the subsequent month = 47/10,000 beneficiaries). Few individuals in the bottom 9 deciles had overdose episodes (0-12/10,000). Machine-learning algorithms integrating claims and social service and criminal justice data modestly improved opioid overdose prediction among Medicaid beneficiaries for a large U.S. county heavily affected by the opioid crisis.

Using machine learning to predict risk of incident opioid use disorder among fee-for-service Medicare beneficiaries: A prognostic study.

OBJECTIVE: To develop and validate a machine-learning algorithm to improve prediction of incident OUD diagnosis among Medicare beneficiaries with ≥1 opioid prescriptions. METHODS: This prognostic study included 361,527 fee-for-service Medicare beneficiaries, without cancer, filling ≥1 opioid prescriptions from 2011-2016. We randomly divided beneficiaries into training, testing, and validation samples. We measured 269 potential predictors including socio-demographics, health status, patterns of opioid use, and provider-level and regional-level factors in 3-month periods, starting from three months before initiating opioids until development of OUD, loss of follow-up or end of 2016. The primary outcome was a recorded OUD diagnosis or initiating methadone or buprenorphine for OUD as proxy of incident OUD. We applied elastic net, random forests, gradient boosting machine, and deep neural network to predict OUD in the subsequent three months. We assessed prediction performance using C-statistics and other metrics (e.g., number needed to evaluate to identify an individual with OUD [NNE]). Beneficiaries were stratified into subgroups by risk-score decile. RESULTS: The training (n = 120,474), testing (n = 120,556), and validation (n = 120,497) samples had similar characteristics (age ≥65 years = 81.1%; female = 61.3%; white = 83.5%; with disability eligibility = 25.5%; 1.5% had incident OUD). In the validation sample, the four approaches had similar prediction performances (C-statistic ranged from 0.874 to 0.882); elastic net required the fewest predictors (n = 48). Using the elastic net algorithm, individuals in the top decile of risk (15.8% [n = 19,047] of validation cohort) had a positive predictive value of 0.96%, negative predictive value of 99.7%, and NNE of 104. Nearly 70% of individuals with incident OUD were in the top two deciles (n = 37,078), having highest incident OUD (36 to 301 per 10,000 beneficiaries). Individuals in the bottom eight deciles (n = 83,419) had minimal incident OUD (3 to 28 per 10,000). CONCLUSIONS: Machine-learning algorithms improve risk prediction and risk stratification of incident OUD in Medicare beneficiaries.

Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions.

Importance: Current approaches to identifying individuals at high risk for opioid overdose target many patients who are not truly at high risk. Objective: To develop and validate a machine-learning algorithm to predict opioid overdose risk among Medicare beneficiaries with at least 1 opioid prescription. Design, Setting, and Participants: A prognostic study was conducted between September 1, 2017, and December 31, 2018. Participants (n = 560 057) included fee-for-service Medicare beneficiaries without cancer who filled 1 or more opioid prescriptions from January 1, 2011, to December 31, 2015. Beneficiaries were randomly and equally divided into training, testing, and validation samples. Exposures: Potential predictors (n = 268), including sociodemographics, health status, patterns of opioid use, and practitioner-level and regional-level factors, were measured in 3-month windows, starting 3 months before initiating opioids until loss of follow-up or the end of observation. Main Outcomes and Measures: Opioid overdose episodes from inpatient and emergency department claims were identified. Multivariate logistic regression (MLR), least absolute shrinkage and selection operator-type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN) were applied to predict overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids. Prediction performance was assessed using the C statistic and other metrics (eg, sensitivity, specificity, and number needed to evaluate [NNE] to identify one overdose). The Youden index was used to identify the optimized threshold of predicted score that balanced sensitivity and specificity. Results: Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics (mean [SD] age of 68.0 [14.5] years, and approximately 63% were female, 82% were white, 35% had disabilities, 41% were dual eligible, and 0.60% had at least 1 overdose episode). In the validation sample, the DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and MLR (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose. At the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, NNE of 542, positive predictive value of 0.18%, and negative predictive value of 99.9%. The DNN classified patients into low-risk (76.2% [142 180] of the cohort), medium-risk (18.6% [34 579] of the cohort), and high-risk (5.2% [9747] of the cohort) subgroups, with only 1 in 10 000 in the low-risk subgroup having an overdose episode. More than 90% of overdose episodes occurred in the high-risk and medium-risk subgroups, although positive predictive values were low, given the rare overdose outcome. Conclusions and Relevance: Machine-learning algorithms appear to perform well for risk prediction and stratification of opioid overdose, especially in identifying low-risk subgroups that have minimal risk of overdose.

Hospital Perceptions of Medicare's Sepsis Quality Reporting Initiative.

BACKGROUND: In October 2015, the Centers for Medicare and Medicaid Services (CMS) implemented the Sepsis CMS Core Measure (SEP-1) program, requiring hospitals to report data on the quality of care for their patients with sepsis. OBJECTIVE: We sought to understand hospital perceptions of and responses to the SEP-1 program. DESIGN: A thematic content analysis of semistructured interviews with hospital quality officials. SETTING: A stratified random sample of short-stay, nonfederal, general acute care hospitals in the United States. PATIENTS: Hospital quality officers, including nurses and physicians. MEASUREMENTS: We completed 29 interviews before reaching content saturation. RESULTS: Hospitals reported a variety of actions in response to SEP-1, including new efforts to collect data, improve sepsis diagnosis and treatment, and manage clinicians' attitudes toward SEP-1. These efforts frequently required dedicated resources to meet the program's requirements for treatment and documentation, which were thought to be complex and not consistently linked to patient-centered outcomes. Most respondents felt that SEP-1 was likely to improve sepsis outcomes. At the same time, they described specific changes that could improve its effectiveness, including allowing hospitals to focus on the treatment processes most directly associated with improved patient outcomes and better aligning the measure's sepsis definitions with current clinical definitions. CONCLUSIONS: Hospitals are responding to the SEP-1 program across a number of domains and in ways that consistently require dedicated resources. Hospitals are interested in further revisions to the program to alleviate the burden of the reporting requirements and help them optimize the effectiveness of their investments in quality-improvement efforts.

ICU Telemedicine and Critical Care Mortality: A National Effectiveness Study.

BACKGROUND: Intensive care unit (ICU) telemedicine is an increasingly common strategy for improving the outcome of critical care, but its overall impact is uncertain. OBJECTIVES: To determine the effectiveness of ICU telemedicine in a national sample of hospitals and quantify variation in effectiveness across hospitals. RESEARCH DESIGN: We performed a multicenter retrospective case-control study using 2001-2010 Medicare claims data linked to a national survey identifying US hospitals adopting ICU telemedicine. We matched each adopting hospital (cases) to up to 3 nonadopting hospitals (controls) based on size, case-mix, and geographic proximity during the year of adoption. Using ICU admissions from 2 years before and after the adoption date, we compared outcomes between case and control hospitals using a difference-in-differences approach. RESULTS: A total of 132 adopting case hospitals were matched to 389 similar nonadopting control hospitals. The preadoption and postadoption unadjusted 90-day mortality was similar in both case hospitals (24.0% vs. 24.3%, P=0.07) and control hospitals (23.5% vs. 23.7%, P<0.01). In the difference-in-differences analysis, ICU telemedicine adoption was associated with a small relative reduction in 90-day mortality (ratio of odds ratios=0.96; 95% CI, 0.95-0.98; P<0.001). However, there was wide variation in the ICU telemedicine effect across individual hospitals (median ratio of odds ratios=1.01; interquartile range, 0.85-1.12; range, 0.45-2.54). Only 16 case hospitals (12.2%) experienced statistically significant mortality reductions postadoption. Hospitals with a significant mortality reduction were more likely to have large annual admission volumes (P<0.001) and be located in urban areas (P=0.04) compared with other hospitals. CONCLUSIONS: Although ICU telemedicine adoption resulted in a small relative overall mortality reduction, there was heterogeneity in effect across adopting hospitals, with large-volume urban hospitals experiencing the greatest mortality reductions.