Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach.

OBJECTIVE: To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators. DESIGN, SETTING AND PARTICIPANTS: This study was conducted in Alberta, Canada during 2018-2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded. EXPOSURE: Each BZRA dispensation from a community pharmacy served as the unit of analysis. MAIN OUTCOMES AND MEASURES: ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data. RESULTS: 65,063 study participants were included which represented 633,333 BZRA dispensation during 2018-2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance. CONCLUSION: Developing ML models using only administrative health data may not provide health regulators with sufficient informative predictions to use as decision aids for potential interventions, especially if considering daily or quarterly classifications of BZRA risks in older adults. ML models may be informative for this context if yearly classifications are preferred. Health regulators should have access to other types of data to improve ML prediction.

Predicting falls-related admissions in older adults in Alberta, Canada: a machine-learning falls prevention tool developed using population administrative health data.

OBJECTIVE: To construct a machine-learning (ML) model for health systems with organised falls prevention programmes to identify older adults at risk for fall-related admissions. DESIGN: This prognostic study used population-level administrative health data to develop an ML prediction model. SETTING: This study took place in Alberta, Canada during 2018-2019. PARTICIPANTS: Albertans aged 65 and older with at least one prior admission. Those with palliative conditions or emigrated out of Alberta were excluded. EXPOSURE: Unit of analysis was the individual person. MAIN OUTCOMES/MEASURES: We identified fall-related admissions. A CatBoost model was developed on 2018 data to predict risk of fall-related emergency department visits or hospitalisations. Temporal validation was done using 2019 data to evaluate model performance. We reported discrimination, calibration and other relevant metrics measured at the end of 2019 on both ranked predictions and predicted probability thresholds. A cost-savings simulation was performed using 2019 data. RESULTS: Final number of study participants was 224 445. The validation set had 203 584 participants with 19 389 fall-related events (9.5% pretest probability) and an ML model c-statistic of 0.70. The highest ranked predictions had post-test probabilities ranging from 40% to 50%. Net benefit analysis presented mixed results with some net benefit using the ML model in the 6%-30% range. The top 50 percentile of predicted risks represented nearly $C60 million in health system costs related to falls. Intervening on the top 25 or 50 percentiles of predicted risk could realise substantial (up to $C16 million) savings. CONCLUSION: ML prediction models based on population-level administrative data can assist health systems with fall prevention programmes identify older adults at risk of fall-related admissions and reduce costs. ML predictions based on ranked predictions or probability thresholds could guide subsequent interventions to mitigate fall risks. Increased access to diverse forms of data could improve ML performance and further reduce costs.

Development and Validation of a Machine Learning Model to Estimate Risk of Adverse Outcomes Within 30 Days of Opioid Dispensation.

Importance: Machine learning approaches can assist opioid stewardship by identifying high-risk opioid prescribing for potential interventions. Objective: To develop a machine learning model for deployment that can estimate the risk of adverse outcomes within 30 days of an opioid dispensation as a potential component of prescription drug monitoring programs using access to real-world data. Design, Setting, and Participants: This prognostic study used population-level administrative health data to construct a machine learning model. This study took place in Alberta, Canada (from January 1, 2018, to December 31, 2019), and included all patients 18 years and older who received at least 1 opioid dispensation from a community pharmacy within the province. Exposures: Each opioid dispensation served as the unit of analysis. Main Outcomes and Measures: Opioid-related adverse outcomes were identified from administrative data sets. An XGBoost model was developed on 2018 data to estimate the risk of hospitalization, an emergency department visit, or mortality within 30 days of an opioid dispensation; validation on 2019 data was done to evaluate model performance. Model discrimination, calibration, and other relevant metrics are reported using daily and weekly predictions on both ranked predictions and predicted probability thresholds using all data from 2019. Results: A total of 853 324 participants represented 6 181 025 opioid dispensations, with 145 016 outcome events reported (2.3%); 46.4% of the participants were men and 53.6% were women, with a mean (SD) age of 49.1 (15.6) years for men and 51.0 (18.0) years for women. Of the outcome events, 77 326 (2.6% pretest probability) occurred within 30 days of a dispensation in the validation set (XGBoost C statistic, 0.82 [95% CI, 0.81-0.82]). The top 0.1 percentile of estimated risk had a positive likelihood ratio (LR) of 28.7, which translated to a posttest probability of 43.1%. In our simulations, the weekly measured predictions had higher positive LRs in both the highest-risk dispensations and percentiles of estimated risk compared with predictions measured daily. Net benefit analysis showed that using machine learning prediction may not add additional benefit over the entire range of probability thresholds. Conclusions and Relevance: These findings suggest that prescription drug monitoring programs can use machine learning classifiers to identify patients at risk of opioid-related adverse outcomes and intervene on high-risk ranked predictions. Better access to available administrative and clinical data could improve the prediction performance of machine learning classifiers and thus expand opioid stewardship efforts.

Predicting 30-Day Readmissions in Patients With Heart Failure Using Administrative Data: A Machine Learning Approach.

BACKGROUND: We sought to develop machine learning (ML) models trained on administrative data which predict risk of readmission in patients with heart failure and to evaluate and compare the ML model with the currently used LaCE score using clinically informative metrics. METHODS AND RESULTS: This prognostic study was conducted in Alberta, Canada, on 9845 patients with confirmed heart failure admitted to hospital between 2012 and 2019. The outcome was unplanned all-cause hospital readmission within 30 days of discharge. We used 80% of the data for the ML model development and 20% for independent validation. We reported, using the validation set, c-statistics (area under the receiver operating characteristic curves)and performance metrics (likelihood ratio, positive predictive values) for the XGBoost model and a modified LaCE score within their respective predictive thresholds. Boosted tree-based classifiers had higher area under the receiver operating characteristic curves (0.65 for XGBoost) compared with others (0.58 for neural networks) and 0.57 for the modified LaCE. Within the predicted threshold range of the XGBoost classifier, the positive likelihood ratio was 1.00 at the low end of predicted risk and 6.12 at the high end, resulting in a positive predictive value (post-test probability) range of 21%-62%; the pretest probability of readmission was 20.9% using prevalence. The corresponding positive likelihood ratios and positive predictive values across LaCE score thresholds were 1.00-1.20 and 21%-24%, respectively. CONCLUSIONS: Despite predicting readmissions better than the LaCE, even the best ML model trained on administrative health data (XGBoost) did not provide substantially informative prediction performance as it only generated a moderate shift from pre to post-test probability. Health systems wishing to deploy such a tool should consider training ML models with additional data. Adding other techniques like natural language processing, along with ML, to use other clinical information (like chart notes) might improve prediction performance.

Safe opioid prescribing: a prognostic machine learning approach to predicting 30-day risk after an opioid dispensation in Alberta, Canada.

OBJECTIVE: To develop machine learning models employing administrative health data that can estimate risk of adverse outcomes within 30 days of an opioid dispensation for use by health departments or prescription monitoring programmes. DESIGN, SETTING AND PARTICIPANTS: This prognostic study was conducted in Alberta, Canada between 2017 and 2018. Participants included all patients 18 years of age and older who received at least one opioid dispensation. Pregnant and cancer patients were excluded. EXPOSURE: Each opioid dispensation served as an exposure. MAIN OUTCOMES/MEASURES: Opioid-related adverse outcomes were identified from linked administrative health data. Machine learning algorithms were trained using 2017 data to predict risk of hospitalisation, emergency department visit and mortality within 30 days of an opioid dispensation. Two validation sets, using 2017 and 2018 data, were used to evaluate model performance. Model discrimination and calibration performance were assessed for all patients and those at higher risk. Machine learning discrimination was compared with current opioid guidelines. RESULTS: Participants in the 2017 training set (n=275 150) and validation set (n=117 829) had similar baseline characteristics. In the 2017 validation set, c-statistics for the XGBoost, logistic regression and neural network classifiers were 0.87, 0.87 and 0.80, respectively. In the 2018 validation set (n=393 023), the corresponding c-statistics were 0.88, 0.88 and 0.82. C-statistics from the Canadian guidelines ranged from 0.54 to 0.69 while the US guidelines ranged from 0.50 to 0.62. The top five percentile of predicted risk for the XGBoost and logistic regression classifiers captured 42% of all events and translated into post-test probabilities of 13.38% and 13.45%, respectively, up from the pretest probability of 1.6%. CONCLUSION: Machine learning classifiers, especially incorporating hospitalisation/physician claims data, have better predictive performance compared with guideline or prescription history only approaches when predicting 30-day risk of adverse outcomes. Prescription monitoring programmes and health departments with access to administrative data can use machine learning classifiers to effectively identify those at higher risk compared with current guideline-based approaches.