External validation of a machine learning classifier to identify unhealthy alcohol use in hospitalized patients.

BACKGROUND AND AIMS: Unhealthy alcohol use (UAU) is one of the leading causes of global morbidity. A machine learning approach to alcohol screening could accelerate best practices when integrated into electronic health record (EHR) systems. This study aimed to validate externally a natural language processing (NLP) classifier developed at an independent medical center. DESIGN: Retrospective cohort study. SETTING: The site for validation was a midwestern United States tertiary-care, urban medical center that has an inpatient structured universal screening model for unhealthy substance use and an active addiction consult service. PARTICIPANTS/CASES: Unplanned admissions of adult patients between October 23, 2017 and December 31, 2019, with EHR documentation of manual alcohol screening were included in the cohort (n = 57 605). MEASUREMENTS: The Alcohol Use Disorders Identification Test (AUDIT) served as the reference standard. AUDIT scores ≥5 for females and ≥8 for males served as cases for UAU. To examine error in manual screening or under-reporting, a post hoc error analysis was conducted, reviewing discordance between the NLP classifier and AUDIT-derived reference. All clinical notes excluding the manual screening and AUDIT documentation from the EHR were included in the NLP analysis. FINDINGS: Using clinical notes from the first 24 hours of each encounter, the NLP classifier demonstrated an area under the receiver operating characteristic curve (AUCROC) and precision-recall area under the curve (PRAUC) of 0.91 (95% CI = 0.89-0.92) and 0.56 (95% CI = 0.53-0.60), respectively. At the optimal cut point of 0.5, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 0.66 (95% CI = 0.62-0.69), 0.98 (95% CI = 0.98-0.98), 0.35 (95% CI = 0.33-0.38), and 1.0 (95% CI = 1.0-1.0), respectively. CONCLUSIONS: External validation of a publicly available alcohol misuse classifier demonstrates adequate sensitivity and specificity for routine clinical use as an automated screening tool for identifying at-risk patients.

Depression Following Operative Treatments for Achilles Ruptures and Ankle Fractures.

BACKGROUND: The sudden and debilitating nature of lower extremity injuries can trigger mood disturbances, including major depressive disorder. METHODS: This prospective study enrolled patients undergoing operative repair of ankle fractures and Achilles ruptures and followed them for 1 year postoperatively. The validated Patient Health Questionnaire (PHQ-9) for depressive symptoms was administered at the preoperative visit and at postoperative weeks 1, 2, 4, 8, 16, 24, 32, 40, and 52. PHQ-9 is scored 0 to 27, with higher values indicating greater depression symptoms. RESULTS: Fifty-eight patients completed 1 year of follow-up. The mean PHQ-9 score was 2.7 (range, 0-20) at the preoperative visit, peaked at postoperative week 1 (4.9; range, 0-16), and reached its low at postoperative week 52 (0.8; range, 0-7). Cumulative incidences of depressive symptoms during the first year following surgery were 51.7% for at least mild depression, 22.4% for at least moderate depression, and 6.9% for severe depression. A history of mental health disorder and the inability to work during the period of postoperative immobilization were independently associated with greater depressive symptoms. CONCLUSION: The majority of patients undergoing operative treatment of Achilles ruptures and ankle fractures develop postoperative symptoms of mild to moderate depression that normalize after several months. Patients with a history of mental health disorder or who cannot work while immobilized postoperatively are at greatest risk. LEVEL OF EVIDENCE: Level II, prospective cohort study.

External validation of an opioid misuse machine learning classifier in hospitalized adult patients.

BACKGROUND: Opioid misuse screening in hospitals is resource-intensive and rarely done. Many hospitalized patients are never offered opioid treatment. An automated approach leveraging routinely captured electronic health record (EHR) data may be easier for hospitals to institute. We previously derived and internally validated an opioid classifier in a separate hospital setting. The aim is to externally validate our previously published and open-source machine-learning classifier at a different hospital for identifying cases of opioid misuse. METHODS: An observational cohort of 56,227 adult hospitalizations was examined between October 2017 and December 2019 during a hospital-wide substance use screening program with manual screening. Manually completed Drug Abuse Screening Test served as the reference standard to validate a convolutional neural network (CNN) classifier with coded word embedding features from the clinical notes of the EHR. The opioid classifier utilized all notes in the EHR and sensitivity analysis was also performed on the first 24 h of notes. Calibration was performed to account for the lower prevalence than in the original cohort. RESULTS: Manual screening for substance misuse was completed in 67.8% (n = 56,227) with 1.1% (n = 628) identified with opioid misuse. The data for external validation included 2,482,900 notes with 67,969 unique clinical concept features. The opioid classifier had an AUC of 0.99 (95% CI 0.99-0.99) across the encounter and 0.98 (95% CI 0.98-0.99) using only the first 24 h of notes. In the calibrated classifier, the sensitivity and positive predictive value were 0.81 (95% CI 0.77-0.84) and 0.72 (95% CI 0.68-0.75). For the first 24 h, they were 0.75 (95% CI 0.71-0.78) and 0.61 (95% CI 0.57-0.64). CONCLUSIONS: Our opioid misuse classifier had good discrimination during external validation. Our model may provide a comprehensive and automated approach to opioid misuse identification that augments current workflows and overcomes manual screening barriers.