Development and Validation of a Machine Learning COVID-19 Veteran (COVet) Deterioration Risk Score.

BACKGROUND AND OBJECTIVE: To develop the COVid Veteran (COVet) score for clinical deterioration in Veterans hospitalized with COVID-19 and further validate this model in both Veteran and non-Veteran samples. No such score has been derived and validated while incorporating a Veteran sample. DERIVATION COHORT: Adults (age ≥ 18 yr) hospitalized outside the ICU with a diagnosis of COVID-19 for model development to the Veterans Health Administration (VHA) (n = 80 hospitals). VALIDATION COHORT: External validation occurred in a VHA cohort of 34 hospitals, as well as six non-Veteran health systems for further external validation (n = 21 hospitals) between 2020 and 2023. PREDICTION MODEL: eXtreme Gradient Boosting machine learning methods were used, and performance was assessed using the area under the receiver operating characteristic curve and compared with the National Early Warning Score (NEWS). The primary outcome was transfer to the ICU or death within 24 hours of each new variable observation. Model predictor variables included demographics, vital signs, structured flowsheet data, and laboratory values. RESULTS: A total of 96,908 admissions occurred during the study period, of which 59,897 were in the Veteran sample and 37,011 were in the non-Veteran sample. During external validation in the Veteran sample, the model demonstrated excellent discrimination, with an area under the receiver operating characteristic curve of 0.88. This was significantly higher than NEWS (0.79; p < 0.01). In the non-Veteran sample, the model also demonstrated excellent discrimination (0.86 vs. 0.79 for NEWS; p < 0.01). The top three variables of importance were eosinophil percentage, mean oxygen saturation in the prior 24-hour period, and worst mental status in the prior 24-hour period. CONCLUSIONS: We used machine learning methods to develop and validate a highly accurate early warning score in both Veterans and non-Veterans hospitalized with COVID-19. The model could lead to earlier identification and therapy, which may improve outcomes.

Development and external validation of deep learning clinical prediction models using variable-length time series data.

OBJECTIVES: To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection). MATERIALS AND METHODS: This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC). RESULTS: The study comprised 373 825 admissions for training and 256 128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks. DISCUSSION: When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC. CONCLUSION: The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

Multicenter Development and Prospective Validation of eCARTv5: A Gradient Boosted Machine Learning Early Warning Score.

Rationale: Early detection of clinical deterioration using early warning scores may improve outcomes. However, most implemented scores were developed using logistic regression, only underwent retrospective internal validation, and were not tested in important patient subgroups. Objectives: To develop a gradient boosted machine model (eCARTv5) for identifying clinical deterioration and then validate externally, test prospectively, and evaluate across patient subgroups. Methods: All adult patients hospitalized on the wards in seven hospitals from 2008- 2022 were used to develop eCARTv5, with demographics, vital signs, clinician documentation, and laboratory values utilized to predict intensive care unit transfer or death in the next 24 hours. The model was externally validated retrospectively in 21 hospitals from 2009-2023 and prospectively in 10 hospitals from February to May 2023. eCARTv5 was compared to the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS) using the area under the receiver operating characteristic curve (AUROC). Measurements and Main Results: The development cohort included 901,491 admissions, the retrospective validation cohort included 1,769,461 admissions, and the prospective validation cohort included 46,330 admissions. In retrospective validation, eCART had the highest AUROC (0.835; 95%CI 0.834, 0.835), followed by NEWS (0.766 (95%CI 0.766, 0.767)), and MEWS (0.704 (95%CI 0.703, 0.704)). eCART's performance remained high (AUROC ≥0.80) across a range of patient demographics, clinical conditions, and during prospective validation. Conclusions: We developed eCARTv5, which accurately identifies early clinical deterioration in hospitalized ward patients. Our model performed better than the NEWS and MEWS retrospectively, prospectively, and across a range of subgroups.

Alcohol Withdrawal Severity Measures for Identifying Patients Requiring High-Intensity Care.

OBJECTIVES: Alcohol withdrawal syndrome (AWS) may progress to require high-intensity care. Approaches to identify hospitalized patients with AWS who received higher level of care have not been previously examined. This study aimed to examine the utility of Clinical Institute Withdrawal Assessment Alcohol Revised (CIWA-Ar) for alcohol scale scores and medication doses for alcohol withdrawal management in identifying patients who received high-intensity care. DESIGN: A multicenter observational cohort study of hospitalized adults with alcohol withdrawal. SETTING: University of Chicago Medical Center and University of Wisconsin Hospital. PATIENTS: Inpatient encounters between November 2008 and February 2022 with a CIWA-Ar score greater than 0 and benzodiazepine or barbiturate administered within the first 24 hours. The primary composite outcome was patients who progressed to high-intensity care (intermediate care or ICU). INTERVENTIONS: None. MAIN RESULTS: Among the 8742 patients included in the study, 37.5% (n = 3280) progressed to high-intensity care. The odds ratio for the composite outcome increased above 1.0 when the CIWA-Ar score was 24. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at this threshold were 0.12 (95% CI, 0.11-0.13), 0.95 (95% CI, 0.94-0.95), 0.58 (95% CI, 0.54-0.61), and 0.64 (95% CI, 0.63-0.65), respectively. The OR increased above 1.0 at a 24-hour lorazepam milligram equivalent dose cutoff of 15 mg. The sensitivity, specificity, PPV, and NPV at this threshold were 0.16 (95% CI, 0.14-0.17), 0.96 (95% CI, 0.95-0.96), 0.68 (95% CI, 0.65-0.72), and 0.65 (95% CI, 0.64-0.66), respectively. CONCLUSIONS: Neither CIWA-Ar scores nor medication dose cutoff points were effective measures for identifying patients with alcohol withdrawal who received high-intensity care. Research studies for examining outcomes in patients who deteriorate with AWS will require better methods for cohort identification.

A novel structured debriefing program for consensus determinations of in-hospital cardiac arrest predictability and preventability.

AIM: Hospital rapid response systems aim to stop preventable cardiac arrests, but defining preventability is a challenge. We developed a multidisciplinary consensus-based process to determine in-hospital cardiac arrest (IHCA) preventability based on objective measures. METHODS: We developed an interdisciplinary ward IHCA debriefing program at an urban quaternary-care academic hospital. This group systematically reviewed all IHCAs weekly, reaching consensus determinations of the IHCA's cause and preventability across three mutually exclusive categories: 1) unpredictable (no evidence of physiologic instability < 1 h prior to and within 24 h of the arrest), 2) predictable but unpreventable (meeting physiologic instability criteria in the setting of either a poor baseline prognosis or a documented goals of care conversation) or 3) potentially preventable (remaining cases). RESULTS: Of 544 arrests between 09/2015 and 11/2023, 339 (61%) were deemed predictable by consensus, with 235 (42% of all IHCAs) considered potentially preventable. Potentially preventable arrests disproportionately occurred on nights and weekends (70% vs 55%, p = 0.002) and were more frequently respiratory than cardiac in etiology (33% vs 15%, p < 0.001). Despite similar rates of ROSC across groups (67-70%), survival to discharge was highest in arrests deemed unpredictable (31%), followed by potentially preventable (21%), and then those deemed predictable but unpreventable which had the lowest survival rate (16%, p = 0.007). CONCLUSIONS: Our IHCA debriefing procedures are a feasible and sustainable means of determining the predictability and potential preventability of ward cardiac arrests. This approach may be useful for improving quality benchmarks and care processes around pre-arrest clinical activities.

Causes, Diagnostic Testing, and Treatments Related to Clinical Deterioration Events among High-Risk Ward Patients.

OBJECTIVE: Timely intervention for clinically deteriorating ward patients requires that care teams accurately diagnose and treat their underlying medical conditions. However, the most common diagnoses leading to deterioration and the relevant therapies provided are poorly characterized. Therefore, we aimed to determine the diagnoses responsible for clinical deterioration, the relevant diagnostic tests ordered, and the treatments administered among high-risk ward patients using manual chart review. DESIGN: Multicenter retrospective observational study. SETTING: Inpatient medical-surgical wards at four health systems from 2006-2020 PATIENTS: Randomly selected patients (1,000 from each health system) with clinical deterioration, defined by reaching the 95th percentile of a validated early warning score, electronic Cardiac Arrest Risk Triage (eCART), were included. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Clinical deterioration was confirmed by a trained reviewer or marked as a false alarm if no deterioration occurred for each patient. For true deterioration events, the condition causing deterioration, relevant diagnostic tests ordered, and treatments provided were collected. Of the 4,000 included patients, 2,484 (62%) had clinical deterioration confirmed by chart review. Sepsis was the most common cause of deterioration (41%; n=1,021), followed by arrhythmia (19%; n=473), while liver failure had the highest in-hospital mortality (41%). The most common diagnostic tests ordered were complete blood counts (47% of events), followed by chest x-rays (42%), and cultures (40%), while the most common medication orders were antimicrobials (46%), followed by fluid boluses (34%), and antiarrhythmics (19%). CONCLUSIONS: We found that sepsis was the most common cause of deterioration, while liver failure had the highest mortality. Complete blood counts and chest x-rays were the most common diagnostic tests ordered, and antimicrobials and fluid boluses were the most common medication interventions. These results provide important insights for clinical decision-making at the bedside, training of rapid response teams, and the development of institutional treatment pathways for clinical deterioration. KEY POINTS: Question: What are the most common diagnoses, diagnostic test orders, and treatments for ward patients experiencing clinical deterioration? Findings: In manual chart review of 2,484 encounters with deterioration across four health systems, we found that sepsis was the most common cause of clinical deterioration, followed by arrythmias, while liver failure had the highest mortality. Complete blood counts and chest x-rays were the most common diagnostic test orders, while antimicrobials and fluid boluses were the most common treatments. Meaning: Our results provide new insights into clinical deterioration events, which can inform institutional treatment pathways, rapid response team training, and patient care.

Predicting transfers to intensive care in children using CEWT and other early warning systems.

Background and Objective: The Children's Early Warning Tool (CEWT), developed in Australia, is widely used in many countries to monitor the risk of deterioration in hospitalized children. Our objective was to compare CEWT prediction performance against a version of the Bedside Pediatric Early Warning Score (Bedside PEWS), Between the Flags (BTF), and the pediatric Calculated Assessment of Risk and Triage (pCART). Methods: We conducted a retrospective observational study of all patient admissions to the Comer Children's Hospital at the University of Chicago between 2009-2019. We compared performance for predicting the primary outcome of a direct ward-to-intensive care unit (ICU) transfer within the next 12 h using the area under the receiver operating characteristic curve (AUC). Alert rates at various score thresholds were also compared. Results: Of 50,815 ward admissions, 1,874 (3.7%) experienced the primary outcome. Among patients in Cohort 1 (years 2009-2017, on which the machine learning-based pCART was trained), CEWT performed slightly worse than Bedside PEWS but better than BTF (CEWT AUC 0.74 vs. Bedside PEWS 0.76, P < 0.001; vs. BTF 0.66, P < 0.001), while pCART performed best for patients in Cohort 2 (years 2018-2019, pCART AUC 0.84 vs. CEWT AUC 0.79, P < 0.001; vs. BTF AUC 0.67, P < 0.001; vs. Bedside PEWS 0.80, P < 0.001). Sensitivity, specificity, and positive predictive values varied across all four tools at the examined thresholds for alerts. Conclusion: CEWT has good discrimination for predicting which patients will likely be transferred to the ICU, while pCART performed the best.

Society of Critical Care Medicine Guidelines on Recognizing and Responding to Clinical Deterioration Outside the ICU: 2023.

RATIONALE: Clinical deterioration of patients hospitalized outside the ICU is a source of potentially reversible morbidity and mortality. To address this, some acute care hospitals have implemented systems aimed at detecting and responding to such patients. OBJECTIVES: To provide evidence-based recommendations for hospital clinicians and administrators to optimize recognition and response to clinical deterioration in non-ICU patients. PANEL DESIGN: The 25-member panel included representatives from medicine, nursing, respiratory therapy, pharmacy, patient/family partners, and clinician-methodologists with expertise in developing evidence-based Clinical Practice Guidelines. METHODS: We generated actionable questions using the Population, Intervention, Control, and Outcomes (PICO) format and performed a systematic review of the literature to identify and synthesize the best available evidence. We used the Grading of Recommendations Assessment, Development, and Evaluation Approach to determine certainty in the evidence and to formulate recommendations and good practice statements (GPSs). RESULTS: The panel issued 10 statements on recognizing and responding to non-ICU patients with critical illness. Healthcare personnel and institutions should ensure that all vital sign acquisition is timely and accurate (GPS). We make no recommendation on the use of continuous vital sign monitoring among unselected patients. We suggest focused education for bedside clinicians in signs of clinical deterioration, and we also suggest that patient/family/care partners' concerns be included in decisions to obtain additional opinions and help (both conditional recommendations). We recommend hospital-wide deployment of a rapid response team or medical emergency team (RRT/MET) with explicit activation criteria (strong recommendation). We make no recommendation about RRT/MET professional composition or inclusion of palliative care members on the responding team but suggest that the skill set of responders should include eliciting patients' goals of care (conditional recommendation). Finally, quality improvement processes should be part of a rapid response system. CONCLUSIONS: The panel provided guidance to inform clinicians and administrators on effective processes to improve the care of patients at-risk for developing critical illness outside the ICU.

Executive Summary: Society of Critical Care Medicine Guidelines on Recognizing and Responding to Clinical Deterioration Outside the ICU.

RATIONALE: Clinical deterioration of patients hospitalized outside the ICU is a source of potentially reversible morbidity and mortality. To address this, some acute care facilities have implemented systems aimed at detecting and responding to such patients. OBJECTIVES: To provide evidence-based recommendations for hospital clinicians and administrators to optimize recognition and response to clinical deterioration in non-ICU patients. PANEL DESIGN: The 25-member panel included representatives from medicine, nursing, respiratory therapy, pharmacy, patient/family partners, and clinician-methodologists with expertise in developing evidence-based clinical practice guidelines. METHODS: We generated actionable questions using the Population, Intervention, Control, and Outcomes format and performed a systematic review of the literature to identify and synthesize the best available evidence. We used the Grading of Recommendations Assessment, Development, and Evaluation approach to determine certainty in the evidence and to formulate recommendations and good practice statements (GPSs). RESULTS: The panel issued 10 statements on recognizing and responding to non-ICU patients with critical illness. Healthcare personnel and institutions should ensure that all vital sign acquisition is timely and accurate (GPS). We make no recommendation on the use of continuous vital sign monitoring among "unselected" patients due to the absence of data regarding the benefit and the potential harms of false positive alarms, the risk of alarm fatigue, and cost. We suggest focused education for bedside clinicians in signs of clinical deterioration, and we also suggest that patient/family/care partners' concerns be included in decisions to obtain additional opinions and help (both conditional recommendations). We recommend hospital-wide deployment of a rapid response team or medical emergency team (RRT/MET) with explicit activation criteria (strong recommendation). We make no recommendation about RRT/MET professional composition or inclusion of palliative care members on the responding team but suggest that the skill set of responders should include eliciting patients' goals of care (conditional recommendation). Finally, quality improvement processes should be part of a rapid response system (GPS). CONCLUSIONS: The panel provided guidance to inform clinicians and administrators on effective processes to improve the care of patients at-risk for developing critical illness outside the ICU.

Temperature Trajectory Subphenotypes in Oncology Patients with Neutropenia and Suspected Infection.

Rationale: Despite etiologic and severity heterogeneity in neutropenic sepsis, management is often uniform. Understanding host response clinical subphenotypes might inform treatment strategies for neutropenic sepsis. Objectives: In this retrospective two-hospital study, we analyzed whether temperature trajectory modeling could identify distinct, clinically relevant subphenotypes among oncology patients with neutropenia and suspected infection. Methods: Among adult oncologic admissions with neutropenia and blood cultures within 24 hours, a previously validated model classified patients' initial 72-hour temperature trajectories into one of four subphenotypes. We analyzed subphenotypes' independent relationships with hospital mortality and bloodstream infection using multivariable models. Measurements and Main Results: Patients (primary cohort n = 1,145, validation cohort n = 6,564) fit into one of four temperature subphenotypes. "Hyperthermic slow resolvers" (pooled n = 1,140 [14.8%], mortality n = 104 [9.1%]) and "hypothermic" encounters (n = 1,612 [20.9%], mortality n = 138 [8.6%]) had higher mortality than "hyperthermic fast resolvers" (n = 1,314 [17.0%], mortality n = 47 [3.6%]) and "normothermic" (n = 3,643 [47.3%], mortality n = 196 [5.4%]) encounters (P < 0.001). Bloodstream infections were more common among hyperthermic slow resolvers (n = 248 [21.8%]) and hyperthermic fast resolvers (n = 240 [18.3%]) than among hypothermic (n = 188 [11.7%]) or normothermic (n = 418 [11.5%]) encounters (P < 0.001). Adjusted for confounders, hyperthermic slow resolvers had increased adjusted odds for mortality (primary cohort odds ratio, 1.91 [P = 0.03]; validation cohort odds ratio, 2.19 [P < 0.001]) and bloodstream infection (primary odds ratio, 1.54 [P = 0.04]; validation cohort odds ratio, 2.15 [P < 0.001]). Conclusions: Temperature trajectory subphenotypes were independently associated with important outcomes among hospitalized patients with neutropenia in two independent cohorts.

Identifying infected patients using semi-supervised and transfer learning.

OBJECTIVES: Early identification of infection improves outcomes, but developing models for early identification requires determining infection status with manual chart review, limiting sample size. Therefore, we aimed to compare semi-supervised and transfer learning algorithms with algorithms based solely on manual chart review for identifying infection in hospitalized patients. MATERIALS AND METHODS: This multicenter retrospective study of admissions to 6 hospitals included "gold-standard" labels of infection from manual chart review and "silver-standard" labels from nonchart-reviewed patients using the Sepsis-3 infection criteria based on antibiotic and culture orders. "Gold-standard" labeled admissions were randomly allocated to training (70%) and testing (30%) datasets. Using patient characteristics, vital signs, and laboratory data from the first 24 hours of admission, we derived deep learning and non-deep learning models using transfer learning and semi-supervised methods. Performance was compared in the gold-standard test set using discrimination and calibration metrics. RESULTS: The study comprised 432 965 admissions, of which 2724 underwent chart review. In the test set, deep learning and non-deep learning approaches had similar discrimination (area under the receiver operating characteristic curve of 0.82). Semi-supervised and transfer learning approaches did not improve discrimination over models fit using only silver- or gold-standard data. Transfer learning had the best calibration (unreliability index P value: .997, Brier score: 0.173), followed by self-learning gradient boosted machine (P value: .67, Brier score: 0.170). DISCUSSION: Deep learning and non-deep learning models performed similarly for identifying infection, as did models developed using Sepsis-3 and manual chart review labels. CONCLUSION: In a multicenter study of almost 3000 chart-reviewed patients, semi-supervised and transfer learning models showed similar performance for model discrimination as baseline XGBoost, while transfer learning improved calibration.

Predicting neurological outcomes after in-hospital cardiac arrests for patients with Coronavirus Disease 2019.

BACKGROUND: Machine learning models are more accurate than standard tools for predicting neurological outcomes in patients resuscitated after cardiac arrest. However, their accuracy in patients with Coronavirus Disease 2019 (COVID-19) is unknown. Therefore, we compared their performance in a cohort of cardiac arrest patients with COVID-19. METHODS: We conducted a retrospective analysis of resuscitation survivors in the Get With The Guidelines®-Resuscitation (GWTG-R) COVID-19 registry between February 2020 and May 2021. The primary outcome was a favorable neurological outcome, indicated by a discharge Cerebral Performance Category score ≤ 2. Pre- and peri-arrest variables were used as predictors. We applied our published logistic regression, neural network, and gradient boosted machine models developed in patients without COVID-19 to the COVID-19 cohort. We also updated the neural network model using transfer learning. Performance was compared between models and the Cardiac Arrest Survival Post-Resuscitation In-Hospital (CASPRI) score. RESULTS: Among the 4,125 patients with COVID-19 included in the analysis, 484 (12 %) patients survived with favorable neurological outcomes. The gradient boosted machine, trained on non-COVID-19 patients was the best performing model for predicting neurological outcomes in COVID-19 patients, significantly better than the CASPRI score (c-statistic: 0.75 vs 0.67, P < 0.001). While calibration improved for the neural network with transfer learning, it did not surpass the gradient boosted machine in terms of discrimination. CONCLUSION: Our gradient boosted machine model developed in non-COVID patients had high discrimination and adequate calibration in COVID-19 resuscitation survivors and may provide clinicians with important information for these patients.

Development and External Validation of a Machine Learning Model for Prediction of Potential Transfer to the PICU.

OBJECTIVES: Unrecognized clinical deterioration during illness requiring hospitalization is associated with high risk of mortality and long-term morbidity among children. Our objective was to develop and externally validate machine learning algorithms using electronic health records for identifying ICU transfer within 12 hours indicative of a child's condition. DESIGN: Observational cohort study. SETTING: Two urban, tertiary-care, academic hospitals (sites 1 and 2). PATIENTS: Pediatric inpatients (age <18 yr). INTERVENTIONS: None. MEASUREMENT AND MAIN RESULTS: Our primary outcome was direct ward to ICU transfer. Using age, vital signs, and laboratory results, we derived logistic regression with regularization, restricted cubic spline regression, random forest, and gradient boosted machine learning models. Among 50,830 admissions at site 1 and 88,970 admissions at site 2, 1,993 (3.92%) and 2,317 (2.60%) experienced the primary outcome, respectively. Site 1 data were split longitudinally into derivation (2009-2017) and validation (2018-2019), whereas site 2 constituted the external test cohort. Across both sites, the gradient boosted machine was the most accurate model and outperformed a modified version of the Bedside Pediatric Early Warning Score that only used physiologic variables in terms of discrimination ( C -statistic site 1: 0.84 vs 0.71, p < 0.001; site 2: 0.80 vs 0.74, p < 0.001), sensitivity, specificity, and number needed to alert. CONCLUSIONS: We developed and externally validated a novel machine learning model that identifies ICU transfers in hospitalized children more accurately than current tools. Our model enables early detection of children at risk for deterioration, thereby creating opportunities for intervention and improvement in outcomes.

The Impact of a Machine Learning Early Warning Score on Hospital Mortality: A Multicenter Clinical Intervention Trial.

OBJECTIVES: To determine the impact of a machine learning early warning risk score, electronic Cardiac Arrest Risk Triage (eCART), on mortality for elevated-risk adult inpatients. DESIGN: A pragmatic pre- and post-intervention study conducted over the same 10-month period in 2 consecutive years. SETTING: Four-hospital community-academic health system. PATIENTS: All adult patients admitted to a medical-surgical ward. INTERVENTIONS: During the baseline period, clinicians were blinded to eCART scores. During the intervention period, scores were presented to providers. Scores greater than or equal to 95th percentile were designated high risk prompting a physician assessment for ICU admission. Scores between the 89th and 95th percentiles were designated intermediate risk, triggering a nurse-directed workflow that included measuring vital signs every 2 hours and contacting a physician to review the treatment plan. MEASUREMENTS AND MAIN RESULTS: The primary outcome was all-cause inhospital mortality. Secondary measures included vital sign assessment within 2 hours, ICU transfer rate, and time to ICU transfer. A total of 60,261 patients were admitted during the study period, of which 6,681 (11.1%) met inclusion criteria (baseline period n = 3,191, intervention period n = 3,490). The intervention period was associated with a significant decrease in hospital mortality for the main cohort (8.8% vs 13.9%; p < 0.0001; adjusted odds ratio [OR], 0.60 [95% CI, 0.52-0.71]). A significant decrease in mortality was also seen for the average-risk cohort not subject to the intervention (0.49% vs 0.26%; p < 0.05; adjusted OR, 0.53 [95% CI, 0.41-0.74]). In subgroup analysis, the benefit was seen in both high- (17.9% vs 23.9%; p = 0.001) and intermediate-risk (2.0% vs 4.0 %; p = 0.005) patients. The intervention period was also associated with a significant increase in ICU transfers, decrease in time to ICU transfer, and increase in vital sign reassessment within 2 hours. CONCLUSIONS: Implementation of a machine learning early warning score-driven protocol was associated with reduced inhospital mortality, likely driven by earlier and more frequent ICU transfer.

Comparison of early warning scores for predicting clinical deterioration and infection in obstetric patients.

BACKGROUND: Early warning scores are designed to identify hospitalized patients who are at high risk of clinical deterioration. Although many general scores have been developed for the medical-surgical wards, specific scores have also been developed for obstetric patients due to differences in normal vital sign ranges and potential complications in this unique population. The comparative performance of general and obstetric early warning scores for predicting deterioration and infection on the maternal wards is not known. METHODS: This was an observational cohort study at the University of Chicago that included patients hospitalized on obstetric wards from November 2008 to December 2018. Obstetric scores (modified early obstetric warning system (MEOWS), maternal early warning criteria (MEWC), and maternal early warning trigger (MEWT)), paper-based general scores (Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS), and a general score developed using machine learning (electronic Cardiac Arrest Risk Triage (eCART) score) were compared using the area under the receiver operating characteristic score (AUC) for predicting ward to intensive care unit (ICU) transfer and/or death and new infection. RESULTS: A total of 19,611 patients were included, with 43 (0.2%) experiencing deterioration (ICU transfer and/or death) and 88 (0.4%) experiencing an infection. eCART had the highest discrimination for deterioration (p < 0.05 for all comparisons), with an AUC of 0.86, followed by MEOWS (0.74), NEWS (0.72), MEWC (0.71), MEWS (0.70), and MEWT (0.65). MEWC, MEWT, and MEOWS had higher accuracy than MEWS and NEWS but lower accuracy than eCART at specific cut-off thresholds. For predicting infection, eCART (AUC 0.77) had the highest discrimination. CONCLUSIONS: Within the limitations of our retrospective study, eCART had the highest accuracy for predicting deterioration and infection in our ante- and postpartum patient population. Maternal early warning scores were more accurate than MEWS and NEWS. While institutional choice of an early warning system is complex, our results have important implications for the risk stratification of maternal ward patients, especially since the low prevalence of events means that small improvements in accuracy can lead to large decreases in false alarms.

Comparison of Machine Learning Methods for Predicting Outcomes After In-Hospital Cardiac Arrest.

OBJECTIVES: Prognostication of neurologic status among survivors of in-hospital cardiac arrests remains a challenging task for physicians. Although models such as the Cardiac Arrest Survival Post-Resuscitation In-hospital score are useful for predicting neurologic outcomes, they were developed using traditional statistical techniques. In this study, we derive and compare the performance of several machine learning models with each other and with the Cardiac Arrest Survival Post-Resuscitation In-hospital score for predicting the likelihood of favorable neurologic outcomes among survivors of resuscitation. DESIGN: Analysis of the Get With The Guidelines-Resuscitation registry. SETTING: Seven-hundred fifty-five hospitals participating in Get With The Guidelines-Resuscitation from January 1, 2001, to January 28, 2017. PATIENTS: Adult in-hospital cardiac arrest survivors. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Of 117,674 patients in our cohort, 28,409 (24%) had a favorable neurologic outcome, as defined as survival with a Cerebral Performance Category score of less than or equal to 2 at discharge. Using patient characteristics, pre-existing conditions, prearrest interventions, and periarrest variables, we constructed logistic regression, support vector machines, random forests, gradient boosted machines, and neural network machine learning models to predict favorable neurologic outcome. Events prior to October 20, 2009, were used for model derivation, and all subsequent events were used for validation. The gradient boosted machine predicted favorable neurologic status at discharge significantly better than the Cardiac Arrest Survival Post-Resuscitation In-hospital score (C-statistic: 0.81 vs 0.73; p < 0.001) and outperformed all other machine learning models in terms of discrimination, calibration, and accuracy measures. Variables that were consistently most important for prediction across all models were duration of arrest, initial cardiac arrest rhythm, admission Cerebral Performance Category score, and age. CONCLUSIONS: The gradient boosted machine algorithm was the most accurate for predicting favorable neurologic outcomes in in-hospital cardiac arrest survivors. Our results highlight the utility of machine learning for predicting neurologic outcomes in resuscitated patients.