Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model.

OBJECTIVES: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN: A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING: Two ICUs in tertiary care centers in The Netherlands. PATIENTS: Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS: In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.

Time trend analysis of long term outcome of patients with haematological malignancies admitted at dutch intensive care units.

A few decades ago, the chances of survival for patients with a haematological malignancy needing Intensive Care Unit (ICU) support were minimal. As a consequence, ICU admission policy was cautious. We hypothesized that the long-term outcome of patients with a haematological malignancy admitted to the ICU has improved in recent years. Furthermore, our objective was to evaluate the predictive value of the Acute Physiology and Chronic Health Evaluation (APACHE) II score. A total of 1095 patients from 5 Dutch university hospitals were included from 2003 until 2015. We studied the prevalence of patients' characteristics over time. By using annual odds ratios, we analysed which patients' characteristics could have had influenced possible trends in time. A approximated mortality rate was compared with the ICU mortality rate, to study the predictive value of the APACHE II score. Overall one-year mortality was 62%. The annual decrease in one-year mortality was 7%, whereas the APACHE II score increased over time. Decreased mortality rates were particularly observed in high-risk patients (acute myeloid leukaemia, old age, low platelet count, bleeding as admission reason and need for mechanical ventilation within 24 h of ICU admission). Furthermore, the APACHE II score overestimates mortality in this patient category.