Multicenter validation of a machine-learning algorithm for 48-h all-cause mortality prediction.

In order to evaluate mortality predictions based on boosted trees, this retrospective study uses electronic medical record data from three academic health centers for inpatients 18 years or older with at least one observation of each vital sign. Predictions were made 12, 24, and 48 hours before death. Models fit to training data from each institution were evaluated using hold-out test data from the same institution, and from the other institutions. Gradient-boosted trees (GBT) were compared to regularized logistic regression (LR) predictions, support vector machine (SVM) predictions, quick Sepsis-Related Organ Failure Assessment (qSOFA), and Modified Early Warning Score (MEWS) using area under the receiver operating characteristic curve (AUROC). For training and testing GBT on data from the same institution, the average AUROCs were 0.96, 0.95, and 0.94 across institutional test sets for 12-, 24-, and 48-hour predictions, respectively. When trained and tested on data from different hospitals, GBT AUROCs achieved up to 0.98, 0.96, and 0.96, for 12-, 24-, and 48-hour predictions, respectively. Average AUROC for 48-hour predictions for LR, SVM, MEWS, and qSOFA were 0.85, 0.79, 0.86 and 0.82, respectively. GBT predictions may help identify patients who would benefit from increased clinical care.

Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs.

OBJECTIVE: Sepsis remains a costly and prevalent syndrome in hospitals; however, machine learning systems can increase timely sepsis detection using electronic health records. This study validates a gradient boosted ensemble machine learning tool for sepsis detection and prediction, and compares its performance to existing methods. MATERIALS AND METHODS: Retrospective data was drawn from databases at the University of California, San Francisco (UCSF) Medical Center and the Beth Israel Deaconess Medical Center (BIDMC). Adult patient encounters without sepsis on admission, and with at least one recording of each of six vital signs (SpO2, heart rate, respiratory rate, temperature, systolic and diastolic blood pressure) were included. We compared the performance of the machine learning algorithm (MLA) to that of commonly used scoring systems. Area under the receiver operating characteristic (AUROC) curve was our primary measure of accuracy. MLA performance was measured at sepsis onset, and at 24 and 48 h prior to sepsis onset. RESULTS: The MLA achieved an AUROC of 0.88, 0.84, and 0.83 for sepsis onset and 24 and 48 h prior to onset, respectively. These values were superior to those of SIRS (0.66), MEWS (0.61), SOFA (0.72), and qSOFA (0.60) at time of onset. When trained on UCSF data and tested on BIDMC data, sepsis onset AUROC was 0.89. DISCUSSION AND CONCLUSION: The MLA predicts sepsis up to 48 h in advance and identifies sepsis onset more accurately than commonly used tools, maintaining high performance for sepsis detection when trained and tested on separate datasets.

Prediction of Acute Kidney Injury With a Machine Learning Algorithm Using Electronic Health Record Data.

BACKGROUND: A major problem in treating acute kidney injury (AKI) is that clinical criteria for recognition are markers of established kidney damage or impaired function; treatment before such damage manifests is desirable. Clinicians could intervene during what may be a crucial stage for preventing permanent kidney injury if patients with incipient AKI and those at high risk of developing AKI could be identified. OBJECTIVE: In this study, we evaluate a machine learning algorithm for early detection and prediction of AKI. DESIGN: We used a machine learning technique, boosted ensembles of decision trees, to train an AKI prediction tool on retrospective data taken from more than 300 000 inpatient encounters. SETTING: Data were collected from inpatient wards at Stanford Medical Center and intensive care unit patients at Beth Israel Deaconess Medical Center. PATIENTS: Patients older than the age of 18 whose hospital stays lasted between 5 and 1000 hours and who had at least one documented measurement of heart rate, respiratory rate, temperature, serum creatinine (SCr), and Glasgow Coma Scale (GCS). MEASUREMENTS: We tested the algorithm's ability to detect AKI at onset and to predict AKI 12, 24, 48, and 72 hours before onset. METHODS: We tested AKI detection and prediction using the National Health Service (NHS) England AKI Algorithm as a gold standard. We additionally tested the algorithm's ability to detect AKI as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) guidelines. We compared the algorithm's 3-fold cross-validation performance to the Sequential Organ Failure Assessment (SOFA) score for AKI identification in terms of area under the receiver operating characteristic (AUROC). RESULTS: The algorithm demonstrated high AUROC for detecting and predicting NHS-defined AKI at all tested time points. The algorithm achieves AUROC of 0.872 (95% confidence interval [CI], 0.867-0.878) for AKI detection at time of onset. For prediction 12 hours before onset, the algorithm achieves an AUROC of 0.800 (95% CI, 0.792-0.809). For 24-hour predictions, the algorithm achieves AUROC of 0.795 (95% CI, 0.785-0.804). For 48-hour and 72-hour predictions, the algorithm achieves AUROC values of 0.761 (95% CI, 0.753-0.768) and 0.728 (95% CI, 0.719-0.737), respectively. LIMITATIONS: Because of the retrospective nature of this study, we cannot draw any conclusions about the impact the algorithm's predictions will have on patient outcomes in a clinical setting. CONCLUSIONS: The results of these experiments suggest that a machine learning-based AKI prediction tool may offer important prognostic capabilities for determining which patients are likely to suffer AKI, potentially allowing clinicians to intervene before kidney damage manifests.

CONTEXTE: Une des principales difficultés liées au traitement de l'insuffisance rénale aiguë (IRA) est le fait que les critères cliniques diagnostiques sont des marqueurs d'une lésion ou d'une dysfonction rénale déjà établie. Il est souhaitable d'intervenir avant une telle issue. En dépistant les patients à risque d'IRA ou atteints d'IRA débutante, les cliniciens seraient en mesure d'intervenir précocement et ainsi prévenir les lésions rénales permanentes. OBJECTIF DE L'ÉTUDE: L'étude visait à évaluer un algorithme d'apprentissage automatique destiné à la prédiction des cas d'IRA et à sa détection précoce. TYPE D'ÉTUDE: Nous avons employé une technique d'apprentissage automatique, soit des ensembles d'arbres décisionnels amplifiés, pour entrainer un outil de prédiction de l'IRA à partir de données rétrospectives provenant de plus de 300 000 consultations auprès de patients hospitalisés. CADRE DE L'ÉTUDE: Les données ont été colligées à partir des dossiers des unités d'hospitalisation du centre médical de l'université Stanford et de l'unité des soins intensifs du centre médical Beth Israel Deaconess. PARTICIPANTS: Ont été inclus dans l'étude tous les patients adultes dont l'hospitalisation avait duré de 5 à 1 000 heures et pour lesquels on disposait d'au moins une mesure parmi les suivantes : pouls, rythme respiratoire, température corporelle, taux de créatinine sérique (SCr) et score de Glasgow. MESURES: Nous avons testé l'efficacité de l'algorithme à détecter l'IRA dès son apparition, et à la prédire 12, 24, 48 et 72 heures avant qu'elle ne se manifeste. MÉTHODOLOGIE: L'algorithme du NHS England a servi de référence pour tester l'efficacité de notre algorithme de prédiction et de détection de l'IRA. Nous avons également testé l'efficacité de notre algorithme à détecter l'IRA telle que définie par les Recommandations de Bonnes Pratiques Cliniques du KDIGO (Kidney Disease: Improving Global Outcomes). Nous avons utilisé la surface sous la courbe ROC (Receiver Operating Characteristic) pour comparer le score SOFA à l'efficacité de validation croisée tripartite de notre algorithme. RÉSULTATS: L'algorithme a démontré une SSROC (surface sous la courbe ROC) élevée pour la détection et la prédiction de l'IRA (telle que définie par le NHS) pour tous les moments testés. En détection de la maladie à son apparition, l'algorithme a obtenu une SSROC de 0,872 (IC 95 % : 0,867-0,878). En prédiction, l'algorithme a obtenu une SSROC de 0,800 (IC 95 % : 0,792-0,809) à 12 heures, de 0,795 à 24 heures (IC 95 % : 0,785-0,804), de 0,761 (IC 95 % : 0,753-0,768) à 48 heures et de 0,728 (IC 95 % : 0,719-0,737) à 72 heures avant l'apparition des premiers symptômes. LIMITES DE L'ÉTUDE: La nature rétrospective de l'étude ne nous permet pas de tirer de conclusions sur les conséquences qu'auront les prédictions de l'algorithme sur les résultats cliniques des patients. CONCLUSION: Les résultats de nos essais laissent supposer qu'un outil de prédiction de l'IRA fondé sur l'apprentissage automatique pourrait offrir d'importantes fonctions pronostiques pour détecter les patients susceptibles de développer une IRA en vue d'une intervention précoce.