Development and Validation of Unplanned Extubation Prediction Models Using Intensive Care Unit Data: Retrospective, Comparative, Machine Learning Study.

BACKGROUND: Patient safety in the intensive care unit (ICU) is one of the most critical issues, and unplanned extubation (UE) is considered the most adverse event for patient safety. Prevention and early detection of such an event is an essential but difficult component of quality care. OBJECTIVE: This study aimed to develop and validate prediction models for UE in ICU patients using machine learning. METHODS: This study was conducted in an academic tertiary hospital in Seoul, Republic of Korea. The hospital had approximately 2000 inpatient beds and 120 ICU beds. As of January 2019, the hospital had approximately 9000 outpatients on a daily basis. The number of annual ICU admissions was approximately 10,000. We conducted a retrospective study between January 1, 2010, and December 31, 2018. A total of 6914 extubation cases were included. We developed a UE prediction model using machine learning algorithms, which included random forest (RF), logistic regression (LR), artificial neural network (ANN), and support vector machine (SVM). For evaluating the model's performance, we used the area under the receiver operating characteristic curve (AUROC). The sensitivity, specificity, positive predictive value, negative predictive value, and F1 score were also determined for each model. For performance evaluation, we also used a calibration curve, the Brier score, and the integrated calibration index (ICI) to compare different models. The potential clinical usefulness of the best model at the best threshold was assessed through a net benefit approach using a decision curve. RESULTS: Among the 6914 extubation cases, 248 underwent UE. In the UE group, there were more males than females, higher use of physical restraints, and fewer surgeries. The incidence of UE was higher during the night shift as compared to the planned extubation group. The rate of reintubation within 24 hours and hospital mortality were higher in the UE group. The UE prediction algorithm was developed, and the AUROC for RF was 0.787, for LR was 0.762, for ANN was 0.763, and for SVM was 0.740. CONCLUSIONS: We successfully developed and validated machine learning-based prediction models to predict UE in ICU patients using electronic health record data. The best AUROC was 0.787 and the sensitivity was 0.949, which was obtained using the RF algorithm. The RF model was well-calibrated, and the Brier score and ICI were 0.129 and 0.048, respectively. The proposed prediction model uses widely available variables to limit the additional workload on the clinician. Further, this evaluation suggests that the model holds potential for clinical usefulness.

Development, validation, and usability evaluation of machine learning algorithms for predicting personalized red blood cell demand among thoracic surgery patients.

INTRODUCTION: Preparing appropriate red blood cells (RBCs) before surgery is crucial for improving both the efficacy of perioperative workflow and patient safety. In particular, thoracic surgery (TS) is a procedure that requires massive transfusion with high variability for each patient. Hence, the precise prediction of RBC requirements for individual patients is becoming increasingly important. This study aimed to 1) develop and validate a machine learning algorithm for personalized RBC predictions for TS patients and 2) assess the usability of a clinical decision support system (CDSS) integrating this artificial intelligence model. METHODS: Adult patients who underwent TS between January 2016 and October 2021 were included in this study. Multiple models were developed by employing both traditional statistical- and machine-learning approaches. The primary outcome evaluated the model's performance in predicting RBC requirements through root mean square error and adjusted R2. Surgeons and informaticians determined the precision MSBOS-Thoracic Surgery (pMSBOS-TS) algorithm through a consensus process. The usability of the pMSBOS-TS was assessed using the System Usability Scale (SUS) survey with 60 clinicians. RESULTS: We identified 7,843 cases (6,200 for training and 1,643 for test sets) of TSs. Among the models with variable performance indices, the extreme gradient boosting model was selected as the pMSBOS-TS algorithm. The pMSBOS-TS model showed statistically significant lower root mean square error (mean: 3.203 and 95% confidence interval [CI]: 3.186-3.220) compared to the calculated Maximum Surgical Blood Ordering Schedule (MSBOS) and a higher adjusted R2 (mean: 0.399 and 95% CI: 0.395-0.403) compared to the calculated MSBOS, while requiring approximately 200 fewer packs for RBC preparation compared to the calculated MSBOS. The SUS score of the pMSBOS-TS CDSS was 72.5 points, indicating good acceptability. CONCLUSIONS: We successfully developed the pMSBOS-TS capable of predicting personalized RBC transfusion requirements for perioperative patients undergoing TS.

The potential for drug incompatibility and its drivers - A hospital wide retrospective descriptive study.

OBJECTIVE: Drug incompatibility, a significant subset of medication errors, threaten patient safety during the medication administration phase. Despite the undeniably high prevalence of drug incompatibility, it is currently poorly understood because previous studies are focused predominantly on intensive care unit (ICU) settings. To enhance patient safety, it is crucial to expand our understanding of this issue from a comprehensive viewpoint. This study aims to investigate the prevalence and mechanism of drug incompatibility by analysing hospital-wide prescription and administration data. METHODS: This retrospective cross-sectional study, conducted at a tertiary academic hospital, included data extracted from the clinical data warehouse of the study institution on patients admitted between January 1, 2021, and May 31, 2021. Potential contacts in drug pairs (PCs) were identified using the study site clinical workflow. Drug incompatibility for each PC was determined by using a commercial drug incompatibility database, the Trissel's™ 2 Clinical Pharmaceutics Database (Trissel's 2 database). Drivers of drug incompatibility were identified, based on a descriptive analysis, after which, multivariate logistic regression was conducted to assess the risk factors for experiencing one or more drug incompatibilities during admission. RESULTS: Among 30,359 patients (representing 40,061 hospitalisations), 24,270 patients (32,912 hospitalisations) with 764,501 drug prescriptions (1,001,685 IV administrations) were analysed, after checking for eligibility. Based on the rule for determining PCs, 5,813,794 cases of PCs were identified. Among these, 25,108 (0.4 %) cases were incompatible PCs: 391 (1.6 %) PCs occurred during the prescription process and 24,717 (98.4 %) PCs during the administration process. By classifying these results, we identified the following drivers contributing to drug incompatibility: incorrect order factor; incorrect administration factor; and lack of related research. In multivariate analysis, the risk of encountering incompatible PCs was higher for patients who were male, older, with longer lengths of stay, with higher comorbidity, and admitted to medical ICUs. CONCLUSIONS: We comprehensively described the current state of drug incompatibility by analysing hospital-wide drug prescription and administration data. The results showed that drug incompatibility frequently occurs in clinical settings.

Healthcare Professionals' Expectations of Medical Artificial Intelligence and Strategies for its Clinical Implementation: A Qualitative Study.

OBJECTIVES: Although medical artificial intelligence (AI) systems that assist healthcare professionals in critical care settings are expected to improve healthcare, skepticism exists regarding whether their potential has been fully actualized. Therefore, we aimed to conduct a qualitative study with physicians and nurses to understand their needs, expectations, and concerns regarding medical AI; explore their expected responses to recommendations by medical AI that contradicted their judgments; and derive strategies to implement medical AI in practice successfully. METHODS: Semi-structured interviews were conducted with 15 healthcare professionals working in the emergency room and intensive care unit in a tertiary teaching hospital in Seoul. The data were interpreted using summative content analysis. In total, 26 medical AI topics were extracted from the interviews. Eight were related to treatment recommendation, seven were related to diagnosis prediction, and seven were related to process improvement. RESULTS: While the participants expressed expectations that medical AI could enhance their patients' outcomes, increase work efficiency, and reduce hospital operating costs, they also mentioned concerns regarding distortions in the workflow, deskilling, alert fatigue, and unsophisticated algorithms. If medical AI decisions contradicted their judgment, most participants would consult other medical staff and thereafter reconsider their initial judgment. CONCLUSIONS: Healthcare professionals wanted to use medical AI in practice and emphasized that artificial intelligence systems should be trustworthy from the standpoint of healthcare professionals. They also highlighted the importance of alert fatigue management and the integration of AI systems into the workflow.

Institutionalizing Health Technology Assessment and Priority Setting in South Korea's Universal Health Coverage Journey.

This study charts the chronological developments of the three institutions that were established in South Korea for priority setting in health. In 2007, the Evidence-based Medicine Team and the Center for New Health Technology Assessment (CnHTA) were established and nested in the Health Insurance Review and Assessment Service (HIRA). In December 2008, the National Evidence-based Healthcare Collaborating Agency (NECA) was launched, to which the CnHTA was transferred in 2010. Since then, non-drug technologies have been reviewed by NECA and drugs have been reviewed by HIRA. Political debates about how to embrace expensive but important health technologies that were not on the benefits list led to the creation of the Participatory Priority Setting Committee (PPSC) in 2012. The PPSC, led by the general public, has played a key role in advancing the path toward universal health coverage by revitalizing the list of essential, yet previously overlooked, medical technologies. PPSC offers these technologies a second chance at coverage. HIRA and NECA served to strengthen evidence-based and efficiency-based decision-making in the health system via CnHTA, and PPSC served to strengthen social value-based decision making via priority setting in Korea. The reassessment by PPSC may be relevant in countries where the economy is growing and citizens want to rapidly expand the benefits list.

Correction: Candidemia Risk Prediction (CanDETEC) Model for Patients With Malignancy: Model Development and Validation in a Single-Center Retrospective Study.

[This corrects the article DOI: 10.2196/24651.].

A Machine Learning-Based Algorithm for the Prediction of Intensive Care Unit Delirium (PRIDE): Retrospective Study.

BACKGROUND: Delirium frequently occurs among patients admitted to the intensive care unit (ICU). There is limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium. Therefore, the early recognition and prevention of delirium are important in the management of critically ill patients. OBJECTIVE: This study aims to develop and validate a delirium prediction model within 24 hours of admission to the ICU using electronic health record data. The algorithm was named the Prediction of ICU Delirium (PRIDE). METHODS: This is a retrospective cohort study performed at a tertiary referral hospital with 120 ICU beds. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU record from the day of ICU admission or if they had a positive Confusion Assessment Method for the ICU record at the time of ICU admission. The algorithm to predict delirium was developed using patient data from the first 2 years of the study period and validated using patient data from the last 6 months. Random forest (RF), Extreme Gradient Boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) were used. The algorithms were externally validated using MIMIC-III data, and the algorithm with the largest area under the receiver operating characteristics (AUROC) curve in the external data set was named the PRIDE algorithm. RESULTS: A total of 37,543 cases were collected. After patient exclusion, 12,409 remained as our study population, of which 3816 (30.8%) patients experienced delirium incidents during the study period. Based on the exclusion criteria, out of the 96,016 ICU admission cases in the MIMIC-III data set, 2061 cases were included, and 272 (13.2%) delirium incidents occurred. The average AUROCs and 95% CIs for internal validation were 0.916 (95% CI 0.916-0.916) for RF, 0.919 (95% CI 0.919-0.919) for XGBoost, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. Regarding the external validation, the best AUROC were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. The Brier score of the RF model is 0.168, indicating that it is well-calibrated. CONCLUSIONS: A machine learning approach based on electronic health record data can be used to predict delirium within 24 hours of ICU admission. RF, XGBoost, DNN, and LR models were used, and they effectively predicted delirium. However, with the potential to advise ICU physicians and prevent ICU delirium, prospective studies are required to verify the algorithm's performance.

Candidemia Risk Prediction (CanDETEC) Model for Patients With Malignancy: Model Development and Validation in a Single-Center Retrospective Study.

BACKGROUND: Appropriate empirical treatment for candidemia is associated with reduced mortality; however, the timely diagnosis of candidemia in patients with sepsis remains poor. OBJECTIVE: We aimed to use machine learning algorithms to develop and validate a candidemia prediction model for patients with cancer. METHODS: We conducted a single-center retrospective study using the cancer registry of a tertiary academic hospital. Adult patients diagnosed with malignancies between January 2010 and December 2018 were included. Our study outcome was the prediction of candidemia events. A stratified undersampling method was used to extract control data for algorithm learning. Multiple models were developed-a combination of 4 variable groups and 5 algorithms (auto-machine learning, deep neural network, gradient boosting, logistic regression, and random forest). The model with the largest area under the receiver operating characteristic curve (AUROC) was selected as the Candida detection (CanDETEC) model after comparing its performance indexes with those of the Candida Score Model. RESULTS: From a total of 273,380 blood cultures from 186,404 registered patients with cancer, we extracted 501 records of candidemia events and 2000 records as control data. Performance among the different models varied (AUROC 0.771- 0.889), with all models demonstrating superior performance to that of the Candida Score (AUROC 0.677). The random forest model performed the best (AUROC 0.889, 95% CI 0.888-0.889); therefore, it was selected as the CanDETEC model. CONCLUSIONS: The CanDETEC model predicted candidemia in patients with cancer with high discriminative power. This algorithm could be used for the timely diagnosis and appropriate empirical treatment of candidemia.

An Automated Fast Healthcare Interoperability Resources-Based 12-Lead Electrocardiogram Mobile Alert System for Suspected Acute Coronary Syndrome.

PURPOSE: For patients with time-critical acute coronary syndrome, reporting electrocardiogram (ECG) findings is the most important component of the treatment process. We aimed to develop and validate an automated Fast Healthcare Interoperability Resources (FHIR)-based 12-lead ECG mobile alert system for use in an emergency department (ED). MATERIALS AND METHODS: An automated FHIR-based 12-lead ECG alert system was developed in the ED of an academic tertiary care hospital. The system was aimed at generating an alert for patients with suspected acute coronary syndrome based on interpretation by the legacy device. The alert is transmitted to physicians both via a mobile application and the patient's electronic medical record (EMR). The automated FHIR-based 12-lead ECG alert system processing interval was defined as the time from ED arrival and 12-lead ECG capture to the time when the FHIR-based notification was transmitted. RESULTS: During the study period, 3812 emergency visits and 1581 12-lead ECGs were recorded. The FHIR system generated 155 alerts for 116 patients. The alerted patients were significantly older [mean (standard deviation): 68.1 (12.4) years vs. 59.6 (16.8) years, p<0.001], and the cardiac-related symptom rate was higher (34.5% vs. 19%, p<0.001). Among the 155 alerts, 146 (94%) were transmitted successfully within 5 minutes. The median interval from 12-lead ECG capture to FHIR notification was 2.7 min [interquartile range (IQR) 2.2-3.1 min] for the group with cardiac-related symptoms and 3.0 min (IQR 2.5-3.4 min) for the group with non-cardiac-related symptoms. CONCLUSION: An automated FHIR-based 12-lead ECG mobile alert system was successfully implemented in an ED.