To warrant clinical adoption AI models require a multi-faceted implementation evaluation.

Despite artificial intelligence (AI) technology progresses at unprecedented rate, our ability to translate these advancements into clinical value and adoption at the bedside remains comparatively limited. This paper reviews the current use of implementation outcomes in randomized controlled trials evaluating AI-based clinical decision support and found limited adoption. To advance trust and clinical adoption of AI, there is a need to bridge the gap between traditional quantitative metrics and implementation outcomes to better grasp the reasons behind the success or failure of AI systems and improve their translation into clinical value.

Optimizing discharge after major surgery using an artificial intelligence-based decision support tool (DESIRE): An external validation study.

BACKGROUND: In the DESIRE study (Discharge aftEr Surgery usIng aRtificial intElligence), we have previously developed and validated a machine learning concept in 1,677 gastrointestinal and oncology surgery patients that can predict safe hospital discharge after the second postoperative day. Despite strong model performance (area under the receiver operating characteristics curve of 0.88) in an academic surgical population, it remains unknown whether these findings can be translated to other hospitals and surgical populations. We therefore aimed to determine the generalizability of the previously developed machine learning concept. METHODS: We externally validated the machine learning concept in gastrointestinal and oncology surgery patients admitted to 3 nonacademic hospitals in The Netherlands between January 2017 and June 2021, who remained admitted 2 days after surgery. Primary outcome was the ability to predict hospital interventions after the second postoperative day, which were defined as unplanned reoperations, radiological interventions, and/or intravenous antibiotics administration. Four forest models were locally trained and evaluated with respect to area under the receiver operating characteristics curve, sensitivity, specificity, positive predictive value, and negative predictive value. RESULTS: All models were trained on 1,693 epsiodes, of which 731 (29.9%) required a hospital intervention and demonstrated strong performance (area under the receiver operating characteristics curve only varied 4%). The best model achieved an area under the receiver operating characteristics curve of 0.83 (95% confidence interval [0.81-0.85]), sensitivity of 77.9% (0.67-0.87), specificity of 79.2% (0.72-0.85), positive predictive value of 61.6% (0.54-0.69), and negative predictive value of 89.3% (0.85-0.93). CONCLUSION: This study showed that a previously developed machine learning concept can predict safe discharge in different surgical populations and hospital settings (academic versus nonacademic) by training a model on local patient data. Given its high accuracy, integration of the machine learning concept into the clinical workflow could expedite surgical discharge and aid hospitals in addressing capacity challenges by reducing avoidable bed-days.

Developing, implementing and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter.

OBJECTIVE: Although the role of artificial intelligence (AI) in medicine is increasingly studied, most patients do not benefit because the majority of AI models remain in the testing and prototyping environment. The development and implementation trajectory of clinical AI models are complex and a structured overview is missing. We therefore propose a step-by-step overview to enhance clinicians' understanding and to promote quality of medical AI research. METHODS: We summarised key elements (such as current guidelines, challenges, regulatory documents and good practices) that are needed to develop and safely implement AI in medicine. CONCLUSION: This overview complements other frameworks in a way that it is accessible to stakeholders without prior AI knowledge and as such provides a step-by-step approach incorporating all the key elements and current guidelines that are essential for implementation, and can thereby help to move AI from bytes to bedside.

Generating insights in uncharted territories: real-time learning from data in critically ill patients-an implementer report.

Introduction In the current situation, clinical patient data are often siloed in multiple hospital information systems. Especially in the intensive care unit (ICU), large volumes of clinical data are routinely collected through continuous patient monitoring. Although these data often contain useful information for clinical decision making, they are not frequently used to improve quality of care. During, but also after, pressing times, data-driven methods can be used to mine treatment patterns from clinical data to determine the best treatment options from a hospitals own clinical data.Methods In this implementer report, we describe how we implemented a data infrastructure that enabled us to learn in real time from consecutive COVID-19 ICU admissions. In addition, we explain our step-by-step multidisciplinary approach to establish such a data infrastructure.Conclusion By sharing our steps and approach, we aim to inspire others, in and outside ICU walls, to make more efficient use of data at hand, now and in the future.

Predicting need for hospital-specific interventional care after surgery using electronic health record data.

BACKGROUND: A significant proportion of surgical inpatients is often admitted longer than necessary. Early identification of patients who do not need care that is strictly provided within hospitals would allow timely discharge of patients to a postoperative nursing home for further recovery. We aimed to develop a model to predict whether a patient needs hospital-specific interventional care beyond the second postoperative day. METHODS: This study included all adult patients discharged from surgical care in the surgical oncology department from June 2017 to February 2020. The primary outcome was to predict whether a patient still needs hospital-specific interventional care beyond the second postoperative day. Hospital-specific care was defined as unplanned reoperations, radiological interventions, and intravenous antibiotics administration. Different analytical methods were compared with respect to the area under the receiver-operating characteristics curve, sensitivity, specificity, positive predictive value, and negative predictive value. RESULTS: Each model was trained on 1,174 episodes. In total, 847 (50.5%) patients required an intervention during postoperative admission. A random forest model performed best with an area under the receiver-operating characteristics curve of 0.88 (95% confidence interval 0.83-0.93), sensitivity of 79.1% (95% confidence interval 0.67-0.92), specificity of 80.0% (0.73-0.87), positive predictive value of 57.6% (0.45-0.70) and negative predictive value of 91.7% (0.87-0.97). CONCLUSION: This proof-of-concept study found that a random forest model could successfully predict whether a patient could be safely discharged to a nursing home and does not need hospital care anymore. Such a model could aid hospitals in addressing capacity challenges and improve patient flow, allowing for timely surgical care.

Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit.

PURPOSE: Due to the increasing demand for intensive care unit (ICU) treatment, and to improve quality and efficiency of care, there is a need for adequate and efficient clinical decision-making. The advancement of artificial intelligence (AI) technologies has resulted in the development of prediction models, which might aid clinical decision-making. This systematic review seeks to give a contemporary overview of the current maturity of AI in the ICU, the research methods behind these studies, and the risk of bias in these studies. METHODS: A systematic search was conducted in Embase, Medline, Web of Science Core Collection and Cochrane Central Register of Controlled Trials databases to identify eligible studies. Studies using AI to analyze ICU data were considered eligible. Specifically, the study design, study aim, dataset size, level of validation, level of readiness, and the outcomes of clinical trials were extracted. Risk of bias in individual studies was evaluated by the Prediction model Risk Of Bias ASsessment Tool (PROBAST). RESULTS: Out of 6455 studies identified through literature search, 494 were included. The most common study design was retrospective [476 studies (96.4% of all studies)] followed by prospective observational [8 (1.6%)] and clinical [10 (2%)] trials. 378 (80.9%) retrospective studies were classified as high risk of bias. No studies were identified that reported on the outcome evaluation of an AI model integrated in routine clinical practice. CONCLUSION: The vast majority of developed ICU-AI models remain within the testing and prototyping environment; only a handful were actually evaluated in clinical practice. A uniform and structured approach can support the development, safe delivery, and implementation of AI to determine clinical benefit in the ICU.

Predicting thromboembolic complications in COVID-19 ICU patients using machine learning.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic is a challenge for intensive care units (ICU) in part due to the failure to identify risks for patients early and the inability to render an accurate prognosis. Previous reports suggest a strong association between hypercoagulability and poor outcome. Factors related to hemostasis may, therefore, serve as tools to improve the management of COVID-19 patients. AIM: The purpose of this report is to develop a model to determine whether it is possible to early identify COVID-19 patients at risk for thromboembolic complications (TCs). METHODS: We analyzed electronic health record data of 108 consecutive COVID-19 patients admitted to the adult ICU of the Erasmus University Medical Center between February 27 and May 20, 2020. By training a decision tree classifier on 66% of the available data, a model for the prediction of TCs was developed. RESULTS: The median (interquartile range) age was 62 (53-70) years and 73% were male. Forty-three patients (40%) developed a TC during their ICU stay. Mortality was higher for patients in the TCs group compared to the control group (26% vs. 8%, P=0.03). Lactate dehydrogenase, standardized bicarbonate, albumin, and leukocytes were identified by the Decision Tree classifier as the most powerful predictors for TCs 2 days before the onset of the TC, with a sensitivity of 73% and a positive likelihood ratio of 2.7 on the test dataset. CONCLUSIONS: Clinically relevant TCs frequently occur in critically ill COVID-19 patients. These can successfully be predicted using a decision tree model. Although this model could be of special importance to aid clinical decision making, its generalizability and clinical impact should be determined in a larger population. RELEVANCE FOR PATIENTS: Recently, severe TCs were observed in COVID-19 patients with progressive respiratory failure warranting ICU treatment. Timely identification of patients at risk of developing TCs is critical inasmuch as it would enable clinicians to initiate potentially salvaging therapeutic anticoagulation.