The use of artificial intelligence to detect voided medication orders in oral and maxillofacial surgery inpatients.

OBJECTIVE: The aim of this study is to determine if supervised machine learning algorithms can accurately predict voided computerized physician order entry in oral and maxillofacial surgery inpatients. METHODS: Data from Electronic Medical Record included patient demographics, comorbidities, procedures, vital signs, laboratory values, and medication orders were retrospectively collected. Predictor variables included patient demographics, comorbidities, procedures, vital signs, and laboratory values. Outcome of interest is if a medication order was voided or not. Data was cleaned and processed using Microsoft Excel and Python v3.12. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes were trained, validated, and tested for accuracy of the prediction of voided medication orders. RESULTS: 37,493 medication orders from 1,204 patient admissions over 5 years were used for this study. 3,892 (10.4%) medication orders were voided. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes had an Area Under the Receiver Operating Curve of 0.802 with 95% CI [0.787, 0.825], 0.746 with 95% CI [0.722, 0.765], 0.685 with 95% CI [0.667, 0.699], and 0.505 with 95% CI [0.489, 0.539], respectively. Area Under the Precision Recall Curve was 0.684 with 95% CI [0.679, 0.702], 0.647 with 95% CI [0.638, 0.664], 0.429 with 95% CI [0.417, 0.434], and 0.551 with 95% CI [0.551, 0.552], respectively. CONCLUSION: Gradient Boosted Decision Trees was the best performing model of the supervised machine learning algorithms with satisfactory outcomes in the test cohort for predicting voided Computerized Physician Order Entry in Oral and Maxillofacial Surgery inpatients.

Effect of an Artificial Intelligence Decision Support Tool on Palliative Care Referral in Hospitalized Patients: A Randomized Clinical Trial.

CONTEXT: Palliative care services are commonly provided to hospitalized patients, but accurately predicting who needs them remains a challenge. OBJECTIVES: To assess the effectiveness on clinical outcomes of an artificial intelligence (AI)/machine learning (ML) decision support tool for predicting patient need for palliative care services in the hospital. METHODS: The study design was a pragmatic, cluster-randomized, stepped-wedge clinical trial in 12 nursing units at two hospitals over a 15-month period between August 19, 2019, and November 17, 2020. Eligible patients were randomly assigned to either a medical service consultation recommendation triggered by an AI/ML tool predicting the need for palliative care services or usual care. The primary outcome was palliative care consultation note. Secondary outcomes included: hospital readmissions, length of stay, transfer to intensive care and palliative care consultation note by unit. RESULTS: A total of 3183 patient hospitalizations were enrolled. Of eligible patients, A total of 2544 patients were randomized to the decision support tool (1212; 48%) and usual care (1332; 52%). Of these, 1717 patients (67%) were retained for analyses. Patients randomized to the intervention had a statistically significant higher incidence rate of palliative care consultation compared to the control group (IRR, 1.44 [95% CI, 1.11-1.92]). Exploratory evidence suggested that the decision support tool group reduced 60-day and 90-day hospital readmissions (OR, 0.75 [95% CI, 0.57, 0.97]) and (OR, 0.72 [95% CI, 0.55-0.93]) respectively. CONCLUSION: A decision support tool integrated into palliative care practice and leveraging AI/ML demonstrated an increased palliative care consultation rate among hospitalized patients and reductions in hospitalizations.

Improving time to palliative care review with predictive modeling in an inpatient adult population: study protocol for a stepped-wedge, pragmatic randomized controlled trial.

BACKGROUND: Palliative care is a medical specialty centered on improving the quality of life (QOL) of patients with complex or life-threatening illnesses. The need for palliative care is increasing and with that the rigorous testing of triage tools that can be used quickly and reliably to identify patients that may benefit from palliative care. METHODS: To that aim, we will conduct a two-armed stepped-wedge cluster randomized trial rolled out to two inpatient hospitals to evaluate whether a machine learning algorithm accurately identifies patients who may benefit from a comprehensive review by a palliative care specialist and decreases time to receiving a palliative care consult in hospital. This is a single-center study which will be conducted from August 2019 to November 2020 at Saint Mary's Hospital & Methodist Hospital both within Mayo Clinic Rochester in Minnesota. Clusters will be nursing units which will be chosen to be a mix of complex patients from Cardiology, Critical Care, and Oncology and had previously established relationships with palliative medicine. The stepped wedge design will have 12 units allocated to a design matrix of 5 treatment wedges. Each wedge will last 75 days resulting in a study period of 12 months of recruitment unless otherwise specified. Data will be analyzed with Bayesian hierarchical models with credible intervals denoting statistical significance. DISCUSSION: This intervention offers a pragmatic approach to delivering specialty palliative care to hospital patients in need using machine learning, thereby leading to high value care and improved outcomes. It is not enough for AI to be utilized by simply publishing research showing predictive performance; clinical trials demonstrating better outcomes are critically needed. Furthermore, the deployment of an AI algorithm is a complex process that requires multiple teams with varying skill sets. To evaluate a deployed AI, a pragmatic clinical trial can accommodate the difficulties of clinical practice while retaining scientific rigor. TRIAL REGISTRATION: ClinicalTrials.gov NCT03976297 . Registered on 6 June 2019, prior to trial start.

A Living, Interactive Systematic Review and Network Meta-analysis of First-line Treatment of Metastatic Renal Cell Carcinoma.

CONTEXT: Identifying the most effective first-line treatment for metastatic renal cell carcinoma (mRCC) is challenging as rapidly evolving data quickly outdate the existing body of evidence, and current approaches to presenting the evidence in user-friendly formats are fraught with limitations. OBJECTIVE: To maintain living evidence for contemporary first-line treatment for previously untreated mRCC. EVIDENCE ACQUISITION: We have created a living, interactive systematic review (LISR) and network meta-analysis for first-line treatment of mRCC using data from randomized controlled trials comparing contemporary treatment options with single-agent tyrosine kinase inhibitors. We applied an advanced programming and artificial intelligence-assisted framework for evidence synthesis to create a living search strategy, facilitate screening and data extraction using a graphical user interface, automate the frequentist network meta-analysis, and display results in an interactive manner. EVIDENCE SYNTHESIS: As of October 22, 2020, the LISR includes data from 14 clinical trials. Baseline characteristics are summarized in an interactive table. The cabozantinib + nivolumab combination (CaboNivo) is ranked the highest for the overall response rate, progression-free survival, and overall survival, whereas ipilimumab + nivolumab (NivoIpi) is ranked the highest for achieving a complete response (CR). NivoIpi, and atezolizumab + bevacizumab (AteBev) were ranked highest (lowest toxicity) and CaboNivo ranked lowest for treatment-related adverse events (AEs). Network meta-analysis results are summarized as interactive tables and plots, GRADE summary-of-findings tables, and evidence maps. CONCLUSIONS: This innovative living and interactive review provides the best current evidence on the comparative effectiveness of multiple treatment options for patients with untreated mRCC. Trial-level comparisons suggest that CaboNivo is likely to cause more AEs but is ranked best for all efficacy outcomes, except NivoIpi offers the best chance of CR. Pembrolizumab + axitinib and NivoIpi are acceptable alternatives, except NivoIpi may not be preferred for patients with favorable risk. Although network meta-analysis provides rankings with statistical adjustments, there are inherent biases in cross-trial comparisons with sparse direct evidence that does not replace randomized comparisons. PATIENT SUMMARY: It is challenging to decide the best option among the several treatment combinations of immunotherapy and targeted treatments for newly diagnosed metastatic kidney cancer. We have created interactive evidence summaries of multiple treatment options that present the benefits and harms and evidence certainty for patient-important outcomes. This evidence is updated as soon as new studies are published.

Improving the delivery of palliative care through predictive modeling and healthcare informatics.

OBJECTIVE: Access to palliative care (PC) is important for many patients with uncontrolled symptom burden from serious or complex illness. However, many patients who could benefit from PC do not receive it early enough or at all. We sought to address this problem by building a predictive model into a comprehensive clinical framework with the aims to (i) identify in-hospital patients likely to benefit from a PC consult, and (ii) intervene on such patients by contacting their care team. MATERIALS AND METHODS: Electronic health record data for 68 349 inpatient encounters in 2017 at a large hospital were used to train a model to predict the need for PC consult. This model was published as a web service, connected to institutional data pipelines, and consumed by a downstream display application monitored by the PC team. For those patients that the PC team deems appropriate, a team member then contacts the patient's corresponding care team. RESULTS: Training performance AUC based on a 20% holdout validation set was 0.90. The most influential variables were previous palliative care, hospital unit, Albumin, Troponin, and metastatic cancer. The model has been successfully integrated into the clinical workflow making real-time predictions on hundreds of patients per day. The model had an "in-production" AUC of 0.91. A clinical trial is currently underway to assess the effect on clinical outcomes. CONCLUSIONS: A machine learning model can effectively predict the need for an inpatient PC consult and has been successfully integrated into practice to refer new patients to PC.

Clinical impact of intraoperative electronic health record downtime on surgical patients.

OBJECTIVE: Despite increased use of electronic health records (EHRs), the clinical impact of system downtime is unknown. MATERIALS AND METHODS: This retrospective matched cohort study evaluated the impact of EHR downtime episodes lasting more than 60 minutes over a 6-year study period. Patients age 18 years or older who underwent surgical procedures at least 60 minutes in duration with an inpatient stay exceeding 24 hours within the study period were eligible for inclusion. Out of 4115 patients exposed to 1 of 176 EHR downtime episodes, 4103 patients were matched to an unexposed cohort in a 1:1 ratio. Multivariable regression analysis, as well as trend analysis for effect of duration of downtime on outcomes, was performed. RESULTS: Downtime-exposed patients had operating room duration 1.1 times longer (p < .001) and postoperative length of stay 1.04 times longer (p = .007) compared to unexposed patients. The 30-day mortality rates were similar between these groups (odds ratio 1.26, p > .05). In trend analysis, there was no association between duration of downtime with respect to evaluated outcomes, postoperative length of stay, and 30-day mortality. CONCLUSION: EHR downtime had no impact on 30-day mortality. Potential associations for increased postoperative length of stay and duration of time spent in the operating room were observed among downtime-exposed patients. No trend effect was observed with respect to duration of downtime and postoperative length of stay and 30-day mortality rates.

Evaluating Muscle Mass by Using Markers of Kidney Function: Development of the Sarcopenia Index.

OBJECTIVES: Sarcopenia is associated with a poor prognosis in the ICU. The purpose of this study was to describe a simple sarcopenia index using routinely available renal biomarkers and evaluate its association with muscle mass and patient outcomes. DESIGN: A retrospective cohort study. SETTING: A tertiary-care medical center. PATIENTS: High-risk adult ICU patients from October 2008 to December 2010. INTERVENTIONS: The gold standard for muscle mass was quantified with the paraspinal muscle surface area at the L4 vertebrae in the subset of individuals with an abdominal CT scan. Using Pearson's correlation coefficient, serum creatinine-to-serum cystatin C ratio was found to be the best performer in the estimation of muscle mass. The relationship between sarcopenia index and hospital and 90-day mortality, and the length of mechanical ventilation was evaluated. MEASUREMENTS AND MAIN RESULTS: Out of 226 enrolled patients, 123 (54%) were female, and 198 (87%) were white. Median (interquartile range) age, body mass index, and body surface area were 68 (57-77) years, 28 (24-34) kg/m, and 1.9 (1.7-2.2) m, respectively. The mean (± SD) Acute Physiology and Chronic Health Evaluation III was 70 (± 22). ICU, hospital, and 90-day mortality rates were 5%, 12%, and 20%, respectively. The correlation (r) between sarcopenia index and muscle mass was 0.62 and coefficient of determination (r) was 0.27 (p < 0.0001). After adjustment for Acute Physiology and Chronic Health Evaluation III, body surface area, and age, sarcopenia index was independently predictive of both hospital (p = 0.001) and 90-day mortality (p < 0.0001). Among the 131 patients on mechanical ventilator, the duration of mechanical ventilation was significantly lower on those with higher sarcopenia index (-1 d for each 10 unit of sarcopenia index [95% CI, -1.4 to -0.2; p = 0.006]). CONCLUSIONS: The sarcopenia index is a fair measure for muscle mass estimation among ICU patients and can modestly predict hospital and 90-day mortality among patients who do not have acute kidney injury at the time of measurement.

Decision Support Tool to Improve Glucose Control Compliance After Cardiac Surgery.

Hyperglycemia control is associated with improved outcomes in patients undergoing cardiac surgery. The Surgical Care Improvement Project metric (SCIP-inf-4) was introduced as a performance measure in surgical patients and included hyperglycemia control. Compliance with the SCIP-inf-4 metric remains suboptimal. A novel real-time decision support tool (DST) with guaranteed feedback that is based on the existing electronic medical record system was developed at a tertiary academic center. Implementation of the DST increased the compliance rate with the SCIP-inf-4 from 87.3% to 96.5%. Changes in tested clinical outcomes were not observed with improved metric compliance. This new framework can serve as a backbone for development of quality control processes for other metrics. Further and, ideally, multicenter studies are required to test if implementation of electronic DSTs will translate into improved resource utilization and outcomes for patients.

The Effect of an Electronic Checklist on Critical Care Provider Workload, Errors, and Performance.

PURPOSE: The strategy used to improve effective checklist use in intensive care unit (ICU) setting is essential for checklist success. This study aimed to test the hypothesis that an electronic checklist could reduce ICU provider workload, errors, and time to checklist completion, as compared to a paper checklist. METHODS: This was a simulation-based study conducted at an academic tertiary hospital. All participants completed checklists for 6 ICU patients: 3 using an electronic checklist and 3 using an identical paper checklist. In both scenarios, participants had full access to the existing electronic medical record system. The outcomes measured were workload (defined using the National Aeronautics and Space Association task load index [NASA-TLX]), the number of checklist errors, and time to checklist completion. Two independent clinician reviewers, blinded to participant results, served as the reference standard for checklist error calculation. RESULTS: Twenty-one ICU providers participated in this study. This resulted in the generation of 63 simulated electronic checklists and 63 simulated paper checklists. The median NASA-TLX score was 39 for the electronic checklist and 50 for the paper checklist (P = .005). The median number of checklist errors for the electronic checklist was 5, while the median number of checklist errors for the paper checklist was 8 (P = .003). The time to checklist completion was not significantly different between the 2 checklist formats (P = .76). CONCLUSION: The electronic checklist significantly reduced provider workload and errors without any measurable difference in the amount of time required for checklist completion. This demonstrates that electronic checklists are feasible and desirable in the ICU setting.

Validation of computerized sniffer for monitoring perioperative normothermia.

INTRODUCTION: The World Health Organization sets a standard to maintain patient core temperature greater than 36°C throughout the perioperative period. Normothermia (defined as >36°C) in the Operating Room (OR) is an important factor to preventing complications in patients (MI, infection, coagulopathy). Randomized studies suggests that maintaining at higher temperatures may further reduce complications in surgery (less complications for group at 36.4°C than the control group at 36.0°C) [1,2]. Perioperative normothermia is an important but often unrecognized element during anesthesia. Early recognition of hypothermia would allow for appropriate interventions and prevent complications. METHODOLOGY: Manual validation of the diagnostic performance a clinical tool (alert) that would automatically measure changes in core temperature to identify patients who fail to be in range of normothermia during surgery. RESULTS: The clinical tool (alert) was found to be 97 % sensitive.

Connecting the dots: rule-based decision support systems in the modern EMR era.

The intensive care unit (ICU) environment is rich in both medical device and electronic medical record (EMR) data. The ICU patient population is particularly vulnerable to medical error or delayed medical intervention both of which are associated with excess morbidity, mortality and cost. The development and deployment of smart alarms, computerized decision support systems (DSS) and "sniffers" within ICU clinical information systems has the potential to improve the safety and outcomes of critically ill hospitalized patients. However, the current generations of alerts, run largely through bedside monitors, are far from ideal and rarely support the clinician in the early recognition of complex physiologic syndromes or deviations from expected care pathways. False alerts and alert fatigue remain prevalent. In the coming era of widespread EMR implementation novel medical informatics methods may be adaptable to the development of next generation, rule-based DSS.

Derivation and validation of automated electronic search strategies to extract Charlson comorbidities from electronic medical records.

OBJECTIVE: To develop and validate automated electronic note search strategies (automated digital algorithm) to identify Charlson comorbidities. PATIENTS AND METHODS: The automated digital algorithm was built by a series of programmatic queries applied to an institutional electronic medical record database. The automated digital algorithm was derived from secondary analysis of an observational cohort study of 1447 patients admitted to the intensive care unit from January 1 through December 31, 2006, and validated in an independent cohort of 240 patients. The sensitivity, specificity, and positive and negative predictive values of the automated digital algorithm and International Classification of Diseases, Ninth Revision (ICD-9) codes were compared with comprehensive medical record review (reference standard) for the Charlson comorbidities. RESULTS: In the derivation cohort, the automated digital algorithm achieved a median sensitivity of 100% (range, 99%-100%) and a median specificity of 99.7% (range, 99%-100%). In the validation cohort, the sensitivity of the automated digital algorithm ranged from 91% to 100%, and the specificity ranged from 98% to 100%. The sensitivity of the ICD-9 codes ranged from 8% for dementia to 100% for leukemia, whereas specificity ranged from 86% for congestive heart failure to 100% for leukemia, dementia, and AIDS. CONCLUSION: Our results suggest that search strategies that use automated electronic search strategies to extract Charlson comorbidities from the clinical notes contained within the electronic medical record are feasible and reliable. Automated digital algorithm outperformed ICD-9 codes in all the Charlson variables except leukemia, with greater sensitivity, specificity, and positive and negative predictive values.

Clinical review: the hospital of the future - building intelligent environments to facilitate safe and effective acute care delivery.

The translation of knowledge into rational care is as essential and pressing a task as the development of new diagnostic or therapeutic devices, and is arguably more important. The emerging science of health care delivery has identified the central role of human factor ergonomics in the prevention of medical error, omission, and waste. Novel informatics and systems engineering strategies provide an excellent opportunity to improve the design of acute care delivery. In this article, future hospitals are envisioned as organizations built around smart environments that facilitate consistent delivery of effective, equitable, and error-free care focused on patient-centered rather than provider-centered outcomes.

Derivation and validation of automated electronic search strategies to identify pertinent risk factors for postoperative acute lung injury.

OBJECTIVE: To develop and validate time-efficient automated electronic search strategies for identifying preoperative risk factors for postoperative acute lung injury. PATIENTS AND METHODS: This secondary analysis of a prospective cohort study included 249 patients undergoing high-risk surgery between November 1, 2005, and August 31, 2006. Two independent data-extraction strategies were compared. The first strategy used a manual review of medical records and the second a Web-based query-building tool. Web-based searches were derived and refined in a derivation cohort of 83 patients and subsequently validated in an independent cohort of 166 patients. Agreement between the 2 search strategies was assessed with percent agreement and Cohen κ statistics. RESULTS: Cohen κ statistics ranged from 0.34 (95% confidence interval, 0.00-0.86) for amiodarone to 0.85 for cirrhosis (95% confidence interval, 0.57-1.00). Agreement between manual and automated electronic data extraction was almost complete for 3 variables (diabetes mellitus, cirrhosis, H(2)-receptor antagonists), substantial for 3 (chronic obstructive pulmonary disease, proton pump inhibitors, statins), moderate for gastroesophageal reflux disease, and fair for 2 variables (restrictive lung disease and amiodarone). Automated electronic queries outperformed manual data collection in terms of sensitivities (median, 100% [range, 77%-100%] vs median, 87% [range, 0%-100%]). The specificities were uniformly high (≥ 96%) for both search strategies. CONCLUSION: Automated electronic query building is an iterative process that ultimately results in accurate, highly efficient data extraction. These strategies may be useful for both clinicians and researchers when determining the risk of time-sensitive conditions such as postoperative acute lung injury.

Timing of the onset of acute respiratory distress syndrome: a population-based study.

BACKGROUND: Many patients with acute lung injury (ALI) and acute respiratory distress syndrome (ARDS) have had recent healthcare interventions prior to developing ALI/ARDS. OBJECTIVE: To determine the timing of ALI/ARDS onset in relation to hospital admission and other healthcare interventions. METHODS: We conducted a population-based observational cohort study with a validated electronic surveillance tool, and identified patients with possible ALI/ARDS among critically ill adults at Mayo Clinic hospitals that provide critical care services for Olmsted County, Minnesota, in 2006. Trained investigators independently reviewed electronic medical records and confirmed the presence and timing of ALI/ARDS based on the American-European consensus definition. RESULTS: Of 124 episodes of ALI in 118 patients, only 5 did not fulfill the ARDS criteria. The syndrome developed a median 30 hours (IQR 10-82 h) after hospital admission in 79 patients (67%). ARDS was present on admission in 39 patients (33%), of whom 14 had recent hospitalization, 6 were transferred from nursing homes, and 3 had recent out-patient contact (1 antibiotic prescription, 1 surgical intervention, and 1 chemotherapy). Only 16 ARDS patients (14%) did not have known recent contact with a healthcare system. Compared to ARDS on admission, hospital-acquired ARDS was more likely to occur in surgery patients (54% vs 15%, P < .001), and had longer adjusted hospital stay (mean difference 8.9 d, 95% CI 0.3-17.4, P = .04). CONCLUSIONS: ARDS in the community most often develops either during hospitalization or in patients who recently had contact with a healthcare system. These findings have important implications for potential preventive strategies.