Validation of an Electronic Health Record-Based Machine Learning Model Compared With Clinical Risk Scores for Gastrointestinal Bleeding.

BACKGROUND & AIMS: Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB. METHODS: The training cohort comprised 2546 patients and internal validation of 850 patients presenting with overt GIB (ie, hematemesis, melena, and hematochezia) to emergency departments of 2 hospitals from 2014 to 2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014 to 2019. The primary outcome was a composite of red blood cell transfusion, hemostatic intervention (ie, endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR, available within 4 hours of presentation, and compared the performance of machine learning models with current guideline-recommended risk scores, Glasgow-Blatchford Score, and Oakland Score. Primary analysis was area under the receiver operating characteristic curve. Secondary analysis was specificity at 99% sensitivity to assess the proportion of patients correctly identified as very low risk. RESULTS: The machine learning model outperformed the Glasgow-Blatchford Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001) and Oakland Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs 18.5% for Glasgow-Blatchford Score and 11.7% for Oakland Score (P < .001 for both comparisons). CONCLUSIONS: An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department.

How Artificial Intelligence Will Transform Clinical Care, Research, and Trials for Inflammatory Bowel Disease.

Artificial intelligence (AI) refers to computer-based methodologies which use data to teach a computer to solve pre-defined tasks; these methods can be applied to identify patterns in large multi-modal data sources. AI applications in inflammatory bowel disease (IBD) includes predicting response to therapy, disease activity scoring of endoscopy, drug discovery, and identifying bowel damage in images. As a complex disease with entangled relationships between genomics, metabolomics, microbiome, and the environment, IBD stands to benefit greatly from methodologies that can handle this complexity. We describe current applications, critical challenges, and propose future directions of AI in IBD.

Achieving Value by Risk Stratification With Machine Learning Model or Clinical Risk Score in Acute Upper Gastrointestinal Bleeding: A Cost Minimization Analysis.

INTRODUCTION: We estimate the economic impact of applying risk assessment tools to identify very low-risk patients with upper gastrointestinal bleeding who can be safely discharged from the emergency department using a cost minimization analysis. METHODS: We compare triage strategies (Glasgow-Blatchford score = 0/0-1 or validated machine learning model) with usual care using a Markov chain model from a US health care payer perspective. RESULTS: Over 5 years, the Glasgow-Blatchford score triage strategy produced national cumulative savings over usual care of more than $2.7 billion and the machine learning strategy of more than $3.4 billion. DISCUSSION: Implementing risk assessment models for upper gastrointestinal bleeding reduces costs, thereby increasing value.

Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes.

Large Language Models (LLMs) are transformer-based neural networks with billions of parameters trained on very large text corpora from diverse sources. LLMs have the potential to improve healthcare due to their capability to parse complex concepts and generate context-based responses. The interest in LLMs has not spared digestive disease academics, who have mainly investigated foundational LLM accuracy, which ranges from 25% to 90% and is influenced by the lack of standardized rules to report methodologies and results for LLM-oriented research. In addition, a critical issue is the absence of a universally accepted definition of accuracy, varying from binary to scalar interpretations, often tied to grader expertise without reference to clinical guidelines. We address strategies and challenges to increase accuracy. In particular, LLMs can be infused with domain knowledge using Retrieval Augmented Generation (RAG) or Supervised Fine-Tuning (SFT) with reinforcement learning from human feedback (RLHF). RAG faces challenges with in-context window limits and accurate information retrieval from the provided context. SFT, a deeper adaptation method, is computationally demanding and requires specialized knowledge. LLMs may increase patient quality of care across the field of digestive diseases, where physicians are often engaged in screening, treatment and surveillance for a broad range of pathologies for which in-context learning or SFT with RLHF could improve clinical decision-making and patient outcomes. However, despite their potential, the safe deployment of LLMs in healthcare still needs to overcome hurdles in accuracy, suggesting a need for strategies that integrate human feedback with advanced model training.

Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review.

BACKGROUND: The number of artificial intelligence (AI) studies in medicine has exponentially increased recently. However, there is no clear quantification of the clinical benefits of implementing AI-assisted tools in patient care. OBJECTIVE: This study aims to systematically review all published randomized controlled trials (RCTs) of AI-assisted tools to characterize their performance in clinical practice. METHODS: CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed were searched to identify relevant RCTs published up to July 2021 and comparing the performance of AI-assisted tools with conventional clinical management without AI assistance. We evaluated the primary end points of each study to determine their clinical relevance. This systematic review was conducted following the updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. RESULTS: Among the 11,839 articles retrieved, only 39 (0.33%) RCTs were included. These RCTs were conducted in an approximately equal distribution from North America, Europe, and Asia. AI-assisted tools were implemented in 13 different clinical specialties. Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy. Most RCTs studied biosignal-based AI-assisted tools, and a minority of RCTs studied AI-assisted tools drawn from clinical data. In 77% (30/39) of the RCTs, AI-assisted interventions outperformed usual clinical care, and clinically relevant outcomes improved with AI-assisted intervention in 70% (21/30) of the studies. Small sample size and single-center design limited the generalizability of these studies. CONCLUSIONS: There is growing evidence supporting the implementation of AI-assisted tools in daily clinical practice; however, the number of available RCTs is limited and heterogeneous. More RCTs of AI-assisted tools integrated into clinical practice are needed to advance the role of AI in medicine. TRIAL REGISTRATION: PROSPERO CRD42021286539; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=286539.

Validation of a Machine Learning Model That Outperforms Clinical Risk Scoring Systems for Upper Gastrointestinal Bleeding.

BACKGROUND & AIMS: Scoring systems are suboptimal for determining risk in patients with upper gastrointestinal bleeding (UGIB); these might be improved by a machine learning model. We used machine learning to develop a model to calculate the risk of hospital-based intervention or death in patients with UGIB and compared its performance with other scoring systems. METHODS: We analyzed data collected from consecutive unselected patients with UGIB from medical centers in 4 countries (the United States, Scotland, England, and Denmark; n = 1958) from March 2014 through March 2015. We used the data to derive and internally validate a gradient-boosting machine learning model to identify patients who met a composite endpoint of hospital-based intervention (transfusion or hemostatic intervention) or death within 30 days. We compared the performance of the machine learning prediction model with validated pre-endoscopic clinical risk scoring systems (the Glasgow-Blatchford score [GBS], admission Rockall score, and AIMS65). We externally validated the machine learning model using data from 2 Asia-Pacific sites (Singapore and New Zealand; n = 399). Performance was measured by area under receiver operating characteristic curve (AUC) analysis. RESULTS: The machine learning model identified patients who met the composite endpoint with an AUC of 0.91 in the internal validation set; the clinical scoring systems identified patients who met the composite endpoint with AUC values of 0.88 for the GBS (P = .001), 0.73 for Rockall score (P < .001), and 0.78 for AIMS65 score (P < .001). In the external validation cohort, the machine learning model identified patients who met the composite endpoint with an AUC of 0.90, the GBS with an AUC of 0.87 (P = .004), the Rockall score with an AUC of 0.66 (P < .001), and the AIMS65 with an AUC of 0.64 (P < .001). At cutoff scores at which the machine learning model and GBS identified patients who met the composite endpoint with 100% sensitivity, the specificity values were 26% with the machine learning model versus 12% with GBS (P < .001). CONCLUSIONS: We developed a machine learning model that identifies patients with UGIB who met a composite endpoint of hospital-based intervention or death within 30 days with a greater AUC and higher levels of specificity, at 100% sensitivity, than validated clinical risk scoring systems. This model could increase identification of low-risk patients who can be safely discharged from the emergency department for outpatient management.

Advancing care for acute gastrointestinal bleeding using artificial intelligence.

The future of gastrointestinal bleeding will include the integration of machine learning algorithms to enhance clinician risk assessment and decision making. Machine learning algorithms have shown promise in outperforming existing clinical risk scores for both upper and lower gastrointestinal bleeding but have not been validated in any prospective clinical trials. The adoption of electronic health records provides an exciting opportunity to deploy risk prediction tools in real time and also to expand the data available to train predictive models. Machine learning algorithms can be used to identify patients with acute gastrointestinal bleeding using data extracted from the electronic health record. This can lead to an automated process to find patients with symptoms of acute gastrointestinal bleeding so that risk prediction tools can be then triggered to consistently provide decision support to the physician. Neural network models can be used to provide continuous risk predictions for patients who are at higher risk, which can be used to guide triage of patients to appropriate levels of care. Finally, the future will likely include neural network-based analysis of endoscopic stigmata of bleeding to help guide best practices for hemostasis during the endoscopic procedure. Machine learning will enhance the delivery of care at every level for patients with acute gastrointestinal bleeding through identifying very low risk patients for outpatient management, triaging high risk patients for higher levels of care, and guiding optimal intervention during endoscopy.

Challenges of developing artificial intelligence-assisted tools for clinical medicine.

Machine learning, a subset of artificial intelligence (AI), is a set of computational tools that can be used to enhance provision of clinical care in all areas of medicine. Gastroenterology and hepatology utilize multiple sources of information, including visual findings on endoscopy, radiologic imaging, manometric testing, genomes, proteomes, and metabolomes. However, clinical care is complex and requires a thoughtful approach to best deploy AI tools to improve quality of care and bring value to patients and providers. On the operational level, AI-assisted clinical management should consider logistic challenges in care delivery, data management, and algorithmic stewardship. There is still much work to be done on a broader societal level in developing ethical, regulatory, and reimbursement frameworks. A multidisciplinary approach and awareness of AI tools will create a vibrant ecosystem for using AI-assisted tools to guide and enhance clinical practice. From optically enhanced endoscopy to clinical decision support for risk stratification, AI tools will potentially transform our practice by leveraging massive amounts of data to personalize care to the right patient, in the right amount, at the right time.

Early identification of patients with acute gastrointestinal bleeding using natural language processing and decision rules.

BACKGROUND AND AIM: Guidelines recommend risk stratification scores in patients presenting with gastrointestinal bleeding (GIB), but such scores are uncommonly employed in practice. Automation and deployment of risk stratification scores in real time within electronic health records (EHRs) would overcome a major impediment. This requires an automated mechanism to accurately identify ("phenotype") patients with GIB at the time of presentation. The goal is to identify patients with acute GIB by developing and evaluating EHR-based phenotyping algorithms for emergency department (ED) patients. METHODS: We specified criteria using structured data elements to create rules for identifying patients and also developed multiple natural language processing (NLP)-based approaches for automated phenotyping of patients, tested them with tenfold cross-validation for 10 iterations (n = 7144) and external validation (n = 2988) and compared them with a standard method to identify patient conditions, the Systematized Nomenclature of Medicine. The gold standard for GIB diagnosis was the independent dual manual review of medical records. The primary outcome was the positive predictive value. RESULTS: A decision rule using GIB-specific terms from ED triage and ED review-of-systems assessment performed better than the Systematized Nomenclature of Medicine on internal validation and external validation (positive predictive value = 85% confidence interval:83%-87% vs 69% confidence interval:66%-72%; P < 0.001). The syntax-based NLP algorithm and Bidirectional Encoder Representation from Transformers neural network-based NLP algorithm had similar performance to the structured-data fields decision rule. CONCLUSIONS: An automated decision rule employing GIB-specific triage and review-of-systems terms can be used to trigger EHR-based deployment of risk stratification models to guide clinical decision making in real time for patients with acute GIB presenting to the ED.

Early Colonoscopy Does Not Improve Outcomes of Patients With Lower Gastrointestinal Bleeding: Systematic Review of Randomized Trials.

BACKGROUND & AIMS: Guidelines recommend colonoscopy evaluation within 24 hours of presentation or admission in patients with high-risk or severe acute lower gastrointestinal bleeding (LGIB). Meta-analyses of the timing of colonoscopy have relied primarily on observational studies that had major potential for bias. We performed a systematic review of randomized trials to determine optimal timing of colonoscopy for patients hospitalized with acute LGIB. METHODS: We searched publication databases through July 2019 and abstracts from gastroenterology meetings through November 2019 for randomized trials of patients with acute LGIB or hematochezia. We searched for studies that compared early colonoscopy (within 24 hours) with elective colonoscopy beyond 24 hours and/or other diagnostic tests. Our primary outcome was further bleeding, defined as persistent or recurrent bleeding after index examination. Secondary outcomes included mortality, diagnostic yield (identifying source of bleeding), endoscopic intervention, and any primary hemostatic intervention (endoscopic, surgical, or interventional radiologic). We performed dual independent review, data extraction, and risk of bias assessments. We performed the meta-analysis using a random-effects model. RESULTS: Our final analysis included data from 4 randomized trials. Further bleeding was not decreased among patients who received early vs later, elective colonoscopy (relative risk [RR] for further bleeding with early colonoscopy, 1.57; 95% CI. 0.74-3.31). We did not find significant differences in the secondary outcomes of mortality (RR, 0.93; 95% CI, 0.05-17.21), diagnostic yield (RR, 1.09; 95% CI, 0.99-1.21), endoscopic intervention (RR, 1.53; 95% CI, 0.67-3.48), or any primary hemostatic intervention (RR, 1.33; 95% CI, 0.92-1.92). CONCLUSIONS: In a meta-analysis of randomized trials, we found that colonoscopy within 24 hours does not reduce further bleeding or mortality in patients hospitalized with acute LGIB. Based on these findings, patients hospitalized with acute LGIB do not generally require early colonoscopy.

Machine Learning Prognostic Models for Gastrointestinal Bleeding Using Electronic Health Record Data.

Risk assessment tools for patients with gastrointestinal bleeding may be used for determining level of care and informing management decisions. Development of models that use data from electronic health records is an important step for future deployment of such tools in clinical practice. Furthermore, machine learning tools have the potential to outperform standard clinical risk assessment tools. The authors developed a new machine learning tool for the outcome of in-hospital mortality and suggested it outperforms the intensive care unit prognostic tool, APACHE IVa. Limitations include lack of generalizability beyond intensive care unit patients, inability to use early in the hospital course, and lack of external validation.

Machine Learning to Predict Outcomes in Patients with Acute Gastrointestinal Bleeding: A Systematic Review.

Risk stratification of patients with gastrointestinal bleeding (GIB) is recommended, but current risk assessment tools have variable performance. Machine learning (ML) has promise to improve risk assessment. We performed a systematic review to evaluate studies utilizing ML techniques for GIB. Bibliographic databases and conference abstracts were searched for studies with a population of overt GIB that used an ML algorithm with outcomes of mortality, rebleeding, hemostatic intervention, and/or hospital stay. Two independent reviewers screened titles and abstracts, reviewed full-text studies, and extracted data from included studies. Risk of bias was assessed with an adapted Quality in Prognosis Studies tool. Area under receiver operating characteristic curves (AUCs) were the primary assessment of performance with AUC ≥ 0.80 predefined as an acceptable threshold of good performance. Fourteen studies with 30 assessments of ML models met inclusion criteria. No study had low risk of bias. Median AUC reported in validation datasets for predefined outcomes of mortality, intervention, or rebleeding was 0.84 (range 0.40-0.98). AUCs were higher with artificial neural networks (median 0.93, range 0.78-0.98) than other ML models (0.81, range 0.40-0.92). ML performed better than clinical risk scores (Glasgow-Blatchford, Rockall, Child-Pugh, MELD) for mortality in upper GIB. Limitations include heterogeneity of ML models, inconsistent comparisons of ML models with clinical risk scores, and high risk of bias. ML generally provided good-excellent prognostic performance in patients with GIB, and artificial neural networks tended to outperform other ML models. ML was better than clinical risk scores for mortality in upper GIB.