The role of artificial intelligence in hepatology research and practice.

PURPOSE OF REVIEW: The use of artificial intelligence (AI) in examining large data sets has recently gained considerable attention to evaluate disease epidemiology, management approaches, and disease outcomes. The purpose of this review is to summarize the current role of AI in contemporary hepatology practice. RECENT FINDINGS: AI was found to be diagnostically valuable in the evaluation of liver fibrosis, detection of cirrhosis, differentiation between compensated and decompensated cirrhosis, evaluation of portal hypertension, detection and differentiation of particular liver masses, preoperative evaluation of hepatocellular carcinoma as well as response to treatment and estimation of graft survival in patients undergoing liver transplantation. AI additionally holds great promise in examination of structured electronic health records data as well as in examination of clinical text (using various natural language processing approaches). Despite its contributions, AI has several limitations, including the quality of existing data, small cohorts with possible sampling bias and the lack of well validated easily reproducible models. SUMMARY: AI and deep learning models have extensive applicability in assessing liver disease. However, multicenter randomized controlled trials are indispensable to validate their utility.

An AI Approach for Identifying Patients With Cirrhosis.

GOAL: The goal of this study was to evaluate an artificial intelligence approach, namely deep learning, on clinical text in electronic health records (EHRs) to identify patients with cirrhosis. BACKGROUND AND AIMS: Accurate identification of cirrhosis in EHR is important for epidemiological, health services, and outcomes research. Currently, such efforts depend on International Classification of Diseases (ICD) codes, with limited success. MATERIALS AND METHODS: We trained several machine learning models using discharge summaries from patients with known cirrhosis from a patient registry and random controls without cirrhosis or its complications based on ICD codes. Models were validated on patients for whom discharge summaries were manually reviewed and used as the gold standard test set. We tested Naive Bayes and Random Forest as baseline models and a deep learning model using word embedding and a convolutional neural network (CNN). RESULTS: The training set included 446 cirrhosis patients and 689 controls, while the gold standard test set included 139 cirrhosis patients and 152 controls. Among the machine learning models, the CNN achieved the highest area under the receiver operating characteristic curve (0.993), with a precision of 0.965 and recall of 0.978, compared with 0.879 and 0.981 for the Naive Bayes and Random Forest, respectively (precision 0.787 and 0.958, and recalls 0.878 and 0.827). The precision by ICD codes for cirrhosis was 0.883 and recall was 0.978. CONCLUSIONS: A CNN model trained on discharge summaries identified cirrhosis patients with high precision and recall. This approach for phenotyping cirrhosis in the EHR may provide a more accurate assessment of disease burden in a variety of studies.

Accurate Identification of Patients with Cirrhosis and Its Complications in the Electronic Health Record.

BACKGROUND: Cirrhosis represents a significant health burden; administrative data provide an important tool for research studies. AIMS: We aimed to understand the validity of current ICD-10 codes compared to previously used ICD-9 codes to identify patients with cirrhosis and its complications. METHODS: We identified 1981 patients presenting to MUSC between 2013 and 2019 with a diagnosis of cirrhosis. To validate the sensitivity of ICD codes, we reviewed the medical records of 200 patients for each associated ICD 9 and 10 codes. Sensitivity, specificity, and positive predictive value for each ICD code (individually or when combined) were calculated and univariate binary logistic models, for cirrhosis and its complications, predicted probabilities were used to calculate C-statistics. RESULTS: Single ICD 9 and 10 codes were similarly insensitive for detection of cirrhosis, with sensitivity ranging from 5 to 94%. However, ICD-9 code combinations (when used as either/or) had high sensitivity and specificity for the detection of cirrhosis, with the combination of either 571.5 (or 456.21) or 571.2 codes having a C-statistic of 0.975. Combinations of ICD-10 codes were only slightly less sensitive and specific than ICD-9 codes for detection of cirrhosis (K76.6, or K70.31, plus K74.60 or K74.69, and K70.30 had a C-statistic of 0.927). CONCLUSIONS: ICD-9 and ICD-10 codes when used alone were inaccurate for identifying cirrhosis. ICD-10 and ICD-9 codes had similar performance characteristics. Combinations of ICD codes exhibited the greatest sensitivity and specificity for detection of cirrhosis, and thus should be used to accurately identify cirrhosis.

Remote Methods for Conducting Tobacco-Focused Clinical Trials.

Most tobacco-focused clinical trials are based on locally conducted studies that face significant challenges to implementation and successful execution. These challenges include the need for large, diverse, yet still representative study samples. This often means a protracted, costly, and inefficient recruitment process. Multisite clinical trials can overcome some of these hurdles but incur their own unique challenges. With recent advances in mobile health and digital technologies, there is now a promising alternative: Remote Trials. These trials are led and coordinated by a local investigative team, but are based remotely, within a given community, state, or even nation. The remote approach affords many of the benefits of multisite trials (more efficient recruitment of larger study samples) without the same barriers (cost, multisite management, and regulatory hurdles). The Coronavirus Disease 2019 (COVID-19) global health pandemic has resulted in rapid requirements to shift ongoing clinical trials to remote delivery and assessment platforms, making methods for the conduct of remote trials even more timely. The purpose of the present review is to provide an overview of available methods for the conduct of remote tobacco-focused clinical trials as well as illustrative examples of how these methods have been implemented across recently completed and ongoing tobacco studies. We focus on key aspects of the clinical trial pipeline including remote: (1) study recruitment and screening, (2) informed consent, (3) assessment, (4) biomarker collection, and (5) medication adherence monitoring. Implications With recent advances in mobile health and digital technologies, remote trials now offer a promising alternative to traditional in-person clinical trials. Remote trials afford expedient recruitment of large, demographically representative study samples, without undo burden to a research team. The present review provides an overview of available methods for the conduct of remote tobacco-focused clinical trials across key aspects of the clinical trial pipeline.

An Exploration of Useful Telemedicine-Based Resources for Clinical Research.

Background: Clinical trials are key to ensuring high-quality, effective, and safe health care interventions, but there are many barriers to their successful and timely implementation. Difficulties with participant recruitment and enrollment are largely affected by difficulties with obtaining informed consent. Teleconsent is a telemedicine- based approach to obtaining informed consent and offers a unique solution to limitations of traditional consent approaches. Methods: We conducted a survey among 134 clinical trial researchers in academic/university-, industry-, and clinically based settings. The survey addressed important aspects of teleconsent, potential teleconsent enhancements, and other telehealth capabilities to support clinical research. Results: The majority of respondents viewed teleconsent as an important approach for obtaining informed consent and indicated that they would likely use teleconsent if available. Consenting participants at remote sites, increasing access to clinical trials, and consenting participants in their homes were viewed as the greatest opportunities for teleconsent. Features for building, validating, and assessing understanding of teleconsent forms, mobile capabilities, three-way teleconsent calls, and direct links to forms via recruitment websites were viewed as important teleconsent enhancements. Other telehealth capabilities to support clinical research, including surveys, file transfer, three-way video, screenshare, and photo capture during telemedicine visits, and proposed telemedicine capabilities such as video call recording, ID information capture, and integration of medical devices, were also viewed as important. Conclusions: Teleconsent and telemedicine are promising solutions to some common challenges to clinical trials. Many barriers to study recruitment and enrollment might be overcome by investing time and resources and further evaluating this technology.

Automatically identifying social isolation from clinical narratives for patients with prostate Cancer.

BACKGROUND: Social isolation is an important social determinant that impacts health outcomes and mortality among patients. The National Academy of Medicine recently recommended that social isolation be documented in electronic health records (EHR). However, social isolation usually is not recorded or obtained as coded data but rather collected from patient self-report or documented in clinical narratives. This study explores the feasibility and effectiveness of natural language processing (NLP) strategy for identifying patients who are socially isolated from clinical narratives. METHOD: We used data from the Medical University of South Carolina (MUSC) Research Data Warehouse. Patients 18 years-of-age or older who were diagnosed with prostate cancer between January 1, 2014 and May 31, 2017 were eligible for this study. NLP pipelines identifying social isolation were developed via extraction of notes on progress, history and physical, consult, emergency department provider, telephone encounter, discharge summary, plan of care, and radiation oncology. Of 4195 eligible prostate cancer patients, we randomly sampled 3138 patients (75%) as a training dataset. The remaining 1057 patients (25%) were used as a test dataset to evaluate NLP algorithm performance. Standard performance measures for the NLP algorithm, including precision, recall, and F-measure, were assessed by expert manual review using the test dataset. RESULTS: A total of 55,516 clinical notes from 3138 patients were included to develop the lexicon and NLP pipelines for social isolation. Of those, 35 unique patients (1.2%) had social isolation mention(s) in 217 notes. Among 24 terms relevant to social isolation, the most prevalent were "lack of social support," "lonely," "social isolation," "no friends," and "loneliness". Among 1057 patients in the test dataset, 17 patients (1.6%) were identified as having social isolation mention(s) in 40 clinical notes. Manual review identified four false positive mentions of social isolation and one false negatives in 154 notes from randomly selected 52 controls. The NLP pipeline demonstrated 90% precision, 97% recall, and 93% F-measure. The major reasons for a false positive included the ambiguities of the experiencer of social isolation, negation, and alternate meaning of words. CONCLUSIONS: Our NLP algorithms demonstrate a highly accurate approach to identify social isolation.

Correction to: Automatically identifying social isolation from clinical narratives for patients with prostate Cancer.

Following publication of the original article [1], the authors reported an error in one of the authors' names.

Automated detection of altered mental status in emergency department clinical notes: a deep learning approach.

BACKGROUND: Machine learning has been used extensively in clinical text classification tasks. Deep learning approaches using word embeddings have been recently gaining momentum in biomedical applications. In an effort to automate the identification of altered mental status (AMS) in emergency department provider notes for the purpose of decision support, we compare the performance of classic bag-of-words-based machine learning classifiers and novel deep learning approaches. METHODS: We used a case-control study design to extract an adequate number of clinical notes with AMS and non-AMS based on ICD codes. The notes were parsed to extract the history of present illness, which was used as the clinical text for the classifiers. The notes were manually labeled by clinicians. As a baseline for comparison, we tested several traditional bag-of-words based classifiers. We then tested several deep learning models using a convolutional neural network architecture with three different types of word embeddings, a pre-trained word2vec model and two models without pre-training but with different word embedding dimensions. RESULTS: We evaluated the models on 1130 labeled notes from the emergency department. The deep learning models had the best overall performance with an area under the ROC curve of 98.5% and an accuracy of 94.5%. Pre-training word embeddings on the unlabeled corpus reduced training iterations and had performance that was statistically no different than the other deep learning models. CONCLUSION: This supervised deep learning approach performs exceedingly well for the detection of AMS symptoms in clinical text in our environment. Further work is needed for the generalizability of these findings, including evaluation of these models in other types of clinical notes and other environments. The results seem promising for the ultimate use of these types of classifiers in combination with other information derived from the electronic health records as input for clinical decision support.

Word2Vec inversion and traditional text classifiers for phenotyping lupus.

BACKGROUND: Identifying patients with certain clinical criteria based on manual chart review of doctors' notes is a daunting task given the massive amounts of text notes in the electronic health records (EHR). This task can be automated using text classifiers based on Natural Language Processing (NLP) techniques along with pattern recognition machine learning (ML) algorithms. The aim of this research is to evaluate the performance of traditional classifiers for identifying patients with Systemic Lupus Erythematosus (SLE) in comparison with a newer Bayesian word vector method. METHODS: We obtained clinical notes for patients with SLE diagnosis along with controls from the Rheumatology Clinic (662 total patients). Sparse bag-of-words (BOWs) and Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs) matrices were produced using NLP pipelines. These matrices were subjected to several different NLP classifiers: neural networks, random forests, naïve Bayes, support vector machines, and Word2Vec inversion, a Bayesian inversion method. Performance was measured by calculating accuracy and area under the Receiver Operating Characteristic (ROC) curve (AUC) of a cross-validated (CV) set and a separate testing set. RESULTS: We calculated the accuracy of the ICD-9 billing codes as a baseline to be 90.00% with an AUC of 0.900, the shallow neural network with CUIs to be 92.10% with an AUC of 0.970, the random forest with BOWs to be 95.25% with an AUC of 0.994, the random forest with CUIs to be 95.00% with an AUC of 0.979, and the Word2Vec inversion to be 90.03% with an AUC of 0.905. CONCLUSIONS: Our results suggest that a shallow neural network with CUIs and random forests with both CUIs and BOWs are the best classifiers for this lupus phenotyping task. The Word2Vec inversion method failed to significantly beat the ICD-9 code classification, but yielded promising results. This method does not require explicit features and is more adaptable to non-binary classification tasks. The Word2Vec inversion is hypothesized to become more powerful with access to more data. Therefore, currently, the shallow neural networks and random forests are the desirable classifiers.

IRB reliance: An informatics approach.

Multi-site Institutional Review Board (IRB) review of clinical research projects is an important but complex and time-consuming activity that is hampered by disparate non-interoperable computer systems for management of IRB applications. This paper describes our work toward harmonizing the workflow and data model of IRB applications through the development of a software-as-a-service shared-IRB platform for five institutions in South Carolina. Several commonalities and differences were recognized across institutions and a core data model that included the data elements necessary for IRB applications across all institutions was identified. We extended and modified the system to support collaborative reviews of IRB proposals within routine workflows of participating IRBs. Overall about 80% of IRB application content was harmonized across all institutions, establishing the foundation for a streamlined cooperative review and reliance. Since going live in 2011, 49 applications that underwent cooperative reviews over a three year period were approved, with the majority involving 2 out of 5 institutions. We believe this effort will inform future work on a common IRB data model that will allow interoperability through a federated approach for sharing IRB reviews and decisions with the goal of promoting reliance across institutions in the translational research community at large.

Health Information Exchange in the ED: What Do ED Clinicians Think?

OBJECTIVES: Our regional health information exchange (HIE), known as Carolina eHealth Alliance (CeHA)-HIE, serves all major hospital systems in our region and is accessible to emergency department (ED) clinicians in those systems. We wanted to understand reasons for low CeHA-HIE utilization and explore options for improving it. METHODS: We implemented a 24-item user survey among ED clinician users of CeHA-HIE to investigate their perceptions of system usability and functionality, the quality of the information available through CeHA-HIE, the value of clinician time spent using CeHA-HIE, the ease of use of CeHA-HIE, and approaches for improving CeHA-HIE. RESULTS: Of the 231 ED clinicians surveyed, 51 responded, and among those, 48 reported having used CeHA-HIE and completed the survey. CONCLUSIONS: Results show most ED clinicians believed that CeHA-HIE was easy to use and added value to their work, but they also desired better integration of information available from CeHA-HIE into their system's electronic medical record.

When Should ED Physicians Use an HIE? Predicting Presence of Patient Data in an HIE.

OBJECTIVES: Health information exchanges (HIEs) make possible the construction of databases to characterize patients as multisystem users (MSUs), those visiting emergency departments (EDs) of more than one hospital system within a region during a 1-year period. HIE data can inform an algorithm highlighting patients for whom information is more likely to be present in the HIE, leading to a higher yield HIE experience for ED clinicians and incentivizing their adoption of HIE. Our objective was to describe patient characteristics that determine which ED patients are likely to be MSUs and therefore have information in an HIE, thereby improving the efficacy of HIE use and increasing ED clinician perception of HIE benefit. METHODS: Data were extracted from a regional HIE involving four hospital systems (11 EDs) in the Charleston, South Carolina area. We used univariate and multivariable regression analyses to develop a predictive model for MSU status. RESULTS: Factors associated with MSUs included younger age groups, dual-payer insurance status, living in counties that are more rural, and one of at least six specific diagnoses: mental disorders; symptoms, signs, and ill-defined conditions; complications of pregnancy, childbirth, and puerperium; diseases of the musculoskeletal system; injury and poisoning; and diseases of the blood and blood-forming organs. For patients with multiple ED visits during 1 year, 43.8% of MSUs had ≥4 visits, compared with 18.0% of non-MSUs (P < 0.0001). CONCLUSIONS: This predictive model accurately identified patients cared for at multiple hospital systems and can be used to increase the likelihood that time spent logging on to the HIE will be a value-added effort for emergency physicians.

A Comprehensive View of Frequent Emergency Department Users Based on Data from a Regional HIE.

OBJECTIVES: A small but significant number of patients make frequent emergency department (ED) visits to multiple EDs within a region. We have a unique health information exchange (HIE) that includes every ED encounter in all hospital systems in our region. Using our HIE we were able to characterize all frequent ED users in our region, regardless of hospital visited or payer class. The objective of our study was to use data from an HIE to characterize patients in a region who are frequent ED users (FEDUs). METHODS: We constructed a database from a cohort of adult patients (18 years old or older) with information in a regional HIE for a 1-year period beginning in April 2012. Patients were defined as FEDUs (those who made four or more visits during the study period) and non-FEDUs (those who made fewer than four ED visits during the study period). Predictor variables included age, race, sex, payer class, county of residence, and International Classification of Diseases, Ninth Revision codes. Bivariate (χ(2)) and multivariate (logistic regression) analyses were performed to determine associations between predictor variables and the outcome of being a FEDU. RESULTS: The database contained 127,672 patients, 12,293 (9.6%) of whom were FEDUs. Logistic regression showed the following patient characteristics to be significantly associated with the outcome of being a FEDU: age 35 to 44 years; African American race; Medicaid, Medicare, and dual-pay payer class; and International Classification of Diseases, Ninth Revision codes 630 to 679 (complications of pregnancy, childbirth, and puerperium), 780 to 799 (ill-defined conditions), 280 to 289 (diseases of the blood), 290-319 (mental disorders), 680 to 709 (diseases of the skin and subcutaneous tissue), 710 to 739 (musculoskeletal and connective tissue disease), 460 to 519 (respiratory disease), and 520 to 579 (digestive disease). No significant differences were noted between men and women. CONCLUSIONS: Data from an HIE can be used to describe all of the patients within a region who are FEDUs, regardless of the hospital system they visited. This information can be used to focus care coordination efforts and link appropriate patients to a medical home. Future studies can be designed to learn the reasons why patients become FEDUs, and interventions can be developed to address deficiencies in health care that result in frequent ED visits.

A Reproducible Model Based on Clinical Text for Predicting Suicidal Behavior.

Suicide risk models are critical for prioritizing patients for intervention. We demonstrate a reproducible approach for training text classifiers to identify patients at risk. The models were effective in phenotyping suicidal behavior (F1=.94) and moderately effective in predicting future events (F1=.63).

Procurement of shared data instruments for Research Electronic Data Capture (REDCap).

REDCap (Research Electronic Data Capture) is a web-based software solution and tool set that allows biomedical researchers to create secure online forms for data capture, management and analysis with minimal effort and training. The Shared Data Instrument Library (SDIL) is a relatively new component of REDCap that allows sharing of commonly used data collection instruments for immediate study use by research teams. Objectives of the SDIL project include: (1) facilitating reuse of data dictionaries and reducing duplication of effort; (2) promoting the use of validated data collection instruments, data standards and best practices; and (3) promoting research collaboration and data sharing. Instruments submitted to the library are reviewed by a library oversight committee, with rotating membership from multiple institutions, which ensures quality, relevance and legality of shared instruments. The design allows researchers to download the instruments in a consumable electronic format in the REDCap environment. At the time of this writing, the SDIL contains over 128 data collection instruments. Over 2500 instances of instruments have been downloaded by researchers at multiple institutions. In this paper we describe the library platform, provide detail about experience gained during the first 25months of sharing public domain instruments and provide evidence of impact for the SDIL across the REDCap consortium research community. We postulate that the shared library of instruments reduces the burden of adhering to sound data collection principles while promoting best practices.

Managing clinical research permissions electronically: A novel approach to enhancing recruitment and managing consents.

BACKGROUND: One mechanism to increase participation in research is to solicit potential research participants' general willingness to be recruited into clinical trials. Such research permissions and consents typically are collected on paper upon patient registration. We describe a novel method of capturing this information electronically. PURPOSE: The objective is to enable the collection of research permissions and informed consent data electronically to permit tracking of potential research participants' interest in current and future research involvement and to provide a foundation for facilitating the research workflow. METHODS: The project involved systematic analysis focused on key areas, including existing business practices, registration processes, and permission collection workflows, and ascertaining best practices for presenting consent information to users via tablet technology and capturing permissions data. Analysis was followed by an iterative software development cycle with feedback from subject matter experts and users. RESULTS: An initial version of the software was piloted at one institution in South Carolina for a period of 1 year, during which consents and permission were collected during 2524 registrations of patients. The captured research permission data were transmitted to a clinical data warehouse. The software was later released as an open-source package that can be adopted for use by other institutions. LIMITATIONS: There are significant ethical, legal, and informatics challenges that must be addressed at an institution to deploy such a system. We have not yet assessed the long-term impact of the system on recruitment of patients to clinical trials. CONCLUSIONS: We propose that by improving the ability to track willing potential research participants, we can improve recruitment into clinical trials and, in the process, improve patient education by introducing multimedia to informed consent documents.

Red Flag/Blue Flag visualization of a common CNN for text classification.

A shallow convolutional neural network (CNN), TextCNN, has become nearly ubiquitous for classification among clinical and medical text. This research presents a novel eXplainable-AI (X-AI) software, Red Flag/Blue Flag (RFBF), designed for binary classification with TextCNN. RFBF visualizes each convolutional filter's discriminative capability. This is a more informative approach than direct assessment of logit contribution, features that overfit to train set nuances on smaller datasets may indiscriminately activate large logits on validation samples from both classes. RFBF enables model diagnosis, term feature verification, and overfit prevention. We present 3 use cases of (1) filter consistency assessment; (2) predictive performance improvement; and (3) estimation of information leakage between train and holdout sets. The use cases derive from experiments on TextCNN for binary prediction of surgical misadventure outcomes from physician-authored operative notes. Due to TextCNN's prevalence, this X-AI can benefit clinical text research, and hence improve patient outcomes.

The effectiveness of a noninterruptive alert to increase prescription of take-home naloxone in emergency departments.

OBJECTIVE: Opioid-related overdose (OD) deaths continue to increase. Take-home naloxone (THN), after treatment for an OD in an emergency department (ED), is a recommended but under-utilized practice. To promote THN prescription, we developed a noninterruptive decision support intervention that combined a detailed OD documentation template with a reminder to use the template that is automatically inserted into a provider's note by decision rules. We studied the impact of the combined intervention on THN prescribing in a longitudinal observational study. METHODS: ED encounters involving an OD were reviewed before and after implementation of the reminder embedded in the physicians' note to use an advanced OD documentation template for changes in: (1) use of the template and (2) prescription of THN. Chi square tests and interrupted time series analyses were used to assess the impact. Usability and satisfaction were measured using the System Usability Scale (SUS) and the Net Promoter Score. RESULTS: In 736 OD cases defined by International Classification of Disease version 10 diagnosis codes (247 prereminder and 489 postreminder), the documentation template was used in 0.0% and 21.3%, respectively (P < .0001). The sensitivity and specificity of the reminder for OD cases were 95.9% and 99.8%, respectively. Use of the documentation template led to twice the rate of prescribing of THN (25.7% vs 50.0%, P < .001). Of 19 providers responding to the survey, 74% of SUS responses were in the good-to-excellent range and 53% of providers were Net Promoters. CONCLUSIONS: A noninterruptive decision support intervention was associated with higher THN prescribing in a pre-post study across a multiinstitution health system.

Scrybe: A Secure Audit Trail for Clinical Trial Data Fusion.

Clinical trials are a multi-billion dollar industry. One of the biggest challenges facing the clinical trial research community is satisfying Part 11 of Title 21 of the Code of Federal Regulations [7] and ISO 27789 [40]. These controls provide audit requirements that guarantee the reliability of the data contained in the electronic records. Context-aware smart devices and wearable IoT devices have become increasingly common in clinical trials. Electronic Data Capture (EDC) and Clinical Data Management Systems (CDMS) do not currently address the new challenges introduced using these devices. The healthcare digital threat landscape is continually evolving, and the prevalence of sensor fusion and wearable devices compounds the growing attack surface. We propose Scrybe, a permissioned blockchain, to store proof of clinical trial data provenance. We illustrate how Scrybe addresses each control and the limitations of the Ethereum-based blockchains. Finally, we provide a proof-of-concept integration with REDCap to show tamper resistance.

Enhanced phenotypes for identifying opioid overdose in emergency department visit electronic health record data.

Background: Accurate identification of opioid overdose (OOD) cases in electronic healthcare record (EHR) data is an important element in surveillance, empirical research, and clinical intervention. We sought to improve existing OOD electronic phenotypes by incorporating new data types beyond diagnostic codes and by applying several statistical and machine learning methods. Materials and Methods: We developed an EHR dataset of emergency department visits involving OOD cases or patients considered at risk for an OOD and ascertained true OOD status through manual chart reviews. We developed and validated prediction models using Random Forest, Extreme Gradient Boost, and Elastic Net models that incorporated 717 features involving primary and second diagnoses, chief complaints, medications prescribed, vital signs, laboratory results, and procedural codes. We also developed models limited to single data types. Results: A total of 1718 records involving 1485 patients were manually reviewed; 541 (36.4%) patients had one or more OOD. Prediction performance was similar for all models; sensitivity varied from 94% to 97%; and area under the receiver operating characteristic curve (AUC) was 98% for all methods. The primary diagnosis and chief complaint were the most important contributors to AUC performance; primary diagnoses and medication class contributed most to sensitivity; chief complaint, primary diagnosis, and vital signs were most important for specificity. Models limited to decision support data types available in real time demonstrated robust prediction performance. Conclusions: Substantial prediction performance improvements were demonstrated for identifying OODs in EHR data. Our e-phenotypes could be applied in surveillance, retrospective empirical applications, or clinical decision support systems.