Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing.

OBJECTIVE: Leverage electronic health record (EHR) audit logs to develop a machine learning (ML) model that predicts which notes a clinician wants to review when seeing oncology patients. MATERIALS AND METHODS: We trained logistic regression models using note metadata and a Term Frequency Inverse Document Frequency (TF-IDF) text representation. We evaluated performance with precision, recall, F1, AUC, and a clinical qualitative assessment. RESULTS: The metadata only model achieved an AUC 0.930 and the metadata and TF-IDF model an AUC 0.937. Qualitative assessment revealed a need for better text representation and to further customize predictions for the user. DISCUSSION: Our model effectively surfaces the top 10 notes a clinician wants to review when seeing an oncology patient. Further studies can characterize different types of clinician users and better tailor the task for different care settings. CONCLUSION: EHR audit logs can provide important relevance data for training ML models that assist with note-writing in the oncology setting.

MIMIC-IV, a freely accessible electronic health record dataset.

Digital data collection during routine clinical practice is now ubiquitous within hospitals. The data contains valuable information on the care of patients and their response to treatments, offering exciting opportunities for research. Typically, data are stored within archival systems that are not intended to support research. These systems are often inaccessible to researchers and structured for optimal storage, rather than interpretability and analysis. Here we present MIMIC-IV, a publicly available database sourced from the electronic health record of the Beth Israel Deaconess Medical Center. Information available includes patient measurements, orders, diagnoses, procedures, treatments, and deidentified free-text clinical notes. MIMIC-IV is intended to support a wide array of research studies and educational material, helping to reduce barriers to conducting clinical research.

Secondary Use of COVID-19 Symptom Incidence Among Hospital Employees as an Example of Syndromic Surveillance of Hospital Admissions Within 7 Days.

Importance: Alternative methods for hospital occupancy forecasting, essential information in hospital crisis planning, are necessary in a novel pandemic when traditional data sources such as disease testing are limited. Objective: To determine whether mandatory daily employee symptom attestation data can be used as syndromic surveillance to estimate COVID-19 hospitalizations in the communities where employees live. Design, Setting, and Participants: This cohort study was conducted from April 2, 2020, to November 4, 2020, at a large academic hospital network of 10 hospitals accounting for a total of 2384 beds and 136 000 discharges in New England. The participants included 6841 employees who worked on-site at hospital 1 and lived in the 10 hospitals' service areas. Exposure: Daily employee self-reported symptoms were collected using an automated text messaging system from a single hospital. Main Outcomes and Measures: Mean absolute error (MAE) and weighted mean absolute percentage error (MAPE) of 7-day forecasts of daily COVID-19 hospital census at each hospital. Results: Among 6841 employees living within the 10 hospitals' service areas, 5120 (74.8%) were female individuals and 3884 (56.8%) were White individuals; the mean (SD) age was 40.8 (13.6) years, and the mean (SD) time of service was 8.8 (10.4) years. The study model had a MAE of 6.9 patients with COVID-19 and a weighted MAPE of 1.5% for hospitalizations for the entire hospital network. The individual hospitals had an MAE that ranged from 0.9 to 4.5 patients (weighted MAPE ranged from 2.1% to 16.1%). For context, the mean network all-cause occupancy was 1286 during this period, so an error of 6.9 is only 0.5% of the network mean occupancy. Operationally, this level of error was negligible to the incident command center. At hospital 1, a doubling of the number of employees reporting symptoms (which corresponded to 4 additional employees reporting symptoms at the mean for hospital 1) was associated with a 5% increase in COVID-19 hospitalizations at hospital 1 in 7 days (regression coefficient, 0.05; 95% CI, 0.02-0.07; P < .001). Conclusions and Relevance: This cohort study found that a real-time employee health attestation tool used at a single hospital could be used to estimate subsequent hospitalizations in 7 days at hospitals throughout a larger hospital network in New England.

A visual representation of microbiological culture data improves comprehension: a randomized controlled trial.

OBJECTIVE: While the judicious use of antibiotics takes past microbiological culture results into consideration, this data's typical format in the electronic health record (EHR) may be unwieldy when incorporated into clinical decision-making. We hypothesize that a visual representation of sensitivities may aid in their comprehension. MATERIALS AND METHODS: A prospective parallel unblinded randomized controlled trial was undertaken at an academic urban tertiary care center. Providers managing emergency department (ED) patients receiving antibiotics and having previous culture sensitivity testing were included. Providers were randomly selected to use standard EHR functionality or a visual representation of patients' past culture data as they answered questions about previous sensitivities. Concordance between provider responses and past cultures was assessed using the kappa statistic. Providers were surveyed about their decision-making and the usability of the tool using Likert scales. RESULTS: 518 ED encounters were screened from 3/5/2018 to 9/30/18, with providers from 144 visits enrolled and analyzed in the intervention arm and 129 in the control arm. Providers using the visualization tool had a kappa of 0.69 (95% CI: 0.65-0.73) when asked about past culture results while the control group had a kappa of 0.16 (95% CI: 0.12-0.20). Providers using the tool expressed improved understanding of previous cultures and found the tool easy to use (P < .001). Secondary outcomes showed no differences in prescribing practices. CONCLUSION: A visual representation of culture sensitivities improves comprehension when compared to standard text-based representations.

Deep Learning to Quantify Pulmonary Edema in Chest Radiographs.

PURPOSE: To develop a machine learning model to classify the severity grades of pulmonary edema on chest radiographs. MATERIALS AND METHODS: In this retrospective study, 369 071 chest radiographs and associated radiology reports from 64 581 patients (mean age, 51.71 years; 54.51% women) from the MIMIC-CXR chest radiograph dataset were included. This dataset was split into patients with and without congestive heart failure (CHF). Pulmonary edema severity labels from the associated radiology reports were extracted from patients with CHF as four different ordinal levels: 0, no edema; 1, vascular congestion; 2, interstitial edema; and 3, alveolar edema. Deep learning models were developed using two approaches: a semisupervised model using a variational autoencoder and a pretrained supervised learning model using a dense neural network. Receiver operating characteristic curve analysis was performed on both models. RESULTS: The area under the receiver operating characteristic curve (AUC) for differentiating alveolar edema from no edema was 0.99 for the semisupervised model and 0.87 for the pretrained models. Performance of the algorithm was inversely related to the difficulty in categorizing milder states of pulmonary edema (shown as AUCs for semisupervised model and pretrained model, respectively): 2 versus 0, 0.88 and 0.81; 1 versus 0, 0.79 and 0.66; 3 versus 1, 0.93 and 0.82; 2 versus 1, 0.69 and 0.73; and 3 versus 2, 0.88 and 0.63. CONCLUSION: Deep learning models were trained on a large chest radiograph dataset and could grade the severity of pulmonary edema on chest radiographs with high performance.Supplemental material is available for this article.See also the commentary by Auffermann in this issue.© RSNA, 2021.

Multimodal Representation Learning via Maximization of Local Mutual Information.

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.

Development and validation of a pancreatic cancer risk model for the general population using electronic health records: An observational study.

AIM: Pancreatic ductal adenocarcinoma (PDAC) is often diagnosed at a late, incurable stage. We sought to determine whether individuals at high risk of developing PDAC could be identified early using routinely collected data. METHODS: Electronic health record (EHR) databases from two independent hospitals in Boston, Massachusetts, providing inpatient, outpatient, and emergency care, from 1979 through 2017, were used with case-control matching. PDAC cases were selected using International Classification of Diseases 9/10 codes and validated with tumour registries. A data-driven feature selection approach was used to develop neural networks and L2-regularised logistic regression (LR) models on training data (594 cases, 100,787 controls) and compared with a published model based on hand-selected diagnoses ('baseline'). Model performance was validated on an external database (408 cases, 160,185 controls). Three prediction lead times (180, 270 and 365 days) were considered. RESULTS: The LR model had the best performance, with an area under the curve (AUC) of 0.71 (confidence interval [CI]: 0.67-0.76) for the training set, and AUC 0.68 (CI: 0.65-0.71) for the validation set, 365 days before diagnosis. Data-driven feature selection improved results over 'baseline' (AUC = 0.55; CI: 0.52-0.58). The LR model flags 2692 (CI 2592-2791) of 156,485 as high risk, 365 days in advance, identifying 25 (CI: 16-36) cancer patients. Risk stratification showed that the high-risk group presented a cancer rate 3 to 5 times the prevalence in our data set. CONCLUSION: A simple EHR model, based on diagnoses, can identify high-risk individuals for PDAC up to one year in advance. This inexpensive, systematic approach may serve as the first sieve for selection of individuals for PDAC screening programs.

Risk of mortality and cardiopulmonary arrest in critical patients presenting to the emergency department using machine learning and natural language processing.

Emergency department triage is the first point in time when a patient's acuity level is determined. The time to assign a priority at triage is short and it is vital to accurately stratify patients at this stage, since under-triage can lead to increased morbidity, mortality and costs. Our aim was to present a model that can assist healthcare professionals in triage decision making, namely in the stratification of patients through the risk prediction of a composite critical outcome-mortality and cardiopulmonary arrest. Our study cohort consisted of 235826 adult patients triaged at a Portuguese Emergency Department from 2012 to 2016. Patients were assigned to emergent, very urgent or urgent priorities of the Manchester Triage System (MTS). Demographics, clinical variables routinely collected at triage and the patients' chief complaint were used. Logistic regression, random forests and extreme gradient boosting were developed using all available variables. The term frequency-inverse document frequency (TF-IDF) natural language processing weighting factor was applied to vectorize the chief complaint. Stratified random sampling was used to split the data into train (70%) and test (30%) data sets. Ten-fold cross validation was performed in train to optimize model hyper-parameters. The performance obtained with the best model was compared against the reference model-a regularized logistic regression trained using only triage priorities. Extreme gradient boosting exhibited good calibration properties and yielded areas under the receiver operating characteristic and precision-recall curves of 0.96 (95% CI 0.95-0.97) and 0.31 (95% CI 0.26-0.36), respectively. The predictors ranked with higher importance by this model were the Glasgow coma score, the patients' age, pulse oximetry and arrival mode. Compared to the reference, the extreme gradient boosting model using clinical variables and the chief complaint presented higher recall for patients assigned MTS-3 and can identify those who are at risk of the composite outcome.

Predicting Intensive Care Unit admission among patients presenting to the emergency department using machine learning and natural language processing.

The risk stratification of patients in the emergency department begins at triage. It is vital to stratify patients early based on their severity, since undertriage can lead to increased morbidity, mortality and costs. Our aim was to present a new approach to assist healthcare professionals at triage in the stratification of patients and in identifying those with higher risk of ICU admission. Adult patients assigned Manchester Triage System (MTS) or Emergency Severity Index (ESI) 1 to 3 from a Portuguese and a United States Emergency Departments were analyzed. Variables routinely collected at triage were used and natural language processing was applied to the patient chief complaint. Stratified random sampling was applied to split the data in train (70%) and test (30%) sets and 10-fold cross validation was performed for model training. Logistic regression, random forests, and a random undersampling boosting algorithm were used. We compared the performance obtained with the reference model-using only triage priorities-with the models using additional variables. For both hospitals, a logistic regression model achieved higher overall performance, yielding areas under the receiver operating characteristic and precision-recall curves of 0.91 (95% CI 0.90-0.92) and 0.30 (95% CI 0.27-0.33) for the United States hospital and of 0.85 (95% CI 0.83-0.86) and 0.06 (95% CI 0.05-0.07) for the Portuguese hospital. Heart rate, pulse oximetry, respiratory rate and systolic blood pressure were the most important predictors of ICU admission. Compared to the reference models, the models using clinical variables and the chief complaint presented higher recall for patients assigned MTS/ESI 3 and can identify patients assigned MTS/ESI 3 who are at risk for ICU admission.

Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment.

We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time. Our experimental results suggest that the joint image-text representation learning improves the performance of pulmonary edema assessment compared to a supervised model trained on images only. We also show the use of the text for explaining the image classification by the joint model. To the best of our knowledge, our approach is the first to leverage free-text radiology reports for improving the image model performance in this application. Our code is available at: https://github.com/RayRuizhiLiao/joint_chestxray.

Derivation and validation of a machine learning record linkage algorithm between emergency medical services and the emergency department.

OBJECTIVE: Linking emergency medical services (EMS) electronic patient care reports (ePCRs) to emergency department (ED) records can provide clinicians access to vital information that can alter management. It can also create rich databases for research and quality improvement. Unfortunately, previous attempts at ePCR and ED record linkage have had limited success. In this study, we use supervised machine learning to derive and validate an automated record linkage algorithm between EMS ePCRs and ED records. MATERIALS AND METHODS: All consecutive ePCRs from a single EMS provider between June 2013 and June 2015 were included. A primary reviewer matched ePCRs to a list of ED patients to create a gold standard. Age, gender, last name, first name, social security number, and date of birth were extracted. Data were randomly split into 80% training and 20% test datasets. We derived missing indicators, identical indicators, edit distances, and percent differences. A multivariate logistic regression model was trained using 5-fold cross-validation, using label k-fold, L2 regularization, and class reweighting. RESULTS: A total of 14 032 ePCRs were included in the study. Interrater reliability between the primary and secondary reviewer had a kappa of 0.9. The algorithm had a sensitivity of 99.4%, a positive predictive value of 99.9%, and an area under the receiver-operating characteristic curve of 0.99 in both the training and test datasets. Date-of-birth match had the highest odds ratio of 16.9, followed by last name match (10.6). Social security number match had an odds ratio of 3.8. CONCLUSIONS: We were able to successfully derive and validate a record linkage algorithm from a single EMS ePCR provider to our hospital EMR.

Robustly Extracting Medical Knowledge from EHRs: A Case Study of Learning a Health Knowledge Graph.

Increasingly large electronic health records (EHRs) provide an opportunity to algorithmically learn medical knowledge. In one prominent example, a causal health knowledge graph could learn relationships between diseases and symptoms and then serve as a diagnostic tool to be refined with additional clinical input. Prior research has demonstrated the ability to construct such a graph from over 270,000 emergency department patient visits. In this work, we describe methods to evaluate a health knowledge graph for robustness. Moving beyond precision and recall, we analyze for which diseases and for which patients the graph is most accurate. We identify sample size and unmeasured confounders as major sources of error in the health knowledge graph. We introduce a method to leverage non-linear functions in building the causal graph to better understand existing model assumptions. Finally, to assess model generalizability, we extend to a larger set of complete patient visits within a hospital system. We conclude with a discussion on how to robustly extract medical knowledge from EHRs.

Assessment of Unintentional Duplicate Orders by Emergency Department Clinicians Before and After Implementation of a Visual Aid in the Electronic Health Record Ordering System.

Importance: Electronic health records allow teams of clinicians to simultaneously care for patients, but an unintended consequence is the potential for duplicate orders of tests and medications. Objective: To determine whether a simple visual aid is associated with a reduction in duplicate ordering of tests and medications. Design, Setting, and Participants: This cohort study used an interrupted time series model to analyze 184 694 consecutive patients who visited the emergency department (ED) of an academic hospital with 55 000 ED visits annually. Patient visits occurred 1 year before and after each intervention, as follows: for laboratory orders, from August 13, 2012, to August 13, 2014; for medication orders, from February 3, 2013, to February 3, 2015; and for radiology orders, from December 12, 2013, to December 12, 2015. Data were analyzed from April to September 2019. Exposure: If an order had previously been placed during the ED visit, a red highlight appeared around the checkbox of that order in the computerized provider order entry system. Main Outcomes and Measures: Number of unintentional duplicate laboratory, medication, and radiology orders. Results: A total of 184 694 patients (mean [SD] age, 51.6 [20.8] years; age range, 0-113.0 years; 99 735 [54.0%] women) who visited the ED were analyzed over the 3 overlapping study periods. After deployment of a noninterruptive nudge in electronic health records, there was an associated 49% decrease in the rate of unintentional duplicate orders for laboratory tests (incidence rate ratio, 0.51; 95% CI, 0.45-0.59), from 4485 to 2731 orders, and an associated 40% decrease in unintentional duplicate orders of radiology tests (incidence rate ratio, 0.60; 95% CI, 0.44-0.82), from 956 to 782 orders. There was not a statistically significant change in unintentional duplicate orders of medications (incidence rate ratio, 1.17; 95% CI, 0.52-2.61), which increased from 225 to 287 orders. The nudge eliminated an estimated 17 936 clicks in our electronic health record. Conclusions and Relevance: In this interrupted time series cohort study, passive visual cues that provided just-in-time decision support were associated with reductions in unintentional duplicate orders for laboratory and radiology tests but not in unintentional duplicate medication orders.

MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.

Chest radiography is an extremely powerful imaging modality, allowing for a detailed inspection of a patient's chest, but requires specialized training for proper interpretation. With the advent of high performance general purpose computer vision algorithms, the accurate automated analysis of chest radiographs is becoming increasingly of interest to researchers. Here we describe MIMIC-CXR, a large dataset of 227,835 imaging studies for 65,379 patients presenting to the Beth Israel Deaconess Medical Center Emergency Department between 2011-2016. Each imaging study can contain one or more images, usually a frontal view and a lateral view. A total of 377,110 images are available in the dataset. Studies are made available with a semi-structured free-text radiology report that describes the radiological findings of the images, written by a practicing radiologist contemporaneously during routine clinical care. All images and reports have been de-identified to protect patient privacy. The dataset is made freely available to facilitate and encourage a wide range of research in computer vision, natural language processing, and clinical data mining.

Improving documentation of presenting problems in the emergency department using a domain-specific ontology and machine learning-driven user interfaces.

OBJECTIVES: To determine the effect of a domain-specific ontology and machine learning-driven user interfaces on the efficiency and quality of documentation of presenting problems (chief complaints) in the emergency department (ED). METHODS: As part of a quality improvement project, we simultaneously implemented three interventions: a domain-specific ontology, contextual autocomplete, and top five suggestions. Contextual autocomplete is a user interface that ranks concepts by their predicted probability which helps nurses enter data about a patient's presenting problems. Nurses were also given a list of top five suggestions to choose from. These presenting problems were represented using a consensus ontology mapped to SNOMED CT. Predicted probabilities were calculated using a previously derived model based on triage vital signs and a brief free text note. We evaluated the percentage and quality of structured data captured using a mixed methods retrospective before-and-after study design. RESULTS: A total of 279,231 consecutive patient encounters were analyzed. Structured data capture improved from 26.2% to 97.2% (p < 0.0001). During the post-implementation period, presenting problems were more complete (3.35 vs 3.66; p = 0.0004) and higher in overall quality (3.38 vs. 3.72; p = 0.0002), but showed no difference in precision (3.59 vs. 3.74; p = 0.1). Our system reduced the mean number of keystrokes required to document a presenting problem from 11.6 to 0.6 (p < 0.0001), a 95% improvement. DISCUSSION: We demonstrated a technique that captures structured data on nearly all patients. We estimate that our system reduces the number of man-hours required annually to type presenting problems at our institution from 92.5 h to 4.8 h. CONCLUSION: Implementation of a domain-specific ontology and machine learning-driven user interfaces resulted in improved structured data capture, ontology usage compliance, and data quality.

Consensus Development of a Modern Ontology of Emergency Department Presenting Problems-The Hierarchical Presenting Problem Ontology (HaPPy).

OBJECTIVE: Numerous attempts have been made to create a standardized "presenting problem" or "chief complaint" list to characterize the nature of an emergency department visit. Previous attempts have failed to gain widespread adoption as they were not freely shareable or did not contain the right level of specificity, structure, and clinical relevance to gain acceptance by the larger emergency medicine community. Using real-world data, we constructed a presenting problem list that addresses these challenges. MATERIALS AND METHODS: We prospectively captured the presenting problems for 180,424 consecutive emergency department patient visits at an urban, academic, Level I trauma center in the Boston metro area. No patients were excluded. We used a consensus process to iteratively derive our system using real-world data. We used the first 70% of consecutive visits to derive our ontology, followed by a 6-month washout period, and the remaining 30% for validation. All concepts were mapped to Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT). RESULTS: Our system consists of a polyhierarchical ontology containing 692 unique concepts, 2,118 synonyms, and 30,613 nonvisible descriptions to correct misspellings and nonstandard terminology. Our ontology successfully captured structured data for 95.9% of visits in our validation data set. DISCUSSION AND CONCLUSION: We present the HierArchical Presenting Problem ontologY (HaPPy). This ontology was empirically derived and then iteratively validated by an expert consensus panel. HaPPy contains 692 presenting problem concepts, each concept being mapped to SNOMED CT. This freely sharable ontology can help to facilitate presenting problem-based quality metrics, research, and patient care.

Mobile device ownership among emergency department patients.

BACKGROUND: The landscape of mobile devices is changing and their present use by patients for healthcare purposes is unknown. An understanding of current attitudes and usage may help increase patient engagement through mobile applications. This study sought to determine characteristics of mobile device ownership among Emergency Department patients, patients' feelings regarding their use in healthcare, and desired functionality in mobile applications. METHODS: A cross-sectional survey was undertaken at a single urban tertiary care academic center. A convenience sample of adult English-speaking patients in the Emergency Department were surveyed from June 21 st, 2017 to December 30th, 2017. A secondary analysis of the data was performed based on demographic and socioeconomic factors. RESULTS: 260 patients were approached for participation, 11 patients declined, and one patient was excluded. The 248 participants had a median age of 49 (interquartile range 28-62) and 54% were female. 91% of those surveyed own smartphones, 58% owned tablets, and 77% of these patients were comfortable using mobile devices. Those without mobile devices were older (p < 0.001) and held less commercial insurance (p = 0.01). A majority of patients were interested in using applications to enter information, track their visit, view results, and communicate with providers during their visit. Following care, there is interest in viewing information about their visit and receiving reminders for appointments and medications. Patients are also interested in using applications for learning about medical conditions and managing medications. Though there are mixed feelings regarding the protection of privacy by apps, they are felt to be safe, effective, useful, and not difficult to use. CONCLUSION: Ownership of smartphones is high across the Emergency Department population and patients are enthusiastic about using mobile devices as part of their care. Further study can elucidate opportunities to further integrate mobile device applications into patient care.