Antibiotic Exposure and Risk for Hospital-Associated Clostridioides difficile Infection.

Clostridioides difficile infection (CDI) is a health care-associated infection associated with significant morbidity and cost, with highly varied risk across populations. More effective, risk-based prevention strategies are needed. Here, we investigate risk factors for hospital-associated CDI in a large integrated health system. In a retrospective cohort of all adult admissions to 21 Intermountain Healthcare hospitals from 2006 to 2012, we identified all symptomatic (i) hospital-onset and (ii) health care-facility-associated, community-onset CDI. We then evaluated the risk associated with antibiotic exposure, including that of specific agents, using multivariable logistic regression. A total of 2,356 cases of CDI among 506,068 admissions were identified (incidence, 46.6 per 10,000). Prior antibiotic use was the dominant risk factor, where for every antibiotic day of therapy prior to the index admission, the odds of subsequent CDI increased by 12.8% (95% confidence interval [CI], 12.2 to 13.4%; P < 0.0001). This was a much stronger association than was inpatient antibiotic exposure (odds ratio [OR], 1.007 [95% CI, 1.005 to 1.009]; P < 0.0001). The highest-risk antibiotics included second-generation and later cephalosporins (especially oral), carbapenems, fluoroquinolones, and clindamycin, while doxycycline and daptomycin were associated with a lower CDI risk. We concluded that cumulative antibiotic exposure prior to admission is the greatest contributor to the risk of subsequent CDI. Most classes of antibiotics carry some risk, which varies by drug and route. This information may be useful for antimicrobial stewardship efforts.

Portable Automated Surveillance of Surgical Site Infections Using Natural Language Processing: Development and Validation.

OBJECTIVES: We present the development and validation of a portable NLP approach for automated surveillance of SSIs. SUMMARY OF BACKGROUND DATA: The surveillance of SSIs is labor-intensive limiting the generalizability and scalability of surgical quality surveillance programs. METHODS: We abstracted patient clinical text notes after surgical procedures from 2 independent healthcare systems using different electronic healthcare records. An SSI detected as part of the American College of Surgeons' National Surgical Quality Improvement Program was used as the reference standard. We developed a rules-based NLP system (Easy Clinical Information Extractor [CIE]-SSI) for operative event-level detection of SSIs using an training cohort (4574 operative events) from 1 healthcare system and then conducted internal validation on a blind cohort from the same healthcare system (1850 operative events) and external validation on a blind cohort from the second healthcare system (15,360 operative events). EasyCIE-SSI performance was measured using sensitivity, specificity, and area under the receiver-operating-curve (AUC). RESULTS: The prevalence of SSI was 4% and 5% in the internal and external validation corpora. In internal validation, EasyCIE-SSI had a sensitivity, specificity, AUC of 94%, 88%, 0.912 for the detection of SSI, respectively. In external validation, EasyCIE-SSI had sensitivity, specificity, AUC of 79%, 92%, 0.852 for the detection of SSI, respectively. The sensitivity of EasyCIE-SSI decreased in clean, skin/subcutaneous, and outpatient procedures in the external validation compared to internal validation. CONCLUSION: Automated surveillance of SSIs can be achieved using NLP of clinical notes with high sensitivity and specificity.

Driving pressure is not associated with mortality in mechanically ventilated patients without ARDS.

BACKGROUND: In patients with acute respiratory distress syndrome (ARDS), low tidal volume ventilation has been associated with reduced mortality. Driving pressure (tidal volume normalized to respiratory system compliance) may be an even stronger predictor of ARDS survival than tidal volume. We sought to study whether these associations hold true in acute respiratory failure patients without ARDS. METHODS: This is a retrospectively cohort analysis of mechanically ventilated adult patients admitted to ICUs from 12 hospitals over 2 years. We used natural language processing of chest radiograph reports and data from the electronic medical record to identify patients who had ARDS. We used multivariable logistic regression and generalized linear models to estimate associations between tidal volume, driving pressure, and respiratory system compliance with adjusted 30-day mortality using covariates of Acute Physiology Score (APS), Charlson Comorbidity Index (CCI), age, and PaO2/FiO2 ratio. RESULTS: We studied 2641 patients; 48% had ARDS (n = 1273). Patients with ARDS had higher mean APS (25 vs. 23, p < .001) but similar CCI (4 vs. 3, p = 0.6) scores. For non-ARDS patients, tidal volume was associated with increased adjusted mortality (OR 1.18 per 1 mL/kg PBW increase in tidal volume, CI 1.04 to 1.35, p = 0.010). We observed no association between driving pressure or respiratory compliance and mortality in patients without ARDS. In ARDS patients, both ΔP (OR1.1, CI 1.06-1.14, p < 0.001) and tidal volume (OR 1.17, CI 1.04-1.31, p = 0.007) were associated with mortality. CONCLUSIONS: In a large retrospective analysis of critically ill non-ARDS patients receiving mechanical ventilation, we found that tidal volume was associated with 30-day mortality, while driving pressure was not.

A Bayesian system to detect and characterize overlapping outbreaks.

Outbreaks of infectious diseases such as influenza are a significant threat to human health. Because there are different strains of influenza which can cause independent outbreaks, and influenza can affect demographic groups at different rates and times, there is a need to recognize and characterize multiple outbreaks of influenza. This paper describes a Bayesian system that uses data from emergency department patient care reports to create epidemiological models of overlapping outbreaks of influenza. Clinical findings are extracted from patient care reports using natural language processing. These findings are analyzed by a case detection system to create disease likelihoods that are passed to a multiple outbreak detection system. We evaluated the system using real and simulated outbreaks. The results show that this approach can recognize and characterize overlapping outbreaks of influenza. We describe several extensions that appear promising.

Impact of an Electronic Clinical Decision Support Tool for Emergency Department Patients With Pneumonia.

STUDY OBJECTIVE: Despite evidence that guideline adherence improves clinical outcomes, management of pneumonia patients varies in emergency departments (EDs). We study the effect of a real-time, ED, electronic clinical decision support tool that provides clinicians with guideline-recommended decision support for diagnosis, severity assessment, disposition, and antibiotic selection. METHODS: This was a prospective, controlled, quasi-experimental trial in 7 Intermountain Healthcare hospital EDs in Utah's urban corridor. We studied adults with International Classification of Diseases, Ninth Revision codes and radiographic evidence for pneumonia during 2 periods: baseline (December 2009 through November 2010) and post-tool deployment (December 2011 through November 2012). The tool was deployed at 4 intervention EDs in May 2011, leaving 3 as usual care controls. We compared 30-day, all-cause mortality adjusted for illness severity, using a mixed-effect, logistic regression model. RESULTS: The study population comprised 4,758 ED pneumonia patients; 14% had health care-associated pneumonia. Median age was 58 years, 53% were female patients, and 59% were admitted to the hospital. Physicians applied the tool for 62.6% of intervention ED study patients. There was no difference overall in severity-adjusted mortality between intervention and usual care EDs post-tool deployment (odds ratio [OR]=0.69; 95% confidence interval [CI] 0.41 to 1.16). Post hoc analysis showed that patients with community-acquired pneumonia experienced significantly lower mortality (OR=0.53; 95% CI 0.28 to 0.99), whereas mortality was unchanged among patients with health care-associated pneumonia (OR=1.12; 95% CI 0.45 to 2.8). Patient disposition from the ED postdeployment adhered more to tool recommendations. CONCLUSION: This study demonstrates the feasibility and potential benefit of real-time electronic clinical decision support for ED pneumonia patients.

Association of Diagnostic Discrepancy with Length of Stay and Mortality in Congestive Heart Failure Patients Admitted to the Emergency Department.

The goal of this study was to analyze diagnostic discrepancies between emergency department (ED) and hospital discharge diagnoses in patients with congestive heart failure admitted to the ED. Using a synthetic dataset from the Department of Veterans Affairs, the patients' primary diagnoses were compared at two levels: diagnostic category and body system. With 12,621 patients and 24,235 admission cases, the study found a 58% mismatch rate at the category level, which was reduced to 30% at the body system level. Diagnostic categories associated with higher levels of mismatch included aplastic anemia, pneumonia, and bacterial infections. In contrast, diagnostic categories associated with lower levels of mismatch included alcohol-related disorders, COVID-19, cardiac dysrhythmias, and gastrointestinal hemorrhage. Further investigation revealed that diagnostic mismatches are associated with longer hospital stays and higher mortality rates. These findings highlight the importance of reducing diagnostic uncertainty, particularly in specific diagnostic categories and body systems, to improve patient care following ED admission.

Natural Language Processing to identify pneumonia from radiology reports.

PURPOSE: This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports. METHODS: We trained one NLP system, ONYX, using radiograph reports from children and adults that were previously manually reviewed. We then assessed its validity on a test set of 5000 reports. We aimed to substantially decrease manual review, not replace it entirely, and so, we classified reports as follows: (1) consistent with pneumonia; (2) inconsistent with pneumonia; or (3) requiring manual review because of complex features. We developed processes tailored either to optimize accuracy or to minimize manual review. Using logistic regression, we jointly modeled sensitivity and specificity of ONYX in relation to patient age, comorbidity, and care setting. We estimated positive and negative predictive value (PPV and NPV) assuming pneumonia prevalence in the source data. RESULTS: Tailored for accuracy, ONYX identified 25% of reports as requiring manual review (34% of true pneumonias and 18% of non-pneumonias). For the remainder, ONYX's sensitivity was 92% (95% CI 90-93%), specificity 87% (86-88%), PPV 74% (72-76%), and NPV 96% (96-97%). Tailored to minimize manual review, ONYX classified 12% as needing manual review. For the remainder, ONYX had sensitivity 75% (72-77%), specificity 95% (94-96%), PPV 86% (83-88%), and NPV 91% (90-91%). CONCLUSIONS: For pneumonia validation, ONYX can replace almost 90% of manual review while maintaining low to moderate misclassification rates. It can be tailored for different outcomes and study needs and thus warrants exploration in other settings.

Standard Vocabularies to Improve Machine Learning Model Transferability With Electronic Health Record Data: Retrospective Cohort Study Using Health Care-Associated Infection.

BACKGROUND: With the widespread adoption of electronic healthcare records (EHRs) by US hospitals, there is an opportunity to leverage this data for the development of predictive algorithms to improve clinical care. A key barrier in model development and implementation includes the external validation of model discrimination, which is rare and often results in worse performance. One reason why machine learning models are not externally generalizable is data heterogeneity. A potential solution to address the substantial data heterogeneity between health care systems is to use standard vocabularies to map EHR data elements. The advantage of these vocabularies is a hierarchical relationship between elements, which allows the aggregation of specific clinical features to more general grouped concepts. OBJECTIVE: This study aimed to evaluate grouping EHR data using standard vocabularies to improve the transferability of machine learning models for the detection of postoperative health care-associated infections across institutions with different EHR systems. METHODS: Patients who underwent surgery from the University of Utah Health and Intermountain Healthcare from July 2014 to August 2017 with complete follow-up data were included. The primary outcome was a health care-associated infection within 30 days of the procedure. EHR data from 0-30 days after the operation were mapped to standard vocabularies and grouped using the hierarchical relationships of the vocabularies. Model performance was measured using the area under the receiver operating characteristic curve (AUC) and F1-score in internal and external validations. To evaluate model transferability, a difference-in-difference metric was defined as the difference in performance drop between internal and external validations for the baseline and grouped models. RESULTS: A total of 5775 patients from the University of Utah and 15,434 patients from Intermountain Healthcare were included. The prevalence of selected outcomes was from 4.9% (761/15,434) to 5% (291/5775) for surgical site infections, from 0.8% (44/5775) to 1.1% (171/15,434) for pneumonia, from 2.6% (400/15,434) to 3% (175/5775) for sepsis, and from 0.8% (125/15,434) to 0.9% (50/5775) for urinary tract infections. In all outcomes, the grouping of data using standard vocabularies resulted in a reduced drop in AUC and F1-score in external validation compared to baseline features (all P<.001, except urinary tract infection AUC: P=.002). The difference-in-difference metrics ranged from 0.005 to 0.248 for AUC and from 0.075 to 0.216 for F1-score. CONCLUSIONS: We demonstrated that grouping machine learning model features based on standard vocabularies improved model transferability between data sets across 2 institutions. Improving model transferability using standard vocabularies has the potential to improve the generalization of clinical prediction models across the health care system.

An alert tool to promote lung protective ventilation for possible acute respiratory distress syndrome.

Objective: Computer-aided decision tools may speed recognition of acute respiratory distress syndrome (ARDS) and promote consistent, timely treatment using lung-protective ventilation (LPV). This study evaluated implementation and service (process) outcomes with deployment and use of a clinical decision support (CDS) synchronous alert tool associated with existing computerized ventilator protocols and targeted patients with possible ARDS not receiving LPV. Materials and Methods: We performed an explanatory mixed methods study from December 2019 to November 2020 to evaluate CDS alert implementation outcomes across 13 intensive care units (ICU) in an integrated healthcare system with >4000 mechanically ventilated patients annually. We utilized quantitative methods to measure service outcomes including CDS alert tool utilization, accuracy, and implementation effectiveness. Attitudes regarding the appropriateness and acceptability of the CDS tool were assessed via an electronic field survey of physicians and advanced practice providers. Results: Thirty-eight percent of study encounters had at least one episode of LPV nonadherence. Addition of LPV treatment detection logic prevented an estimated 1812 alert messages (41%) over use of disease detection logic alone. Forty-eight percent of alert recommendations were implemented within 2 h. Alert accuracy was estimated at 63% when compared to gold standard ARDS adjudication, with sensitivity of 85% and positive predictive value of 62%. Fifty-seven percent of survey respondents observed one or more benefits associated with the alert. Conclusion: Introduction of a CDS alert tool based upon ARDS risk factors and integrated with computerized ventilator protocol instructions increased visibility to gaps in LPV use and promoted increased adherence to LPV.

Natural language processing for the surveillance of postoperative venous thromboembolism.

BACKGROUND: The objective of this study was to develop a portal natural language processing approach to aid in the identification of postoperative venous thromboembolism events from free-text clinical notes. METHODS: We abstracted clinical notes from 25,494 operative events from 2 independent health care systems. A venous thromboembolism detected as part of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) was used as the reference standard. A natural language processing engine, easy clinical information extractor-pulmonary embolism/deep vein thrombosis (EasyCIE-PEDVT), was trained to detect pulmonary embolism and deep vein thrombosis from clinical notes. International Classification of Diseases (ICD) discharge diagnosis codes for venous thromboembolism were used as baseline comparators. The classification performance of EasyCIE-PEDVT was compared with International Classification of Diseases codes using sensitivity, specificity, area under the receiver operating characteristic curve, using an internal and external validation cohort. RESULTS: To detect pulmonary embolism, EasyCIE-PEDVT had a sensitivity of 0.714 and 0.815 in internal and external validation, respectively. To detect deep vein thrombosis, EasyCIE-PEDVT had a sensitivity of 0.846 and 0.849 in internal and external validation, respectively. EasyCIE-PEDVT had significantly higher discrimination for deep vein thrombosis compared with International Classification of Diseases codes in internal validation (area under the receiver operating characteristic curve: 0.920 vs 0.761; P < .001) and external validation (area under the receiver operating characteristic curve: 0.921 vs 0.794; P < .001). There was no significant difference in the discrimination for pulmonary embolism between EasyCIE-PEDVT and ICD codes. CONCLUSION: Accurate surveillance of postoperative venous thromboembolism may be achieved using natural language processing on clinical notes in 2 independent health care systems. These findings suggest natural language processing may augment manual chart abstraction for large registries such as NSQIP.

Ascertainment of Veterans With Metastatic Prostate Cancer in Electronic Health Records: Demonstrating the Case for Natural Language Processing.

PURPOSE: Prostate cancer (PCa) is among the leading causes of cancer deaths. While localized PCa has a 5-year survival rate approaching 100%, this rate drops to 31% for metastatic prostate cancer (mPCa). Thus, timely identification of mPCa is a crucial step toward measuring and improving access to innovations that reduce PCa mortality. Yet, methods to identify patients diagnosed with mPCa remain elusive. Cancer registries provide detailed data at diagnosis but are not updated throughout treatment. This study reports on the development and validation of a natural language processing (NLP) algorithm deployed on oncology, urology, and radiology clinical notes to identify patients with a diagnosis or history of mPCa in the Department of Veterans Affairs. PATIENTS AND METHODS: Using a broad set of diagnosis and histology codes, the Veterans Affairs Corporate Data Warehouse was queried to identify all Veterans with PCa. An NLP algorithm was developed to identify patients with any history or progression of mPCa. The NLP algorithm was prototyped and developed iteratively using patient notes, grouped into development, training, and validation subsets. RESULTS: A total of 1,144,610 Veterans were diagnosed with PCa between January 2000 and October 2020, among which 76,082 (6.6%) were identified by NLP as having mPCa at some point during their care. The NLP system performed with a specificity of 0.979 and sensitivity of 0.919. CONCLUSION: Clinical documentation of mPCa is highly reliable. NLP can be leveraged to improve PCa data. When compared to other methods, NLP identified a significantly greater number of patients. NLP can be used to augment cancer registry data, facilitate research inquiries, and identify patients who may benefit from innovations in mPCa treatment.

A Bayesian approach for detecting a disease that is not being modeled.

Over the past decade, outbreaks of new or reemergent viruses such as severe acute respiratory syndrome (SARS) virus, Middle East respiratory syndrome (MERS) virus, and Zika have claimed thousands of lives and cost governments and healthcare systems billions of dollars. Because the appearance of new or transformed diseases is likely to continue, the detection and characterization of emergent diseases is an important problem. We describe a Bayesian statistical model that can detect and characterize previously unknown and unmodeled diseases from patient-care reports and evaluate its performance on historical data.

Determination of Marital Status of Patients from Structured and Unstructured Electronic Healthcare Data.

Social Determinants of Health, including marital status, are becoming increasingly identified as key drivers of health care utilization. This paper describes a robust method to determine the marital status of patients using structured and unstructured electronic healthcare data from a single academic institution in the United States. We developed and validated a natural language processing pipeline (NLP) for the ascertainment of marital status from clinical notes and compared the performance against two baseline methods: a machine learning n-gram model, and structured data obtained from the electronic health record. Overall our NLP engine had excellent performance on both document-level (F1 0.97) and patient-level (F1 0.95) classification. The NLP Engine had superior performance compared with a baseline machine learning n-gram model. We also observed a good correlation between the marital status obtained from our NLP engine and the baseline structured electronic healthcare data (κ 0.6).

Use of Computerized Provider Order Entry Events for Postoperative Complication Surveillance.

Importance: Conventional approaches for tracking postoperative adverse events requires manual medical record review, thus limiting the scalability of such efforts. Objective: To determine if a surveillance system using computerized provider order entry (CPOE) events for selected medications as well as laboratory, microbiologic, and radiologic orders can decrease the manual medical record review burden for surveillance of postoperative complications. Design, Setting, and Participants: This cohort study reviewed the medical records of 21 775 patients who underwent surgical procedures at a university-based tertiary referral center (University of Utah, Salt Lake City) from July 1, 2007, to August 31, 2017. Patients were included if their case was selected for review by a surgical clinical reviewer as part of the National Surgical Quality Improvement Program. Patients were excluded if they had incomplete follow-up data. Main Outcomes and Measures: Thirty-day postoperative occurrences of superficial surgical site infection, deep surgical site infection, organ space surgical site infection, urinary tract infection, pneumonia, sepsis, septic shock, deep vein thrombosis requiring therapy, and pulmonary embolism, as defined by the National Surgical Quality Improvement Program. A logistic regression model was developed for each postoperative complication using CPOE features as predictors on a development set, and performance was measured on a holdout internal validation set. The models were internally validated using bootstrapping with 10 000 replications to determine the sensitivity, specificity, positive predictive value, and negative predictive value of CPOE-based surveillance system. Results: The study included 21 775 patients who underwent surgical procedures. Among these patients, 11 855 (54.4%) were women and 9920 (45.6%) were men, with a mean (SD) age of 51.7 (16.8) years. Overall, the prevalence of postoperative complications was low, ranging from 0.2% (pulmonary embolism) to 2.6% (superficial surgical site infection). Use of CPOE events to detect patients who experienced at least 1 complication had a sensitivity of 74.8% (95% CI, 71.1%-78.4%), specificity of 86.8% (95% CI, 85.5%-88.3%), positive predictive value of 33.8% (95% CI, 31.2%-36.4%), negative predictive value of 97.5% (95% CI, 97.1%-97.8%), and area under the curve of 0.808 (95% CI, 0.791-0.824). The negative predictive value for individual complications ranged from 98.7% to 100%. Use of CPOE events to screen for adverse events was estimated to diminish the burden of manual medical record review by 55.4% to 90.3%. A CPOE-based surveillance system performed well for both inpatient and outpatient procedures. Conclusions and Relevance: A CPOE-based surveillance of postoperative complications has high negative predictive value, which demonstrates that this approach can augment the currently used, resource-intensive manual medical record review process.

The design and evaluation of a Bayesian system for detecting and characterizing outbreaks of influenza.

The prediction and characterization of outbreaks of infectious diseases such as influenza remains an open and important problem. This paper describes a framework for detecting and characterizing outbreaks of influenza and the results of testing it on data from ten outbreaks collected from two locations over five years. We model outbreaks with compartment models and explicitly model non-influenza influenza-like illnesses.

Using Natural Language Processing to improve EHR Structured Data-based Surgical Site Infection Surveillance.

Surgical Site Infection surveillance in healthcare systems is labor intensive and plagued by underreporting as current methodology relies heavily on manual chart review. The rapid adoption of electronic health records (EHRs) has the potential to allow the secondary use of EHR data for quality surveillance programs. This study aims to investigate the effectiveness of integrating natural language processing (NLP) outputs with structured EHR data to build machine learning models for SSI identification using real-world clinical data. We examined a set of models using structured data with and without NLP document-level, mention-level, and keyword features. The top-performing model was based on a Random Forest classifier enhanced with NLP document-level features achieving a 0.58 sensitivity, 0.97 specificity, 0.54 PPV, 0.98 NPV, and 0.52 F0.5 score. We further interrogated the feature contributions, analyzed the errors, and discussed future directions.

Epidemiology and Clinical Features of Invasive Fungal Infection in a US Health Care Network.

BACKGROUND: A better understanding of the epidemiology and clinical features of invasive fungal infection (IFI) is integral to improving outcomes. We describe a novel case-finding methodology, reporting incidence, clinical features, and outcomes of IFI in a large US health care network. METHODS: All available records in the Intermountain Healthcare Enterprise Data Warehouse from 2006 to 2015 were queried for clinical data associated with IFI. The resulting data were overlaid in 124 different combinations to identify high-probability IFI cases. The cohort was manually reviewed, and exclusions were applied. European Organization for Research and Treatment of Cancer/Invasive Fungal Infections Cooperative Group and the National Institute of Allergy and Infectious Diseases Mycoses Study Group Consensus Group definitions were adapted to categorize IFI in a broad patient population. Linear regression was used to model variation in incidence over time. RESULTS: A total of 3374 IFI episodes occurred in 3154 patients. The mean incidence was 27.2 cases/100 000 patients per year, and there was a mean annual increase of 0.24 cases/100 000 patients (P = .21). Candidiasis was the most common (55%). Dimorphic fungi, primarily Coccidioides spp., comprised 25.1% of cases, followed by Aspergillus spp. (8.9%). The median age was 55 years, and pediatric cases accounted for 13%; 26.1% of patients were on immunosuppression, 14.9% had autoimmunity or immunodeficiency, 13.3% had active malignancy, and 5.9% were transplant recipients. Lymphopenia preceded IFI in 22.1% of patients. Hospital admission occurred in 76.2%. The median length of stay was 16 days. All-cause mortality was 17.0% at 42 days and 28.8% at 1 year. Forty-two-day mortality was highest in Aspergillus spp. (27.5%), 20.5% for Candida, and lowest for dimorphic fungi (7.5%). CONCLUSIONS: In this population, IFI was not uncommon, affected a broad spectrum of patients, and was associated with high crude mortality.

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients.

Importance: To improve patient safety, health care systems need reliable methods to detect adverse events in large patient populations. Events are often described in clinical notes, rather than structured data, which make them difficult to identify on a large scale. Objective: To develop and compare 2 natural language processing methods, a rules-based approach and a machine learning (ML) approach, for identifying bleeding events in clinical notes. Design, Setting, and Participants: This diagnostic study used deidentified notes from the Medical Information Mart for Intensive Care, which spans 2001 to 2012. A training set of 990 notes and a test set of 660 notes were randomly selected. Physicians classified each note as present or absent for a clinically relevant bleeding event during the hospitalization. A bleeding dictionary was developed for the rules-based approach; bleeding mentions were then aggregated to arrive at a classification for each note. Three ML models (support vector machine, extra trees, and convolutional neural network) were developed and trained using the 990-note training set. Another instance of each ML model was also trained on a sample of 450 notes, with equal numbers of bleeding-present and bleeding-absent notes. The notes were represented using term frequency-inverse document frequency vectors and global vectors for word representation. Main Outcomes and Measures: The main outcomes were accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for each model. Following training, the models were tested on the test set and sensitivities were compared using a McNemar test. Results: The 990-note training set represented 769 patients (296 [38.5%] female; mean [SD] age, 67.42 [14.7] years). The 660-note test set represented 527 patients (211 [40.0%] female; mean [SD] age, 67.86 [14.7] years). Bleeding was present in 146 notes (22.1%). The extra trees down-sampled model and rules-based approaches were similarly sensitive (93.8% vs 91.1%; difference, 2.7%; 95% CI, -3.8% to 7.9%; P = .44). The positive predictive value for the extra trees model, however, was 48.6%. The rules-based model had the best performance overall, with 84.6% specificity, 62.7% positive predictive value, and 97.1% negative predictive value. Conclusions and Relevance: Bleeding is a common complication in health care, and these results demonstrate an automated and scalable detection method. The rules-based natural language processing approach, compared with ML, had the best performance in identifying bleeding, with high sensitivity and negative predictive value.

The effects of natural language processing on cross-institutional portability of influenza case detection for disease surveillance.

OBJECTIVES: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance. METHODS: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis. RESULTS: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser). CONCLUSION: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.

A study of the transferability of influenza case detection systems between two large healthcare systems.

OBJECTIVES: This study evaluates the accuracy and transferability of Bayesian case detection systems (BCD) that use clinical notes from emergency department (ED) to detect influenza cases. METHODS: A BCD uses natural language processing (NLP) to infer the presence or absence of clinical findings from ED notes, which are fed into a Bayesain network classifier (BN) to infer patients' diagnoses. We developed BCDs at the University of Pittsburgh Medical Center (BCDUPMC) and Intermountain Healthcare in Utah (BCDIH). At each site, we manually built a rule-based NLP and trained a Bayesain network classifier from over 40,000 ED encounters between Jan. 2008 and May. 2010 using feature selection, machine learning, and expert debiasing approach. Transferability of a BCD in this study may be impacted by seven factors: development (source) institution, development parser, application (target) institution, application parser, NLP transfer, BN transfer, and classification task. We employed an ANOVA analysis to study their impacts on BCD performance. RESULTS: Both BCDs discriminated well between influenza and non-influenza on local test cases (AUCs > 0.92). When tested for transferability using the other institution's cases, BCDUPMC discriminations declined minimally (AUC decreased from 0.95 to 0.94, p<0.01), and BCDIH discriminations declined more (from 0.93 to 0.87, p<0.0001). We attributed the BCDIH decline to the lower recall of the IH parser on UPMC notes. The ANOVA analysis showed five significant factors: development parser, application institution, application parser, BN transfer, and classification task. CONCLUSION: We demonstrated high influenza case detection performance in two large healthcare systems in two geographically separated regions, providing evidentiary support for the use of automated case detection from routinely collected electronic clinical notes in national influenza surveillance. The transferability could be improved by training Bayesian network classifier locally and increasing the accuracy of the NLP parser.