Generalizability and portability of natural language processing system to extract individual social risk factors.

OBJECTIVE: The objective of this study is to validate and report on portability and generalizability of a Natural Language Processing (NLP) method to extract individual social factors from clinical notes, which was originally developed at a different institution. MATERIALS AND METHODS: A rule-based deterministic state machine NLP model was developed to extract financial insecurity and housing instability using notes from one institution and was applied on all notes written during 6 months at another institution. 10% of positively-classified notes by NLP and the same number of negatively-classified notes were manually annotated. The NLP model was adjusted to accommodate notes at the new site. Accuracy, positive predictive value, sensitivity, and specificity were calculated. RESULTS: More than 6 million notes were processed at the receiving site by the NLP model, which resulted in about 13,000 and 19,000 classified as positive for financial insecurity and housing instability, respectively. The NLP model showed excellent performance on the validation dataset with all measures over 0.87 for both social factors. DISCUSSION: Our study illustrated the need to accommodate institution-specific note-writing templates as well as clinical terminology of emergent diseases when applying NLP model for social factors. A state machine is relatively simple to port effectively across institutions. Our study. showed superior performance to similar generalizability studies for extracting social factors. CONCLUSION: Rule-based NLP model to extract social factors from clinical notes showed strong portability and generalizability across organizationally and geographically distinct institutions. With only relatively simple modifications, we obtained promising performance from an NLP-based model.

Natural language processing-driven state machines to extract social factors from unstructured clinical documentation.

Objective: This study sought to create natural language processing algorithms to extract the presence of social factors from clinical text in 3 areas: (1) housing, (2) financial, and (3) unemployment. For generalizability, finalized models were validated on data from a separate health system for generalizability. Materials and Methods: Notes from 2 healthcare systems, representing a variety of note types, were utilized. To train models, the study utilized n-grams to identify keywords and implemented natural language processing (NLP) state machines across all note types. Manual review was conducted to determine performance. Sampling was based on a set percentage of notes, based on the prevalence of social need. Models were optimized over multiple training and evaluation cycles. Performance metrics were calculated using positive predictive value (PPV), negative predictive value, sensitivity, and specificity. Results: PPV for housing rose from 0.71 to 0.95 over 3 training runs. PPV for financial rose from 0.83 to 0.89 over 2 training iterations, while PPV for unemployment rose from 0.78 to 0.88 over 3 iterations. The test data resulted in PPVs of 0.94, 0.97, and 0.95 for housing, financial, and unemployment, respectively. Final specificity scores were 0.95, 0.97, and 0.95 for housing, financial, and unemployment, respectively. Discussion: We developed 3 rule-based NLP algorithms, trained across health systems. While this is a less sophisticated approach, the algorithms demonstrated a high degree of generalizability, maintaining >0.85 across all predictive performance metrics. Conclusion: The rule-based NLP algorithms demonstrated consistent performance in identifying 3 social factors within clinical text. These methods may be a part of a strategy to measure social factors within an institution.

A randomized study on the usefulness of an electronic outpatient hypoglycemia risk calculator for clinicians of patients with diabetes in a safety-net institution.

Objective: Hypoglycemia (HG) occurs in up to 60% of patients with diabetes mellitus (DM) each year. We assessed a HG alert tool in an electronic health record system, and determined its effect on clinical practice and outcomes.Methods: The tool applied a statistical model, yielding patient-specific information about HG risk. We randomized outpatient primary-care providers (PCPs) to see or not see the alerts. Patients were assigned to study group according to the first PCP seen during four months. We assessed prescriptions, testing, and HG. Variables were compared by multinomial, logistic, or linear model. ClinicalTrials.gov ID: NCT04177147 (registered on 22 November 2019).Results: Patients (N = 3350) visited 123 intervention PCPs; 3395 patients visited 220 control PCPs. Intervention PCPs were shown 18,645 alerts (mean of 152 per PCP). Patients' mean age was 55 years, with 61% female, 49% black, and 49% Medicaid recipients. Mean baseline A1c and body mass index were similar between groups. During follow-up, the number of A1c and glucose tests, and number of new, refilled, changed, or discontinued insulin prescriptions, were highest for patients with highest risk. Per 100 patients on average, the intervention group had fewer sulfonylurea refills (6 vs. 8; p < .05) and outpatient encounters (470 vs. 502; p < .05), though the change in encounters was not significant. Frequency of HG events was unchanged.Conclusions: Informing PCPs about risk of HG led to fewer sulfonylurea refills and visits. Longer-term studies are needed to assess potential for long-term benefits.

Predictive modeling of hypoglycemia for clinical decision support in evaluating outpatients with diabetes mellitus.

Objective: Hypoglycemia occurs in 20-60% of patients with diabetes mellitus. Identifying at-risk patients can facilitate interventions to lower risk. We sought to develop a hypoglycemia prediction model. Methods: In this retrospective cohort study, urban adults prescribed a diabetes drug between 2004 and 2013 were identified. Demographic and clinical data were extracted from an electronic medical record (EMR). Laboratory tests, diagnostic codes and natural language processing (NLP) identified hypoglycemia. We compared multiple logistic regression, classification and regression trees (CART), and random forest. Models were evaluated on an independent test set or through cross-validation. Results: The 38,780 patients had mean age 57 years; 56% were female, 40% African-American and 39% uninsured. Hypoglycemia occurred in 8128 (539 identified only by NLP). In logistic regression, factors positively associated with hypoglycemia included infection, non-long-acting insulin, dementia and recent hypoglycemia. Negatively associated factors included long-acting insulin plus sulfonylurea, and age 75 or older. The models' area under curve was similar (logistic regression, 89%; CART, 88%; random forest, 90%, with ten-fold cross-validation). Conclusions: NLP improved identification of hypoglycemia. Non-long-acting insulin was an important risk factor. Decreased risk with age may reflect treatment or diminished awareness of hypoglycemia. More complex models did not improve prediction.

Designing a system for patients controlling providers' access to their electronic health records: organizational and technical challenges.

BACKGROUND: Electronic health records (EHRs) are proliferating, and financial incentives encourage their use. Applying Fair Information Practice principles to EHRs necessitates balancing patients' rights to control their personal information with providers' data needs to deliver safe, high-quality care. We describe the technical and organizational challenges faced in capturing patients' preferences for patient-controlled EHR access and applying those preferences to an existing EHR. METHODS: We established an online system for capturing patients' preferences for who could view their EHRs (listing all participating clinic providers individually and categorically-physicians, nurses, other staff) and what data to redact (none, all, or by specific categories of sensitive data or patient age). We then modified existing data-viewing software serving a state-wide health information exchange and a large urban health system and its primary care clinics to allow patients' preferences to guide data displays to providers. RESULTS: Patients could allow or restrict data displays to all clinicians and staff in a demonstration primary care clinic, categories of providers (physicians, nurses, others), or individual providers. They could also restrict access to all EHR data or any or all of five categories of sensitive data (mental and reproductive health, sexually transmitted diseases, HIV/AIDS, and substance abuse) and for specific patient ages. The EHR viewer displayed data via reports, data flowsheets, and coded and free text data displayed by Google-like searches. Unless patients recorded restrictions, by default all requested data were displayed to all providers. Data patients wanted restricted were not displayed, with no indication they were redacted. Technical barriers prevented redacting restricted information in free textnotes. The program allowed providers to hit a "Break the Glass" button to override patients' restrictions, recording the date, time, and next screen viewed. Establishing patient-control over EHR data displays was complex and required ethical, clinical, database, and programming expertise and difficult choices to overcome technical and health system constraints. CONCLUSIONS: Assessing patients' preferences for access to their EHRs and applying them in clinical practice requires wide-ranging technical, clinical, and bioethical expertise, to make tough choices to overcome significant technical and organization challenges.