Disparities Associated With Electronic Behavioral Alerts for Safety and Violence Concerns in the Emergency Department.

STUDY OBJECTIVE: Although electronic behavioral alerts are placed as an alert flag in the electronic health record to notify staff of previous behavioral and/or violent incidents in emergency departments (EDs), they have the potential to reinforce negative perceptions of patients and contribute to bias. We provide characterization of ED electronic behavioral alerts using electronic health record data across a large, regional health care system. METHODS: We conducted a retrospective cross-sectional study of adult patients presenting to 10 adult EDs within a Northeastern United States health care system from 2013 to 2022. Electronic behavioral alerts were manually screened for safety concerns and then categorized by the type of concern. In our patient-level analyses, we included patient data at the time of the first ED visit where an electronic behavioral alert was triggered or, if a patient had no electronic behavioral alerts, the earliest visit in the study period. We performed a mixed-effects regression analysis to identify patient-level risk factors associated with safety-related electronic behavioral alert deployment. RESULTS: Of the 2,932,870 ED visits, 6,775 (0.2%) had associated electronic behavioral alerts across 789 unique patients and 1,364 unique electronic behavioral alerts. Of the encounters with electronic behavioral alerts, 5,945 (88%) were adjudicated as having a safety concern involving 653 patients. In our patient-level analysis, the median age for patients with safety-related electronic behavioral alerts was 44 years (interquartile range 33 to 55 years), 66% were men, and 37% were Black. Visits with safety-related electronic behavioral alerts had higher rates of discontinuance of care (7.8% vs 1.5% with no alert; P<.001) as defined by the patient-directed discharge, left-without-being-seen, or elopement-type dispositions. The most common topics in the electronic behavioral alerts were physical (41%) or verbal (36%) incidents with staff or other patients. In the mixed-effects logistic analysis, Black non-Hispanic patients (vs White non-Hispanic patients: adjusted odds ratio 2.60; 95% confidence interval [CI] 2.13 to 3.17), aged younger than 45 (vs aged 45-64 years: adjusted odds ratio 1.41; 95% CI 1.17 to 1.70), male (vs female: adjusted odds ratio 2.09; 95% CI 1.76 to 2.49), and publicly insured patients (Medicaid: adjusted odds ratio 6.18; 95% CI 4.58 to 8.36; Medicare: adjusted odds ratio 5.63; 95% CI 3.96 to 8.00 vs commercial) were associated with a higher risk of a patient having at least 1 safety-related electronic behavioral alert deployment during the study period. CONCLUSION: In our analysis, younger, Black non-Hispanic, publicly insured, and male patients were at a higher risk of having an ED electronic behavioral alert. Although our study is not designed to reflect causality, electronic behavioral alerts may disproportionately affect care delivery and medical decisions for historically marginalized populations presenting to the ED, contribute to structural racism, and perpetuate systemic inequities.

Identifying opioid-related electronic health record phenotypes for emergency care research and surveillance: An expert consensus driven concept mapping process.

Background: Case identification for many areas of opioid research and surveillance in the emergency department (ED) is challenging as patients are often undifferentiated with nonspecific symptoms and diagnostic codes have proven to be inaccurate. Opioid-related phenotypes based on combinations of electronic health record data are a promising method to address this gap but lack a consensus-based conceptual framework to aid organization and prioritization. Methods: To achieve consensus around opioid-related phenotype topics in the ED, we used a hybrid scheme that employed modified Delphi and conceptual mapping methods. The combined iterative process used three rounds of electronic meetings and questionnaires to generate consensus recommendations and concept mappings based on the opinions and feedback of the 9 member Delphi panel. Mean importance and feasibility scores based on 5-point Likert scales (1 = relatively unimportant (infeasible) to 5 = extremely important (feasible)) for each statement/phenotype were calculated. We used multidimensional scaling to produce a point map of the phenotype concepts and hierarchical cluster analysis to generate concept maps. Results: After the first round, 120 initial phenotype concepts were proposed which were reduced to 73 concepts after normalization by the research team. Opioid overdose (9.54, SD = 0.9) had the highest combined importance and feasibility score. A final labeled 12-cluster solution was determined to be the most parsimonious description of the content by the research team. Three key groups emerged: opioid overdose, other opioid-specific phenotypes (opioid use disorder, opioid misuse, and opioid withdrawal) with significant concept overlap and opioid use-related phenotypes (homelessness, falls, infections, and suicidality). Conclusions: Using an expert consensus driven concept mapping process we identified specific opioid phenotype concepts within an overlapping schema that carry high priority for development and validation to advance emergency care opioid-related research and surveillance.

Forty years of emergency medicine research: Uncovering research themes and trends through topic modeling.

STUDY OBJECTIVE: Topic identification can facilitate knowledge curation, discover thematic relationships, trends, and predict future direction. We aimed to determine through an unsupervised, machine learning approach to topic modeling the most common research themes in emergency medicine over the last 40 years and summarize their trends and characteristics. METHODS: We retrieved the complete reference entries including article abstracts from Ovid for all original research articles from 1980 to 2019 within emergency medicine for six widely-cited journals. Abstracts were processed through a natural language pipeline and analyzed by a latent Dirichlet allocation topic modeling algorithm for unsupervised topic discovery. Topics were further examined through trend analysis, word associations, co-occurrence metrics, and two-dimensional embeddings. RESULTS: We retrieved 47,158 articles during the defined time period that were filtered to 20,528 articles for further analysis. Forty topics covering methodologic and clinical areas were discovered. These topics separated into distinct clusters when embedded in two-dimensional space and exhibited consistent patterns of interaction. We observed the greatest increase in popularity in research themes involving risk factors (0.4% to 5.2%), health utilization (1.2% to 5.0%), and ultrasound (0.7% to 3.3%), and a relative decline in research involving basic science (8.9% to 1.1%), cardiac arrest (6.5% to 2.2%), and vitals (6.3% to 1.3%) over the past 40 years. Our data show only very modest growth in mental health and substance abuse research (1.0% to 1.6%), despite ongoing crises. CONCLUSIONS: Topic modeling via unsupervised machine learning applied to emergency medicine abstracts discovered coherent topics, trends, and patterns of interaction.

Association between patient-physician gender concordance and patient experience scores. Is there gender bias?

BACKGROUND: Patient satisfaction, a commonly measured indicator of quality of care and patient experience, is often used in physician performance reviews and promotion decisions. Patient satisfaction surveys may introduce gender-related bias. OBJECTIVE: Examine the effect of patient and physician gender concordance on patient satisfaction with emergency care. METHODS: We performed a cross-sectional analysis of electronic health record and Press Ganey patient satisfaction survey data of adult patients discharged from the emergency department (2015-2018). Logistic regression models were used to examine relationships between physician gender, patient gender, and physician-patient gender dyads. Binary outcomes included: perfect care provider score and perfect overall assessment score. RESULTS: Female patients returned surveys more often (n=7 612; 61.55%) and accounted for more visits (n=232 024; 55.26%). Female patients had lower odds of perfect scores for provider score and overall assessment score (OR: 0.852, 95% CI: 0.790, 0.918; OR: 0.782, 95% CI: 0.723, 0.846). Female physicians had 1.102 (95% CI: 1.001, 1.213) times the odds of receiving a perfect provider score. Physician gender did not influence male patients' odds of reporting a perfect care provider score (95% CI: 0.916, 1.158) whereas female patients treated by female physicians had 1.146 times the odds (95% CI: 1.019, 1.289) of a perfect provider score. CONCLUSION: Female patients prefer female emergency physicians but were less satisfied with their physician and emergency department visit overall. Over-representation of female patients on patient satisfaction surveys introduces bias. Patient satisfaction surveys should be deemphasized from physician compensation and promotion decisions.

Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation.

STUDY OBJECTIVE: The goal of this study is to create a predictive, interpretable model of early hospital respiratory failure among emergency department (ED) patients admitted with coronavirus disease 2019 (COVID-19). METHODS: This was an observational, retrospective, cohort study from a 9-ED health system of admitted adult patients with severe acute respiratory syndrome coronavirus 2 (COVID-19) and an oxygen requirement less than or equal to 6 L/min. We sought to predict respiratory failure within 24 hours of admission as defined by oxygen requirement of greater than 10 L/min by low-flow device, high-flow device, noninvasive or invasive ventilation, or death. Predictive models were compared with the Elixhauser Comorbidity Index, quick Sequential [Sepsis-related] Organ Failure Assessment, and the CURB-65 pneumonia severity score. RESULTS: During the study period, from March 1 to April 27, 2020, 1,792 patients were admitted with COVID-19, 620 (35%) of whom had respiratory failure in the ED. Of the remaining 1,172 admitted patients, 144 (12.3%) met the composite endpoint within the first 24 hours of hospitalization. On the independent test cohort, both a novel bedside scoring system, the quick COVID-19 Severity Index (area under receiver operating characteristic curve mean 0.81 [95% confidence interval {CI} 0.73 to 0.89]), and a machine-learning model, the COVID-19 Severity Index (mean 0.76 [95% CI 0.65 to 0.86]), outperformed the Elixhauser mortality index (mean 0.61 [95% CI 0.51 to 0.70]), CURB-65 (0.50 [95% CI 0.40 to 0.60]), and quick Sequential [Sepsis-related] Organ Failure Assessment (0.59 [95% CI 0.50 to 0.68]). A low quick COVID-19 Severity Index score was associated with a less than 5% risk of respiratory decompensation in the validation cohort. CONCLUSION: A significant proportion of admitted COVID-19 patients progress to respiratory failure within 24 hours of admission. These events are accurately predicted with bedside respiratory examination findings within a simple scoring system.

Physical Restraint Use in Adult Patients Presenting to a General Emergency Department.

STUDY OBJECTIVE: The prevalence of agitation among emergency department (ED) patients is increasing. Physical restraints are routinely used to prevent self-harm and to protect staff, but are associated with serious safety risks. To date, characterization of physical restraint use in the emergency setting has been limited. We thus aim to describe restraint patterns in the general ED to guide future investigation in the management of behavioral disorders. METHODS: We conducted a cross-sectional study of adult patients presenting to 5 adult EDs within a large regional health system for 2013 to 2015, and with a physical restraint order during their visit. We undertook descriptive analyses and cluster analysis to determine unique meaningful groups within our sample. RESULTS: In 956,153 total ED visits, 4,661 patients (0.5%) had associated restraint orders, representing 3,739 unique patients. The median age was 47 years (interquartile range 32 to 59 years), 66.7% of patients were men, 61.9% had a psychiatric history, and 91.1% arrived by ambulance. For chief complaints, 33.7% were alcohol or drug use, 45.4% medical, 12.3% psychiatric, and 8.5% trauma. Cluster analysis identified 2 distinct cohorts. A younger, predominantly male population presented with alcohol or drug use, whereas an older group arrived with medical complaints. CONCLUSION: Our data found strong association of alcohol or drug use with physical restraints and identified a unique elderly population with behavioral disturbances in the ED. Further characterization of causal links and safer practices to manage agitation for these vulnerable populations are needed.

US-Pro: An Application Enabling Efficient, High-Throughput Ultrasound Video Processing.

We describe a new graphical user interface-based application, US-Pro, designed to enable customized, high-throughput ultrasound video anonymization and dynamic cropping before output to video or high-efficiency disk storage. This application is distributed in a Docker container environment, which supports facile software installation on the most commonly used operating systems, as well as local processing of data sets, precluding the external transfer of electronic protected health information. The US-Pro application will facilitate the reproducible production of large-scale ultrasound video data sets for varied applications, including machine-learning analysis, educational distribution, and quality assurance review.

The Impact of Risk Standardization on Variation in CT Use and Emergency Physician Profiling.

OBJECTIVE: The purpose of this study is to use detailed electronic health record data to profile the use of condition-specific, risk-standardized imaging by emergency physicians. MATERIALS AND METHODS: CT utilization in four emergency departments in a single health care system was retrospectively analyzed. The primary outcome for analysis was indication-specific, risk-standardized CT utilization. We constructed seven clinical cohorts on the basis of the presence or absence of a traumatic indication for the most frequently performed CT studies. Risk standardization was performed using machine learning algorithms and hierarchic logistic regression models. Variation in CT utilization for each cohort was analyzed using coefficients of variation and box plots, the effect of risk standardization on physician profiling was determined using slope diagrams and kappa values, and within-physician correlation was assessed using correlation coefficients and matrices. RESULTS: For the seven cohorts, the number of physicians ordering more than 25 CT studies for a particular indication ranged from 70 to 88, and the number of ED visits ranged from 17,458 to 117,489. The unadjusted variation was large for each indication (coefficient of variation, 30.2-57.9). Risk standardization resulted in reduced but persistent variation for all indications (coefficient of variation, 12.3-22.3). Among indication-specific models, risk standardization resulted in reclassification by two or more deciles for 14.0-39.1% of physicians. The R value for within-physician correlation varied from 0.02 to 0.80 and was highest between chest and abdominal imaging for trauma. CONCLUSION: In this multisite study of CT utilization, risk standardization had a substantial impact on variation in CT utilization and emergency physician profiling. Administrators and payers should include risk standardization in future measures of physician imaging to ensure valid assessment of performance and achieve improvements in emergency care value.

Impact of point-of-care ultrasonography on ED time to disposition for patients with nontraumatic shock.

STUDY OBJECTIVE: Nontraumatic shock in the emergency department (ED) has multiple causes and carries in-hospital mortality approaching 20%, underscoring the need for prompt diagnosis and treatment. Diagnostic ultrasonography at the point of care is one method that may improve the ability of ED physicians to quickly diagnose and treat. This study assesses the effect of the use and timing of point-of-care (POC) ultrasonography on time to disposition request. METHODS: This retrospective study across 4 Connecticut EDs compared propensity score matched shock patients who did and did not receive POC ultrasonography. Two propensity score matches were performed: the first using covariates of time to disposition from previous literature and the second using 25 novel covariates identified from electronic health records using machine learning to reduce variable selection biases. RESULTS: A total of 3834 unique patients presented with shock during an 18-month period, and 703 (18.3%) patients received POC ultrasonography. Mean time to disposition for all patients was 255.4minutes (interquartile range, 163.8). After propensity score matching, patients had a mean reduction of 26.7minutes (95% confidence interval [CI], 2.8-58.3) in time to disposition when POC ultrasonography was performed within 1hour of ED arrival and a lesser reduction of 16.7minutes (95% CI, -2.8 to 35.5) when POC ultrasonography was performed within 2hours. There was no evidence of reduction in time to disposition when ultrasonography was performed after 2hours (16.7minutes; 95% CI, -14.3 to 29.9). Propensity score models using machine learning-selected variables yielded similar results. CONCLUSION: Performance of POC ultrasonography likely improves time to disposition when performed early on ED patients with shock.

Emergency physician focused cardiac ultrasound improves diagnosis of ascending aortic dissection.

STUDY OBJECTIVE: Ascending aortic dissection (AAD) is an uncommon, time-sensitive, and deadly diagnosis with a nonspecific presentation. Ascending aortic dissection is associated with aortic dilation, which can be determined by emergency physician focused cardiac ultrasound (EP FOCUS). We seek to determine if patients who receive EP FOCUS have reduced time to diagnosis for AAD. METHODS: We performed a retrospective review of patients treated at 1 of 3 affiliated emergency departments, March 1, 2013, to May 1, 2015, diagnosed as having AAD. All autopsies were reviewed for missed cases. Primary outcome measure was time to diagnosis. Secondary outcomes were time to disposition, misdiagnosis rate, and mortality. RESULTS: Of 386547 ED visits, targeted review of 123 medical records and 194 autopsy reports identified 32 patients for inclusion. Sixteen patients received EP FOCUS and 16 did not. Median time to diagnosis in the EP FOCUS group was 80 (interquartile range [IQR], 46-157) minutes vs 226 (IQR, 109-1449) minutes in the non-EP FOCUS group (P = .023). Misdiagnosis was 0% (0/16) in the EP FOCUS group vs 43.8% (7/16) in the non-EP FOCUS group (P = .028). Mortality, adjusted for do-not-resuscitate status, for EP FOCUS vs non-EP FOCUS was 15.4% vs 37.5% (P = .24). Median rooming time to disposition was 134 (IQR, 101-195) minutes for EP FOCUS vs 205 (IQR, 114-342) minutes for non-EP FOCUS (P = .27). CONCLUSIONS: Patients who receive EP FOCUS are diagnosed faster and misdiagnosed less compared with patients who do not receive EP FOCUS. We recommend assessment of the thoracic aorta be performed routinely during cardiac ultrasound in the emergency department.

Use of Point-of-Care Ultrasound in the Emergency Department: Insights From the 2012 Medicare National Payment Data Set.

OBJECTIVES: Point-of-care ultrasound is a valuable tool with potential to expedite diagnoses and improve patient outcomes in the emergency department. However, little is known about national patterns of adoption. This study examined nationwide point-of-care ultrasound reimbursement among emergency medicine (EM) practitioners and examined regional and practitioner level variations. METHODS: Data from the 2012 Center for Medicare and Medicaid Services Fee-for-Service Provider Utilization and Payment Data include all practitioners who received more than 10 Medicare Part B fee-for-service reimbursements for any Healthcare Common Procedure Coding System code in 2012. Odds ratios (ORs) and descriptive statistics were calculated to assess relationships between ultrasound reimbursement and practice location, nearby presence of an EM residency, and time elapsed since practitioner graduation. RESULTS: Of 52,928 unique EM practitioners, 391 (0.7%) received limited ultrasound reimbursements for a total of 16,389 scans in 2012. Urban counties had an OR of 5.4 (95% confidence interval, 3.8-7.8) for receiving point-of-care ultrasound reimbursements compared to rural counties. Counties with an EM residency had an OR of 84.7 (95% confidence interval, 42.6-178.8) for reimbursement compared to counties without. The OR for receiving reimbursement was independent of medical school graduation year (P = .83); however, recent graduates performed more scans (P = .02). CONCLUSIONS: A small minority of EM practitioners received reimbursements for point-of-care ultrasound from Medicare beneficiaries. These practitioners were more likely to reside in urban and academic settings. Future efforts should assess the degree to which our findings reflect either low point-of-care ultrasound use or low rates of billing for ultrasound examinations that are performed.

Improving emergency physician performance using audit and feedback: a systematic review.

BACKGROUND: Audit and feedback can decrease variation and improve the quality of care in a variety of health care settings. There is a growing literature on audit and feedback in the emergency department (ED) setting. Because most studies have been small and not focused on a single clinical process, systematic assessment could determine the effectiveness of audit and feedback interventions in the ED and which specific characteristics improve the quality of emergency care. OBJECTIVE: The objective of the study is to assess the effect of audit and feedback on emergency physician performance and identify features critical to success. METHODS: We adhered to the PRISMA statement to conduct a systematic review of the literature from January 1994 to January 2014 related to audit and feedback of physicians in the ED. We searched Medline, EMBASE, PsycINFO, and PubMed databases. We included studies that were conducted in the ED and reported quantitative outcomes with interventions using both audit and feedback. For included studies, 2 reviewers independently assessed methodological quality using the validated Downs and Black checklist for nonrandomized studies. Treatment effect and heterogeneity were to be reported via meta-analysis and the I2 inconsistency index. RESULTS: The search yielded 4332 articles, all of which underwent title review; 780 abstracts and 131 full-text articles were reviewed. Of these, 24 studies met inclusion criteria with an average Downs and Black score of 15.6 of 30 (range, 6-22). Improved performance was reported in 23 of the 24 studies. Six studies reported sufficient outcome data to conduct summary analysis. Pooled data from studies that included 41,124 patients yielded an average treatment effect among physicians of 36% (SD, 16%) with high heterogeneity (I2=83%). CONCLUSION: The literature on audit and feedback in the ED reports positive results for interventions across numerous clinical conditions but without standardized reporting sufficient for meta-analysis. Characteristics of audit and feedback interventions that were used in a majority of studies were feedback that targeted errors of omission and that was explicit with measurable instruction and a plan for change delivered in the clinical setting greater than 1 week after the audited performance using a combination of media and types at both the individual and group levels. Future work should use standardized reporting to identify the specific aspects of audit or feedback that drive effectiveness in the ED.

Redefining Overuse to Include Costs: A Decision Analysis for Computed Tomography in Minor Head Injury.

BACKGROUND: A study was conducted to (1) determine the testing threshold for head computed tomography (CT) in minor head injury in the emergency department using decision analysis with and without costs included in the analysis, (2) to determine which variables have significant impact on the testing threshold, and (3) to compare this calculated testing threshold to the pretest risk estimate previously reported when the Canadian CT Head Rule (CCHR) was applied. It was hypothesized that the CCHR might not identify all patients above the testing threshold. METHODS: A decision analytic model was constructed using commercially available software and data from published literature. Outcomes were assigned values on the basis of quality-adjusted life-years (QALYs) and cost. Two testing thresholds were calculated, the first based only on the effectiveness of either strategy, the second on the overall net monetary benefit. Two-way sensitivity analyses were performed to determine which variables most affected the testing threshold. RESULTS: When only effectiveness (QALYs) was considered, the testing threshold for obtaining head CT was 0.039%. This threshold increased to 0.421% when the net monetary benefit was considered in lieu of QALYs. Age, probability of lesion on CT requiring neurosurgery, and cost of CT were the main drivers of the model. CONCLUSION: If only effectiveness is considered, current clinical decision rules might not provide a sufficient degree of certainty to ensure identification of all patients for whom the benefits of CT outweigh its risks. However, inclusion of cost in the analysis increases the testing threshold by an order of magnitude and well outside the range of uncertainty of current clinical decision rules. These results suggest that the term overuse should be redefined to include the provision of medical services with no benefits or for which harms including cost outweigh benefits.

The Scope of Multimorbidity in Family Medicine: Identifying Age Patterns Across the Lifespan.

INTRODUCTION: Multimorbidity rates are both increasing in prevalence across age ranges, and also increasing in diagnostic importance within and outside the family medicine clinic. Here we aim to describe the course of multimorbidity across the lifespan. METHODS: This was a retrospective cohort study across 211,953 patients from a large northeastern health care system. Past medical histories were collected in the form of ICD-10 diagnostic codes. Rates of multimorbidity were calculated from comorbid diagnoses defined from the ICD10 codes identified in the past medical histories. RESULTS: We identify 4 main age groups of diagnosis and multimorbidity. Ages 0 to 10 contain diagnoses which are infectious or respiratory, whereas ages 10 to 40 are related to mental health. From ages 40 to 70 there is an emergence of alcohol use disorders and cardiometabolic disorders. And ages 70 to 90 are predominantly long-term sequelae of the most common cardiometabolic disorders. The mortality of the whole population over the study period was 5.7%, whereas the multimorbidity with the highest mortality across the study period was Circulatory Disorders-Circulatory Disorders at 23.1%. CONCLUSION: The results from this study provide a comparison for the presence of multimorbidity within age cohorts longitudinally across the population. These patterns of comorbidity can assist in the allocation to practice resources that will best support the common conditions that patients need assistance with, especially as the patients transition between pediatric, adult, and geriatric care. Future work examining and comparing multimorbidity indices is warranted.

Computationally derived transition points across phases of clinical care.

The objective of this study is to use statistical techniques for the identification of transition points along the life course, aiming to identify fundamental changes in patient multimorbidity burden across phases of clinical care. This retrospective cohort analysis utilized 5.2 million patient encounters from 2013 to 2022, collected from a large academic institution and its affiliated hospitals. Structured information was systematically gathered for each encounter and three methodologies - clustering analysis, False Nearest Neighbor, and transitivity analysis - were employed to pinpoint transitions in patients' clinical phase. Clustering analysis identified transition points at age 2, 17, 41, and 66, FNN at 4.27, 5.83, 5.85, 14.12, 20.62, 24.30, 25.10, 29.08, 33.12, 35.7, 38.69, 55.66, 70.03, and transitivity analysis at 7.27, 23.58, 29.04, 35.00, 61.29, 67.03, 77.11. Clustering analysis identified transition points that align with the current clinical gestalt of pediatric, adult, and geriatric phases of care. Notably, over half of the transition points identified by FNN and transitivity analysis were between ages 20 and 40, a population that is traditionally considered to be clinically homogeneous. Few transition points were identified between ages 3 and 17. Despite large social and developmental transition at those ages, the burden of multimorbidities may be consistent across the age range. Transition points derived through unsupervised machine learning approaches identify changes in the clinical phase that align with true differences in underlying multimorbidity burden. These transitions may be different from conventional pediatric and geriatric phases, which are often influenced by policy rather than clinical changes.

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models.

BACKGROUND: Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms. OBJECTIVES: This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes. METHODS: The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level. RESULTS: A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model. CONCLUSIONS: The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development.

Objectives: This study presents a design framework to enhance the accuracy by which large language models (LLMs), like ChatGPT can extract insights from clinical notes. We highlight this framework via prompt refinement for the automated determination of HEART (History, ECG, Age, Risk factors, Troponin risk algorithm) scores in chest pain evaluation. Methods: We developed a pipeline for LLM prompt testing, employing stochastic repeat testing and quantifying response errors relative to physician assessment. We evaluated the pipeline for automated HEART score determination across a limited set of 24 synthetic clinical notes representing four simulated patients. To assess whether iterative prompt design could improve the LLMs' ability to extract complex clinical concepts and apply rule-based logic to translate them to HEART subscores, we monitored diagnostic performance during prompt iteration. Results: Validation included three iterative rounds of prompt improvement for three HEART subscores with 25 repeat trials totaling 1200 queries each for GPT-3.5 and GPT-4. For both LLM models, from initial to final prompt design, there was a decrease in the rate of responses with erroneous, non-numerical subscore answers. Accuracy of numerical responses for HEART subscores (discrete 0-2 point scale) improved for GPT-4 from the initial to final prompt iteration, decreasing from a mean error of 0.16-0.10 (95% confidence interval: 0.07-0.14) points. Conclusion: We established a framework for iterative prompt design in the clinical space. Although the results indicate potential for integrating LLMs in structured clinical note analysis, translation to real, large-scale clinical data with appropriate data privacy safeguards is needed.

Identifying incarceration status in the electronic health record using large language models in emergency department settings.

Background: Incarceration is a significant social determinant of health, contributing to high morbidity, mortality, and racialized health inequities. However, incarceration status is largely invisible to health services research due to inadequate clinical electronic health record (EHR) capture. This study aims to develop, train, and validate natural language processing (NLP) techniques to more effectively identify incarceration status in the EHR. Methods: The study population consisted of adult patients (≥ 18 y.o.) who presented to the emergency department between June 2013 and August 2021. The EHR database was filtered for notes for specific incarceration-related terms, and then a random selection of 1,000 notes was annotated for incarceration and further stratified into specific statuses of prior history, recent, and current incarceration. For NLP model development, 80% of the notes were used to train the Longformer-based and RoBERTa algorithms. The remaining 20% of the notes underwent analysis with GPT-4. Results: There were 849 unique patients across 989 visits in the 1000 annotated notes. Manual annotation revealed that 559 of 1000 notes (55.9%) contained evidence of incarceration history. ICD-10 code (sensitivity: 4.8%, specificity: 99.1%, F1-score: 0.09) demonstrated inferior performance to RoBERTa NLP (sensitivity: 78.6%, specificity: 73.3%, F1-score: 0.79), Longformer NLP (sensitivity: 94.6%, specificity: 87.5%, F1-score: 0.93), and GPT-4 (sensitivity: 100%, specificity: 61.1%, F1-score: 0.86). Conclusions: Our advanced NLP models demonstrate a high degree of accuracy in identifying incarceration status from clinical notes. Further research is needed to explore their scaled implementation in population health initiatives and assess their potential to mitigate health disparities through tailored system interventions.