Moving Biosurveillance Beyond Coded Data Using AI for Symptom Detection From Physician Notes: Retrospective Cohort Study.

BACKGROUND: Real-time surveillance of emerging infectious diseases necessitates a dynamically evolving, computable case definition, which frequently incorporates symptom-related criteria. For symptom detection, both population health monitoring platforms and research initiatives primarily depend on structured data extracted from electronic health records. OBJECTIVE: This study sought to validate and test an artificial intelligence (AI)-based natural language processing (NLP) pipeline for detecting COVID-19 symptoms from physician notes in pediatric patients. We specifically study patients presenting to the emergency department (ED) who can be sentinel cases in an outbreak. METHODS: Subjects in this retrospective cohort study are patients who are 21 years of age and younger, who presented to a pediatric ED at a large academic children's hospital between March 1, 2020, and May 31, 2022. The ED notes for all patients were processed with an NLP pipeline tuned to detect the mention of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria. For a gold standard, 3 subject matter experts labeled 226 ED notes and had strong agreement (F1-score=0.986; positive predictive value [PPV]=0.972; and sensitivity=1.0). F1-score, PPV, and sensitivity were used to compare the performance of both NLP and the International Classification of Diseases, 10th Revision (ICD-10) coding to the gold standard chart review. As a formative use case, variations in symptom patterns were measured across SARS-CoV-2 variant eras. RESULTS: There were 85,678 ED encounters during the study period, including 4% (n=3420) with patients with COVID-19. NLP was more accurate at identifying encounters with patients that had any of the COVID-19 symptoms (F1-score=0.796) than ICD-10 codes (F1-score =0.451). NLP accuracy was higher for positive symptoms (sensitivity=0.930) than ICD-10 (sensitivity=0.300). However, ICD-10 accuracy was higher for negative symptoms (specificity=0.994) than NLP (specificity=0.917). Congestion or runny nose showed the highest accuracy difference (NLP: F1-score=0.828 and ICD-10: F1-score=0.042). For encounters with patients with COVID-19, prevalence estimates of each NLP symptom differed across variant eras. Patients with COVID-19 were more likely to have each NLP symptom detected than patients without this disease. Effect sizes (odds ratios) varied across pandemic eras. CONCLUSIONS: This study establishes the value of AI-based NLP as a highly effective tool for real-time COVID-19 symptom detection in pediatric patients, outperforming traditional ICD-10 methods. It also reveals the evolving nature of symptom prevalence across different virus variants, underscoring the need for dynamic, technology-driven approaches in infectious disease surveillance.

Coming in Hot: A quality improvement approach to improving care of febrile infants.

Background and Objectives: Significant practice variation exists in managing young infants with fever. Quality improvement strategies can aid in risk stratification and standardization of best care practices, along with a reduction of unnecessary interventions. The aim of this initiative was to safely reduce unnecessary admissions, antibiotics, and lumbar punctures (LPs) by 10% in low-risk, febrile infants aged 29 to 90 days presenting to the emergency department (ED) over a 12-month period. Methods: Using the Model for Improvement, a multidisciplinary team developed a multipronged intervention: an updated clinical decision tool (CDT), procalcitonin (PCT) adoption, education, a feedback tool, and best practice advisory (BPA) banner. Outcome measures included the proportion of low-risk infants that were admitted, received antibiotics, and had LPs. Process measures were adherence to the CDT and percentage of PCT ordered. Missed bacterial infections and return visits were balancing measures. The analysis was completed using descriptive statistics and statistical process control methods. Results: Five hundred and sixteen patients less than 90 days of age were included in the study, with 403 patients in the 29- to 90-day old subset of primary interest. In the low-risk group, a reduction in hospital admissions from a mean of 24.1% to 12.0% and a reduction in antibiotics from a mean of 15.2% to 1.3% was achieved. The mean proportion of LPs performed decreased in the intervention period from 7.5% to 1.8%, but special cause variation was not detected. Adherence to the CDT increased from 70.4% to 90.9% and PCT was ordered in 92.3% of cases. The proportion of missed bacterial infections was 0.3% at baseline and 0.5% in the intervention period while return visits were 6.7% at baseline and 5.0% in the intervention period. Conclusions: The implementation of a quality improvement strategy, including an updated evidence-based CDT for young infant fever incorporating PCT, safely reduced unnecessary care in low-risk, febrile infants aged 29 to 90 days in the ED. Purpose: To develop and implement a multipronged improvement strategy including an evidence-based CDT utilizing PCT to maximize value of care delivered to well-appearing, febrile infants presenting to EDs.