Use of a Stroke Alert Protocol and Outcomes at a Quaternary Children's Hospital.

OBJECTIVE: To assess for improvement in diagnostic efficiency following implementation of an institutional pediatric stroke alert protocol at a quaternary children's hospital, and to compare characteristics of in-hospital (IH) and out-of-hospital (OH) stroke alert activations. STUDY DESIGN: We retrospectively reviewed data from pediatric stroke alerts called for children between age 1 month to 21 years of age at our quaternary children's hospital between October 2016 and October 2022 after implementation of an institutional stroke alert protocol. Generalized linear models assessed code-to-image (CTI) time over the study period, with and without interaction terms for alert location. Demographic, clinical, and imaging characteristics between IH and OH alerts were compared using Fisher's exact test or Mann-Whitney U test. RESULTS: Of 206 total stroke activations, 129 (62.6%) occurred IH and 77 (37.4%) occurred OH. Overall mean CTI time decreased by 4.56 minutes per year (p = 0.007) after adjusting for confounders. The association between year and mean CTI time was significantly stronger for IH alerts (decrease of 8.33 minutes/year) compared with OH alerts (increase of 1.90 minutes/year). Subgroup analyses showed that CTI for computed tomography (CT) ± CT angiography and magnetic resonance imaging (MRI) without sedation improved, although CTI time for MRI with sedation did not change over time. IH/OH divergent trends were consistent for CT ± CTA and non-sedated MRI. CONCLUSION: After implementation of a pediatric stroke alert protocol, we observed a steady and significant improvement in CTI times for IH, but not OH alerts.

Leveraging a Large Language Model to Assess Quality-of-Care: Monitoring ADHD Medication Side Effects.

Objective: To assess the accuracy of a large language model (LLM) in measuring clinician adherence to practice guidelines for monitoring side effects after prescribing medications for children with attention-deficit/hyperactivity disorder (ADHD). Methods: Retrospective population-based cohort study of electronic health records. Cohort included children aged 6-11 years with ADHD diagnosis and ≥2 ADHD medication encounters (stimulants or non-stimulants prescribed) between 2015-2022 in a community-based primary healthcare network (n=1247). To identify documentation of side effects inquiry, we trained, tested, and deployed an open-source LLM (LLaMA) on all clinical notes from ADHD-related encounters (ADHD diagnosis or ADHD medication prescription), including in-clinic/telehealth and telephone encounters (n=15,593 notes). Model performance was assessed using holdout and deployment test sets, compared to manual chart review. Results: The LLaMA model achieved excellent performance in classifying notes that contain side effects inquiry (sensitivity= 87.2%, specificity=86.3/90.3%, area under curve (AUC)=0.93/0.92 on holdout/deployment test sets). Analyses revealed no model bias in relation to patient age, sex, or insurance. Mean age (SD) at first prescription was 8.8 (1.6) years; patient characteristics were similar across patients with and without documented side effects inquiry. Rates of documented side effects inquiry were lower in telephone encounters than in-clinic/telehealth encounters (51.9% vs. 73.0%, p<0.01). Side effects inquiry was documented in 61% of encounters following stimulant prescriptions and 48% of encounters following non-stimulant prescriptions (p<0.01). Conclusions: Deploying an LLM on a variable set of clinical notes, including telephone notes, offered scalable measurement of quality-of-care and uncovered opportunities to improve psychopharmacological medication management in primary care.