Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study.

BACKGROUND: Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. OBJECTIVE: The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. METHODS: Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient's care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. RESULTS: GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. CONCLUSIONS: Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.

A risk prediction score to identify patients at low risk for COVID-19 infection.

Introduction: Singapore's enhanced surveillance programme for COVID-19 identifies and isolates hospitalised patients with acute respiratory symptoms to prevent nosocomial spread. We developed risk prediction models to identify patients with low risk for COVID-19 from this cohort of hospitalised patients with acute respiratory symptoms. Methods: This was a single-centre retrospective observational study. Patients admitted to our institution's respiratory surveillance wards from 10 February to 30 April 2020 contributed data for analysis. Prediction models for COVID-19 were derived from a training cohort using variables based on demographics, clinical symptoms, exposure risks and blood investigations fitted into logistic regression models. The derived prediction models were subsequently validated on a test cohort. Results: Of the 1,228 patients analysed, 52 (4.2%) were diagnosed with COVID-19. Two prediction models were derived, the first based on age, presence of sore throat, dormitory residence, blood haemoglobin level (Hb), and total white blood cell counts (TW), and the second based on presence of headache, contact with infective patients, Hb and TW. Both models had good diagnostic performance with areas under the receiver operating characteristic curve of 0.934 and 0.866, respectively. Risk score cut-offs of 0.6 for Model 1 and 0.2 for Model 2 had 100% sensitivity, allowing identification of patients with low risk for COVID-19. Limiting COVID-19 screening to only elevated-risk patients reduced the number of isolation days for surveillance patients by up to 41.7% and COVID-19 swab testing by up to 41.0%. Conclusion: Prediction models derived from our study were able to identify patients at low risk for COVID-19 and rationalise resource utilisation.