Evaluating large language models on medical evidence summarization.

Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study demonstrates that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.

Evaluating Large Language Models on Medical Evidence Summarization.

Recent advances in large language models (LLMs) have demonstrated remarkable successes in zero- and few-shot performance on various downstream tasks, paving the way for applications in high-stakes domains. In this study, we systematically examine the capabilities and limitations of LLMs, specifically GPT-3.5 and ChatGPT, in performing zero-shot medical evidence summarization across six clinical domains. We conduct both automatic and human evaluations, covering several dimensions of summary quality. Our study has demonstrated that automatic metrics often do not strongly correlate with the quality of summaries. Furthermore, informed by our human evaluations, we define a terminology of error types for medical evidence summarization. Our findings reveal that LLMs could be susceptible to generating factually inconsistent summaries and making overly convincing or uncertain statements, leading to potential harm due to misinformation. Moreover, we find that models struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts.

Metformin Is Associated With a Lower Risk of Atrial Fibrillation and Ventricular Arrhythmias Compared With Sulfonylureas: An Observational Study.

[Figure: see text].

Clinical and cardiac characteristics of COVID-19 mortalities in a diverse New York City Cohort.

INTRODUCTION: Electrocardiographic characteristics in COVID-19-related mortality have not yet been reported, particularly in racial/ethnic minorities. METHODS AND RESULTS: We reviewed demographics, laboratory and cardiac tests, medications, and cardiac rhythm proximate to death or initiation of comfort care for patients hospitalized with a positive SARS-CoV-2 reverse-transcriptase polymerase chain reaction in three New York City hospitals between March 1 and April 3, 2020 who died. We described clinical characteristics and compared factors contributing toward arrhythmic versus nonarrhythmic death. Of 1258 patients screened, 133 died and were enrolled. Of these, 55.6% (74/133) were male, 69.9% (93/133) were racial/ethnic minorities, and 88.0% (117/133) had cardiovascular disease. The last cardiac rhythm recorded was VT or fibrillation in 5.3% (7/133), pulseless electrical activity in 7.5% (10/133), unspecified bradycardia in 0.8% (1/133), and asystole in 26.3% (35/133). Most 74.4% (99/133) died receiving comfort measures only. The most common abnormalities on admission electrocardiogram included abnormal QRS axis (25.8%), atrial fibrillation/flutter (14.3%), atrial ectopy (12.0%), and right bundle branch block (11.9%). During hospitalization, an additional 17.6% developed atrial ectopy, 14.7% ventricular ectopy, 10.1% atrial fibrillation/flutter, and 7.8% a right ventricular abnormality. Arrhythmic death was confirmed or suspected in 8.3% (11/133) associated with age, coronary artery disease, asthma, vasopressor use, longer admission corrected QT interval, and left bundle branch block (LBBB). CONCLUSIONS: Conduction, rhythm, and electrocardiographic abnormalities were common during COVID-19-related hospitalization. Arrhythmic death was associated with age, coronary artery disease, asthma, longer admission corrected QT interval, LBBB, ventricular ectopy, and usage of vasopressors. Most died receiving comfort measures.