Can large language models reason about medical questions?

Although large language models often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether closed- and open-source models (GPT-3.5, Llama 2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-US Medical Licensing Examination [USMLE], MedMCQA, and PubMedQA) and multiple prompting scenarios: chain of thought (CoT; think step by step), few shot, and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason, and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions but also reaches the passing score on three datasets: MedQA-USMLE (60.2%), MedMCQA (62.7%), and PubMedQA (78.2%). Open-source models are closing the gap: Llama 2 70B also passed the MedQA-USMLE with 62.5% accuracy.

ThoughtSource: A central hub for large language model reasoning data.

Large language models (LLMs) such as GPT-4 have recently demonstrated impressive results across a wide range of tasks. LLMs are still limited, however, in that they frequently fail at complex reasoning, their reasoning processes are opaque, they are prone to 'hallucinate' facts, and there are concerns about their underlying biases. Letting models verbalize reasoning steps as natural language, a technique known as chain-of-thought prompting, has recently been proposed as a way to address some of these issues. Here we present ThoughtSource, a meta-dataset and software library for chain-of-thought (CoT) reasoning. The goal of ThoughtSource is to improve future artificial intelligence systems by facilitating qualitative understanding of CoTs, enabling empirical evaluations, and providing training data. This first release of ThoughtSource integrates seven scientific/medical, three general-domain and five math word question answering datasets.

Estimation of SARS-CoV-2 Infection Fatality Rate by Real-time Antibody Screening of Blood Donors.

BACKGROUND: The pandemic due to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has tremendous consequences for our societies. Knowledge of the seroprevalence of SARS-CoV-2 is needed to accurately monitor the spread of the epidemic and to calculate the infection fatality rate (IFR). These measures may help the authorities make informed decisions and adjust the current societal interventions. The objective was to perform nationwide real-time seroprevalence surveying among blood donors as a tool to estimate previous SARS-CoV-2 infections and the population-based IFR. METHODS: Danish blood donors aged 17-69 years giving blood 6 April to 3 May were tested for SARS-CoV-2 immunoglobulin M and G antibodies using a commercial lateral flow test. Antibody status was compared between geographical areas, and an estimate of the IFR was calculated. Seroprevalence was adjusted for assay sensitivity and specificity taking the uncertainties of the test validation into account when reporting the 95% confidence intervals (CIs). RESULTS: The first 20 640 blood donors were tested, and a combined adjusted seroprevalence of 1.9% (95% CI, .8-2.3) was calculated. The seroprevalence differed across areas. Using available data on fatalities and population numbers, a combined IFR in patients <70 years is estimated at 89 per 100 000 (95% CI, 72-211) infections. CONCLUSIONS: The IFR was estimated to be slightly lower than previously reported from other countries not using seroprevalence data. The IFR is likely severalfold lower than the current estimate. We have initiated real-time nationwide anti-SARS-CoV-2 seroprevalence surveying of blood donations as a tool in monitoring the epidemic.