Your browser doesn't support javascript.
loading
Almanac - Retrieval-Augmented Language Models for Clinical Medicine.
Zakka, Cyril; Shad, Rohan; Chaurasia, Akash; Dalal, Alex R; Kim, Jennifer L; Moor, Michael; Fong, Robyn; Phillips, Curran; Alexander, Kevin; Ashley, Euan; Boyd, Jack; Boyd, Kathleen; Hirsch, Karen; Langlotz, Curt; Lee, Rita; Melia, Joanna; Nelson, Joanna; Sallam, Karim; Tullis, Stacey; Vogelsong, Melissa Ann; Cunningham, John Patrick; Hiesinger, William.
Afiliação
  • Zakka C; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Shad R; Division of Cardiovascular Surgery, Penn Medicine, Philadelphia.
  • Chaurasia A; Department of Computer Science, Stanford University, Stanford, CA.
  • Dalal AR; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Kim JL; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Moor M; Department of Computer Science, Stanford University, Stanford, CA.
  • Fong R; Department of Computer Science, Stanford University, Stanford, CA.
  • Phillips C; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Alexander K; Division of Cardiovascular Medicine, Stanford Medicine, Stanford, CA.
  • Ashley E; Division of Cardiovascular Medicine, Stanford Medicine, Stanford, CA.
  • Boyd J; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Boyd K; Department of Pediatrics, Stanford Medicine, Stanford, CA.
  • Hirsch K; Department of Neurology, Stanford Medicine, Stanford, CA.
  • Langlotz C; Department of Radiology and Biomedical Informatics, Stanford Medicine, Stanford, CA.
  • Lee R; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Melia J; Division of Gastroenterology and Hepatology, Johns Hopkins Medicine, Baltimore.
  • Nelson J; Division of Infectious Diseases, Stanford Medicine, Stanford, CA.
  • Sallam K; Division of Cardiovascular Medicine, Stanford Medicine, Stanford, CA.
  • Tullis S; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
  • Vogelsong MA; Division of Anesthesia, Stanford Medicine, Stanford, CA.
  • Cunningham JP; Department of Statistics, Columbia University, New York.
  • Hiesinger W; Department of Cardiothoracic Surgery, Stanford Medicine, Stanford, CA.
NEJM AI ; 1(2)2024 Feb.
Article em En | MEDLINE | ID: mdl-38343631
ABSTRACT

BACKGROUND:

Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements.

METHODS:

We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties.

RESULTS:

Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety.

CONCLUSIONS:

Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).

Texto completo: 1 Base de dados: MEDLINE Tipo de estudo: Guideline / Prognostic_studies Idioma: En Ano de publicação: 2024 Tipo de documento: Article

Texto completo: 1 Base de dados: MEDLINE Tipo de estudo: Guideline / Prognostic_studies Idioma: En Ano de publicação: 2024 Tipo de documento: Article