Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Influenza vaccine effectiveness against hospitalized SARS-CoV-2 infection.

Background: Some studies conducted before the Delta and Omicron variant-dominant periods have indicated that influenza vaccination provided protection against COVID-19 infection or hospitalization, but these results were limited by small study cohorts and a lack of comprehensive data on patient characteristics. No studies have examined this question during the Delta and Omicron periods (08/01/2021 to 2/22/2022). Methods: We conducted a retrospective cohort study of influenza-vaccinated and unvaccinated patients in the Corewell Health East(CHE, formerly known as Beaumont Health), Corewell Health West(CHW, formerly known as Spectrum Health) and Michigan Medicine (MM) healthcare system during the Delta-dominant and Omicron-dominant periods. We used a test-negative, case-control analysis to assess the effectiveness of the influenza vaccine against hospitalized SARS-CoV-2 outcome in adults, while controlling for individual characteristics as well as pandameic severity and waning immunity of COVID-19 vaccine. Results: The influenza vaccination has shown to provided some protection against SARS-CoV-2 hospitalized outcome across three main healthcare systems. CHE site (odds ratio [OR]=0.73, vaccine effectiveness [VE]=27%, 95% confidence interval [CI]: [18-35], p<0.001), CHW site (OR=0.85, VE=15%, 95% CI: [6-24], p<0.001), MM (OR=0.50, VE=50%, 95% CI: [40-58], p <0.001) and overall (OR=0.75, VE=25%, 95% CI: [20-30], p <0.001). Conclusion: The influenza vaccine provides a small degree of protection against SARS-CoV-2 infection across our study sites.

BGLM: big data-guided LOINC mapping with multi-language support.

Motivation: Mapping internal, locally used lab test codes to standardized logical observation identifiers names and codes (LOINC) terminology has become an essential step in harmonizing electronic health record (EHR) data across different institutions. However, most existing LOINC code mappers are based on text-mining technology and do not provide robust multi-language support. Materials and methods: We introduce a simple, yet effective tool called big data-guided LOINC code mapper (BGLM), which leverages the large amount of patient data stored in EHR systems to perform LOINC coding mapping. Distinguishing from existing methods, BGLM conducts mapping based on distributional similarity. Results: We validated the performance of BGLM with real-world datasets and showed that high mapping precision could be achieved under proper false discovery rate control. In addition, we showed that the mapping results of BGLM could be used to boost the performance of Regenstrief LOINC Mapping Assistant (RELMA), one of the most widely used LOINC code mappers. Conclusions: BGLM paves a new way for LOINC code mapping and therefore could be applied to EHR systems without the restriction of languages. BGLM is freely available at https://github.com/Bin-Chen-Lab/BGLM.

MetBERT: a generalizable and pre-trained deep learning model for the prediction of metastatic cancer from clinical notes.

Distant metastasis is the major cause of cancer-related deaths; however, early diagnosis of cancer metastasis remains a significant challenge. The recent advances in pre-trained natural language processing models coupled with the accumulation of publicly available Electronic Health Records (EHR) data provide an unprecedented opportunity to computationally tackle the challenge. Here, we fine-tuned multiple state-of-the-art BERT-based models using discharge summaries from the open MIMIC-III dataset and derived MetBERT, a novel model tailored to predict cancer metastasis from clinical notes. MetBERT achieved high performance (AUC=0.94) on our in-house validation dataset, suggesting its high generalizability. In addition, MetBERT enabled determining the date of cancer metastasis using the rich information in clinical notes and therefore could be potentially deployed as a tool for early diagnosis. Finally, we interpreted MetBERT at different scales and revealed a possible association between radiation therapy and metastasis risk in multiple cancer types.