Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Influenza vaccine effectiveness against hospitalized SARS-CoV-2 infection.

Background: Some studies conducted before the Delta and Omicron variant-dominant periods have indicated that influenza vaccination provided protection against COVID-19 infection or hospitalization, but these results were limited by small study cohorts and a lack of comprehensive data on patient characteristics. No studies have examined this question during the Delta and Omicron periods (08/01/2021 to 2/22/2022). Methods: We conducted a retrospective cohort study of influenza-vaccinated and unvaccinated patients in the Corewell Health East(CHE, formerly known as Beaumont Health), Corewell Health West(CHW, formerly known as Spectrum Health) and Michigan Medicine (MM) healthcare system during the Delta-dominant and Omicron-dominant periods. We used a test-negative, case-control analysis to assess the effectiveness of the influenza vaccine against hospitalized SARS-CoV-2 outcome in adults, while controlling for individual characteristics as well as pandameic severity and waning immunity of COVID-19 vaccine. Results: The influenza vaccination has shown to provided some protection against SARS-CoV-2 hospitalized outcome across three main healthcare systems. CHE site (odds ratio [OR]=0.73, vaccine effectiveness [VE]=27%, 95% confidence interval [CI]: [18-35], p<0.001), CHW site (OR=0.85, VE=15%, 95% CI: [6-24], p<0.001), MM (OR=0.50, VE=50%, 95% CI: [40-58], p <0.001) and overall (OR=0.75, VE=25%, 95% CI: [20-30], p <0.001). Conclusion: The influenza vaccine provides a small degree of protection against SARS-CoV-2 infection across our study sites.

Differential Transcriptomic Signatures of Small Airway Cell Cultures Derived from IPF and COVID-19-Induced Exacerbation of Interstitial Lung Disease.

Idiopathic pulmonary fibrosis (IPF) is a pathological condition wherein lung injury precipitates the deposition of scar tissue, ultimately leading to a decline in pulmonary function. Existing research indicates a notable exacerbation in the clinical prognosis of IPF patients following infection with COVID-19. This investigation employed bulk RNA-sequencing methodologies to describe the transcriptomic profiles of small airway cell cultures derived from IPF and post-COVID fibrosis patients. Differential gene expression analysis unveiled heightened activation of pathways associated with microtubule assembly and interferon signaling in IPF cell cultures. Conversely, post-COVID fibrosis cell cultures exhibited distinctive characteristics, including the upregulation of pathways linked to extracellular matrix remodeling, immune system response, and TGF-ß1 signaling. Notably, BMP signaling levels were elevated in cell cultures derived from IPF patients compared to non-IPF control and post-COVID fibrosis samples. These findings underscore the molecular distinctions between IPF and post-COVID fibrosis, particularly in the context of signaling pathways associated with each condition. A better understanding of the underlying molecular mechanisms holds the promise of identifying potential therapeutic targets for future interventions in these diseases.

Targeting ATP12A, a Nongastric Proton Pump α Subunit, for Idiopathic Pulmonary Fibrosis Treatment.

Idiopathic pulmonary fibrosis (IPF) is a pathological condition of unknown etiology that results from injury to the lung and an ensuing fibrotic response that leads to the thickening of the alveolar walls and obliteration of the alveolar space. The pathogenesis is not clear, and there are currently no effective therapies for IPF. Small airway disease and mucus accumulation are prominent features in IPF lungs, similar to cystic fibrosis lung disease. The ATP12A gene encodes the α-subunit of the nongastric H+, K+-ATPase, which functions to acidify the airway surface fluid and impairs mucociliary transport function in patients with cystic fibrosis. It is hypothesized that the ATP12A protein may play a role in the pathogenesis of IPF. The authors' studies demonstrate that ATP12A protein is overexpressed in distal small airways from the lungs of patients with IPF compared with normal human lungs. In addition, overexpression of the ATP12A protein in mouse lungs worsened bleomycin induced experimental pulmonary fibrosis. This was prevented by a potassium competitive proton pump blocker, vonoprazan. These data support the concept that the ATP12A protein plays an important role in the pathogenesis of lung fibrosis. Inhibition of the ATP12A protein has potential as a novel therapeutic strategy in IPF treatment.

BGLM: big data-guided LOINC mapping with multi-language support.

Motivation: Mapping internal, locally used lab test codes to standardized logical observation identifiers names and codes (LOINC) terminology has become an essential step in harmonizing electronic health record (EHR) data across different institutions. However, most existing LOINC code mappers are based on text-mining technology and do not provide robust multi-language support. Materials and methods: We introduce a simple, yet effective tool called big data-guided LOINC code mapper (BGLM), which leverages the large amount of patient data stored in EHR systems to perform LOINC coding mapping. Distinguishing from existing methods, BGLM conducts mapping based on distributional similarity. Results: We validated the performance of BGLM with real-world datasets and showed that high mapping precision could be achieved under proper false discovery rate control. In addition, we showed that the mapping results of BGLM could be used to boost the performance of Regenstrief LOINC Mapping Assistant (RELMA), one of the most widely used LOINC code mappers. Conclusions: BGLM paves a new way for LOINC code mapping and therefore could be applied to EHR systems without the restriction of languages. BGLM is freely available at https://github.com/Bin-Chen-Lab/BGLM.

MetBERT: a generalizable and pre-trained deep learning model for the prediction of metastatic cancer from clinical notes.

Distant metastasis is the major cause of cancer-related deaths; however, early diagnosis of cancer metastasis remains a significant challenge. The recent advances in pre-trained natural language processing models coupled with the accumulation of publicly available Electronic Health Records (EHR) data provide an unprecedented opportunity to computationally tackle the challenge. Here, we fine-tuned multiple state-of-the-art BERT-based models using discharge summaries from the open MIMIC-III dataset and derived MetBERT, a novel model tailored to predict cancer metastasis from clinical notes. MetBERT achieved high performance (AUC=0.94) on our in-house validation dataset, suggesting its high generalizability. In addition, MetBERT enabled determining the date of cancer metastasis using the rich information in clinical notes and therefore could be potentially deployed as a tool for early diagnosis. Finally, we interpreted MetBERT at different scales and revealed a possible association between radiation therapy and metastasis risk in multiple cancer types.

High-Density Blood Transcriptomics Reveals Precision Immune Signatures of SARS-CoV-2 Infection in Hospitalized Individuals.

The immune response to COVID-19 infection is variable. How COVID-19 influences clinical outcomes in hospitalized patients needs to be understood through readily obtainable biological materials, such as blood. We hypothesized that a high-density analysis of host (and pathogen) blood RNA in hospitalized patients with SARS-CoV-2 would provide mechanistic insights into the heterogeneity of response amongst COVID-19 patients when combined with advanced multidimensional bioinformatics for RNA. We enrolled 36 hospitalized COVID-19 patients (11 died) and 15 controls, collecting 74 blood PAXgene RNA tubes at multiple timepoints, one early and in 23 patients after treatment with various therapies. Total RNAseq was performed at high-density, with >160 million paired-end, 150 base pair reads per sample, representing the most sequenced bases per sample for any publicly deposited blood PAXgene tube study. There are 770 genes significantly altered in the blood of COVID-19 patients associated with antiviral defense, mitotic cell cycle, type I interferon signaling, and severe viral infections. Immune genes activated include those associated with neutrophil mechanisms, secretory granules, and neutrophil extracellular traps (NETs), along with decreased gene expression in lymphocytes and clonal expansion of the acquired immune response. Therapies such as convalescent serum and dexamethasone reduced many of the blood expression signatures of COVID-19. Severely ill or deceased patients are marked by various secondary infections, unique gene patterns, dysregulated innate response, and peripheral organ damage not otherwise found in the cohort. High-density transcriptomic data offers shared gene expression signatures, providing unique insights into the immune system and individualized signatures of patients that could be used to understand the patient's clinical condition. Whole blood transcriptomics provides patient-level insights for immune activation, immune repertoire, and secondary infections that can further guide precision treatment.

Development of a Center for Personalized Cancer Care at a Regional Cancer Center: Feasibility Trial of an Institutional Tumor Sequencing Advisory Board.

Next-generation sequencing (NGS) capabilities can affect therapeutic decisions in patients with complex, advanced, or refractory cancer. We report the feasibility of a tumor sequencing advisory board at a regional cancer center. Specimens were analyzed for approximately 2800 mutations in 50 genes. Outcomes of interest included tumor sequencing advisory board function and processes, timely discussion of results, and proportion of reports having potentially actionable mutations. NGS results were successfully generated for 15 patients, with median time from tissue processing to reporting of 11.6 days (range, 5 to 21 days), and presented at a biweekly multidisciplinary tumor sequencing advisory board. Attendance averaged 19 participants (range, 12 to 24) at 20 days after patient enrollment (range, 10 to 30 days). Twenty-seven (range, 1 to 4 per patient) potentially actionable mutations were detected in 11 of 15 patients: TP53 (n = 6), KRAS (n = 4), MET (n = 3), APC (n = 3), CDKN2A (n = 2), PTEN (n = 2), PIK3CA, FLT3, NRAS, VHL, BRAF, SMAD4, and ATM. The Hotspot Panel is now offered as a clinically available test at our institution. NGS results can be obtained by in-house high-throughput sequencing and reviewed in a multidisciplinary tumor sequencing advisory board in a clinically relevant manner. The essential components of a center for personalized cancer care can support clinical decisions outside the university.