Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes.

The emerging large language models (LLMs) are actively evaluated in various fields including healthcare. Most studies have focused on established benchmarks and standard parameters; however, the variation and impact of prompt engineering and fine-tuning strategies have not been fully explored. This study benchmarks GPT-3.5 Turbo, GPT-4, and Llama-7B against BERT models and medical fellows' annotations in identifying patients with metastatic cancer from discharge summaries. Results revealed that clear, concise prompts incorporating reasoning steps significantly enhanced performance. GPT-4 exhibited superior performance among all models. Notably, one-shot learning and fine-tuning provided no incremental benefit. The model's accuracy sustained even when keywords for metastatic cancer were removed or when half of the input tokens were randomly discarded. These findings underscore GPT-4's potential to substitute specialized models, such as PubMedBERT, through strategic prompt engineering, and suggest opportunities to improve open-source models, which are better suited to use in clinical settings.

Symptomatic Colorectal Cancer Is Associated With Stage IV Diagnosis in Two Disparate Populations.

In the United States, individuals of Black/African Ancestry (AA) have a higher incidence and mortality from colorectal cancer (CRC) compared to individuals of White/European Ancestry (EA). In order to develop an approach towards disentangling the complex effects of associated race and socioeconomic factors on CRC outcome, we have conducted a manual chart review of sporadic CRC pathological diagnoses (total n = 334) at an urban public hospital (UH) and a suburban university hospital (SH). There were significant differences between the SH and UH CRC patients with respect to Black/AA race (4.2% vs. 89.1%, p < 0.0001) and Medicaid/Self-pay insurance status (14.9% vs. 85.0%, p < 0.0001). While a higher proportion of newly diagnosed CRC patients presented with metastatic stage 4 CRC at the UH (21%) than the SH (12.5%), only the presence of symptoms was significantly associated with stage 4 CRC (odds ratio, OR 7.94, 95% confidence interval, CI 1.83- 34.54, p = 0.0057) in a multivariable generalized linear model (GLM). The proportion of asymptomatic CRC patients was ~20% at both institutions, suggesting that the UH has contributed to reducing CRC disparities. Initiation of CRC screening at the recommended age at both institutions could reduce the proportion of CRC patients presenting with metastatic spread.

Metagenomic and bile acid metabolomic analysis of fecal microbiota transplantation for recurrent Clostridiodes difficile and/or inflammatory bowel diseases.

Background: Fecal microbiota transplantation (FMT) is an effective treatment of recurrent Clostridioides difficile infections (rCDI), but has more limited efficacy in treating either ulcerative colitis (UC) or Crohn's disease (CD), two major forms of inflammatory bowel diseases (IBD). We hypothesize that FMT recipients with rCDI and/or IBD have baseline fecal bile acid (BA) compositions that differ significantly from that of their healthy donors and that FMT will normalize the BA compositions. Aim: To study the effect of single colonoscopic FMT on microbial composition and function in four recipient groups: 1.) rCDI patients without IBD (rCDI-IBD); 2.) rCDI with IBD (rCDI+IBD); 3.) UC patients without rCDI (UC-rCDI); 4.) CD patients without rCDI (CD-rCDI). Methods: We performed 16S rRNA gene sequence, shotgun DNA sequence and quantitative bile acid metabolomic analyses on stools collected from 55 pairs of subjects and donors enrolled in two prospective single arm FMT clinical trials (Clinical Trials.gov ID NCT03268213, 479696, UC no rCDI ≥ 2x IND 1564 and NCT03267238, IND 16795). Fitted linear mixed models were used to examine the effects of four recipient groups, FMT status (Donor, pre-FMT, 1-week post-FMT, 3-months post-FMT) and first order Group*FMT interactions on microbial diversity and composition, bile acid metabolites and bile acid metabolizing enzyme gene abundance. Results: The pre-FMT stools collected from rCDI ± IBD recipients had reduced α-diversity compared to the healthy donor stools and was restored post-FMT. The α-diversity in the pre-FMT stools collected from UC-rCDI or CD-rCDI recipients did not differ significantly from donor stools. FMT normalized some recipient/donor ratios of genus level taxa abundance in the four groups. Fecal secondary BA levels, including some of the secondary BA epimers that exhibit in vitro immunomodulatory activities, were lower in rCDI±IBD and CD-rCDI but not UC-rCDI recipients compared to donors. FMT restored secondary BA levels. Metagenomic baiE gene and some of the eight bile salt hydrolase (BSH) phylotype abundances were significantly correlated with fecal BA levels. Conclusion: Restoration of multiple secondary BA levels, including BA epimers implicated in immunoregulation, are associated with restoration of fecal baiE gene counts, suggesting that the 7-α-dehydroxylation step is rate-limiting.

Impact of type 2 diabetes on adenoma detection in screening colonoscopies performed in disparate populations.

BACKGROUND: The Black/African Ancestry (AA) population has a higher prevalence of type 2 diabetes mellitus (T2DM) and a higher incidence and mortality rate for colorectal cancer (CRC) than all other races in the United States. T2DM has been shown to increase adenoma risk in predominantly white/European ancestry (EA) populations, but the effect of T2DM on adenoma risk in Black/AA individuals is less clear. We hypothesize that T2DM has a significant effect on adenoma risk in a predominantly Black/AA population. AIM: To investigate the effect of T2DM and race on the adenoma detection rate (ADR) in screening colonoscopies in two disparate populations. METHODS: A retrospective cohort study was conducted on ADR during index screening colonoscopies (age 45-75) performed at an urban public hospital serving a predominantly Black/AA population (92%) (2017-2018, n = 1606). Clinical metadata collected included basic demographics, insurance, body mass index (BMI), family history of CRC, smoking, diabetes diagnosis, and aspirin use. This dataset was combined with a recently reported parallel retrospective cohort data set collected at a suburban university hospital serving a predominantly White/EA population (87%) (2012-2015, n = 2882). RESULTS: The ADR was higher in T2DM patients than in patients without T2DM or prediabetes (35.2% vs 27.9%, P = 0.0166, n = 981) at the urban public hospital. Multivariable analysis of the combined datasets showed that T2DM [odds ratio (OR) = 1.29, 95% confidence interval (CI): 1.08-1.55, P = 0.0049], smoking (current vs never OR = 1.47, 95%CI: 1.18-1.82, current vs past OR = 1.32, 95%CI: 1.02-1.70, P = 0.0026), older age (OR = 1.05 per year, 95%CI: 1.04-1.06, P < 0.0001), higher BMI (OR = 1.02 per unit, 95%CI: 1.01-1.03, P = 0.0003), and male sex (OR = 1.87, 95%CI: 1.62-2.15, P < 0.0001) were associated with increased ADR in the combined datasets, but race, aspirin use and insurance were not. CONCLUSION: T2DM, but not race, is significantly associated with increased ADR on index screening colonoscopy while controlling for other factors.

Single-Pass vs 2-Pass Endoscopic Ultrasound-Guided Fine-Needle Biopsy Sample Collection for Creation of Pancreatic Adenocarcinoma Organoids.

Pancreatic ductal adenocarcinoma (PDAC) has one of the poorest prognoses of all malignancies, with a 5-year survival rate <8%.1,2 Suspicious lesions are typically diagnosed via endoscopic ultrasound-guided fine-needle aspiration or endoscopic ultrasound-guided fine-needle biopsy (EUS-FNB).3 Fewer needle passes decreases the risk of postprocedure complications, including pancreatitis and hemorrhage, while allowing additional needle passes to be used for adjuvant tissue testing, such as organoid creation and DNA sequencing.