Development of a liver disease-specific large language model chat interface using retrieval-augmented generation.

BACKGROUND AND AIMS: Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows the embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. APPROACH AND RESULTS: We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. RESULTS: We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. CONCLUSIONS: In this demonstration, we built disease-specific and protected health information-compliant LLMs using RAG. While LiVersa demonstrated higher accuracy in answering questions related to hepatology, there were some deficiencies due to limitations set by the number of documents used for RAG. LiVersa will likely require further refinement before potential live deployment. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical use cases.

Development of a Liver Disease-Specific Large Language Model Chat Interface using Retrieval Augmented Generation.

Background: Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating incorrect or hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. Methods: We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information (PHI)-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases (AASLD) guidelines and guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by comparing its responses versus those of trainees from a previously published knowledge assessment study regarding hepatitis B (HBV) treatment and hepatocellular carcinoma (HCC) surveillance. Results: LiVersa answered all 10 questions correctly when forced to provide a "yes" or "no" answer. Full detailed responses with justifications and rationales, however, were not completely correct for three of the questions. Discussions: In this study, we demonstrated the ability to build disease-specific and PHI-compliant LLMs using RAG. While our LLM, LiVersa, demonstrated more specificity in answering questions related to clinical hepatology - there were some knowledge deficiencies due to limitations set by the number and types of documents used for RAG. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical uses and a potential strategy to realize personalized medicine in the future.

Association of Geriatric Comanagement and 90-Day Postoperative Mortality Among Patients Aged 75 Years and Older With Cancer.

Importance: Collaboration between geriatricians and surgeons in the perioperative treatment of older patients has been associated with improved outcomes in several nononcologic specialties. Similar associations may be possible among older patients with cancer. Objective: To investigate the associations of geriatric comanagement of care for older patients undergoing cancer-related surgical treatment with 90-day postoperative mortality, rate of adverse surgical events, and postoperative use of inpatient supportive care services. Design, Setting, and Participants: This retrospective cohort study assessed outcomes of patients who received geriatric comanaged care vs those who did not using multivariable logistic regression analysis, with 90-day mortality as the outcome and geriatric comanagement of care as the main variable, with adjustment for age, sex, American Society of Anesthesiology score, Memorial Sloan Kettering Frailty Index score, preoperative albumin level, operative time, and estimated blood loss. A similar model was used to assess the association of geriatric comanagement with adverse surgical events, defined as any major complication, readmission, or emergency department visit within 30 days. Patients aged 75 years and older who underwent an elective surgical procedure with a hospital stay of at least 1 day at a single tertiary-care cancer center between February 2015 and February 2018 were included. Data were analyzed from January to July 2019. Exposures: Postoperative care comanaged by the geriatrics service and surgical service (geriatric comanagement group) vs by the surgical service only (surgical service group). Main Outcomes and Measures: 90-day mortality, adverse surgical events, and use of supportive care services. Results: Of 1892 patients included, 1020 (53.9%) received geriatric comanagement of care; these patients, compared with those who received care managed by the surgery service only, were older (mean [SD] age, 81 [4] years vs 80 [4] years; P < .001), had longer operative time (mean [SD], 203 [146] minutes vs 138 [112] minutes; P < .001), and longer length of stay (median [interquartile range], 5 [3-8] days vs 4 [2-7] days; P < .001). There were no differences in the proportions of men (488 [47.8%] men vs 450 [51.6%] men; P = .11). Adverse surgical events were not significantly different between groups (odds ratio, 0.93 [95% CI, 0.73-1.18]; P = .54). However, the adjusted probability of death within 90 days after surgical treatment was 4.3% for the geriatric comanagement group vs 8.9% for the surgical service group (difference, 4.6% [95% CI, 2.3%-6.9%]; P < .001). Additionally, compared with patients who received postoperative care management from the surgery service only, a higher proportion of patients in the geriatric comanagement group received inpatient supportive care services, including physical therapy (555 patients [63.6%] vs 820 patients [80.4%]; P < .001), occupational therapy (220 patients [25.2%] vs 385 patients [37.7%]; P < .001), speech and swallow rehabilitation (42 patients [4.8%] vs 86 patients [8.4%]; P = .002), and nutrition services (637 patients [73.1%] vs 803 patients [78.7%]; P = .004). Conclusions and Relevance: This cohort study found that geriatric comanagement was associated with significantly lower 90-day postoperative mortality among older patients with cancer. These findings suggest that such patients may benefit from geriatric comanagement, which could improve their ability to survive adverse postoperative events.

The Relationship Between Shiftwork and Violence Against Nurses: A Case Control Study.

This study investigated the potential relationship between shiftwork and work-related physical assault (PA) against nurses who are at high risk of violence globally. Nurses (6,300), randomly selected from the licensing database and working in Minnesota, were surveyed regarding PA experiences. Through a nested case-control study, nurses who reported a PA in the previous 12 months and controls who were randomly selected from their assault-free working months, respectively, identified exposures experienced during the month prior to the assault month (cases) and the random non-PA months (controls). Comparing case and control exposures, shiftwork was examined relevant to PA. Among 310 cases and 946 controls, most worked 8 hours or less (87%, 88%) during day shifts (44%, 70%). Multivariable analyses (odds ratios [ORs] and 95% confidence intervals [CIs]) revealed increased risk of PA for nurses working evening (OR = 1.55, 95% CI = [1.05, 2.27]), night (OR = 3.54, 95% CI = [2.31, 5.44]), and rotating day and evening (OR = 2.88, 95% CI = [1.22, 6.80]) shifts, which provides a basis for intervention opportunities.

Direct medical costs and medication compliance among fibromyalgia patients: duloxetine initiators vs. pregabalin initiators.

OBJECTIVES: To assess and compare direct medical costs and medication compliance between patients with fibromyalgia who initiated duloxetine and patients with fibromyalgia who initiated pregabalin in 2008. METHODS: A retrospective cohort study design was used based on a large US national commercial claims database (2006 to 2009). Patients with fibromyalgia aged 18 to 64 who initiated duloxetine or pregabalin in 2008 and who had continuous health insurance 1 year preceding and 1 year following the initiation were selected into duloxetine cohort or pregabalin cohort based on their initiated agent. Medication compliance was measured by total supply days, medication possession ratio (MPR), and proportion of patients with MPR ≥ 0.8. Direct medical costs were measured by annual costs per patient and compared between the cohorts in the year following the initiation. Propensity score stratification and bootstrapping methods were used to adjust for distribution bias, as well as cross-cohort differences in demographic, clinical and economic characteristics, and medication history prior to the initiation. RESULTS: Both the duloxetine (n = 3,033) and pregabalin (n = 4,838) cohorts had a mean initiation age around 49 years, 89% were women. During the postindex year, compared to the pregabalin cohort, the duloxetine cohort had higher totally annual supply days (273.5 vs. 176.6, P < 0.05), higher MPR (0.7 vs. 0.5, P < 0.05), and more patients with MPR ≥ 0.8 (45.1% vs. 29.4%, P < 0.05). Further, relative to pregabalin cohort, duloxetine cohort had lower inpatient costs ($2,994.9 vs. $4,949.6, P < 0.05), lower outpatient costs ($8,259.6 vs. $10,312.2, P < 0.05), similar medication costs ($5,214.6 vs. $5,290.8, P > 0.05), and lower total medical costs ($16,469.1 vs. $20,552.6, P < 0.05) in the postinitiation year. CONCLUSIONS: In a real-world setting, patients with fibromyalgia who initiated duloxetine in 2008 had better medication compliance and consumed less inpatient, outpatient, and total medical costs than those who initiated pregabalin.