Development of a liver disease-specific large language model chat interface using retrieval-augmented generation.

BACKGROUND AND AIMS: Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows the embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. APPROACH AND RESULTS: We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. RESULTS: We evaluated LiVersa's performance by conducting 2 rounds of testing. First, we compared LiVersa's outputs versus those of trainees from a previously published knowledge assessment. LiVersa answered all 10 questions correctly. Second, we asked 15 hepatologists to evaluate the outputs of 10 hepatology topic questions generated by LiVersa, OpenAI's ChatGPT 4, and Meta's Large Language Model Meta AI 2. LiVersa's outputs were more accurate but were rated less comprehensive and safe compared to those of ChatGPT 4. CONCLUSIONS: In this demonstration, we built disease-specific and protected health information-compliant LLMs using RAG. While LiVersa demonstrated higher accuracy in answering questions related to hepatology, there were some deficiencies due to limitations set by the number of documents used for RAG. LiVersa will likely require further refinement before potential live deployment. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical use cases.

Development of a Liver Disease-Specific Large Language Model Chat Interface using Retrieval Augmented Generation.

Background: Large language models (LLMs) have significant capabilities in clinical information processing tasks. Commercially available LLMs, however, are not optimized for clinical uses and are prone to generating incorrect or hallucinatory information. Retrieval-augmented generation (RAG) is an enterprise architecture that allows embedding of customized data into LLMs. This approach "specializes" the LLMs and is thought to reduce hallucinations. Methods: We developed "LiVersa," a liver disease-specific LLM, by using our institution's protected health information (PHI)-complaint text embedding and LLM platform, "Versa." We conducted RAG on 30 publicly available American Association for the Study of Liver Diseases (AASLD) guidelines and guidance documents to be incorporated into LiVersa. We evaluated LiVersa's performance by comparing its responses versus those of trainees from a previously published knowledge assessment study regarding hepatitis B (HBV) treatment and hepatocellular carcinoma (HCC) surveillance. Results: LiVersa answered all 10 questions correctly when forced to provide a "yes" or "no" answer. Full detailed responses with justifications and rationales, however, were not completely correct for three of the questions. Discussions: In this study, we demonstrated the ability to build disease-specific and PHI-compliant LLMs using RAG. While our LLM, LiVersa, demonstrated more specificity in answering questions related to clinical hepatology - there were some knowledge deficiencies due to limitations set by the number and types of documents used for RAG. The LiVersa prototype, however, is a proof of concept for utilizing RAG to customize LLMs for clinical uses and a potential strategy to realize personalized medicine in the future.

Rolling Up the Sleeve: Equitable, Efficient, and Safe COVID-19 Mass Immunization for Academic Medical Center Employees.

BACKGROUND: Novel coronavirus disease 2019 (COVID-19) vaccine administration has faced distribution barriers across the United States. We sought to delineate our vaccine delivery experience in the first week of vaccine availability, and our effort to prioritize employees based on risk with a goal of providing an efficient infrastructure to optimize speed and efficiency of vaccine delivery while minimizing risk of infection during the immunization process. OBJECTIVE: This article aims to evaluate an employee prioritization/invitation/scheduling system, leveraging an integrated electronic health record patient portal framework for employee COVID-19 immunizations at an academic medical center. METHODS: We conducted an observational cross-sectional study during January 2021 at a single urban academic center. All employees who met COVID-19 allocation vaccine criteria for phase 1a.1 to 1a.4 were included. We implemented a prioritization/invitation/scheduling framework and evaluated time from invitation to scheduling as a proxy for vaccine interest and arrival to vaccine administration to measure operational throughput. RESULTS: We allotted vaccines for 13,753 employees but only 10,662 employees with an active patient portal account received an invitation. Of those with an active account, 6,483 (61%) scheduled an appointment and 6,251 (59%) were immunized in the first 7 days. About 66% of invited providers were vaccinated in the first 7 days. In contrast, only 41% of invited facility/food service employees received the first dose of the vaccine in the first 7 days (p < 0.001). At the vaccination site, employees waited 5.6 minutes (interquartile range [IQR]: 3.9-8.3) from arrival to vaccination. CONCLUSION: We developed a system of early COVID-19 vaccine prioritization and administration in our health care system. We saw strong early acceptance in those with proximal exposure to COVID-19 but noticed significant difference in the willingness of different employee groups to receive the vaccine.

Rapid Development of Specialty Population Registries and Quality Measures from Electronic Health Record Data*. An Agile Framework.

BACKGROUND: Creation of a new electronic health record (EHR)-based registry often can be a "one-off" complex endeavor: first developing new EHR data collection and clinical decision support tools, followed by developing registry-specific data extractions from the EHR for analysis. Each development phase typically has its own long development and testing time, leading to a prolonged overall cycle time for delivering one functioning registry with companion reporting into production. The next registry request then starts from scratch. Such an approach will not scale to meet the emerging demand for specialty registries to support population health and value-based care. OBJECTIVE: To determine if the creation of EHR-based specialty registries could be markedly accelerated by employing (a) a finite core set of EHR data collection principles and methods, (b) concurrent engineering of data extraction and data warehouse design using a common dimensional data model for all registries, and (c) agile development methods commonly employed in new product development. METHODS: We adopted as guiding principles to (a) capture data as a byproduct of care of the patient, (b) reinforce optimal EHR use by clinicians, (c) employ a finite but robust set of EHR data capture tool types, and (d) leverage our existing technology toolkit. Registries were defined by a shared condition (recorded on the Problem List) or a shared exposure to a procedure (recorded on the Surgical History) or to a medication (recorded on the Medication List). Any EHR fields needed - either to determine registry membership or to calculate a registry-associated clinical quality measure (CQM) - were included in the enterprise data warehouse (EDW) shared dimensional data model. Extract-transform-load (ETL) code was written to pull data at defined "grains" from the EHR into the EDW model. All calculated CQM values were stored in a single Fact table in the EDW crossing all registries. Registry-specific dashboards were created in the EHR to display both (a) real-time patient lists of registry patients and (b) EDW-generated CQM data. Agile project management methods were employed, including co-development, lightweight requirements documentation with User Stories and acceptance criteria, and time-boxed iterative development of EHR features in 2-week "sprints" for rapid-cycle feedback and refinement. RESULTS: Using this approach, in calendar year 2015 we developed a total of 43 specialty chronic disease registries, with 111 new EHR data collection and clinical decision support tools, 163 new clinical quality measures, and 30 clinic-specific dashboards reporting on both real-time patient care gaps and summarized and vetted CQM measure performance trends. CONCLUSIONS: This study suggests concurrent design of EHR data collection tools and reporting can quickly yield useful EHR structured data for chronic disease registries, and bodes well for efforts to migrate away from manual abstraction. This work also supports the view that in new EHR-based registry development, as in new product development, adopting agile principles and practices can help deliver valued, high-quality features early and often.

Quality of life in older Chinese-speaking adults with hearing impairment.

PURPOSE: General and hearing-specific health-related quality of life (HRQoL) was examined in elderly Chinese with hearing impairment. METHODS: Sixty-four Chinese speakers aged ≥65 years and did not use hearing aids were evaluated using Chinese versions of the Short-Form 36 health survey (SF-36) and the Hearing Handicap Inventory for the Elderly (Screening Version) (HHIE-S). Results on the SF-36 were compared to norms obtained in a general elderly Chinese population. The relationships between HRQoL and degree of hearing impairment, and between SF-36 and HHIE-S were also evaluated. RESULTS: Elderly Chinese speakers with hearing impairment rated six of the eight scales of the SF-36 poorer, compared to a general elderly Chinese population. When average hearing impairment in the better ear exceeded 40 dB HL, SF-36 ratings were poorer than those with better hearing. Poorer better ear hearing was significantly related to poorer ratings on the Vitality scale of the SF-36 and the three scales of the HHIE-S, after controlling for age, gender and number of coexisting chronic health problems. Ratings on SF-36 and HHIE-S did not correlate. CONCLUSION: Elderly Chinese who are hearing impaired experienced poorer general and hearing-specific HRQoL, and HRQoL is reduced further among those with greater hearing impairment.