Retrieval Augmented Generation Enabled Generative Pre-Trained Transformer 4 (GPT-4) Performance for Clinical Trial Screening.

Background: Subject screening is a key aspect of all clinical trials; however, traditionally, it is a labor-intensive and error-prone task, demanding significant time and resources. With the advent of large language models (LLMs) and related technologies, a paradigm shift in natural language processing capabilities offers a promising avenue for increasing both quality and efficiency of screening efforts. This study aimed to test the Retrieval-Augmented Generation (RAG) process enabled Generative Pretrained Transformer Version 4 (GPT-4) to accurately identify and report on inclusion and exclusion criteria for a clinical trial. Methods: The Co-Operative Program for Implementation of Optimal Therapy in Heart Failure (COPILOT-HF) trial aims to recruit patients with symptomatic heart failure. As part of the screening process, a list of potentially eligible patients is created through an electronic health record (EHR) query. Currently, structured data in the EHR can only be used to determine 5 out of 6 inclusion and 5 out of 17 exclusion criteria. Trained, but non-licensed, study staff complete manual chart review to determine patient eligibility and record their assessment of the inclusion and exclusion criteria. We obtained the structured assessments completed by the study staff and clinical notes for the past two years and developed a workflow of clinical note-based question answering system powered by RAG architecture and GPT-4 that we named RECTIFIER (RAG-Enabled Clinical Trial Infrastructure for Inclusion Exclusion Review). We used notes from 100 patients as a development dataset, 282 patients as a validation dataset, and 1894 patients as a test set. An expert clinician completed a blinded review of patients' charts to answer the eligibility questions and determine the "gold standard" answers. We calculated the sensitivity, specificity, accuracy, and Matthews correlation coefficient (MCC) for each question and screening method. We also performed bootstrapping to calculate the confidence intervals for each statistic. Results: Both RECTIFIER and study staff answers closely aligned with the expert clinician answers across criteria with accuracy ranging between 97.9% and 100% (MCC 0.837 and 1) for RECTIFIER and 91.7% and 100% (MCC 0.644 and 1) for study staff. RECTIFIER performed better than study staff to determine the inclusion criteria of "symptomatic heart failure" with an accuracy of 97.9% vs 91.7% and an MCC of 0.924 vs 0.721, respectively. Overall, the sensitivity and specificity of determining eligibility for the RECTIFIER was 92.3% (CI) and 93.9% (CI), and study staff was 90.1% (CI) and 83.6% (CI), respectively. Conclusion: GPT-4 based solutions have the potential to improve efficiency and reduce costs in clinical trial screening. When incorporating new tools such as RECTIFIER, it is important to consider the potential hazards of automating the screening process and set up appropriate mitigation strategies such as final clinician review before patient engagement.

Communicating new knowledge on previously reported genetic variants.

Genetic tests often identify variants whose significance cannot be determined at the time they are reported. In many situations, it is critical that clinicians be informed when new information emerges on these variants. It is already extremely challenging for laboratories to provide these updates. These challenges will grow rapidly as an increasing number of clinical genetic tests are ordered and as the amount of patient DNA assayed per test expands; the challenges will need to be addressed before whole-genome sequencing is used on a widespread basis.Information technology infrastructure can be useful in this context. We have deployed an infrastructure enabling clinicians to receive knowledge updates when a laboratory changes the classification of a variant. We have gathered statistics from this deployment regarding the frequency of both variant classification changes and the effects of these classification changes on patients. We report on the system's functionality as well as the statistics derived from its use.Genet Med advance online publication 5 April 2012.

Usability of a novel clinician interface for genetic results.

The complexity and rapid growth of genetic data demand investment in information technology to support effective use of this information. Creating infrastructure to communicate genetic information to healthcare providers and enable them to manage that data can positively affect a patient's care in many ways. However, genetic data are complex and present many challenges. We report on the usability of a novel application designed to assist providers in receiving and managing a patient's genetic profile, including ongoing updated interpretations of the genetic variants in those patients. Because these interpretations are constantly evolving, managing them represents a challenge. We conducted usability tests with potential users of this application and reported findings to the application development team, many of which were addressed in subsequent versions. Clinicians were excited about the value this tool provides in pushing out variant updates to providers and overall gave the application high usability ratings, but had some difficulty interpreting elements of the interface. Many issues identified required relatively little development effort to fix suggesting that consistently incorporating this type of analysis in the development process can be highly beneficial. For genetic decision support applications, our findings suggest the importance of designing a system that can deliver the most current knowledge and highlight the significance of new genetic information for clinical care. Our results demonstrate that using a development and design process that is user focused helped optimize the value of this application for personalized medicine.

The GeneInsight Suite: a platform to support laboratory and provider use of DNA-based genetic testing.

The future of personalized medicine will hinge on effective management of patient genetic profiles. Molecular diagnostic testing laboratories need to track knowledge surrounding an increasingly large number of genetic variants, incorporate this knowledge into interpretative reports, and keep ordering clinicians up to date as this knowledge evolves. Treating clinicians need to track which variants have been identified in each of their patients along with the significance of these variants. The GeneInsight(SM) Suite assists in these areas. The suite also provides a basis for interconnecting laboratories and clinicians in a manner that increases the scalability of personalized medicine processes.

Workflow Automation for a Virtual Hypertension Management Program.

OBJECTIVES: Hypertension is a modifiable risk factor for numerous comorbidities and treating hypertension can greatly improve health outcomes. We sought to increase the efficiency of a virtual hypertension management program through workflow automation processes. METHODS: We developed a customer relationship management (CRM) solution at our institution for the purpose of improving processes and workflow for a virtual hypertension management program and describe here the development, implementation, and initial experience of this CRM system. RESULTS: Notable system features include task automation, patient data capture, multi-channel communication, integration with our electronic health record (EHR), and device integration (for blood pressure cuffs). In the five stages of our program (intake and eligibility screening, enrollment, device configuration/setup, medication titration, and maintenance), we describe some of the key process improvements and workflow automations that are enabled using our CRM platform, like automatic reminders to capture blood pressure data and present these data to our clinical team when ready for clinical decision making. We also describe key limitations of CRM, like balancing out-of-the-box functionality with development flexibility. Among our first group of referred patients, 76% (39/51) preferred email as their communication method, 26/51 (51%) were able to enroll electronically, and 63% of those enrolled (32/51) were able to transmit blood pressure data without phone support. CONCLUSION: A CRM platform could improve clinical processes through multiple pathways, including workflow automation, multi-channel communication, and device integration. Future work will examine the operational improvements of this health information technology solution as well as assess clinical outcomes.

Evaluation: A Qualitative Pilot Study of Novel Information Technology Infrastructure to Communicate Genetic Variant Updates.

BACKGROUND: Partners HealthCare Personalized Medicine developed GeneInsight Clinic (GIC), a tool designed to communicate updated variant information from laboratory geneticists to treating clinicians through automated alerts, categorized by level of variant interpretation change. OBJECTIVES: The study aimed to evaluate feedback from the initial users of the GIC, including the advantages and challenges to receiving this variant information and using this technology at the point of care. METHODS: Healthcare professionals from two clinics that ordered genetic testing for cardiomyopathy and related disorders were invited to participate in one-hour semi-structured interviews and/ or a one-hour focus group. Using a Grounded Theory approach, transcript concepts were coded and organized into themes. RESULTS: Two genetic counselors and two physicians from two treatment clinics participated in individual interviews. Focus group participants included one genetic counselor and four physicians. Analysis resulted in 8 major themes related to structuring and communicating variant knowledge, GIC's impact on the clinic, and suggestions for improvements. The interview analysis identified longitudinal patient care, family data, and growth in genetic testing content as potential challenges to optimization of the GIC infrastructure. DISCUSSION: Participants agreed that GIC implementation increased efficiency and effectiveness of the clinic through increased access to genetic variant information at the point of care. CONCLUSION: Development of information technology (IT) infrastructure to aid in the organization and management of genetic variant knowledge will be critical as the genetic field moves towards whole exome and whole genome sequencing. Findings from this study could be applied to future development of IT support for genetic variant knowledge management that would serve to improve clinicians' ability to manage and care for patients.

A novel clinician interface to improve clinician access to up-to-date genetic results.

OBJECTIVES: To understand the impact of GeneInsight Clinic (GIC), a web-based tool designed to manage genetic information and facilitate communication of test results and variant updates from the laboratory to the clinics, we measured the use of GIC and the time it took for new genetic knowledge to be available to clinicians. METHODS: Usage data were collected across four study sites for the GIC launch and post-GIC implementation time periods. The primary outcome measures were the time (average number of days) between variant change approval and notification of clinic staff, and the time between notification and viewing the patient record. RESULTS: Post-GIC, time between a variant change approval and provider notification was shorter than at launch (average days at launch 503.8, compared to 4.1 days post-GIC). After e-mail alerts were sent at launch, providers clicked into the patient record associated with 91% of these alerts. In the post period, clinic providers clicked into the patient record associated with 95% of the alerts, on average 12 days after the e-mail was sent. DISCUSSION: We found that GIC greatly increased the likelihood that a provider would receive updated variant information as well as reduced the time associated with distributing that variant information, thus providing a more efficient process for incorporating new genetic knowledge into clinical care. CONCLUSIONS: Our study results demonstrate that health information technology systems have the potential effectively to assist providers in utilizing genetic information in patient care.