A validation of models for prediction of pathogenic variants in mismatch repair genes.

PURPOSE: Models used to predict the probability of an individual having a pathogenic homozygous or heterozygous variant in a mismatch repair gene, such as MMRpro, are widely used. Recently, MMRpro was updated with new colorectal cancer penetrance estimates. The purpose of this study was to evaluate the predictive performance of MMRpro and other models for individuals with a family history of colorectal cancer. METHODS: We performed a validation study of 4 models, Leiden, MMRpredict, PREMM5, and MMRpro, using 784 members of clinic-based families from the United States. Predicted probabilities were compared with germline testing results and evaluated for discrimination, calibration, and predictive accuracy. We analyzed several strategies to combine models and improve predictive performance. RESULTS: MMRpro with additional tumor information (MMRpro+) and PREMM5 outperformed the other models in discrimination and predictive accuracy. MMRpro+ was the best calibrated with an observed to expected ratio of 0.98 (95% CI = 0.89-1.08). The combination models showed improvement over PREMM5 and performed similar to MMRpro+. CONCLUSION: MMRpro+ and PREMM5 performed well in predicting the probability of having a pathogenic homozygous or heterozygous variant in a mismatch repair gene. They serve as useful clinical decision tools for identifying individuals who would benefit greatly from screening and prevention strategies.

Illuminating the landscape of high-level clinical trial opportunities in the All of Us Research Program.

OBJECTIVE: With its size and diversity, the All of Us Research Program has the potential to power and improve representation in clinical trials through ancillary studies like Nutrition for Precision Health. We sought to characterize high-level trial opportunities for the diverse participants and sponsors of future trial investment. MATERIALS AND METHODS: We matched All of Us participants with available trials on ClinicalTrials.gov based on medical conditions, age, sex, and geographic location. Based on the number of matched trials, we (1) developed the Trial Opportunities Compass (TOC) to help sponsors assess trial investment portfolios, (2) characterized the landscape of trial opportunities in a phenome-wide association study (PheWAS), and (3) assessed the relationship between trial opportunities and social determinants of health (SDoH) to identify potential barriers to trial participation. RESULTS: Our study included 181 529 All of Us participants and 18 634 trials. The TOC identified opportunities for portfolio investment and gaps in currently available trials across federal, industrial, and academic sponsors. PheWAS results revealed an emphasis on mental disorder-related trials, with anxiety disorder having the highest adjusted increase in the number of matched trials (59% [95% CI, 57-62]; P < 1e-300). Participants from certain communities underrepresented in biomedical research, including self-reported racial and ethnic minorities, had more matched trials after adjusting for other factors. Living in a nonmetropolitan area was associated with up to 13.1 times fewer matched trials. DISCUSSION AND CONCLUSION: All of Us data are a valuable resource for identifying trial opportunities to inform trial portfolio planning. Characterizing these opportunities with consideration for SDoH can provide guidance on prioritizing the most pressing barriers to trial participation.

Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models.

Purpose: Phenotyping is critical for informing rare disease diagnosis and treatment, but disease phenotypes are often embedded in unstructured text. While natural language processing (NLP) can automate extraction, a major bottleneck is developing annotated corpora. Recently, prompt learning with large language models (LLMs) has been shown to lead to generalizable results without any (zero-shot) or few annotated samples (few-shot), but none have explored this for rare diseases. Our work is the first to study prompt learning for identifying and extracting rare disease phenotypes in the zero- and few-shot settings. Methods: We compared the performance of prompt learning with ChatGPT and fine-tuning with BioClinicalBERT. We engineered novel prompts for ChatGPT to identify and extract rare diseases and their phenotypes (e.g., diseases, symptoms, and signs), established a benchmark for evaluating its performance, and conducted an in-depth error analysis. Results: Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.610 in the zero- and few-shot settings, respectively). However, ChatGPT achieved higher accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725). Conversational, sentence-based prompts generally achieved higher accuracy than structured lists. Conclusion: Prompt learning using ChatGPT has the potential to match or outperform fine-tuning BioClinicalBERT at extracting rare diseases and signs with just one annotated sample. Given its accessibility, ChatGPT could be leveraged to extract these entities without relying on a large, annotated corpus. While LLMs can support rare disease phenotyping, researchers should critically evaluate model outputs to ensure phenotyping accuracy.

Impact of financial compensation on enrollment and participation in a remote, mobile-app based research study.

Background: There is no consensus on how to determine appropriate financial compensation for research recruitment. Selecting incentive amounts that are reasonable and respectful, without undue inducement, remains challenging. Previously, we demonstrated that incentive amount significantly impacts participants' willingness to complete various hypothetical research activities. Here we further explore this relationship in a mock decentralized study. Methods: Adult ResearchMatch volunteers were invited to join a prospective study where interested individuals were given an opportunity to view details for a study along with participation requirements, then offered a randomly generated compensation amount between $0 and $50 to enroll and participate. Individuals agreeing to participate were then asked to complete tasks using a remote mobile application (MyCap), for two weeks. Tasks included a weekly survey, a daily gratitude journal and daily phone tapping task. Results: Willingness to participate was 85% across all incentive levels but not significantly impacted by amount. Task completion appeared to increase as a function of compensation until a plateau at $25. While participants described the study as low burden and reported that compensation was moderately important to their decision to join, only 31% completed all study tasks. Conclusion: While offering compensation in this study did not have a strong effect on enrollment rate, this work provides insight into participant motivation when joining and participating in studies employing mobile applications.

Leveraging artificial intelligence to summarize abstracts in lay language for increasing research accessibility and transparency.

OBJECTIVE: Returning aggregate study results is an important ethical responsibility to promote trust and inform decision making, but the practice of providing results to a lay audience is not widely adopted. Barriers include significant cost and time required to develop lay summaries and scarce infrastructure necessary for returning them to the public. Our study aims to generate, evaluate, and implement ChatGPT 4 lay summaries of scientific abstracts on a national clinical study recruitment platform, ResearchMatch, to facilitate timely and cost-effective return of study results at scale. MATERIALS AND METHODS: We engineered prompts to summarize abstracts at a literacy level accessible to the public, prioritizing succinctness, clarity, and practical relevance. Researchers and volunteers assessed ChatGPT-generated lay summaries across five dimensions: accuracy, relevance, accessibility, transparency, and harmfulness. We used precision analysis and adaptive random sampling to determine the optimal number of summaries for evaluation, ensuring high statistical precision. RESULTS: ChatGPT achieved 95.9% (95% CI, 92.1-97.9) accuracy and 96.2% (92.4-98.1) relevance across 192 summary sentences from 33 abstracts based on researcher review. 85.3% (69.9-93.6) of 34 volunteers perceived ChatGPT-generated summaries as more accessible and 73.5% (56.9-85.4) more transparent than the original abstract. None of the summaries were deemed harmful. We expanded ResearchMatch's technical infrastructure to automatically generate and display lay summaries for over 750 published studies that resulted from the platform's recruitment mechanism. DISCUSSION AND CONCLUSION: Implementing AI-generated lay summaries on ResearchMatch demonstrates the potential of a scalable framework generalizable to broader platforms for enhancing research accessibility and transparency.

Empowering the biomedical research community: Innovative SAS deployment on the All of Us Researcher Workbench.

OBJECTIVES: The All of Us Research Program is a precision medicine initiative aimed at establishing a vast, diverse biomedical database accessible through a cloud-based data analysis platform, the Researcher Workbench (RW). Our goal was to empower the research community by co-designing the implementation of SAS in the RW alongside researchers to enable broader use of All of Us data. MATERIALS AND METHODS: Researchers from various fields and with different SAS experience levels participated in co-designing the SAS implementation through user experience interviews. RESULTS: Feedback and lessons learned from user testing informed the final design of the SAS application. DISCUSSION: The co-design approach is critical for reducing technical barriers, broadening All of Us data use, and enhancing the user experience for data analysis on the RW. CONCLUSION: Our co-design approach successfully tailored the implementation of the SAS application to researchers' needs. This approach may inform future software implementations on the RW.