Empowering the biomedical research community: Innovative SAS deployment on the All of Us Researcher Workbench.

OBJECTIVES: The All of Us Research Program is a precision medicine initiative aimed at establishing a vast, diverse biomedical database accessible through a cloud-based data analysis platform, the Researcher Workbench (RW). Our goal was to empower the research community by co-designing the implementation of SAS in the RW alongside researchers to enable broader use of All of Us data. MATERIALS AND METHODS: Researchers from various fields and with different SAS experience levels participated in co-designing the SAS implementation through user experience interviews. RESULTS: Feedback and lessons learned from user testing informed the final design of the SAS application. DISCUSSION: The co-design approach is critical for reducing technical barriers, broadening All of Us data use, and enhancing the user experience for data analysis on the RW. CONCLUSION: Our co-design approach successfully tailored the implementation of the SAS application to researchers' needs. This approach may inform future software implementations on the RW.

Leveraging artificial intelligence to summarize abstracts in lay language for increasing research accessibility and transparency.

OBJECTIVE: Returning aggregate study results is an important ethical responsibility to promote trust and inform decision making, but the practice of providing results to a lay audience is not widely adopted. Barriers include significant cost and time required to develop lay summaries and scarce infrastructure necessary for returning them to the public. Our study aims to generate, evaluate, and implement ChatGPT 4 lay summaries of scientific abstracts on a national clinical study recruitment platform, ResearchMatch, to facilitate timely and cost-effective return of study results at scale. MATERIALS AND METHODS: We engineered prompts to summarize abstracts at a literacy level accessible to the public, prioritizing succinctness, clarity, and practical relevance. Researchers and volunteers assessed ChatGPT-generated lay summaries across five dimensions: accuracy, relevance, accessibility, transparency, and harmfulness. We used precision analysis and adaptive random sampling to determine the optimal number of summaries for evaluation, ensuring high statistical precision. RESULTS: ChatGPT achieved 95.9% (95% CI, 92.1-97.9) accuracy and 96.2% (92.4-98.1) relevance across 192 summary sentences from 33 abstracts based on researcher review. 85.3% (69.9-93.6) of 34 volunteers perceived ChatGPT-generated summaries as more accessible and 73.5% (56.9-85.4) more transparent than the original abstract. None of the summaries were deemed harmful. We expanded ResearchMatch's technical infrastructure to automatically generate and display lay summaries for over 750 published studies that resulted from the platform's recruitment mechanism. DISCUSSION AND CONCLUSION: Implementing AI-generated lay summaries on ResearchMatch demonstrates the potential of a scalable framework generalizable to broader platforms for enhancing research accessibility and transparency.

Impact of financial compensation on enrollment and participation in a remote, mobile-app based research study.

Background: There is no consensus on how to determine appropriate financial compensation for research recruitment. Selecting incentive amounts that are reasonable and respectful, without undue inducement, remains challenging. Previously, we demonstrated that incentive amount significantly impacts participants' willingness to complete various hypothetical research activities. Here we further explore this relationship in a mock decentralized study. Methods: Adult ResearchMatch volunteers were invited to join a prospective study where interested individuals were given an opportunity to view details for a study along with participation requirements, then offered a randomly generated compensation amount between $0 and $50 to enroll and participate. Individuals agreeing to participate were then asked to complete tasks using a remote mobile application (MyCap), for two weeks. Tasks included a weekly survey, a daily gratitude journal and daily phone tapping task. Results: Willingness to participate was 85% across all incentive levels but not significantly impacted by amount. Task completion appeared to increase as a function of compensation until a plateau at $25. While participants described the study as low burden and reported that compensation was moderately important to their decision to join, only 31% completed all study tasks. Conclusion: While offering compensation in this study did not have a strong effect on enrollment rate, this work provides insight into participant motivation when joining and participating in studies employing mobile applications.

Selecting trial centers using a standardized, automated site assessment survey instrument (SASI).

BACKGROUND: To improve the site selection process for clinical trials, we expanded a site survey to include standardized assessments of site commitment time, team experience, feasibility of tight timelines, and local medical community equipoise as factors that might better predict performance. We also collected contact information about institutional research services ahead of site onboarding. AIM: As a first step, we wanted to confirm that an expanded survey could be feasible and generalizable-that asking site teams for more details upfront was acceptable and that the survey could be completed in a reasonable amount of time, despite the assessment length. METHODS: A standardized, two-part Site Assessment Survey Instrument (SASI), examining qualitative components and with multiple contact list sections, was developed using a publicly accessible dashboard and later transferred to a REDCap platform. After multiple rounds of internal testing, the SASI was deployed 11 times for multicenter trials. Follow-up questionnaires were sent to site teams to confirm that an expanded survey instrument is acceptable to the research community and could be completed during a brief work shift. RESULTS: Respondents thought the SASI collected useful and relevant information about their sites (100%). Sites were "comfortable" (90%) supplying detailed information early in the site selection process and 57% completed the SASI in one to two hours. CONCLUSIONS: Coordinating centers and sites found the SASI tool to be acceptable and helpful when collecting data in consideration of multicenter trial site selection.

Classification of Breast Implant Malposition.

BACKGROUND: Implant malposition is a well-recognized complication of prosthetic breast implants. However, to date, no objective classification system has been described. OBJECTIVES: The aim of this study was to perform a prospective trial of an objective and reproducible classification system for implant malposition formulated by analyzing retrospective data from a large cohort of patients with implant malposition. METHODS: The authors retrospectively analyzed the degree of medial/lateral and inferior/superior implant malposition relative to their optimal position within the breast footprint in a series of 189 breasts (n = 100 patients). An objective classification system for implant malposition was devised and then applied to a prospective cohort of 53 breasts in 28 patients with implant malposition. RESULTS: The degree of malposition in a single or combination of axes was categorized according to the distance (measured in centimeters) from the ideal breast footprint. The classification system incorporated the axis of malposition and distance to generate a treatment decision-making guide. Cases of Grade 1 malposition did not warrant surgical intervention, whereas surgical correction was warranted in all Grade 3 cases. In the combined patient cohort (n = 242 breasts, 128 patients), lateral, inferior, medial, and superior displacement ranged between Grades 1 and 3. There was no interobserver variability in the grades assigned to 9 out of 10 patients in the prospective group. CONCLUSIONS: A simple and reproducible classification system for implant malposition has been created that allows surgeons to objectively record the extent of malposition, guides surgical decision-making, and can be used to document the results of any intervention.

Illuminating the landscape of high-level clinical trial opportunities in the All of Us Research Program.

OBJECTIVE: With its size and diversity, the All of Us Research Program has the potential to power and improve representation in clinical trials through ancillary studies like Nutrition for Precision Health. We sought to characterize high-level trial opportunities for the diverse participants and sponsors of future trial investment. MATERIALS AND METHODS: We matched All of Us participants with available trials on ClinicalTrials.gov based on medical conditions, age, sex, and geographic location. Based on the number of matched trials, we (1) developed the Trial Opportunities Compass (TOC) to help sponsors assess trial investment portfolios, (2) characterized the landscape of trial opportunities in a phenome-wide association study (PheWAS), and (3) assessed the relationship between trial opportunities and social determinants of health (SDoH) to identify potential barriers to trial participation. RESULTS: Our study included 181 529 All of Us participants and 18 634 trials. The TOC identified opportunities for portfolio investment and gaps in currently available trials across federal, industrial, and academic sponsors. PheWAS results revealed an emphasis on mental disorder-related trials, with anxiety disorder having the highest adjusted increase in the number of matched trials (59% [95% CI, 57-62]; P < 1e-300). Participants from certain communities underrepresented in biomedical research, including self-reported racial and ethnic minorities, had more matched trials after adjusting for other factors. Living in a nonmetropolitan area was associated with up to 13.1 times fewer matched trials. DISCUSSION AND CONCLUSION: All of Us data are a valuable resource for identifying trial opportunities to inform trial portfolio planning. Characterizing these opportunities with consideration for SDoH can provide guidance on prioritizing the most pressing barriers to trial participation.

Physical Activity and Incident Obesity Across the Spectrum of Genetic Risk for Obesity.

Importance: Despite consistent public health recommendations, obesity rates in the US continue to increase. Physical activity recommendations do not account for individual genetic variability, increasing risk of obesity. Objective: To use activity, clinical, and genetic data from the All of Us Research Program (AoURP) to explore the association of genetic risk of higher body mass index (BMI) with the level of physical activity needed to reduce incident obesity. Design, Setting, and Participants: In this US population-based retrospective cohort study, participants were enrolled in the AoURP between May 1, 2018, and July 1, 2022. Enrollees in the AoURP who were of European ancestry, owned a personal activity tracking device, and did not have obesity up to 6 months into activity tracking were included in the analysis. Exposure: Physical activity expressed as daily step counts and a polygenic risk score (PRS) for BMI, calculated as weight in kilograms divided by height in meters squared. Main Outcome and Measures: Incident obesity (BMI ≥30). Results: A total of 3124 participants met inclusion criteria. Among 3051 participants with available data, 2216 (73%) were women, and the median age was 52.7 (IQR, 36.4-62.8) years. The total cohort of 3124 participants walked a median of 8326 (IQR, 6499-10 389) steps/d over a median of 5.4 (IQR, 3.4-7.0) years of personal activity tracking. The incidence of obesity over the study period increased from 13% (101 of 781) to 43% (335 of 781) in the lowest and highest PRS quartiles, respectively (P = 1.0 × 10-20). The BMI PRS demonstrated an 81% increase in obesity risk (P = 3.57 × 10-20) while mean step count demonstrated a 43% reduction (P = 5.30 × 10-12) when comparing the 75th and 25th percentiles, respectively. Individuals with a PRS in the 75th percentile would need to walk a mean of 2280 (95% CI, 1680-3310) more steps per day (11 020 total) than those at the 50th percentile to have a comparable risk of obesity. To have a comparable risk of obesity to individuals at the 25th percentile of PRS, those at the 75th percentile with a baseline BMI of 22 would need to walk an additional 3460 steps/d; with a baseline BMI of 24, an additional 4430 steps/d; with a baseline BMI of 26, an additional 5380 steps/d; and with a baseline BMI of 28, an additional 6350 steps/d. Conclusions and Relevance: In this cohort study, the association between daily step count and obesity risk across genetic background and baseline BMI were quantified. Population-based recommendations may underestimate physical activity needed to prevent obesity among those at high genetic risk.

ResearchMatch on FHIR: Development and evaluation of a recruitment registry and electronic health record system interface for volunteer profile completion.

Background: Obtaining complete and accurate information in recruitment registries is essential for matching potential participants to research studies for which they qualify. Since electronic health record (EHR) systems are required to make patient data available to external systems, an interface between EHRs and recruitment registries may improve accuracy and completeness of volunteers' profiles. We tested this hypothesis on ResearchMatch (RM), a disease- and institution-neutral recruitment registry with 1357 studies across 255 institutions. Methods: We developed an interface where volunteers signing up for RM can authorize transfer of demographic data, medical conditions, and medications from the EHR into a registration form. We obtained feedback from a panel of community members to determine acceptability of the planned integration. We then developed the EHR interface and performed an evaluation study of 100 patients to determine whether RM profiles generated with EHR-assisted adjudication included more conditions and medications than those without the EHR connection. Results: Community member feedback revealed that members of the public were willing to authenticate into the EHR from RM with proper messaging about choice and privacy. The evaluation study showed that out of 100 participants, 75 included more conditions and 69 included more medications in RM profiles completed with the EHR connection than those without. Participants also completed the EHR-connected profiles in 16 fewer seconds than non-EHR-connected profiles. Conclusions: The EHR to RM integration could lead to more complete profiles, less participant burden, and better study matches for many of the over 148,000 volunteers who participate in ResearchMatch.

Leveraging the Expertise of the CTSA Program to Increase the Impact and Efficiency of Clinical Trials.

Importance: Multicenter clinical trials play a critical role in the translational processes that enable new treatments to reach all people and improve public health. However, conducting multicenter randomized clinical trials (mRCT) presents challenges. The Trial Innovation Network (TIN), established in 2016 to partner with the Clinical and Translational Science Award (CTSA) Consortium of academic medical institutions in the implementation of mRCTs, consists of 3 Trial Innovation Centers (TICs) and 1 Recruitment Innovation Center (RIC). This unique partnership has aimed to address critical roadblocks that impede the design and conduct of mRCTs, in expectation of accelerating the translation of novel interventions to clinical practice. The TIN's challenges and achievements are described in this article, along with examples of innovative resources and processes that may serve as useful models for other clinical trial networks providing operational and recruitment support. Observations: The TIN has successfully integrated more than 60 CTSA institution program hubs into a functional network for mRCT implementation and optimization. A unique support system for investigators has been created that includes the development and deployment of novel tools, operational and recruitment services, consultation models, and rapid communication pathways designed to reduce delays in trial start-up, enhance recruitment, improve engagement of diverse research participants and communities, and streamline processes that improve the quality, efficiency, and conduct of mRCTs. These resources and processes span the clinical trial spectrum and enable the TICs and RIC to serve as coordinating centers, data centers, and recruitment specialists to assist trials across the National Institutes of Health and other agencies. The TIN's impact has been demonstrated through its response to both historical operational challenges and emerging public health emergencies, including the national opioid public health crisis and the COVID-19 pandemic. Conclusions and Relevance: The TIN has worked to reduce barriers to implementing mRCTs and to improve mRCT processes and operations by providing needed clinical trial infrastructure and resources to CTSA investigators. These resources have been instrumental in more quickly and efficiently translating research discoveries into beneficial patient treatments.

Decentralized clinical trials in the trial innovation network: Value, strategies, and lessons learned.

New technologies and disruptions related to Coronavirus disease-2019 have led to expansion of decentralized approaches to clinical trials. Remote tools and methods hold promise for increasing trial efficiency and reducing burdens and barriers by facilitating participation outside of traditional clinical settings and taking studies directly to participants. The Trial Innovation Network, established in 2016 by the National Center for Advancing Clinical and Translational Science to address critical roadblocks in clinical research and accelerate the translational research process, has consulted on over 400 research study proposals to date. Its recommendations for decentralized approaches have included eConsent, participant-informed study design, remote intervention, study task reminders, social media recruitment, and return of results for participants. Some clinical trial elements have worked well when decentralized, while others, including remote recruitment and patient monitoring, need further refinement and assessment to determine their value. Partially decentralized, or "hybrid" trials, offer a first step to optimizing remote methods. Decentralized processes demonstrate potential to improve urban-rural diversity, but their impact on inclusion of racially and ethnically marginalized populations requires further study. To optimize inclusive participation in decentralized clinical trials, efforts must be made to build trust among marginalized communities, and to ensure access to remote technology.

The RIC Recruitment & Retention Materials Toolkit - a resource for developing community-informed study materials.

Clinical trials face many challenges with meeting projected enrollment and retention goals. A study's recruitment materials and messaging convey necessary key information and therefore serve as a critical first impression with potential participants. Yet study teams often lack the resources and skills needed to develop engaging, culturally tailored, and professional-looking recruitment materials. To address this gap, the Recruitment Innovation Center recently developed a Recruitment & Retention Materials Content and Design Toolkit, which offers research teams guidance, actionable tips, resources, and customizable templates for creating trial-specific study materials. This paper seeks to describe the creation and contents of this new toolkit.

The All of Us Data and Research Center: Creating a Secure, Scalable, and Sustainable Ecosystem for Biomedical Research.

The All of Us Research Program's Data and Research Center (DRC) was established to help acquire, curate, and provide access to one of the world's largest and most diverse datasets for precision medicine research. Already, over 500,000 participants are enrolled in All of Us, 80% of whom are underrepresented in biomedical research, and data are being analyzed by a community of over 2,300 researchers. The DRC created this thriving data ecosystem by collaborating with engaged participants, innovative program partners, and empowered researchers. In this review, we first describe how the DRC is organized to meet the needs of this broad group of stakeholders. We then outline guiding principles, common challenges, and innovative approaches used to build the All of Us data ecosystem. Finally, we share lessons learned to help others navigate important decisions and trade-offs in building a modern biomedical data platform.

De-black-boxing health AI: demonstrating reproducible machine learning computable phenotypes using the N3C-RECOVER Long COVID model in the All of Us data repository.

Machine learning (ML)-driven computable phenotypes are among the most challenging to share and reproduce. Despite this difficulty, the urgent public health considerations around Long COVID make it especially important to ensure the rigor and reproducibility of Long COVID phenotyping algorithms such that they can be made available to a broad audience of researchers. As part of the NIH Researching COVID to Enhance Recovery (RECOVER) Initiative, researchers with the National COVID Cohort Collaborative (N3C) devised and trained an ML-based phenotype to identify patients highly probable to have Long COVID. Supported by RECOVER, N3C and NIH's All of Us study partnered to reproduce the output of N3C's trained model in the All of Us data enclave, demonstrating model extensibility in multiple environments. This case study in ML-based phenotype reuse illustrates how open-source software best practices and cross-site collaboration can de-black-box phenotyping algorithms, prevent unnecessary rework, and promote open science in informatics.

Daily Step Counts Before and After the COVID-19 Pandemic Among All of Us Research Participants.

This cohort study of US adults examines changes in physical activity following the onset of the COVID-19 pandemic.

Using a multicultural and multilingual awareness-raising strategy to enhance enrollment of racially underrepresented minoritized communities - the PassITON trial.

Racially and ethnically minoritized populations have been historically excluded and underrepresented in research. This paper will describe best practices in multicultural and multilingual awareness-raising strategies used by the Recruitment Innovation Center to increase minoritized enrollment into clinical trials. The Passive Immunity Trial for Our Nation will be used as a primary example to highlight real-world application of these methods to raise awareness, engage community partners, and recruit diverse study participants.

Evaluating automated electronic case report form data entry from electronic health records.

Background: Many clinical trials leverage real-world data. Typically, these data are manually abstracted from electronic health records (EHRs) and entered into electronic case report forms (CRFs), a time and labor-intensive process that is also error-prone and may miss information. Automated transfer of data from EHRs to eCRFs has the potential to reduce data abstraction and entry burden as well as improve data quality and safety. Methods: We conducted a test of automated EHR-to-CRF data transfer for 40 participants in a clinical trial of hospitalized COVID-19 patients. We determined which coordinator-entered data could be automated from the EHR (coverage), and the frequency with which the values from the automated EHR feed and values entered by study personnel for the actual study matched exactly (concordance). Results: The automated EHR feed populated 10,081/11,952 (84%) coordinator-completed values. For fields where both the automation and study personnel provided data, the values matched exactly 89% of the time. Highest concordance was for daily lab results (94%), which also required the most personnel resources (30 minutes per participant). In a detailed analysis of 196 instances where personnel and automation entered values differed, both a study coordinator and a data analyst agreed that 152 (78%) instances were a result of data entry error. Conclusions: An automated EHR feed has the potential to significantly decrease study personnel effort while improving the accuracy of CRF data.

A community-informed recruitment plan template to increase recruitment of racial and ethnic groups historically excluded and underrepresented in clinical research.

INTRODUCTION: Engaging communities in research planning and implementation can enhance recruitment and retention (R&R) of racial and ethnic groups historically excluded and underrepresented in clinical research; however, most studies do not use community-informed approaches. This paper describes the formative research process used to design a Community-Informed Recruitment Plan Template for racial and ethnic groups historically excluded and underrepresented in clinical research. METHODS: Using an existing R&R template as a starting point, we iteratively developed and refined the community-informed template through a 3-phase process to achieve cultural-appropriateness. Phase 1 included a literature review, 34 community engagement (CE) studios to review recommendations, community advisory board (CAB) review, and survey data from minority recruitment experts. Phase 2 involved integration of content into existing R&R template. Phase 3 was a final review and revision using input of the CAB and researchers' panel. Survey data collected in Phase 1 were analyzed using descriptives (i.e., frequencies and percentages). Open-ended survey responses were analyzed using inductive, qualitative thematic analysis. RESULTS: The final 8-section template can help develop effective grant or proposal language where study R&R plans are requested. They include: 1) Recruitment Strategy; 2) A Stakeholder Communication Plan; 3) Evidence of Recruitment Feasibility; 4) Recruitment and Retention Team; 5) Recruitment and Retention Methods; 6) Recruitment and Retention Timeline; 7) Evaluation; and 8) Budget. CONCLUSIONS: Incorporating multiple perspectives into this formative research process enhances the cultural appropriateness of this community-informed R&R template to help research teams achieve R&R goals for individuals historically excluded and underrepresented in clinical research.

Association of Longitudinal Activity Measures and Diabetes Risk: An Analysis From the National Institutes of Health All of Us Research Program.

CONTEXT: Prior studies of the relationship between physical activity and incident type 2 diabetes mellitus (T2DM) relied primarily on questionnaires at a single time point. OBJECTIVE: We sought to investigate the relationship between physical activity and incident T2DM with an innovative approach using data from commercial wearable devices linked to electronic health records in a real-world population. METHODS: Using All of Us participants' accelerometer data from their personal Fitbit devices, we used a time-varying Cox proportional hazards models with repeated measures of physical activity for the outcome of incident T2DM. We evaluated for effect modification with age, sex, body mass index (BMI), and sedentary time using multiplicative interaction terms. RESULTS: From 5677 participants in the All of Us Research Program (median age 51 years; 74% female; 89% White), there were 97 (2%) cases of incident T2DM over a median follow-up period of 3.8 years between 2010 to 2021. In models adjusted for age, sex, and race, the hazard of incident diabetes was reduced by 44% (95% CI, 15%-63%; P = 0.01) when comparing those with an average daily step count of 10 700 to those with 6000. Similar benefits were seen comparing groups based on average duration of various intensities of activity (eg, lightly active, fairly active, very active). There was no evidence for effect modification by age, sex, BMI, or sedentary time. CONCLUSION: Greater time in any type of physical activity intensity was associated with lower risk of T2DM irrespective of age, sex, BMI, or sedentary time.

Development, implementation, and dissemination of operational innovations across the trial innovation network.

Improving the quality and conduct of multi-center clinical trials is essential to the generation of generalizable knowledge about the safety and efficacy of healthcare treatments. Despite significant effort and expense, many clinical trials are unsuccessful. The National Center for Advancing Translational Science launched the Trial Innovation Network to address critical roadblocks in multi-center trials by leveraging existing infrastructure and developing operational innovations. We provide an overview of the roadblocks that led to opportunities for operational innovation, our work to develop, define, and map innovations across the network, and how we implemented and disseminated mature innovations.

Selecting EHR-driven recruitment strategies: An evidence-based decision guide.

Participant recruitment for research is a persistent bottleneck that can be improved by leveraging electronic health records (EHRs). Despite emerging evidence for various EHR-driven approaches, guidance for those attempting to select and use such approaches is limited. The national Recruitment Innovation Center established the EHR Recruitment Consult Resource (ERCR) service line to support multisite studies through implementation of EHR-driven recruitment strategies. As the ERCR, we evolved a guide through 17 consultations over 3 years with multisite studies recruiting in diverse biomedical research domains. We assessed literature and engaged domain experts to identify five key EHR-driven recruitment strategies: direct to patient messages, candidate lists for mailings/calls, direct to research alerts, point of care alerts, and participant registries. Differentiating factors were grouped into factors of study population, study protocol and recruitment workflows, and recruitment site capabilities. The decision matrix indicates acceptable or preferred strategies based on the differentiating factors. Across the ERCR consultations, candidate lists for mailing or calls were most common, participant registries were least frequently recommended, and for some studies no EHR-driven recruitment was recommended. Comparative effectiveness research is needed to refine further evidence for these and potentially new strategies to come.