The All of Us Research Program: Data quality, utility, and diversity.

The All of Us Research Program seeks to engage at least one million diverse participants to advance precision medicine and improve human health. We describe here the cloud-based Researcher Workbench that uses a data passport model to democratize access to analytical tools and participant information including survey, physical measurement, and electronic health record (EHR) data. We also present validation study findings for several common complex diseases to demonstrate use of this novel platform in 315,000 participants, 78% of whom are from groups historically underrepresented in biomedical research, including 49% self-reporting non-White races. Replication findings include medication usage pattern differences by race in depression and type 2 diabetes, validation of known cancer associations with smoking, and calculation of cardiovascular risk scores by reported race effects. The cloud-based Researcher Workbench represents an important advance in enabling secure access for a broad range of researchers to this large resource and analytical tools.

Remote smartphone monitoring of Parkinson's disease and individual response to therapy.

Remote health assessments that gather real-world data (RWD) outside clinic settings require a clear understanding of appropriate methods for data collection, quality assessment, analysis and interpretation. Here we examine the performance and limitations of smartphones in collecting RWD in the remote mPower observational study of Parkinson's disease (PD). Within the first 6 months of study commencement, 960 participants had enrolled and performed at least five self-administered active PD symptom assessments (speeded tapping, gait/balance, phonation or memory). Task performance, especially speeded tapping, was predictive of self-reported PD status (area under the receiver operating characteristic curve (AUC) = 0.8) and correlated with in-clinic evaluation of disease severity (r = 0.71; P < 1.8 × 10-6) when compared with motor Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Although remote assessment requires careful consideration for accurate interpretation of RWD, our results support the use of smartphones and wearables in objective and personalized disease assessments.

Empirical validation of an automated approach to data use oversight.

The current paradigm for data use oversight of biomedical datasets is onerous, extending the timescale and resources needed to obtain access for secondary analyses, thus hindering scientific discovery. For a researcher to utilize a controlled-access dataset, a data access committee must review her research plans to determine whether they are consistent with the data use limitations (DULs) specified by the informed consent form. The newly created GA4GH data use ontology (DUO) holds the potential to streamline this process by making data use oversight computable. Here, we describe an open-source software platform, the Data Use Oversight System (DUOS), that connects with DUO terminology to enable automated data use oversight. We analyze dbGaP data acquired since 2006, finding an exponential increase in data access requests, which will not be sustainable with current manual oversight review. We perform an empirical evaluation of DUOS and DUO on selected datasets from the Broad Institute's data repository. We were able to structure 118/123 of the evaluated DULs (96%) and 52/52 (100%) of research proposals using DUO terminology, and we find that DUOS' automated data access adjudication in all cases agreed with the DAC manual review. This first empirical evaluation of the feasibility of automated data use oversight demonstrates comparable accuracy to human-based data access oversight in real-world data governance.

Assessment of the All of Us research program's informed consent process.

Informed consent is the gateway to research participation. We report on the results of the formative evaluation that follows the electronic informed consent process for the All of Us Research Program. Of the nearly 250,000 participants included in this analysis, more than 95% could correctly answer questions distinguishing the program from medical care, the voluntary nature of participation, and the right to withdraw; comparatively, participants were less sure of privacy risk of the program. We also report on a small mixed-methods study of the experience of persons of very low health literacy with All of Us informed consent materials. Of note, many of the words commonly employed in the consent process were unfamiliar to or differently defined by informants. In combination, these analyses may inform participant-centered development and highlight areas for refinement of informed consent materials for the All of Us Research Program and similar studies.

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment.

OBJECTIVE: Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers. MATERIALS AND METHODS: The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics. RESULTS: Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access. CONCLUSIONS: The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

Bringing Code to Data: Do Not Forget Governance.

Developing or independently evaluating algorithms in biomedical research is difficult because of restrictions on access to clinical data. Access is restricted because of privacy concerns, the proprietary treatment of data by institutions (fueled in part by the cost of data hosting, curation, and distribution), concerns over misuse, and the complexities of applicable regulatory frameworks. The use of cloud technology and services can address many of the barriers to data sharing. For example, researchers can access data in high performance, secure, and auditable cloud computing environments without the need for copying or downloading. An alternative path to accessing data sets requiring additional protection is the model-to-data approach. In model-to-data, researchers submit algorithms to run on secure data sets that remain hidden. Model-to-data is designed to enhance security and local control while enabling communities of researchers to generate new knowledge from sequestered data. Model-to-data has not yet been widely implemented, but pilots have demonstrated its utility when technical or legal constraints preclude other methods of sharing. We argue that model-to-data can make a valuable addition to our data sharing arsenal, with 2 caveats. First, model-to-data should only be adopted where necessary to supplement rather than replace existing data-sharing approaches given that it requires significant resource commitments from data stewards and limits scientific freedom, reproducibility, and scalability. Second, although model-to-data reduces concerns over data privacy and loss of local control when sharing clinical data, it is not an ethical panacea. Data stewards will remain hesitant to adopt model-to-data approaches without guidance on how to do so responsibly. To address this gap, we explored how commitments to open science, reproducibility, security, respect for data subjects, and research ethics oversight must be re-evaluated in a model-to-data context.

Electronic Informed Consent in Mobile Applications Research.

The article covers electronic informed consent (eIC) from different dimensions so that practitioners might understand the history, regulation, and current status of eIC. It covers the transition of informed consent to electronic screens and the implications of that transition in terms of design, costs, and data analysis. The article explores the limits of regulation mandating eIC for mobile application research, and addresses some of the broader social context around eIC.

Unregulated Health Research Using Mobile Devices: Ethical Considerations and Policy Recommendations.

Mobile devices with health apps, direct-to-consumer genetic testing, crowd-sourced information, and other data sources have enabled research by new classes of researchers. Independent researchers, citizen scientists, patient-directed researchers, self-experimenters, and others are not covered by federal research regulations because they are not recipients of federal financial assistance or conducting research in anticipation of a submission to the FDA for approval of a new drug or medical device. This article addresses the difficult policy challenge of promoting the welfare and interests of research participants, as well as the public, in the absence of regulatory requirements and without discouraging independent, innovative scientific inquiry. The article recommends a series of measures, including education, consultation, transparency, self-governance, and regulation to strike the appropriate balance.

Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants.

Digital technologies such as smartphones are transforming the way scientists conduct biomedical research. Several remotely conducted studies have recruited thousands of participants over a span of a few months allowing researchers to collect real-world data at scale and at a fraction of the cost of traditional research. Unfortunately, remote studies have been hampered by substantial participant attrition, calling into question the representativeness of the collected data including generalizability of outcomes. We report the findings regarding recruitment and retention from eight remote digital health studies conducted between 2014-2019 that provided individual-level study-app usage data from more than 100,000 participants completing nearly 3.5 million remote health evaluations over cumulative participation of 850,000 days. Median participant retention across eight studies varied widely from 2-26 days (median across all studies = 5.5 days). Survival analysis revealed several factors significantly associated with increase in participant retention time, including (i) referral by a clinician to the study (increase of 40 days in median retention time); (ii) compensation for participation (increase of 22 days, 1 study); (iii) having the clinical condition of interest in the study (increase of 7 days compared with controls); and (iv) older age (increase of 4 days). Additionally, four distinct patterns of daily app usage behavior were identified by unsupervised clustering, which were also associated with participant demographics. Most studies were not able to recruit a sample that was representative of the race/ethnicity or geographical diversity of the US. Together these findings can help inform recruitment and retention strategies to enable equitable participation of populations in future digital health research.

Navigating ethical quandaries with the privacy dilemma of biomedical datasets.

With decreasing cost of biomedical technologies, the scale of the genetic and healthcare data have exponentially increased and become available to wider audiences. Hence, privacy of patients and study participants has garnered the attention of researchers and regulators alike. Availability of genetic and health care information for uses not anticipated at the time of collection gives rise to privacy concerns such that people suffer dignitary harm when their data is used in ways they did not desire or intend, even if no concrete economic damage results. In this workshop, we explore the issues surrounding data use to advance human health from a privacy perspective. Broadly this field can be considered in two encompassing areas: (1) Ethics and regulation of privacy: The ethical and regulatory frames through which we can consider privacy, the existing regulations regarding privacy and what is on the horizon, and implementation of such ethical considerations for data with the new Common Rule. (2) Approaches to ensuring privacy using technology: The technologies that allow responsible use and sharing of data such as encryption and the quantification of privacy leakages in publicly available data through privacy attacks for better risk-assessment tools.

The Rise of Citizen Science in Health and Biomedical Research.

Citizen science models of public participation in scientific research represent a growing area of opportunity for health and biomedical research, as well as new impetus for more collaborative forms of engagement in large-scale research. However, this also surfaces a variety of ethical issues that both fall outside of and build upon the standard human subjects concerns in bioethics. This article provides background on citizen science, examples of current projects in the field, and discussion of established and emerging ethical issues for citizen science in health and biomedical research.

Implementing a universal informed consent process for the All of Us Research Program.

The United States' All of Us Research Program is a longitudinal research initiative with ambitious national recruitment goals, including of populations traditionally underrepresented in biomedical research, many of whom have high geographic mobility. The program has a distributed infrastructure, with key programmatic resources spread across the US. Given its planned duration and geographic reach both in terms of recruitment and programmatic resources, a diversity of state and territory laws might apply to the program over time as well as to the determination of participants' rights. Here we present a listing and discussion of state and territory guidance and regulation of specific relevance to the program, and our approach to their incorporation within the program's informed consent processes.

Importance of Participant-Centricity and Trust for a Sustainable Medical Information Commons.

Drawing on a landscape analysis of existing data-sharing initiatives, in-depth interviews with expert stakeholders, and public deliberations with community advisory panels across the U.S., we describe features of the evolving medical information commons (MIC). We identify participant-centricity and trustworthiness as the most important features of an MIC and discuss the implications for those seeking to create a sustainable, useful, and widely available collection of linked resources for research and other purposes.

The Role of Government in Precision Medicine, Precision Public Health and the Intersection With Healthy Living.

This paper focuses on the significant role of government in promoting precision medicine and public health and the potential intersection with healthy living (HL) and population health. Recent research has highlighted the interplay between genes, environments and different exposures individuals and populations experience over a lifetime. These interactions between longitudinal behaviors, epigenetics, and expression of the human genome have the potential to transform health and well-being, even within a single generation. Precision medicine can elucidate these longitudinal interactions with a granularity that has not been previously possible across the exposome. Understanding the interactions between genes, epigenetics, proteins, metabolites, and the exposome may inform more evidence-based, effective policy, system, and environmental change to optimize individual and population health. Government has an important role in helping to fund primary research in precision medicine and precision public health, as well as creating and enforcing standards related to food systems, air quality, and access to health care, defining and optimizing measures of health care quality and safety, and ensuring data privacy standards and protections, interoperability, and integration with surveillance systems. Government partnership and collaboration with the non-profit and private sectors can optimize precision medicine and precision public health for the benefit of the United States and global population.

Responsible sharing of biomedical data and biospecimens via the "Automatable Discovery and Access Matrix" (ADA-M).

Given the data-rich nature of modern biomedical research, there is a pressing need for a systematic, structured, computer-readable way to capture, communicate, and manage sharing rules that apply to biomedical resources. This is essential for responsible recording, versioning, communication, querying, and actioning of resource sharing plans. However, lack of a common "information model" for rules and conditions that govern the sharing of materials, methods, software, data, and knowledge creates a fundamental barrier. Without this, it can be virtually impossible for Research Ethics Committees (RECs), Institutional Review Boards (IRBs), Data Access Committees (DACs), biobanks, and end users to confidently track, manage, and interpret applicable legal and ethical requirements. This raises costs and burdens of data stewardship and decreases efficient and responsible access to data, biospecimens, and other resources. To address this, the GA4GH and IRDiRC organizations sponsored the creation of the Automatable Discovery and Access Matrix (ADA-M, read simply as "Adam"). ADA-M is a comprehensive information model that provides the basis for producing structured metadata "Profiles" of regulatory conditions, thereby enabling efficient application of those conditions across regulatory spheres. Widespread use of ADA-M will aid researchers in globally searching and prescreening potential data and/or biospecimen resources for compatibility with their research plans in a responsible and efficient manner, increasing likelihood of timely DAC approvals while also significantly reducing time and effort DACs, RECs, and IRBs spend evaluating resource requests and research proposals. Extensive online documentation, software support, video guides, and an Application Programming Interface (API) for ADA-M have been made available.

Creating a data resource: what will it take to build a medical information commons?

National and international public-private partnerships, consortia, and government initiatives are underway to collect and share genomic, personal, and healthcare data on a massive scale. Ideally, these efforts will contribute to the creation of a medical information commons (MIC), a comprehensive data resource that is widely available for both research and clinical uses. Stakeholder participation is essential in clarifying goals, deepening understanding of areas of complexity, and addressing long-standing policy concerns such as privacy and security and data ownership. This article describes eight core principles proposed by a diverse group of expert stakeholders to guide the formation of a successful, sustainable MIC. These principles promote formation of an ethically sound, inclusive, participant-centric MIC and provide a framework for advancing the policy response to data-sharing opportunities and challenges.

An open source pharma roadmap.

In an Essay, Matthew Todd and colleagues discuss an open source approach to drug development.

Formative Evaluation of Participant Experience With Mobile eConsent in the App-Mediated Parkinson mPower Study: A Mixed Methods Study.

BACKGROUND: To fully capitalize on the promise of mobile technology to enable scalable, participant-centered research, we must develop companion self-administered electronic informed consent (eConsent) processes. As we do so, we have an ethical obligation to ensure that core tenants of informed consent-informedness, comprehension, and voluntariness-are upheld. Furthermore, we should be wary of recapitulating the pitfalls of "traditional" informed consent processes. OBJECTIVE: Our objective was to describe the essential qualities of participant experience, including delineation of common and novel themes relating to informed consent, with a self-administered, smartphone-based eConsent process. We sought to identify participant responses related to informedness, comprehension, and voluntariness as well as to capture any emergent themes relating to the informed consent process in an app-mediated research study. METHODS: We performed qualitative thematic analysis of participant responses to a daily general prompt collected over a 6-month period within the Parkinson mPower app. We employed a combination of a priori and emergent codes for our analysis. A priori codes focused on the core concepts of informed consent; emergent codes were derived to capture additional themes relating to self-administered consent processes. We used self-reported demographic information from the study's baseline survey to characterize study participants and respondents. RESULTS: During the study period, 9846 people completed the eConsent process and enrolled in the Parkinson mPower study. In total, 2758 participants submitted 7483 comments; initial categorization identified a subset of 3875 germane responses submitted by 1678 distinct participants. Respondents were more likely to self-report a Parkinson disease diagnosis (30.21% vs 11.10%), be female (28.26% vs 20.18%), be older (42.89 years vs 34.47 years), and have completed more formal education (66.23% with a 4-year college degree or more education vs 55.77%) than all the mPower participants (P<.001 for all values). Within our qualitative analysis, 3 conceptual domains emerged. First, consistent with fully facilitated in-person informed consent settings, we observed a broad spectrum of comprehension of core research concepts following eConsent. Second, we identified new consent themes born out of the remote mobile research setting, for example the impact of the study design on the engagement of controls and the misconstruction of the open response field as a method for responsive communication with researchers, that bear consideration for inclusion within self-administered eConsent. Finally, our findings highlighted participants' desire to be empowered as partners. CONCLUSIONS: Our study serves as a formative evaluation of participant experience with a self-administered informed consent process via a mobile app. Areas for future investigation include direct comparison of the efficacy of self-administered eConsent with facilitated informed consent processes, exploring the potential benefits and pitfalls of smartphone user behavioral habits on participant engagement in research, and developing best practices to increase informedness, comprehension, and voluntariness via participant coengagement in the research endeavor.