When Will Real-World Data Fulfill Its Promise to Provide Timely Insights in Oncology?

Randomized trials provide high-quality, internally consistent data on selected clinical questions, but lack generalizability for the aging population who are most often diagnosed with cancer and have comorbid conditions that may affect the interpretation of treatment benefit. The need for high-quality, relevant, and timely data is greater than ever. Promising solutions lie in the collection and analysis of real-world data (RWD), which can potentially provide timely insights about the patient's course during and after initial treatment and the outcomes of important subgroups such as the elderly, rural populations, children, and patients with greater social health needs. However, to inform practice and policy, real-world evidence must be created from trustworthy and comprehensive sources of RWD; these may include pragmatic clinical trials, registries, prospective observational studies, electronic health records (EHRs), administrative claims, and digital technologies. There are unique challenges in oncology since key parameters (eg, cancer stage, biomarker status, genomic assays, imaging response, side effects, quality of life) are not recorded, siloed in inaccessible documents, or available only as free text or unstructured reports in the EHR. Advances in analytics, such as artificial intelligence, may greatly enhance the ability to obtain more granular information from EHRs and support integrated diagnostics; however, they will need to be validated purpose by purpose. We recommend a commitment to standardizing data across sources and building infrastructures that can produce fit-for-purpose RWD that will provide timely understanding of the effectiveness of individual interventions.

The Next Horizon of Drug Development: External Control Arms and Innovative Tools to Enrich Clinical Trial Data.

Conducting clinical trials (CTs) has become increasingly costly and complex in terms of designing and operationalizing. These challenges exist in running CTs on novel therapies, particularly in oncology and rare diseases, where CTs increasingly target narrower patient groups. In this study, we describe external control arms (ECA) and other relevant tools, such as virtualization and decentralized clinical trials (DCTs), and the ability to follow the clinical trial subjects in the real world using tokenization. ECAs are typically constructed by identifying appropriate external sources of data, then by cleaning and standardizing it to create an analysis-ready data file, and finally, by matching subjects in the external data with the subjects in the CT of interest. In addition, ECA tools also include subject-level meta-analysis and simulated subjects' data for analyses. By implementing the recent advances in digital health technologies and devices, virtualization, and DCTs, realigning of CTs from site-centric designs to virtual, decentralized, and patient-centric designs can be done, which reduces the patient burden to participate in the CTs and encourages diversity. Tokenization technology allows linking the CT data with real-world data (RWD), creating more comprehensive and longitudinal outcome measures. These tools provide robust ways to enrich the CT data for informed decision-making, reduce the burden on subjects and costs of trial operations, and augment the insights gained for the CT data.

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force.

PROBLEM: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. WHAT WE DID: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force.

OBJECTIVES: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. METHODS: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials.

BACKGROUND: There have been ongoing efforts to understand when and how data from observational studies can be applied to clinical and regulatory decision making. The objective of this review was to assess the comparability of relative treatment effects of pharmaceuticals from observational studies and randomized controlled trials (RCTs). METHODS: We searched PubMed and Embase for systematic literature reviews published between January 1, 1990, and January 31, 2020, that reported relative treatment effects of pharmaceuticals from both observational studies and RCTs. We extracted pooled relative effect estimates from observational studies and RCTs for each outcome, intervention-comparator, or indication assessed in the reviews. We calculated the ratio of the relative effect estimate from observational studies over that from RCTs, along with the corresponding 95% confidence interval (CI) for each pair of pooled RCT and observational study estimates, and we evaluated the consistency in relative treatment effects. RESULTS: Thirty systematic reviews across 7 therapeutic areas were identified from the literature. We analyzed 74 pairs of pooled relative effect estimates from RCTs and observational studies from 29 reviews. There was no statistically significant difference (based on the 95% CI) in relative effect estimates between RCTs and observational studies in 79.7% of pairs. There was an extreme difference (ratio < 0.7 or > 1.43) in 43.2% of pairs, and, in 17.6% of pairs, there was a significant difference and the estimates pointed in opposite directions. CONCLUSIONS: Overall, our review shows that while there is no significant difference in the relative risk ratios between the majority of RCTs and observational studies compared, there is significant variation in about 20% of comparisons. The source of this variation should be the subject of further inquiry to elucidate how much of the variation is due to differences in patient populations versus biased estimates arising from issues with study design or analytical/statistical methods.

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA.

The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

The joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making: an interview with Marc Berger.

Marc L Berger, MD, is a retired, part-time consultant. He recently became Chair of the Real World Evidence Advisory Board for SHYFT Analytics. Over a 25-year industry career, Marc has held senior-level positions at Pfizer, Inc., OptumInsight, Eli Lilly and Company, and Merck & Co., Inc. His professional activities have included serving on committees for Center for Medicare and Medicaid Services (CMS), Agency for Healthcare Research and Quality (AHRQ), Patient-Centered Outcomes Research Institute (PCORI), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), Drug Information Association (DIA), and the editorial advisory boards of several journals. Marc has written or co-written more than 100 peer-reviewed articles, book chapters, and other publications on a range of topics including health services research, outcomes research, health economics, health policy, and the analysis of real-world data.

Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Utilization of Positive and Negative Controls to Examine Comorbid Associations in Observational Database Studies.

BACKGROUND: Opportunities to leverage observational data for precision medicine research are hampered by underlying sources of bias and paucity of methods to handle resulting uncertainty. We outline an approach to account for bias in identifying comorbid associations between 2 rare genetic disorders and type 2 diabetes (T2D) by applying a positive and negative control disease paradigm. RESEARCH DESIGN: Association between 10 common and 2 rare genetic disorders [Hereditary Fructose Intolerance (HFI) and α-1 antitrypsin deficiency] and T2D was compared with the association between T2D and 7 negative control diseases with no established relationship with T2D in 4 observational databases. Negative controls were used to estimate how much bias and variance existed in datasets when no effect should be observed. RESULTS: Unadjusted association for common and rare genetic disorders and T2D was positive and variable in magnitude and distribution in all 4 databases. However, association between negative controls and T2D was 200% greater than expected indicating the magnitude and confidence intervals for comorbid associations are sensitive to systematic bias. A meta-analysis using this method demonstrated a significant association between HFI and T2D but not for α-1 antitrypsin deficiency. CONCLUSIONS: For observational studies, when covariate data are limited or ambiguous, positive and negative controls provide a method to account for the broadest level of systematic bias, heterogeneity, and uncertainty. This provides greater confidence in assessing associations between diseases and comorbidities. Using this approach we were able to demonstrate an association between HFI and T2D. Leveraging real-world databases is a promising approach to identify and corroborate potential targets for precision medicine therapies.

Big Data: transforming drug development and health policy decision making.

The explosion of data sources, accompanied by the evolution of technology and analytical techniques, has created considerable challenges and opportunities for drug development and healthcare resource utilization. We present a systematic overview these phenomena, and suggest measures to be taken for effective integration of the new developments in the traditional medical research paradigm and health policy decision making. Special attention is paid to pertinent issues in emerging areas, including rare disease drug development, personalized medicine, Comparative Effectiveness Research, and privacy and confidentiality concerns.

Opportunities and challenges in leveraging electronic health record data in oncology.

The widespread adoption of electronic health records (EHRs) and the growing wealth of digitized information sources about patients is ushering in an era of 'Big Data' that may revolutionize clinical research in oncology. Research will likely be more efficient and potentially more accurate than the current gold standard of manual chart review studies. However, EHRs as they exist today have significant limitations: important data elements are missing or are only captured in free text or PDF documents. Using two case studies, we illustrate the challenges of leveraging the data that are routinely collected by the healthcare system in EHRs (e.g., real-world data), specific challenges encountered in the cancer domain and opportunities that can be achieved when these are overcome.

Optimizing the leveraging of real-world data to improve the development and use of medicines.

Health research, including health outcomes and comparative effectiveness research, is on the cusp of a golden era of access to digitized real-world data, catalyzed by the adoption of electronic health records and the integration of clinical and biological information with other data. This era promises more robust insights into what works in health care. Several barriers, however, will need to be addressed if the full potential of these new data are fully realized; these will involve both policy solutions and stakeholder cooperation. Although a number of these issues have been widely discussed, we focus on the one we believe is the most important-the facilitation of greater openness among public and private stakeholders to collaboration, connecting information and data sharing, with the goal of making robust and complete data accessible to all researchers. In this way, we can better understand the consequences of health care delivery, improve the effectiveness and efficiency of health care systems, and develop advancements in health technologies. Early real-world data initiatives illustrate both potential and the need for future progress, as well as the essential role of collaboration and data sharing. Health policies critical to progress will include those that promote open source data standards, expand access to the data, increase data capture and connectivity, and facilitate communication of findings.

A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.

Evidence-based health care decisions are best informed by comparisons of all relevant interventions used to treat conditions in specific patient populations. Observational studies are being performed to help fill evidence gaps. Widespread adoption of evidence from observational studies, however, has been limited because of various factors, including the lack of consensus regarding accepted principles for their evaluation and interpretation. Two task forces were formed to develop questionnaires to assist decision makers in evaluating observational studies, with one Task Force addressing retrospective research and the other Task Force addressing prospective research. The intent was to promote a structured approach to reduce the potential for subjective interpretation of evidence and drive consistency in decision making. Separately developed questionnaires were combined into a single questionnaire consisting of 33 items. These were divided into two domains: relevance and credibility. Relevance addresses the extent to which findings, if accurate, apply to the setting of interest to the decision maker. Credibility addresses the extent to which the study findings accurately answer the study question. The questionnaire provides a guide for assessing the degree of confidence that should be placed from observational studies and promotes awareness of the subtleties involved in evaluating those.

Big data, advanced analytics and the future of comparative effectiveness research.

The intense competition that accompanied the growth of internet-based companies ushered in the era of 'big data' characterized by major innovations in processing of very large amounts of data and the application of advanced analytics including data mining and machine learning. Healthcare is on the cusp of its own era of big data, catalyzed by the changing regulatory and competitive environments, fueled by growing adoption of electronic health records, as well as efforts to integrate medical claims, electronic health records and other novel data sources. Applying the lessons from big data pioneers will require healthcare and life science organizations to make investments in new hardware and software, as well as in individuals with different skills. For life science companies, this will impact the entire pharmaceutical value chain from early research to postcommercialization support. More generally, this will revolutionize comparative effectiveness research.