HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force.

PROBLEM: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. WHAT WE DID: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force.

OBJECTIVES: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. METHODS: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials.

BACKGROUND: There have been ongoing efforts to understand when and how data from observational studies can be applied to clinical and regulatory decision making. The objective of this review was to assess the comparability of relative treatment effects of pharmaceuticals from observational studies and randomized controlled trials (RCTs). METHODS: We searched PubMed and Embase for systematic literature reviews published between January 1, 1990, and January 31, 2020, that reported relative treatment effects of pharmaceuticals from both observational studies and RCTs. We extracted pooled relative effect estimates from observational studies and RCTs for each outcome, intervention-comparator, or indication assessed in the reviews. We calculated the ratio of the relative effect estimate from observational studies over that from RCTs, along with the corresponding 95% confidence interval (CI) for each pair of pooled RCT and observational study estimates, and we evaluated the consistency in relative treatment effects. RESULTS: Thirty systematic reviews across 7 therapeutic areas were identified from the literature. We analyzed 74 pairs of pooled relative effect estimates from RCTs and observational studies from 29 reviews. There was no statistically significant difference (based on the 95% CI) in relative effect estimates between RCTs and observational studies in 79.7% of pairs. There was an extreme difference (ratio < 0.7 or > 1.43) in 43.2% of pairs, and, in 17.6% of pairs, there was a significant difference and the estimates pointed in opposite directions. CONCLUSIONS: Overall, our review shows that while there is no significant difference in the relative risk ratios between the majority of RCTs and observational studies compared, there is significant variation in about 20% of comparisons. The source of this variation should be the subject of further inquiry to elucidate how much of the variation is due to differences in patient populations versus biased estimates arising from issues with study design or analytical/statistical methods.

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA.

The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

Utilization of Positive and Negative Controls to Examine Comorbid Associations in Observational Database Studies.

BACKGROUND: Opportunities to leverage observational data for precision medicine research are hampered by underlying sources of bias and paucity of methods to handle resulting uncertainty. We outline an approach to account for bias in identifying comorbid associations between 2 rare genetic disorders and type 2 diabetes (T2D) by applying a positive and negative control disease paradigm. RESEARCH DESIGN: Association between 10 common and 2 rare genetic disorders [Hereditary Fructose Intolerance (HFI) and α-1 antitrypsin deficiency] and T2D was compared with the association between T2D and 7 negative control diseases with no established relationship with T2D in 4 observational databases. Negative controls were used to estimate how much bias and variance existed in datasets when no effect should be observed. RESULTS: Unadjusted association for common and rare genetic disorders and T2D was positive and variable in magnitude and distribution in all 4 databases. However, association between negative controls and T2D was 200% greater than expected indicating the magnitude and confidence intervals for comorbid associations are sensitive to systematic bias. A meta-analysis using this method demonstrated a significant association between HFI and T2D but not for α-1 antitrypsin deficiency. CONCLUSIONS: For observational studies, when covariate data are limited or ambiguous, positive and negative controls provide a method to account for the broadest level of systematic bias, heterogeneity, and uncertainty. This provides greater confidence in assessing associations between diseases and comorbidities. Using this approach we were able to demonstrate an association between HFI and T2D. Leveraging real-world databases is a promising approach to identify and corroborate potential targets for precision medicine therapies.

Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Opportunities and challenges in leveraging electronic health record data in oncology.

The widespread adoption of electronic health records (EHRs) and the growing wealth of digitized information sources about patients is ushering in an era of 'Big Data' that may revolutionize clinical research in oncology. Research will likely be more efficient and potentially more accurate than the current gold standard of manual chart review studies. However, EHRs as they exist today have significant limitations: important data elements are missing or are only captured in free text or PDF documents. Using two case studies, we illustrate the challenges of leveraging the data that are routinely collected by the healthcare system in EHRs (e.g., real-world data), specific challenges encountered in the cancer domain and opportunities that can be achieved when these are overcome.

Optimizing the leveraging of real-world data to improve the development and use of medicines.

Health research, including health outcomes and comparative effectiveness research, is on the cusp of a golden era of access to digitized real-world data, catalyzed by the adoption of electronic health records and the integration of clinical and biological information with other data. This era promises more robust insights into what works in health care. Several barriers, however, will need to be addressed if the full potential of these new data are fully realized; these will involve both policy solutions and stakeholder cooperation. Although a number of these issues have been widely discussed, we focus on the one we believe is the most important-the facilitation of greater openness among public and private stakeholders to collaboration, connecting information and data sharing, with the goal of making robust and complete data accessible to all researchers. In this way, we can better understand the consequences of health care delivery, improve the effectiveness and efficiency of health care systems, and develop advancements in health technologies. Early real-world data initiatives illustrate both potential and the need for future progress, as well as the essential role of collaboration and data sharing. Health policies critical to progress will include those that promote open source data standards, expand access to the data, increase data capture and connectivity, and facilitate communication of findings.

A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.

Evidence-based health care decisions are best informed by comparisons of all relevant interventions used to treat conditions in specific patient populations. Observational studies are being performed to help fill evidence gaps. Widespread adoption of evidence from observational studies, however, has been limited because of various factors, including the lack of consensus regarding accepted principles for their evaluation and interpretation. Two task forces were formed to develop questionnaires to assist decision makers in evaluating observational studies, with one Task Force addressing retrospective research and the other Task Force addressing prospective research. The intent was to promote a structured approach to reduce the potential for subjective interpretation of evidence and drive consistency in decision making. Separately developed questionnaires were combined into a single questionnaire consisting of 33 items. These were divided into two domains: relevance and credibility. Relevance addresses the extent to which findings, if accurate, apply to the setting of interest to the decision maker. Credibility addresses the extent to which the study findings accurately answer the study question. The questionnaire provides a guide for assessing the degree of confidence that should be placed from observational studies and promotes awareness of the subtleties involved in evaluating those.

When Will Real-World Data Fulfill Its Promise to Provide Timely Insights in Oncology?

Randomized trials provide high-quality, internally consistent data on selected clinical questions, but lack generalizability for the aging population who are most often diagnosed with cancer and have comorbid conditions that may affect the interpretation of treatment benefit. The need for high-quality, relevant, and timely data is greater than ever. Promising solutions lie in the collection and analysis of real-world data (RWD), which can potentially provide timely insights about the patient's course during and after initial treatment and the outcomes of important subgroups such as the elderly, rural populations, children, and patients with greater social health needs. However, to inform practice and policy, real-world evidence must be created from trustworthy and comprehensive sources of RWD; these may include pragmatic clinical trials, registries, prospective observational studies, electronic health records (EHRs), administrative claims, and digital technologies. There are unique challenges in oncology since key parameters (eg, cancer stage, biomarker status, genomic assays, imaging response, side effects, quality of life) are not recorded, siloed in inaccessible documents, or available only as free text or unstructured reports in the EHR. Advances in analytics, such as artificial intelligence, may greatly enhance the ability to obtain more granular information from EHRs and support integrated diagnostics; however, they will need to be validated purpose by purpose. We recommend a commitment to standardizing data across sources and building infrastructures that can produce fit-for-purpose RWD that will provide timely understanding of the effectiveness of individual interventions.

The Next Horizon of Drug Development: External Control Arms and Innovative Tools to Enrich Clinical Trial Data.

Conducting clinical trials (CTs) has become increasingly costly and complex in terms of designing and operationalizing. These challenges exist in running CTs on novel therapies, particularly in oncology and rare diseases, where CTs increasingly target narrower patient groups. In this study, we describe external control arms (ECA) and other relevant tools, such as virtualization and decentralized clinical trials (DCTs), and the ability to follow the clinical trial subjects in the real world using tokenization. ECAs are typically constructed by identifying appropriate external sources of data, then by cleaning and standardizing it to create an analysis-ready data file, and finally, by matching subjects in the external data with the subjects in the CT of interest. In addition, ECA tools also include subject-level meta-analysis and simulated subjects' data for analyses. By implementing the recent advances in digital health technologies and devices, virtualization, and DCTs, realigning of CTs from site-centric designs to virtual, decentralized, and patient-centric designs can be done, which reduces the patient burden to participate in the CTs and encourages diversity. Tokenization technology allows linking the CT data with real-world data (RWD), creating more comprehensive and longitudinal outcome measures. These tools provide robust ways to enrich the CT data for informed decision-making, reduce the burden on subjects and costs of trial operations, and augment the insights gained for the CT data.

Prospective observational studies to assess comparative effectiveness: the ISPOR good research practices task force report.

OBJECTIVE: In both the United States and Europe there has been an increased interest in using comparative effectiveness research of interventions to inform health policy decisions. Prospective observational studies will undoubtedly be conducted with increased frequency to assess the comparative effectiveness of different treatments, including as a tool for "coverage with evidence development," "risk-sharing contracting," or key element in a "learning health-care system." The principle alternatives for comparative effectiveness research include retrospective observational studies, prospective observational studies, randomized clinical trials, and naturalistic ("pragmatic") randomized clinical trials. METHODS: This report details the recommendations of a Good Research Practice Task Force on Prospective Observational Studies for comparative effectiveness research. Key issues discussed include how to decide when to do a prospective observational study in light of its advantages and disadvantages with respect to alternatives, and the report summarizes the challenges and approaches to the appropriate design, analysis, and execution of prospective observational studies to make them most valuable and relevant to health-care decision makers. RECOMMENDATIONS: The task force emphasizes the need for precision and clarity in specifying the key policy questions to be addressed and that studies should be designed with a goal of drawing causal inferences whenever possible. If a study is being performed to support a policy decision, then it should be designed as hypothesis testing-this requires drafting a protocol as if subjects were to be randomized and that investigators clearly state the purpose or main hypotheses, define the treatment groups and outcomes, identify all measured and unmeasured confounders, and specify the primary analyses and required sample size. Separate from analytic and statistical approaches, study design choices may strengthen the ability to address potential biases and confounding in prospective observational studies. The use of inception cohorts, new user designs, multiple comparator groups, matching designs, and assessment of outcomes thought not to be impacted by the therapies being compared are several strategies that should be given strong consideration recognizing that there may be feasibility constraints. The reasoning behind all study design and analytic choices should be transparent and explained in study protocol. Execution of prospective observational studies is as important as their design and analysis in ensuring that results are valuable and relevant, especially capturing the target population of interest, having reasonably complete and nondifferential follow-up. Similar to the concept of the importance of declaring a prespecified hypothesis, we believe that the credibility of many prospective observational studies would be enhanced by their registration on appropriate publicly accessible sites (e.g., clinicaltrials.gov and encepp.eu) in advance of their execution.

Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part I.

OBJECTIVES: Health insurers, physicians, and patients worldwide need information on the comparative effectiveness and safety of prescription drugs in routine care. Nonrandomized studies of treatment effects using secondary databases may supplement the evidence based from randomized clinical trials and prospective observational studies. Recognizing the challenges to conducting valid retrospective epidemiologic and health services research studies, a Task Force was formed to develop a guidance document on state of the art approaches to frame research questions and report findings for these studies. METHODS: The Task Force was commissioned and a Chair was selected by the International Society for Pharmacoeconomics and Outcomes Research Board of Directors in October 2007. This Report, the first of three reported in this issue of the journal, addressed issues of framing the research question and reporting and interpreting findings. RESULTS: The Task Force Report proposes four primary characteristics-relevance, specificity, novelty, and feasibility while defining the research question. Recommendations included: the practice of a priori specification of the research question; transparency of prespecified analytical plans, provision of justifications for any subsequent changes in analytical plan, and reporting the results of prespecified plans as well as results from significant modifications, structured abstracts to report findings with scientific neutrality; and reasoned interpretations of findings to help inform policy decisions. CONCLUSIONS: Comparative effectiveness research in the form of nonrandomized studies using secondary databases can be designed with rigorous elements and conducted with sophisticated statistical methods to improve causal inference of treatment effects. Standardized reporting and careful interpretation of results can aid policy and decision-making.

Getting real performance out of pay-for-performance.

CONTEXT: Most private and public health insurers are implementing pay-for-performance (P4P) programs in an effort to improve the quality of medical care. This article offers a paradigm for evaluating how P4P programs should be structured and how effective they are likely to be. METHODS: This article assesses the current comprehensiveness of evidence-based medicine by estimating the percentage of outpatient medical spending for eighteen medical processes recommended by the Institute of Medicine. FINDINGS: Three conditions must be in place for outcomes-based P4P programs to improve the quality of care: (1) health insurers must not fully understand what medical processes improve health (i.e., the health production function); (2) providers must know more about the health production function than insurers do; and (3) health insurers must be able to measure a patient's risk-adjusted health. Only two of these conditions currently exist. Payers appear to have incomplete knowledge of the health production function, and providers appear to know more about the health production function than payers do, but accurate methods of adjusting the risk of a patient's health status are still being developed. CONCLUSIONS: This article concludes that in three general situations, P4P will have a different impact on quality and costs and so should be structured differently. When information about patients' health and the health production function is incomplete, as is currently the case, P4P payments should be kept small, should be based on outcomes rather than processes, and should target physicians' practices and health systems. As information improves, P4P incentive payments could be increased, and P4P may become more powerful. Ironically, once information becomes complete, P4P can be replaced entirely by "optimal fee-for-service."

The joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making: an interview with Marc Berger.

Marc L Berger, MD, is a retired, part-time consultant. He recently became Chair of the Real World Evidence Advisory Board for SHYFT Analytics. Over a 25-year industry career, Marc has held senior-level positions at Pfizer, Inc., OptumInsight, Eli Lilly and Company, and Merck & Co., Inc. His professional activities have included serving on committees for Center for Medicare and Medicaid Services (CMS), Agency for Healthcare Research and Quality (AHRQ), Patient-Centered Outcomes Research Institute (PCORI), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), Drug Information Association (DIA), and the editorial advisory boards of several journals. Marc has written or co-written more than 100 peer-reviewed articles, book chapters, and other publications on a range of topics including health services research, outcomes research, health economics, health policy, and the analysis of real-world data.