Evaluation of US oncology electronic health record real-world data to reduce uncertainty in health technology appraisals: a retrospective cohort study.

OBJECTIVES: Examine whether data from early access to medicines in the USA can be used to inform National Institute for Health and Care Excellence (NICE) health technology assessments (HTA) in oncology. DESIGN: Retrospective cohort study. SETTING: Oncology-based community and academic treatment centres in the USA. PARTICIPANTS: Patients present in a nationwide electronic health record (EHR)-derived deidentified database. INTERVENTIONS: Cancer drugs that underwent NICE technology appraisal (TA) between 2014 and 2019. PRIMARY AND SECONDARY OUTCOME MEASURES: The count and follow-up time of US patients, available in the EHR, who were exposed to cancer drugs of interest in the period between Food and Drug Administration (FDA) approval and dates relevant to the NICE appraisal process. RESULTS: In 59 of 60 TAs analysed, the cancer therapy was approved in the USA before the final appraisal by NICE. The median time from FDA approval to the publication of NICE recommendations was 18.5 months, at which time the US EHR-derived database had, on average, 269 patients (SD=356) exposed to the new therapy, with a median of 75.3 person-years (IQR: 13.1-173) in time-at-risk. A case study generated evidence on real-world overall survival and treatment duration. CONCLUSIONS: Across different cancer therapies, there was substantial variability in US real-world data accumulated between FDA approval and NICE decision milestones. The applicability of these data to generate evidence for HTA decision-making should be assessed on a case-by-case basis depending on the intended HTA use case.

Perspectives on Patient-Reported Outcome Data After Treatment Discontinuation in Cancer Clinical Trials.

OBJECTIVES: Patient-reported outcome (PRO) data are critical in understanding treatments from the patient perspective in cancer clinical trials. The potential benefits and methodological approaches to the collection of PRO data after treatment discontinuation (eg, because of progressive disease or unacceptable drug toxicity) are less clear. The purpose of this article is to describe the Food and Drug Administration's Oncology Center of Excellence and the Critical Path Institute cosponsored 2-hour virtual roundtable, held in 2020, to discuss this specific issue. METHODS: We summarize key points from this discussion with 16 stakeholders representing academia, clinical practice, patients, international regulatory agencies, health technology assessment bodies/payers, industry, and PRO instrument development. RESULTS: Stakeholders recognized that any PRO data collection after treatment discontinuation should have clearly defined objectives to ensure that data can be analyzed and reported. CONCLUSIONS: Data collection after discontinuation without a justification for its use wastes patients' time and effort and is unethical.

Environmental impact assessment in health technology assessment: principles, approaches, and challenges.

To reduce harm to the environment resulting from the production, use, and disposal of health technologies, there are different options for how health technology assessment (HTA) agencies can consider environmental information. We identified four approaches that HTA agencies can use to take environmental information into account in healthcare decision making and the challenges associated with each approach. Republishing data that is in the public domain or has been submitted to an HTA agency we term the "information conduit" approach. Analyzing and presenting environmental data separately from established health economic analyses is described as "parallel evaluation." Integrating environmental impact into HTAs by identifying or creating new methods that allow clinical, financial, and environmental information to be combined in a single quantitative analysis is "integrated evaluation." Finally, evidence synthesis and analysis of health technologies that are not expected to improve health-related outcomes but claim to have relative environmental benefits are termed "environment-focused evaluation."

Advancing the use of real world evidence in health technology assessment: insights from a multi-stakeholder workshop.

Introduction: Real-world evidence (RWE) in health technology assessment (HTA) holds significant potential for informing healthcare decision-making. A multistakeholder workshop was organised by the European Health Data and Evidence Network (EHDEN) and the GetReal Institute to explore the status, challenges, and opportunities in incorporating RWE into HTA, with a focus on learning from regulatory initiatives such as the European Medicines Agency (EMA) Data Analysis and Real World Interrogation Network (DARWIN EU®). Methods: The workshop gathered key stakeholders from regulatory agencies, HTA organizations, academia, and industry for three panel discussions on RWE and HTA integration. Insights and recommendations were collected through panel discussions and audience polls. The workshop outcomes were reviewed by authors to identify key themes, challenges, and recommendations. Results: The workshop discussions revealed several important findings relating to the use of RWE in HTA. Compared with regulatory processes, its adoption in HTA to date has been slow. Barriers include limited trust in RWE, data quality concerns, and uncertainty about best practices. Facilitators include multidisciplinary training, educational initiatives, and stakeholder collaboration, which could be facilitated by initiatives like EHDEN and the GetReal Institute. Demonstrating the impact of "driver projects" could promote RWE adoption in HTA. Conclusion: To enhance the integration of RWE in HTA, it is crucial to address known barriers through comprehensive training, stakeholder collaboration, and impactful exemplar research projects. By upskilling users and beneficiaries of RWE and those that generate it, promoting collaboration, and conducting "driver projects," can strengthen the HTA evidence base for more informed healthcare decisions.

Transferability of real-world data across borders for regulatory and health technology assessment decision-making.

Recently, there has been increased consideration of real-world data (RWD) and real-world evidence (RWE) in regulatory and health technology assessment (HTA) decision-making. Due to challenges in identifying high-quality and relevant RWD sources, researchers and regulatory/HTA bodies may turn to RWD generated in locales outside of the locale of interest (referred to as "transferring RWD"). We therefore performed a review of stakeholder guidance as well as selected case studies to identify themes for researchers to consider when transferring RWD from one jurisdiction to another. Our review highlighted that there is limited consensus on defining decision-grade, transferred RWD; certain stakeholders have issued relevant guidance, but the recommendations are high-level and additional effort is needed to generate comprehensive guidance. Additionally, the case studies revealed that RWD transferability has not been a consistent concern for regulatory/HTA bodies and that more focus has been put on the evaluation of internal validity. To help develop transferability best practices (alongside internal validity best practices), we suggest that researchers address the following considerations in their justification for transferring RWD: treatment pathways, nature of the healthcare system, incidence/prevalence of indication, and patient demographics. We also recommend that RWD transferability should garner more attention as the use of imported RWD could open doors to high-quality data sources and potentially reduce methodological issues that often arise in the use of local RWD; we thus hope this review provides a foundation for further dialogue around the suitability and utility of transferred RWD in the regulatory/HTA decision-making space.

Can we use existing guidance to support the development of robust real-world evidence for health technology assessment/payer decision-making?

Advances in the digitization of health systems and expedited regulatory approvals of innovative treatments have led to increased potential for the use of real-world data (RWD) to generate real-world evidence (RWE) to complement evidence from clinical trials. However, health technology assessment (HTA) bodies and payers have concerns about the ability to generate RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness. Consequently, there is a growing need for HTA bodies and payers to develop guidance for the industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing. We therefore sought to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of published guidance; and (iii) review findings with selected HTA/payer bodies. Four themes considered key to shaping the generation of robust RWE for HTA bodies and payers were identified as: (i) data (availability, governance, and quality); (ii) methodology (design and analytics); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. A range of guidance documents were found from trusted sources that could address these themes. These were discussed with HTA experts. This commentary summarizes the potential guidance solutions available to help resolve issues faced by HTA decision-makers in the adoption of RWD/RWE. It shows that there is alignment among stakeholders about the areas that need improvement in the development of RWE and that the key priority to move forward is better collaboration to make data usable for multiple purposes.

Using primary care data to assess comparative effectiveness and safety of apixaban and rivaroxaban in patients with nonvalvular atrial fibrillation in the UK: an observational cohort study.

OBJECTIVE: To compare real-world effectiveness and safety of direct oral anticoagulants (DOACs) in patients with nonvalvular atrial fibrillation (AFib) for prevention of stroke. STUDY DESIGN AND SETTING: A comparative cohort study in UK general practice data from The Health Improvement Network database. PARTICIPANTS AND INTERVENTIONS: Before matching, 5655 patients ≥18 years with nonvalvular AFib who initiated at least one DOAC between 1 July 2014 and 31 December 2020 were included. DOACs of interest included apixaban, rivaroxaban, edoxaban and dabigatran, with the primary comparison between apixaban and rivaroxaban. Initiators of DOACs were defined as new users with no record of prescription for any DOAC during 12 months before index date. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was stroke (ischaemic or haemorrhagic). Secondary outcomes included the occurrence of all-cause mortality, myocardial infarction (MI), transient ischaemic attacks (TIA), major bleeding events and a composite angina/MI/stroke (AMS) endpoint. RESULTS: Compared with rivaroxaban, patients initiating apixaban showed similar rates of stroke (HR: 0.93; 95% CI 0.64 to 1.34), all-cause mortality (HR: 1.03; 95% CI 0.87 to 1.22), MI (HR: 0.95; 95% CI 0.54 to 1.68), TIA (HR: 1.03; 95% CI 0.61 to 1.72) and AMS (HR: 0.96; 95% CI 0.72 to 1.27). Apixaban initiators showed lower rates of major bleeding events (HR: 0.60; 95% CI 0.47 to 0.75). CONCLUSIONS: Among patients with nonvalvular AFib, apixaban was as effective as rivaroxaban in reducing rate of stroke and safer in terms of major bleeding episodes. This head-to-head comparison supports conclusions drawn from indirect comparisons of DOAC trials against warfarin and demonstrates the potential for real-world evidence to fill evidence gaps and reduce uncertainty in both health technology assessment decision-making and clinical guideline development.

Variations in comorbidity burden in people with type 2 diabetes over disease duration: A population-based analysis of real world evidence.

Background: The prevalence of type 2 diabetes (T2DM) is increasing, but increasing longevity among persons with diagnosed diabetes may be is associated with more extensive and diverse types of morbidity. The extent and breadth of morbidity and how this varies across sub-groups is unclear and could have important clinical and public health implications. We aimed to estimate comorbidity profiles in people with T2DM and variations across sub-groups and over time. Methods: We identified approximately 224,000 people with T2DM in the Discover-NOW dataset, a real-world primary care database from 2000 to 2020 covering 2.5 million people across North-West London, England, linked to hospital records. We generated a mixed prevalence and incidence study population through repeated annual cross sections, and included a broad set of 35 comorbidities covering traditional T2DM conditions, emerging T2DM conditions and other common conditions.We estimated annual age-standardised prevalence of comorbidities, over the course of the disease in people with T2DM and several sub-groups. Findings: Multimorbidity (two or more chronic conditions) is common in people with T2DM and increasing, but the comorbidity profiles of people with T2DM vary substantially. Nearly 30% of T2DM patients had three or more comorbidities at diagnosis, increasing to 60% of patients ten years later. Two of the five commonest comorbidities at diagnosis were traditional T2DM conditions (hypertension (37%) and ischaemic heart disease (10%)) the other three were not (depression (15%), back pain (25%) and osteoarthritis (11%)). The prevalence of each increased during the course of the disease, with more than one in three patients having back pain and one in four having depression ten years post diagnosis.People with five or more comorbidities at diagnosis had higher prevalence of each of the 35 comorbidities. Hypertension (73%) was the commonest comorbidity at diagnosis in this group; followed by back pain (69%), depression (67%), asthma (45%) and osteoarthritis (36%). People with obesity at diagnosis had substantially different comorbidity profiles to those without, and the five commonest comorbidities were 50% more common in this group. Interpretation: Preventative and clinical interventions alongside care pathways for people with T2DM should transition to reflect the diverse set of causes driving persistent morbidity. This would benefit both patients and healthcare systems alike. Funding: The study was funded by the National Institute for Health and Care Excellence (NICE).

Machine Learning Methods in Health Economics and Outcomes Research-The PALISADE Checklist: A Good Practices Report of an ISPOR Task Force.

Advances in machine learning (ML) and artificial intelligence offer tremendous potential benefits to patients. Predictive analytics using ML are already widely used in healthcare operations and care delivery, but how can ML be used for health economics and outcomes research (HEOR)? To answer this question, ISPOR established an emerging good practices task force for the application of ML in HEOR. The task force identified 5 methodological areas where ML could enhance HEOR: (1) cohort selection, identifying samples with greater specificity with respect to inclusion criteria; (2) identification of independent predictors and covariates of health outcomes; (3) predictive analytics of health outcomes, including those that are high cost or life threatening; (4) causal inference through methods, such as targeted maximum likelihood estimation or double-debiased estimation-helping to produce reliable evidence more quickly; and (5) application of ML to the development of economic models to reduce structural, parameter, and sampling uncertainty in cost-effectiveness analysis. Overall, ML facilitates HEOR through the meaningful and efficient analysis of big data. Nevertheless, a lack of transparency on how ML methods deliver solutions to feature selection and predictive analytics, especially in unsupervised circumstances, increases risk to providers and other decision makers in using ML results. To examine whether ML offers a useful and transparent solution to healthcare analytics, the task force developed the PALISADE Checklist. It is a guide for balancing the many potential applications of ML with the need for transparency in methods development and findings.

The Sandbox Approach and its Potential for Use in Health Technology Assessment: A Literature Review.

BACKGROUND: The concept of the regulatory sandbox-a safe space for testing new regulatory processes-was first used within the financial technologies (FinTech) sector, but has since expanded into other sectors, including healthcare. OBJECTIVES: This review aims to describe the extent of use of sandboxes in healthcare and assess the potential for the sandbox approach to be used to test and develop emerging health technology assessment (HTA) methods, policies and processes for innovative technologies. METHODS: A systematic literature review was undertaken to identify published papers and reports that described and/or assessed the use of sandboxes in the healthcare sector. Searches were conducted in Medline, Embase, Econlit, Social Policy and Practice, and Health Management Information Consortium databases from inception to March 2020. Free-text Google search was also conducted to identify relevant grey literature. Only papers and reports discussing or evaluating the use of sandboxes in healthcare settings and published in English were included. Included studies were qualitatively summarised using a thematic analysis approach. RESULTS: Overall, 46 papers and reports were included. The topics covered were classified into 4 major themes: history of the regulatory sandbox, the sandbox as a testing environment, the sandbox as a regulatory approach, examples of using sandboxes in healthcare. Findings show that the use of regulatory sandboxes in healthcare is relatively new and primarily used in high-income countries to support the adoption of new technologies, particularly those related to digital health. Recommendations are made based on these findings to guide its use in HTA policy and methods development. CONCLUSIONS: Sandboxes are increasingly used within healthcare regulation. Despite its potential, this approach has not been used in HTA policy and methodological developments to date. HTA agencies should consider this approach to facilitate developing policies, methods and processes for innovative and disruptive health technologies. Transferability to low- and middle-income countries' settings, however, should be assessed.

The use of nonrandomized evidence to estimate treatment effects in health technology assessment.

Health technology assessment (HTA) is increasingly informed by nonrandomized studies, but there is limited guidance from HTA bodies on expectations around evidence quality and study conduct. We developed recommendations to support the appropriate use of such evidence based on a pragmatic literature review and a workshop involving 16 experts from eight countries as part of the EU's Horizon-2020 IMPACT-HTA program (work package six). To ensure HTA processes remain rigorous and robust, HTA bodies should demand clear, extensive and structured reporting of nonrandomized studies, including an in-depth assessment of the risk of bias. In recognition of the additional uncertainty imparted by nonrandomized designs in estimates of treatment effects, HTA bodies should strengthen early scientific advice and engage in collaborative efforts to improve use of real-world data.

Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance.

Decision-makers have become increasingly interested in incorporating real-world evidence (RWE) into their decision-making process. Due to concerns regarding the reliability and quality of RWE, stakeholders have issued numerous recommendation documents to assist in setting RWE standards. The fragmented nature of these documents poses a challenge to researchers and decision-makers looking for guidance on what is 'high-quality' RWE and how it can be used in decision-making. We offer researchers and decision-makers a structure to organize the landscape of RWE recommendations and identify consensus and gaps in the current recommendations. To provide researchers with a much needed pathway for generating RWE, we discuss how decision-makers can move from fragmented recommendations to comprehensive guidance.

Common Problems, Common Data Model Solutions: Evidence Generation for Health Technology Assessment.

There is growing interest in using observational data to assess the safety, effectiveness, and cost effectiveness of medical technologies, but operational, technical, and methodological challenges limit its more widespread use. Common data models and federated data networks offer a potential solution to many of these problems. The open-source Observational and Medical Outcomes Partnerships (OMOP) common data model standardises the structure, format, and terminologies of otherwise disparate datasets, enabling the execution of common analytical code across a federated data network in which only code and aggregate results are shared. While common data models are increasingly used in regulatory decision making, relatively little attention has been given to their use in health technology assessment (HTA). We show that the common data model has the potential to facilitate access to relevant data, enable multidatabase studies to enhance statistical power and transfer results across populations and settings to meet the needs of local HTA decision makers, and validate findings. The use of open-source and standardised analytics improves transparency and reduces coding errors, thereby increasing confidence in the results. Further engagement from the HTA community is required to inform the appropriate standards for mapping data to the common data model and to design tools that can support evidence generation and decision making.

Improving Transparency to Build Trust in Real-World Secondary Data Studies for Hypothesis Testing-Why, What, and How: Recommendations and a Road Map from the Real-World Evidence Transparency Initiative.

Real-world data (RWD) and the derivations of these data into real-world evidence (RWE) are rapidly expanding from informing healthcare decisions at the patient and health system level to influencing major health policy decisions, including regulatory approvals and coverage. Recent examples include the approval of palbociclib in combination with endocrine therapy for male breast cancer and the inclusion of RWE in the label of paliperidone palmitate for schizophrenia. This interest has created an urgency to develop processes that promote trust in the evidence-generation process. Key stakeholders and decision-makers include patients and their healthcare providers; learning health systems; health technology assessment bodies and payers; pharmacoepidemiologists and other clinical reseachers, and policy makers interested in bioethical and regulatory issues. A key to optimal uptake of RWE is transparency of the research process to enable decision-makers to evaluate the quality of the methods used and the applicability of the evidence that results from the RWE studies. Registration of RWE studies-particularly for hypothesis evaluating treatment effectiveness (HETE) studies-has been proposed to improve transparency, trust, and research replicability. Although registration would not guarantee better RWE studies would be conducted, it would encourage the prospective disclosure of study plans, timing, and rationale for modifications. A joint task force of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) recommended that investigators preregister their RWE studies and post their study protocols in a publicly available forum before starting studies to reduce publication bias and improve the transparency of research methods. Recognizing that published recommendations alone are insufficient, especially without accessible registration options and with no incentives, a group of experts gathered on February 25 and 26, 2019, in National Harbor, Maryland, to explore the structural and practical challenges to the successful implementation of the recommendations of the ISPOR/ISPE task force for preregistration. This positioning article describes a plan for making registration of HETE RWE studies routine. The plan includes specifying the rationale for registering HETE RWE studies, the studies that should be registered, where and when these studies should be registered, how and when analytic deviations from protocols should be reported, how and when to publish results, and incentives to encourage registration. Table 1 summarizes the rationale, goals, and potential solutions that increase transparency, in addition to unique concerns about secondary data studies. Definitions of terms used throughout this report are provided in Table 2.

Improving transparency to build trust in real-world secondary data studies for hypothesis testing-Why, what, and how: recommendations and a road map from the real-world evidence transparency initiative.

Real-world data (RWD) and the derivations of these data into real-world evidence (RWE) are rapidly expanding from informing healthcare decisions at the patient and health system level to influencing major health policy decisions, including regulatory approvals and coverage. Recent examples include the approval of palbociclib in combination with endocrine therapy for male breast cancer and the inclusion of RWE in the label of paliperidone palmitate for schizophrenia. This interest has created an urgency to develop processes that promote trust in the evidence-generation process. Key stakeholders and decision-makers include patients and their healthcare providers; learning health systems; health technology assessment bodies and payers; pharmacoepidemiologists and other clinical reseachers, and policy makers interested in bioethical and regulatory issues. A key to optimal uptake of RWE is transparency of the research process to enable decision-makers to evaluate the quality of the methods used and the applicability of the evidence that results from the RWE studies. Registration of RWE studies-particularly for hypothesis evaluating treatment effectiveness (HETE) studies-has been proposed to improve transparency, trust, and research replicability. Although registration would not guarantee better RWE studies would be conducted, it would encourage the prospective disclosure of study plans, timing, and rationale for modifications. A joint task force of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) recommended that investigators preregister their RWE studies and post their study protocols in a publicly available forum before starting studies to reduce publication bias and improve the transparency of research methods. Recognizing that published recommendations alone are insufficient, especially without accessible registration options and with no incentives, a group of experts gathered on February 25 and 26, 2019, in National Harbor, Maryland, to explore the structural and practical challenges to the successful implementation of the recommendations of the ISPOR/ISPE task force for preregistration. This positioning article describes a plan for making registration of HETE RWE studies routine. The plan includes specifying the rationale for registering HETE RWE studies, the studies that should be registered, where and when these studies should be registered, how and when analytic deviations from protocols should be reported, how and when to publish results, and incentives to encourage registration. Table 1 summarizes the rationale, goals, and potential solutions that increase transparency, in addition to unique concerns about secondary data studies. Definitions of terms used throughout this report are provided in Table 2.

Reweighting Randomized Controlled Trial Evidence to Better Reflect Real Life - A Case Study of the Innovative Medicines Initiative.

Evidence from randomized controlled trials available for timely health technology assessments of new pharmacological treatments and regulatory decision making may not be generalizable to local patient populations, often resulting in decisions being made under uncertainty. In recent years, several reweighting approaches have been explored to address this important question of generalizability to a target population. We present a case study of the Innovative Medicines Initiative to illustrate the inverse propensity score reweighting methodology, which may allow us to estimate the expected treatment benefit if a clinical trial had been run in a broader real-world target population. We learned that identifying treatment effect modifiers, understanding and managing differences between patient characteristic data sets, and balancing the closeness of trial and target patient populations with effective sample size are key to successfully using this methodology and potentially mitigating some of this uncertainty around local decision making.

Transparent Reporting on Research Using Unstructured Electronic Health Record Data to Generate 'Real World' Evidence of Comparative Effectiveness and Safety.

Research that makes secondary use of administrative and clinical healthcare databases is increasingly influential for regulatory, reimbursement, and other healthcare decision-making. Consequently, there are numerous guidance documents on reporting for studies that use 'real-world' data captured in administrative claims and electronic health record (EHR) databases. These guidance documents are intended to improve transparency, reproducibility, and the ability to evaluate validity and relevance of design and analysis decisions. However, existing guidance does not differentiate between structured and unstructured information contained in EHRs, registries, or other healthcare data sources. While unstructured text is convenient and readily interpretable in clinical practice, it can be difficult to use for investigation of causal questions, e.g., comparative effectiveness and safety, until data have been cleaned and algorithms applied to extract relevant information to structured fields for analysis. The goal of this paper is to increase transparency for healthcare decision makers and causal inference researchers by providing general recommendations for reporting on steps taken to make unstructured text-based data usable for comparative effectiveness and safety research. These recommendations are designed to be used as an adjunct for existing reporting guidance. They are intended to provide sufficient context and supporting information for causal inference studies involving use of natural language processing- or machine learning-derived data fields, so that researchers, reviewers, and decision makers can be confident in their ability to evaluate the validity and relevance of derived measures for exposures, inclusion/exclusion criteria, covariates, and outcomes for the causal question of interest.