Application of Multi-Criteria Decision Analysis (MCDA) to Prioritize Real-World Evidence Studies for Health Technology Management: Outcomes and Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration.

Multi-criteria decision analysis (MCDA) is a value assessment tool designed to help support complex decision-making by incorporating multiple factors and perspectives in a transparent, structured approach. We developed an MCDA rating tool, consisting of seven criteria evaluating the importance and feasibility of conducting potential real-world evidence (RWE) studies aimed at addressing uncertainties stemming from initial cancer drug funding recommendations. In collaboration with the Canadian Agency for Drugs and Technologies in Health's Provincial Advisory Group, a validation exercise was conducted to further evaluate the application of the rating tool using RWE proposals varying in complexity. Through this exercise, we aimed to gain insight into consensus building and deliberation processes and to identify efficiencies in the application of the rating tool. An experienced facilitator led a multidisciplinary committee, consisting of 11 Canadian experts, through consensus building, deliberation, and prioritization. A total of nine RWE proposals were evaluated and prioritized as low (n = 4), medium (n = 3), or high (n = 2) priority. Through an iterative process, efficiencies and recommendations to improve the rating tool and associated procedures were identified. The refined MCDA rating tool can help decision-makers prioritize important and feasible RWE studies for research and can enable the use of RWE for the life-cycle evaluation of cancer drugs.

Cost and value of cancer medicines in a single-payer public health system in Ontario, Canada: a cross-sectional study.

BACKGROUND: The financial impact of cancer medicines on health systems is not well known. We describe temporal trends in expenditure on cancer medicines within the single-payer health system of Ontario, Canada, and the extent of clinical benefit these treatments offer. METHODS: In this cross-sectional study, we identified cancer medicines and expenditures from formularies and costing databases (the New Drug Funding Program, Ontario Drug Benefit Program, and The High-Cost Therapy Funding Program) during 10 consecutive years (April 1, 2012, to March 31, 2022) in Ontario, Canada. For intravenous medicines, we applied the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS) to identify expenditures associated with substantial clinical benefit. We also identified treatments associated with improved overall survival or quality of life. FINDINGS: 69 intravenous and 98 oral or injectable medicines were funded during 2012-22. Annual expenditure on cancer medicines increased by approximately 15% per year during 2012-22; the increase was more rapid in the most recent 4 years. Total expenditure on cancer medicines in the 2021-22 financial year was CA$1·7 billion. Immune checkpoint inhibitors were the single biggest expense by class ($284 million), representing 17% of the entire cancer medicine annual budget. Drugs with the highest individual costs were lenalidomide ($178 million) and pembrolizumab ($163 million), each accounting for around 10% of the entire budget. 29 (76%) of 38 indications eligible for ESMO-MCBS scoring met the threshold for substantial clinical benefit. Eight (21%) indications had no randomised trial evidence of improved overall survival, and only four (11%) were associated with improved QOL. $346 million (67% of the expenditure on intravenous cancer medicines) was spent on drugs that improved median overall survival by more than 6 months, $82 million (16%) was spent on medicines with overall survival gains of 3-6 months, and $32 million (6%) was spent on medicines with overall survival gains of less than 3 months. $53 million (10%) was spent on medicines with no established improvement in overall survival. INTERPRETATION: Costs of cancer medicines to the Canadian health system are increasing rapidly. Most funded indications met thresholds for substantial clinical benefit and two-thirds of the expenditure were for medicines that improve survival by more than 6 months. Whether this cost trajectory can be maintained in a sustainable, equitable, high-quality health system is unclear. Efforts are needed to ensure the price of medicines with substantial benefit is affordable and funding of treatments with very modest benefit might need to be re-assessed, particularly when alternative supportive and palliative therapies are available. FUNDING: None.

Development of a Multi-Criteria Decision Analysis Rating Tool to Prioritize Real-World Evidence Questions for the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration.

The Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration developed an MCDA rating tool to assess and prioritize potential post-market real-world evidence (RWE) questions/uncertainties emerging from public drug funding decisions in Canada. In collaboration with a group of multidisciplinary stakeholders from across Canada, the rating tool was developed following a three-step process: (1) selection of criteria to assess the importance and feasibility of an RWE question; (2) development of rating scales, application of weights and calculating aggregate scores; and (3) validation testing. An initial MCDA rating tool was developed, composed of seven criteria, divided into two groups. Group A criteria assess the importance of an RWE question by examining the (1) drug's perceived clinical benefit, (2) magnitude of uncertainty identified, and (3) relevance of the uncertainty to decision-makers. Group B criteria assess the feasibility of conducting an RWE analysis including the (1) feasibility of identifying a comparator, (2) ability to identify cases, (3) availability of comprehensive data, and (4) availability of necessary expertise and methodology. Future directions include partnering with the Canadian Agency for Drugs and Technology in Health's Provincial Advisory Group for further tool refinement and to gain insight into incorporating the tool into drug funding deliberations.

Cost-effectiveness of second-line ipilimumab for metastatic melanoma: A real-world population-based cohort study of resource utilization.

BACKGROUND: The efficacy-effectiveness gap between randomized trial and real-world evidence regarding the clinical benefit of ipilimumab for metastatic melanoma (MM) has been well characterized by previous literature, consistent with initial concerns raised by health technology assessment agencies (HTAs). As these differences can significantly impact cost-effectiveness, it is critical to assess the real-world cost-effectiveness of second-line ipilimumab versus non-ipilimumab treatments for MM. METHODS: This was a population-based retrospective cohort study of patients who received second-line non-ipilimumab therapies between 2008 and 2012 versus ipilimumab treatment between 2012 and 2015 (after public reimbursement) for MM in Ontario. Using a 5-year time horizon, censor-adjusted and discounted (1.5%) costs (from the public payer's perspective in Canadian dollars) and effectiveness were used to calculate incremental cost-effectiveness ratios (ICERs) in life-years gained (LYGs) and quality-adjusted life years (QALYs), with bootstrapping to capture uncertainty. Varying the discount rate and reducing the price of ipilimumab were done as sensitivity analyses. RESULTS: In total, 329 MM were identified (Treated: 189; Controls: 140). Ipilimumab was associated with an incremental effectiveness of 0.59 LYG, incremental cost of $91,233, and ICER of $153,778/LYG. ICERs were not sensitive to discounting rate. Adjusting for quality of life using utility weights resulted in an ICER of $225,885/QALY, confirming the original HTA estimate prior to public reimbursement. Reducing the price of ipilimumab by 100% resulted in an ICER of $111,728/QALY. CONCLUSION: Despite its clinical benefit, ipilimumab as second-line monotherapy for MM patients is not cost-effective in the real world as projected by HTA under conventional willingness-to-pay thresholds.

Building a National Reassessment Process for Oncology Drugs: Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration through a Simulated Reassessment Exercise.

The CanREValue Collaboration established the Reassessment & Uptake Working Group to develop a preliminary process to reassess funded cancer drugs in Canada. A simulated exercise was conducted to evaluate the proposed reassessment process using a real-world case. We invited 32 attendees including representatives from Health Canada and Health Technology Assessment (HTA) agencies, along with payers, clinicians, academics, and patient representatives. A case was developed using a real-world study on a publicly funded cancer drug. In facilitated group sessions, participants were asked to deliberate upon the evidence presented in the case to issue reassessment recommendations. Several themes were identified through the deliberation discussions. While the generalizability of real-world evidence (RWE) is perceived as a strength, trust in the RWE depends largely on the source of the real-world data. The attendees suggested several improvements to the proposed reassessment process including evidence requirement for reassessment, recommendation categories, and a priori study protocols. This exercise generated important insights on the evidence required for conducting reassessment and considerations for improvements of the proposed reassessment process. Building upon lessons from this exercise, future work would continue to refine the reassessment process as part of the overall CanREValue framework.

Considerations for Developing a Reassessment Process: Report from the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration's Reassessment and Uptake Working Group.

The Canadian Real-world Evidence for Value in Cancer Drugs (CanREValue) Collaboration was established to develop a framework for generating and using real-world evidence (RWE) to inform the reassessment of cancer drugs following initial health technology assessment (HTA). The Reassessment and Uptake Working Group (RWG) is one of the five established CanREValue Working Groups. The RWG aims to develop considerations for incorporating RWE for HTA reassessment and strategies for using RWE to reassess drug funding decisions. Between February 2018 and December 2019, the RWG attended four teleconferences (with follow-up surveys) and two in-person meetings to discuss recommendations for the development of a reassessment process and potential barriers and facilitators. Modified Delphi methods were used to gather input. A draft report of recommendations (to December 2018) was shared for public consultation (December 2019 to January 2020). Initial considerations for developing a reassessment process were proposed. Specifically, reassessment can be initiated by diverse stakeholders, including decision makers from public drug plans or industry stakeholders. The reassessment process should be modelled after existing deliberation and recommendation frameworks used by HTA agencies. Proposed reassessment outcome categories include maintaining status quo, revisiting funding criteria, renegotiating price, or disinvesting. Overall, these initial considerations will serve as the basis for future advancements by the Collaboration.

Clinician Perspectives of COVID-19-Related Cancer Drug Funding Measures in Ontario.

The COVID-19 pandemic has a significant impact on cancer patients and the delivery of cancer care. To allow clinicians to adapt treatment plans for patients, Ontario Health (Cancer Care Ontario) issued a series of interim funding measures for the province's New Drug Funding Program (NDFP), which covers the cost of most hospital-delivered cancer drugs. To assess the utility of the measures and the need for their continuation, we conducted an online survey of Ontario oncology clinicians. The survey was open 3-25 September 2020 and generated 105 responses. Between April and June 2020, 46% of respondents changed treatment plans for more than 25% of their cancer patients due to the pandemic. Clinicians report broad use of interim funding measures. The most frequently reported strategies used were treatment breaks for stable patients (62%), extending dosing intervals (59%), and deferring routine imaging (56%). Most clinicians anticipate continuing to use these interim funding measures in the coming months. The survey showed that adapting cancer drug funding policies has supported clinical care in Ontario during the pandemic.

Use of real-world evidence in cancer drug funding decisions in Canada: a qualitative study of stakeholders' perspectives.

BACKGROUND: Real-world evidence (RWE) can provide postmarket data to inform whether funded cancer drugs yield expected outcomes and value for money, but it is unclear how to incorporate RWE into Canadian cancer drug funding decisions. As part of the Canadian Real-World Evidence Value for Cancer Drugs (CanREValue) Collaboration, this study aimed to explore stakeholder perspectives on the current state of RWE in Canada to inform a Canadian framework for use of RWE in cancer drug funding decisions. METHODS: This was a qualitative descriptive study. Qualitative semistructured interviews were conducted from April to July 2018. Participants were Canadian and international stakeholders who had experience with RWE and drug funding decision-making. Thematic analysis was used to analyze data. RESULTS: Thirty stakeholders participated in the study. Five themes were identified. Stakeholders indicated that RWE had value in cancer drug funding decisions. However, a cultural shift is needed to adopt RWE in decision-making. Further, the Canadian infrastructure for real-world data is currently inadequate for decision-making, and there is a need for committed investment in building capacity to collect and analyze RWE. Finally, there is a need for increased collaboration among key stakeholders. INTERPRETATION: The findings of this study suggest that if RWE is to be used in drug funding decisions, there is a need for a cultural shift, improved data infrastructure, committed investment in capacity building and increased stakeholder collaboration. Together with local stakeholder engagement, application of these findings may contribute to optimizing implementation of RWE.

Are We Making a Difference? A Qualitative Study of Patient Engagement at the pan-Canadian Oncology Drug Review: Perspectives of Patient Groups.

OBJECTIVES: Despite wide support for patient involvement in health technology assessments (HTA), determining meaningful engagement is complex. This article explores experiences and perceptions among patient groups participating in the Canadian Agency for Drugs and Technologies in Health (CADTH)'s pan-Canadian Oncology Drug Review (pCODR) process. METHODS: We created a qualitative interview study comprising 22 semi-structured telephone interviews with individuals representing 21 different patient groups registered with the pCODR process. The analysis used a qualitative descriptive approach employing techniques from grounded theory. RESULTS: Patient groups view the ability to make submissions to the pCODR process as a meaningful activity closely aligned with organizational priorities. Concurrently, they face substantial resource challenges to prepare submissions, including high opportunity costs and difficulty accessing needed literature and finding relevant patients. Although patient groups felt that CADTH is committed to transparency, they expressed considerable uncertainty around the direct impact of their submissions and desired additional avenues for engagement. CONCLUSIONS: This study suggests a strong commitment by patient groups to participate in the pCODR process despite uncertainty about how their submissions are used to inform HTA recommendations. Identifying opportunities to provide both financial and nonfinancial resources to patient groups is crucial to encouraging and supporting their meaningful participation in HTA processes.

Impact of rarity on Canadian oncology health technology assessment and funding.

OBJECTIVES: The pan-Canadian Oncology Drug Review (pCODR) evaluates new cancer drugs for public funding recommendations. While pCODR's deliberative framework evaluates overall clinical benefit and includes considerations for exceptional circumstances, rarity of indication is not explicitly addressed. Given the high unmet need that typically accompanies these indications, we explored the impact of rarity on oncology HTA recommendations and funding decisions. METHODS: We examined pCODR submissions with final recommendations from 2012 to 2017. Incidence rates were calculated using pCODR recommendation reports and statistics from the Canadian Cancer Society. Indications were classified as rare if the incidence rate was lower than 1/100,000 diagnoses, a definition referenced by the Canadian Agency for Drugs and Technologies in Health. Each pCODR final report was examined for the funding recommendation/justification, level of supporting evidence (presence of a randomized control trial [RCT]), and time to funding (if applicable). RESULTS: Of the ninety-six pCODR reviews examined, 16.6 percent were classified as rare indications per above criteria. While the frequency of positive funding recommendations were similar between rare and nonrare indication (78.6 vs. 75 percent), rare indications were less likely to be presented with evidence from RCT (50 vs. 90 percent). The average time to funding did not differ significantly across provinces. CONCLUSION: Rare indications appear to be associated with weaker clinical evidence. There appears to be no association between rarity, positive funding recommendations, and time to funding. Further work will evaluate factors associated with positive recommendations and the real-world utilization of funded treatments for rare indications.

Developing a framework to incorporate real-world evidence in cancer drug funding decisions: the Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration.

BACKGROUND: Oncology therapy is becoming increasingly more expensive and challenging the affordability and sustainability of drug programmes around the world. When new drugs are evaluated, health technology assessment organisations rely on clinical trials to inform funding decisions. However, clinical trials are not able to assess overall survival and generalises evidence in a real-world setting. As a result, policy makers have little information on whether drug funding decisions based on clinical trials ultimately yield the outcomes and value for money that might be expected. OBJECTIVE: The Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration, consisting of researchers, recommendation-makers, decision makers, payers, patients and caregivers, are developing and testing a framework for Canadian provinces to generate and use real-world evidence (RWE) for cancer drug funding in a consistent and integrated manner. STRATEGY: The CanREValue collaboration has established five formal working groups (WGs) to focus on specific processes in the generation and use of RWE for cancer drug funding decisions in Canada. The different RWE WGs are: (1) Planning and Drug Selection; (2) Methods; (3) Data; (4) Reassessment and Uptake; (5) Engagement. These WGs are acting collaboratively to develop a framework for RWE evaluation, validate the framework through the multiprovince RWE projects and help to integrate the final RWE framework into the Canadian healthcare system. OUTCOMES: The framework will enable the reassessment of cancer drugs, refinement of funding recommendations and use of novel funding mechanisms by decision-makers/payers across Canada to ensure the healthcare system is providing clinical benefits and value for money.