Application of Multi-Criteria Decision Analysis (MCDA) to Prioritize Real-World Evidence Studies for Health Technology Management: Outcomes and Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration.

Multi-criteria decision analysis (MCDA) is a value assessment tool designed to help support complex decision-making by incorporating multiple factors and perspectives in a transparent, structured approach. We developed an MCDA rating tool, consisting of seven criteria evaluating the importance and feasibility of conducting potential real-world evidence (RWE) studies aimed at addressing uncertainties stemming from initial cancer drug funding recommendations. In collaboration with the Canadian Agency for Drugs and Technologies in Health's Provincial Advisory Group, a validation exercise was conducted to further evaluate the application of the rating tool using RWE proposals varying in complexity. Through this exercise, we aimed to gain insight into consensus building and deliberation processes and to identify efficiencies in the application of the rating tool. An experienced facilitator led a multidisciplinary committee, consisting of 11 Canadian experts, through consensus building, deliberation, and prioritization. A total of nine RWE proposals were evaluated and prioritized as low (n = 4), medium (n = 3), or high (n = 2) priority. Through an iterative process, efficiencies and recommendations to improve the rating tool and associated procedures were identified. The refined MCDA rating tool can help decision-makers prioritize important and feasible RWE studies for research and can enable the use of RWE for the life-cycle evaluation of cancer drugs.

Uncertainty tolerance among experts involved in drug reimbursement recommendations: Qualitative evidence from HTA committees in Canada and Poland.

OBJECTIVES: Drug reimbursement decisions often rely on health technology assessment (HTA). Increasingly, new drugs have limited clinical evidence and uncertain clinical benefit. Our goal was to describe how members of drug advisory committees and other stakeholders conceptualize and tolerate uncertainty and how they rationalize uncertainty tolerance. METHODS: Our triangulated parallel design applied two qualitative methods. We interviewed 31 members of drug advisory committees in Canada and Poland about their information needs and included hypothetical scenarios with uncertain clinical benefits. Respondents speculated about their likely reimbursement recommendation. We analyzed written recommendations of the pan Canadian Oncology Drug Review for drugs with uncertain benefit and compared initial recommendations to the responses from patient and clinician groups. RESULTS: Uncertainty tolerance varied among committee members and across jurisdictions. In the scenario analysis, 7 Canadian and 11 Polish respondents leaned against recommending a hypothetical drug with uncertain clinical benefit, whereas 5 Canadian and 5 Polish respondents leaned in favour. Those against rationalized that uncertainty increases potential harm; those in favour rationalized that patients often have no alternatives. The document analysis revealed that patients had higher uncertainty tolerance in general. CONCLUSIONS: Uncertainty tolerance varies among committee members and other stakeholders depending on their backgrounds and on the decision contexts. We argue that policy guidance around uncertainty management could improve the transparency and consistency of recommendations.

Ranking the Criteria Used in the Appraisal of Drugs for Reimbursement: A Stated Preferences Elicitation With Health Technology Assessment Stakeholders Across Jurisdictional Contexts.

OBJECTIVES: Our goal was to estimate the relative importance assigned to health technology assessment (HTA) criteria by stakeholders involved in the HTA process. HTA is an increasingly common framework used in the appraisal of drugs for public reimbursement. It identifies clinical, economic, social, and organizational criteria to be considered. The criteria can vary across jurisdictions and are typically appraised by multidisciplinary expert committees. Guidance on the relative weighing of criteria is often absent. METHODS: We elicited stakeholders' preferences using a single-scenario discrete choice experiment and a best-worst scaling model with conviction scores to assess the weights assigned to selected criteria by HTA stakeholders. We recruited 111 HTA stakeholders across multiple jurisdictions, including members of expert committees, clinical and economic experts, patients, and public payer representatives. Each judged twelve hypothetical cancer drug profiles for suitability for public funding and identified which characteristics were best and worst. In addition to standard discrete choice experiment and best-worst scaling models, we estimated a hybrid model to obtain a ranking of criteria by importance they played in the appraisal. RESULTS: A strong clinical benefit proved the most important criterion, followed by cost considerations, presence of adverse events, and availability of other treatments. The importance of clinical benefit was moderated by unmet need, adverse events, and number of patients. CONCLUSION: Policymakers might want to consider providing an explicit weighing scheme, or moving to a 2-stage selection process with an assessment of the quality of clinical evidence as a gatekeeping step for a full HTA review.

Drug attributes associated with the selection of drugs for reimbursement: a pilot stated preferences experiment with Canadian stakeholders.

INTRODUCTION: Health Technologies Assessment requires that evidence about clinical, economic, social, and organizational aspects be considered and weighted in the selection of drugs for reimbursement. We investigate how evidence is balanced by committee members in Canada, where neither explicit weighing schemes nor thresholds are provided. METHODS: Thirty-six past and present members of cancer drug appraisal committees participated in an online stated preferences experiment. The experiment included a ranking of drug attributes, a discrete choice experiment asking to vote in favor or against the funding of drugs described using five attributes, and a best-worst scaling experiment using the same drug descriptions. RESULTS: Respondents focused on the clinical attributes of drugs, particularly on the survival benefit relative to a comparator. As a second criterion, respondents either consider economic attributes or they consider patient relevant attributes, depending on how questions are framed. The small sample size is a limitation to generalizability. CONCLUSION: Understanding how individuals involved in HTA weigh evidence is important to the development of policy guidelines for the drug selection process. Our pilot results suggest that non-clinical criteria can become marginalized in the appraisal process in the absence of clear guidelines to their use. Avenues for further research are discussed.