Suitability of the Current Health Technology Assessment of Innovative Artificial Intelligence-Based Medical Devices: Scoping Literature Review.

BACKGROUND: Artificial intelligence (AI)-based medical devices have garnered attention due to their ability to revolutionize medicine. Their health technology assessment framework is lacking. OBJECTIVE: This study aims to analyze the suitability of each health technology assessment (HTA) domain for the assessment of AI-based medical devices. METHODS: We conducted a scoping literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We searched databases (PubMed, Embase, and Cochrane Library), gray literature, and HTA agency websites. RESULTS: A total of 10.1% (78/775) of the references were included. Data quality and integration are vital aspects to consider when describing and assessing the technical characteristics of AI-based medical devices during an HTA process. When it comes to implementing specialized HTA for AI-based medical devices, several practical challenges and potential barriers could be highlighted and should be taken into account (AI technological evolution timeline, data requirements, complexity and transparency, clinical validation and safety requirements, regulatory and ethical considerations, and economic evaluation). CONCLUSIONS: The adaptation of the HTA process through a methodological framework for AI-based medical devices enhances the comparability of results across different evaluations and jurisdictions. By defining the necessary expertise, the framework supports the development of a skilled workforce capable of conducting robust and reliable HTAs of AI-based medical devices. A comprehensive adapted HTA framework for AI-based medical devices can provide valuable insights into the effectiveness, cost-effectiveness, and societal impact of AI-based medical devices, guiding their responsible implementation and maximizing their benefits for patients and health care systems.

Challenges and Opportunities in Interdisciplinary Research and Real-World Data for Treatment Sequences in Health Technology Assessments.

With an ever-increasing number of treatment options, the assessment of treatment sequences has become crucial in health technology assessment (HTA). This review systematically explores the multifaceted challenges inherent in evaluating sequences, delving into their interplay and nuances that go beyond economic model structures. We synthesised a 'roadmap' of literature from key methodological studies, highlighting the evolution of recent advances and emerging research themes. These insights were compared against HTA guidelines to identify potential avenues for future research. Our findings reveal a spectrum of challenges in sequence evaluation, encompassing selecting appropriate decision-analytic modelling approaches and comparators, deriving appropriate clinical effectiveness evidence in the face of data scarcity, scrutinising effectiveness assumptions and statistical adjustments, considering treatment displacement, and optimising model computations. Integrating methodologies from diverse disciplines-statistics, epidemiology, causal inference, operational research and computer science-has demonstrated promise in addressing these challenges. An updated review of application studies is warranted to provide detailed insights into the extent and manner in which these methodologies have been implemented. Data scarcity on the effectiveness of treatment sequences emerged as a dominant concern, especially because treatment sequences are rarely compared in clinical trials. Real-world data (RWD) provide an alternative means for capturing evidence on effectiveness and future research should prioritise harnessing causal inference methods, particularly Target Trial Emulation, to evaluate treatment sequence effectiveness using RWD. This approach is also adaptable for analysing trials harbouring sequencing information and adjusting indirect comparisons when collating evidence from heterogeneous sources. Such investigative efforts could lend support to reviews of HTA recommendations and contribute to synthesising external control arms involving treatment sequences.

The impact of willingness-to-pay threshold on price reduction recommendations for oncology drugs: a review of assessments conducted by the Canadian Agency for Drugs and Technologies in Health.

Since late 2020, the Canadian Agency of Drugs and Technologies in Health (CADTH) has been using a threshold of $50,000 (CAD) per quality-adjusted life-year (QALY) for both oncology and non-oncology drugs. When used for oncology products, this threshold is hypothesized to have a higher impact on the time to access these drugs in Canada. We studied the impact of price reductions on time to engagement and negotiation with the pan-Canadian Pharmaceutical Alliance for oncology drugs reviewed by CADTH between January 2020 and December 2022. Overall, 103 assessments reported data on price reductions recommended by CADTH to meet the cost-effectiveness threshold for reimbursement. Of these assessments, 57% (59/103) recommendations included a price reduction of greater than 70% off the list price. Eight percent (8/103) were not cost-effective even at a 100% price reduction. Of the 47 assessments that had a clear benefit, in 21 (45%) CADTH recommended a price reduction of at least 70%. The median time to price negotiation (not including time to engagement) for assessments that received at least 70% vs >70% price reduction was 2.6 vs 4.8 months. This study showed that there is a divergence between drug sponsor's incremental cost-effectiveness ratio (ICER) and CADTH revised ICER leading to a price reduction to meet the $50,000/QALY threshold. For the submissions with clear clinical benefit the median length of engagement (2.5 vs 3.3 months) and median length of negotiation (3.1 vs 3.6 months) were slightly shorter compared with the submissions where uncertainties were noted in the clinical benefit according to CADTH. This study shows that using a $50,000 per QALY threshold for oncology products potentially impacts timely access to life saving medications.

Application of Multi-Criteria Decision Analysis (MCDA) to Prioritize Real-World Evidence Studies for Health Technology Management: Outcomes and Lessons Learned by the Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration.

Multi-criteria decision analysis (MCDA) is a value assessment tool designed to help support complex decision-making by incorporating multiple factors and perspectives in a transparent, structured approach. We developed an MCDA rating tool, consisting of seven criteria evaluating the importance and feasibility of conducting potential real-world evidence (RWE) studies aimed at addressing uncertainties stemming from initial cancer drug funding recommendations. In collaboration with the Canadian Agency for Drugs and Technologies in Health's Provincial Advisory Group, a validation exercise was conducted to further evaluate the application of the rating tool using RWE proposals varying in complexity. Through this exercise, we aimed to gain insight into consensus building and deliberation processes and to identify efficiencies in the application of the rating tool. An experienced facilitator led a multidisciplinary committee, consisting of 11 Canadian experts, through consensus building, deliberation, and prioritization. A total of nine RWE proposals were evaluated and prioritized as low (n = 4), medium (n = 3), or high (n = 2) priority. Through an iterative process, efficiencies and recommendations to improve the rating tool and associated procedures were identified. The refined MCDA rating tool can help decision-makers prioritize important and feasible RWE studies for research and can enable the use of RWE for the life-cycle evaluation of cancer drugs.

R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 15.

In this latest update we discuss real-world evidence (RWE) guidance from the leading oncology professional societies, the American Society of Clinical Oncology and the European Society for Medical Oncology, and the PRINCIPLED practical guide on the design and analysis of causal RWE studies.

Cross-sectional analysis of use of real-world data in single technology appraisals of oncological medicine by the National Institute for Health and Care Excellence in 2011-2021.

OBJECTIVES: This study aims to identify how real-world data (RWD) have been used in single technology appraisals (STAs) of cancer drugs by the National Institute for Health and Care Excellence (NICE). DESIGN: Cross-sectional study of NICE technology appraisals of cancer drugs for which guidance was issued between January 2011 and December 2021 (n=229). The appraisals were reviewed following a published protocol to extract the data about the use of RWD. The use of RWD was analysed by reviewing the specific ways in which RWD were used and by identifying different patterns of use. PRIMARY OUTCOME MEASURE: The number of appraisals where RWD are used in the economic modelling. RESULTS: Most appraisals used RWD in their economic models. The parametric use of RWD was commonly made in the economic models (76% of the included appraisals), whereas non-parametric use was less common (41%). Despite widespread use of RWD, there was no dominant pattern of use. Three sources of RWD (registries, administrative data, chart reviews) were found across the three important parts of the economic model (choice of comparators, overall survival and volume of treatment). CONCLUSIONS: NICE has had a long-standing interest in the use of RWD in STAs. A systematic review of oncology appraisals suggests that RWD have been widely used in diverse parts of the economic models. Between 2011 and 2021, parametric use was more commonly found in economic models than non-parametric use. Nonetheless, there was no clear pattern in the way these data were used. As each appraisal involves a different decision problem and the ability of RWD to provide the information required for the economic modelling varies, appraisals will continue to differ with respect to their use of RWD.

Real-world data: a comprehensive literature review on the barriers, challenges, and opportunities associated with their inclusion in the health technology assessment process.

Objective: This review aimed to assess the current use and acceptance of real-world data (RWD) and real-world evidence (RWE) in health technology assessment (HTA) process. It additionally aimed to discern stakeholders' viewpoints concerning RWD and RWE in HTA and illuminate the obstacles, difficulties, prospects, and consequences associated with the incorporation of RWD and RWE into the realm of HTA. Methods: A comprehensive PRISMA-based systematic review was performed in July 2022 in PubMed/Medline, Scopus, IDEAS-RePEc, International HTA database, and Centre for Reviews and Dissemination with ad hoc supplementary search in Google Scholar and international organization websites. The review included pre-determined inclusion criteria while the selection of eligible studies, the data extraction process and quality assessment were carried out using standardized and transparent methods. Results: Twenty-nine (n = 29) studies were included in the review out of 2,115 studies identified by the search strategy. In various global contexts, disparities in RWD utilization were evident, with randomized controlled trials (RCTs) serving as the primary evidence source. RWD and RWE played pivotal roles, surpassing relative effectiveness assessments (REAs) and significantly influencing decision-making and cost-effectiveness analyses. Identified challenges impeding RWD integration into HTA encompassed limited local data access, complexities in non-randomized trial design, data quality, privacy, and fragmentation. Addressing these is imperative for optimal RWD utilization. Incorporating RWD/RWE in HTA yields multifaceted advantages, enhancing understanding of treatment efficacy, resource utilization, and cost analysis, particularly via patient registries. RWE complements assessments of advanced therapy medicinal products (ATMPs) and rare diseases. Local data utilization strengthens HTA, bridging gaps when RCT data is lacking. RWD aids medical device decision-making, cancer drug reassessment, and indirect treatment comparisons. Challenges include data availability, stakeholder acceptance, expertise, and privacy. However, standardization, training, collaboration, and guidance can surmount these barriers, fostering enhanced RWD utilization in HTA. Conclusion: This study highlights the intricate global landscape of RWD and RWE acceptance in HTA. Recognizing regional nuances, addressing methodological challenges, and promoting collaboration are pivotal, among others, for leveraging RWD and RWE effectively in healthcare decision-making.

A literature review of quality assessment and applicability to HTA of risk prediction models of coronary heart disease in patients with diabetes.

This literature review had two objectives: to identify models for predicting the risk of coronary heart diseases in patients with diabetes (DM); and to assess model quality in terms of risk of bias (RoB) and applicability for the purpose of health technology assessment (HTA). We undertook a targeted review of journal articles published in English, Dutch, Chinese, or Spanish in 5 databases from 1st January 2016 to 18th December 2022, and searched three systematic reviews for the models published after 2012. We used PROBAST (Prediction model Risk Of Bias Assessment Tool) to assess RoB, and used findings from Betts et al. 2019, which summarized recommendations and criticisms of HTA agencies on cardiovascular risk prediction models, to assess model applicability for the purpose of HTA. As a result, 71 % and 67 % models reporting C-index showed good discrimination abilities (C-index >= 0.7). Of the 26 model studies and 30 models identified, only one model study showed low RoB in all domains, and no model was fully applicable for HTA. Since the major cause of high RoB is inappropriate use of analysis method, we advise clinicians to carefully examine the model performance declared by model developers, and to trust a model if all PROBAST domains except analysis show low RoB and at least one validation study conducted in the same setting (e.g. country) is available. Moreover, since general model applicability is not informative for HTA, novel adapted tools may need to be developed.

Minimally Invasive Bleb Surgery for Glaucoma: A Health Technology Assessment.

Background: Glaucoma is the term for a group of eye disorders that causes progressive damage to the optic nerve, which can lead to visual impairment and, potentially, irreversible blindness. Minimally invasive bleb surgery (MIBS) reduces eye pressure through the implantation of a device that creates a new subconjunctival outflow pathway for eye fluid drainage. MIBS is a less invasive alternative to conventional/incisional glaucoma surgery (e.g., trabeculectomy). We conducted a health technology assessment of MIBS for people with glaucoma, which included an evaluation of effectiveness, safety, the budget impact of publicly funding MIBS, and patient preferences and values. Methods: We performed a systematic literature search of the clinical evidence. We assessed the risk of bias of each included study using the Cochrane Risk of Bias 1.0 tool for randomized controlled trials (RCTs) and the Risk of Bias Assessment tool for Nonrandomized Studies (RoBANS) for comparative observational studies, and the quality of the body of evidence according to the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group criteria. We conducted an economic literature search and we estimated the budget impact of publicly funding MIBS in Ontario. We did not conduct a primary economic evaluation due to the limited long-term effectiveness data. We summarized the preferences and values evidence from previous health technology assessments to understand the perspectives and experiences of patients with glaucoma. Results: We included 41 studies (2 RCTs and 39 comparative observational studies) in the clinical evidence review. MIBS may reduce intraocular pressure and the number of medications used, but we are uncertain if MIBS results in outcomes similar to trabeculectomy (GRADE: Moderate to Very low). Compared with trabeculectomy, MIBS may result in fewer follow-up visits and interventions, and adverse events (GRADE: Moderate to Very Low). MIBS may also reduce intraocular pressure and the number of antiglaucoma medications used, compared with other glaucoma treatments, but the evidence is uncertain (GRADE: Very low). Our economic evidence review identified two directly applicable studies. The results of these studies indicate that the cost-effectiveness of MIBS is highly uncertain, and the cost of glaucoma interventions are likely to vary across provinces. The annual budget impact of publicly funding MIBS in Ontario ranged from $0.11 million in year 1 to $0.67 million in year 5, for a total 5-year budget impact estimate of $1.93 million. Preferences and values evidence suggests that fear of ultimate blindness and difficulty managing medication for glaucoma led patients to explore other treatment options such as MIBS. Glaucoma patients found minimally invasive glaucoma surgery (MIGS) procedure beneficial, with minimal side effects and recovery time. Conclusions: Minimally invasive bleb surgery reduces intraocular eye pressure and the number of antiglaucoma medications needed, but we are uncertain if the outcomes are similar to trabeculectomy (GRADE: Moderate to Very low). However, MIBS may be safer than trabeculectomy (GRADE: Moderate to Very low) and result in fewer follow-ups (GRADE: Moderate to Very low). MIBS may also improve glaucoma symptoms compared with other glaucoma treatments, but the evidence is very uncertain (GRADE: Very low).We estimate that publicly funding MIBS would result in an additional cost of $1.93 million over 5 years. Patients who underwent MIGS procedures found them to be generally successful and beneficial, with minimal side effects and recovery time. We could not draw conclusions about specific MIBS procedures or long-term outcomes.

A Step Toward the Development of the First National Multi-Criteria Decision Analysis Framework to Support Healthcare Decision Making in Saudi Arabia.

OBJECTIVES: To assess the feasibility of implementing multi-criteria decision analysis (MCDA) and to select the criteria for preparing a national MCDA framework for health technology assessment of orphan drugs in the Kingdom of Saudi Arabia (KSA). METHODS: The study was conducted in 3 phases. In phase I, a targeted literature review was performed to gather relevant information on the implementation of MCDA in healthcare decision making. Phase II was a cross-sectional survey, conducted to obtain insights from different stakeholders and key opinion leaders on specific topics from the KSA perspective. Phase III included a round-table discussion involving experts to validate the results obtained in the phase II survey and further elaborate on specific requirements that may be critical for developing the first national MCDA framework in the KSA. RESULTS: All the key opinion leaders involved in the study acknowledged the importance of implementing MCDA in the KSA. The Ministry of Health was assigned the responsibility of chairing the MCDA decision process. The experts selected the quantitative, qualitative, and economic criteria to be considered for the MCDA framework. The stakeholders decided to initiate a pilot phase using the deliberative MCDA methodology for the assessment of orphan drugs based on the selected criteria for a period of 1 year and then reevaluate the need to adapt the pragmatic MCDA model. CONCLUSION: This article describes the novel initiative that examined the feasibility and process required for the development of the first MCDA framework in the KSA to support healthcare decision making.

Intrathecal Drug Delivery Systems for Cancer Pain: A Health Technology Assessment.

Background: Pain is a common and very distressing symptom for adults and children with cancer. Compared with other routes of delivery, infusing pain medication directly into the intrathecal space around the spinal cord may reduce the incidence of systemic side effects and allow for more rapid and effective pain relief. We conducted a health technology assessment of intrathecal drug delivery systems (IDDSs) for adults and children with cancer pain, which included an evaluation of effectiveness, safety, cost-effectiveness, the budget impact of publicly funding IDDSs, patient preferences and values, and ethical considerations. Methods: We performed a systematic literature search of the clinical evidence to retrieve systematic reviews, and we selected and reported results from 2 recent reviews that were relevant to our research questions. We complemented the chosen systematic reviews with a literature search to identify primary studies published after December 2020. We used the Risk of Bias in Systematic Reviews (ROBIS) tool to assess the risk of bias of each included systematic review. We assessed the quality of the body of evidence according to the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Working Group criteria. We performed a systematic economic literature search and conducted a cost-effectiveness analysis comparing IDDSs with standard care (i.e., non-IDDS methods of pain management) from a public payer perspective. We also analyzed the budget impact of publicly funding IDDSs in Ontario. To contextualize the potential value of IDDSs, we spoke with patients with cancer pain and with caregivers of patients with cancer pain. We explored ethical considerations from a review of published literature on the use of IDDSs for the management of cancer pain in adults and children as well as a review of the other components of this health technology assessment to identify ethical considerations relevant to the Ontario context. Results: We included 2 systematic reviews (1 on adults and 1 on children) in the clinical evidence review. In adults with cancer pain who have a life expectancy greater than 6 months, intrathecal drug delivery was associated with a significant reduction in pain intensity compared with before implantation up to a 1-year follow-up (GRADE: Moderate to Low). Improved pain management appeared to be maintained beyond a 4-week follow-up. IDDSs likely decrease the use of systemic opioids (GRADE: Moderate to Low). They may also improve health-related quality of life (GRADE: Low), functional outcomes (GRADE: Low), and survival (GRADE: Low to Very low). In children with cancer pain, IDDSs may reduce pain intensity, improve functional outcomes, and improve survival, but the evidence is very uncertain (all GRADEs: Very low). IDDS implantation carries certain rare risks related to mechanical errors, drug-related side effects, and surgical complications. There are inherent limitations in conducting research in patients with refractory cancer pain; therefore, it is unlikely that higher-quality evidence will emerge in the next few years. Our primary economic evaluation found that IDDSs are more effective and more costly than standard care. The incremental cost-effectiveness ratio of IDDSs compared with standard care is $57,314 per quality-adjusted life-year (QALY) gained. The probability of IDDSs being cost-effective versus standard care is 43.46% at a willingness-to-pay of $50,000 per QALY gained and 72.54% at a willingness-to-pay of $100,000 per QALY gained. Publicly funding IDDSs in Ontario would cost an additional $0.27 million per year, for a total of $1.34 million over the next 5 years. The patients with cancer pain and caregivers with whom we spoke described the debilitating nature of cancer pain and the difficulty of finding effective pain management options. Patients with experience of an IDDS spoke of its effectiveness and its positive impact on their quality of life and mental health. Implementing IDDSs for patients with cancer pain raises several ethical and equity considerations related to the experiences and management of cancer pain, how limitations in evidence may entail uncertainties in clinical and health system decision-making, as well as clinical, geographic, and health system access barriers. Conclusions: Intrathecal drug delivery likely reduces pain intensity and decreases the use of systemic opioids in adults with cancer pain who have a life expectancy greater than 6 months. It may also improve health-related quality of life, functional outcomes, and survival, although the evidence for survival is very uncertain. The clinical evidence in children with cancer pain is very uncertain. IDDS implantation is reasonably safe. Intrathecal drug delivery is more effective and more costly than standard care. We estimate that funding IDDSs in Ontario will result in additional costs of $0.27 million per year, for a total of $1.34 million over the next 5 years. Considerations related to funding and implementing IDDSs for patients with cancer pain in Ontario will require explicit and focused attention to considerations of equity and access in the diagnosis and management of cancer pain and in the use, clinical uptake, and delivery of IDDS pain management.

Mapping of health technology assessment in China: a comparative study between 2016 and 2021.

BACKGROUND: Health Technology Assessment (HTA) in China has recently expanded from purely academic research to include policy or decision-oriented practice, especially after HTA evidence was used to update the National Reimbursement Drug List for the first time in 2017. This study aims to identify the progress and challenges of HTA development from 2016 to 2021 and inform policies and decisions to promote further HTA development in China. METHODS: We conducted a cross-sectional web-based survey with policy makers, researchers and industry-providers in China in 2016 and 2021 respectively. The 'Mapping of HTA Instrument', was utilized to assess the HTA development across eight domains: Institutionalization, Identification, Priority setting, Assessment, Appraisal, Reporting, Dissemination of findings and conclusions, and Implementation in policy and practice. To reduce the influence of confounders and compare the mapping outcomes between the 2016 and 2021 groups, we conducted 1:1 Propensity Score Matching (PSM). Univariate analysis was conducted to compare the differences between the two groups. The overall results were further compared with those of a mapping study that included ten countries. RESULTS: In total, 212 and 255 respondents completed the survey in 2016 and 2021, respectively. The total score of the HTA development level in China in 2021 was higher than that in 2016 before PSM (89.38 versus 83.96). Following PSM, 183 respondents from the 2016 and 2021 groups were matched. Overall, the mean scores for most indicators in the Institutionalization domain and Dissemination domain in 2021 were higher than those in 2016 (P < 0.05). The Appraisal domain in 2021 was more explicit, transparent and replicable than that in 2016 (t = -3.279, P < 0.05). However, the mean scores of most indicators in the Assessment domain were higher in 2016 than those in 2021 (P < 0.05). CONCLUSIONS: Our study suggest that the level of HTA development in China progressed significantly from 2016 to 2021. However, before engaging in HTA activities, further efforts are required to enhance the assessment process. For instance, it is important to establish a clear goal and scope for HTA; adapt standardized methodologies for evaluating the performance of systematic reviews or meta-analyses; and provide comprehensive descriptions of the safety, clinical effectiveness, cost, and cost-effectiveness of the assessed technologies, thus improving the development of HTA in China.

Adapting Health Technology Assessment agency standards for surrogate outcomes in early stage cancer trials: what needs to happen?

INTRODUCTION: An avalanche of early stage cancer clinical trials is coming. The majority of these solely use surrogate outcomes that have not been validated against a target outcome of interest (e.g. overall survival). Current HTA guidance on surrogate outcome validation are not methodologically or practically conducive to this scenario. AREAS COVERED: We provide a high-level overview of methods, approaches, and conceptual thinking for making better use of limited evidence within early stage cancer HTA submissions. We outline regulatory and HTA issues and emphasize how evidence transitions from one to another, what major gaps currently exist, and how these may be bridged. We summarize current methodologies and practices, their pros and cons. We outline how complementary measurements strengthen evaluations and address fallacies and biases of conventional statistical methods for surrogate outcomes validation. The value of real-world data to support some of the necessary validity components is discussed. Lastly, we address the importance of the patient voice for better understanding which surrogate outcomes may appropriately inform HTA. EXPERT OPINION: Conventional surrogate outcome validation represents a fraught and sub-optimal framework for HTA purposes, particularly for early stage cancer. Tools for optimizing use of limited evidence exist. Education of stakeholders is highly needed.

Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases.

OBJECTIVE: This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. METHODS: We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. RESULTS: The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. CONCLUSION: Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Differences in evidentiary requirements for oncology drug effectiveness assessments among six European health technology assessment bodies - can alignment be improved?

OBJECTIVES: Evidentiary requirements for relative effectiveness assessment vary among European health technology assessment (HTA) bodies, affecting the time to HTA decision-making and potentially delaying time to patient access. Improved alignment may reduce this time; therefore, we aim to analyze the differences in evidentiary requirements for oncology drug assessments among European HTA bodies and provide recommendations toward an increased alignment. METHODS: Interviews were conducted with stakeholders in drug assessments of Italy, the Netherlands, Poland, Portugal, England and Wales, and Sweden about evidentiary requirements for several subdomains to identify differences and obtain recommendations for addressing differences. The interview results were analyzed on degrees of evidence acceptability per HTA body and alignment on evidentiary requirements among HTA bodies. RESULTS: Subdomains demonstrating noteworthy differences concerned the acceptability of extrapolation to other populations, class effects, progression-free survival and (other) surrogate endpoints as outcomes, the absence of quality-of-life data, single-arm trials, cross-over trial designs, short trial duration, and the clinical relevance of effect size. CONCLUSION: Alignment can be enhanced to reduce time to decision-making and to improve equity in patient access. Proposed recommendations to achieve this included joint early dialogues, intensified collaboration and exchange between countries, joint relative effectiveness assessments, and the use of access agreements.

Opportunities and Challenges for Decentralized Clinical Trial Approaches: European Health Technology Assessment Perspective.

OBJECTIVES: Decentralized clinical trial (DCT) approaches are clinical trials in which some or all trial activities take place closer to participants' proximities instead of a traditional investigative site. Data from DCTs may be used for clinical and economic evaluations by health technology assessment (HTA) bodies to support reimbursement decision making. This study aimed to explore the opportunities and challenges for DCT approaches from an HTA perspective by interviewing representatives from European HTA bodies. METHODS: We conducted semistructured interviews with 25 European HTA representatives between September 2022 and February 2023, and transcripts were analyzed after thematic analysis. RESULTS: Two main themes were identified from the data relating to (1) DCT approaches in HTA and (2) trial-level acceptance and relevance. Experience with assessing DCTs was limited and a variety of knowledge about DCTs was observed. The respondents recognized the opportunity of DCTs to reduce recall bias when participant-reported outcome data can be collected more frequently and conveniently from home. Concerns were expressed about the data quality when participants become responsible for data collection. Despite this challenge, the respondents recognized the potential of DCTs to increase the generalizability of results because data can be collected in a setting reflective of the everyday situation potentially from a more diverse participant group. CONCLUSIONS: DCTs could generate relevant results for HTA decision making when data are collected in a real-world setting from a diverse participant group. Increased awareness of the opportunities and challenges could help HTA assessors in their appraisal of DCT approaches.

Making recommendations to subsidize new health technologies in Australia: A qualitative study of decision-makers' perspectives on committee processes.

OBJECTIVES: To explore experiences of, and perspectives on, health technology assessment (HTA) processes used to produce recommendations about subsidizing new medicines, and medical technologies in Australia, from the perspectives of those experienced in these processes. METHODS: Semi-structured interviews with a diverse group of 18 informants currently or previously members of the Pharmaceutical Benefits Advisory Committee (PBAC) or the Medical Services Advisory Committee (MSAC). Participants were interviewed September 2021-February 2022. Transcripts were analyzed using reflexive thematic analysis. RESULTS: 3 major themes were identified: contrasting technical and decision-making stages, resisting reductionist approaches, and navigating decision-making trade-offs. Participants discussed the complexities of the evaluative HTA process, especially when considering uncertainty in the evidence. As part of the current process, a deliberative decision-making stage was considered essential, allowing a flexible approach to decision making to consider factors beyond strength and quality of quantifiable data in the technical evaluation. Participants acknowledged these less-quantifiable factors were sometimes considered implicitly or were difficult to describe and this, paired with commercial in confidence requirements, presented challenges with respect to the desire to increase transparency. CONCLUSION (S): As HTA processes for new medicines and medical technologies in Australia continue to be reviewed, the balance between retaining flexibility during deliberation, confidentiality for sponsors and the public's desire for greater transparency may be a fruitful area for continuing research.

On the normativity of evidence - Lessons from philosophy of science and the "VALIDATE" project.

"Evidence" is a key term in medicine and health services research, including Health Technology Assessment (HTA). Randomized clinical trials (RCTs) have undoubtedly dominated the scene of generating evidence for a long period of time, becoming the hallmark of evidence-based medicine (EBM). However, due to a number of misunderstandings, the lay audience and some researchers have sometimes placed too much trust in RCTs compared to other methods of investigation. One of the principal misunderstandings is to consider RCTs findings as isolated and self-apparent pieces of information. In other words, what has been essentially lacking was the awareness of the value-context of the evidence and, in particular, the value- and theory-ladenness (normativity) of scientific knowledge. This paper aims to emphasize the normativity that exists in the production of scientific knowledge, and in particular in the conduct of RCTs as well as in the performance of HTA. The work is based on some lessons learned from Philosophy of Science and the European project "VALIDATE" (VALues In Doing Assessments of healthcare TEchnologies"). VALIDATE was a three-year EU Erasmus+ strategic partnerships project (2018-2021), in which training in the field of HTA was further optimized by using insights from political science and ethics (in accordance with the recent definition of HTA). Our analysis may reveal useful insights for addressing some challenges that HTA is going to face in the future.

Advancing hospital-based health technology assessment: evaluating genomic panel contracting strategies for blood tumors through a multimethodology.

INTRODUCTION: The adoption of genomic technologies in the context of hospital-based health technology assessment presents multiple practical and organizational challenges. OBJECTIVE: This study aimed to assist the Instituto Português de Oncologia de Lisboa Francisco Gentil (IPO Lisboa) decision makers in analyzing which acute myeloid leukemia (AML) genomic panel contracting strategies had the highest value-for-money. METHODS: A tailored, three-step approach was developed, which included: mapping clinical pathways of AML patients, building a multicriteria value model using the MACBETH approach to evaluate each genomic testing contracting strategy, and estimating the cost of each strategy through Monte Carlo simulation modeling. The value-for-money of three contracting strategies - "Standard of care (S1)," "FoundationOne Heme test (S2)," and "New diagnostic test infrastructure (S3)" - was then analyzed through strategy landscape and value-for-money graphs. RESULTS: Implementing a larger gene panel (S2) and investing in a new diagnostic test infrastructure (S3) were shown to generate extra value, but also to entail extra costs in comparison with the standard of care, with the extra value being explained by making available additional genetic information that enables more personalized treatment and patient monitoring (S2 and S3), access to a broader range of clinical trials (S2), and more complete databases to potentiate research (S3). CONCLUSION: The proposed multimethodology provided IPO Lisboa decision makers with comprehensive and insightful information regarding each strategy's value-for-money, enabling an informed discussion on whether to move from the current Strategy S1 to other competing strategies.

Opportunities to improve the adoption of health-related quality of life evidence as part of the French Health Technology Assessment process.

OBJECTIVES: Patient's health-related quality of life (HRQoL) is an important outcome measure that is considered by many payers and health technology assessment (HTA) bodies in the evaluation of treatments. We aimed to identify opportunities for HRQoL to be further incorporated into the assessment of the French HTA by comparing three health systems. We put forward recommendations that could bring further innovations to French patients. METHODS: We reviewed methodologies by the French, German and British HTA, and conducted a systematic review of all French (n = 312) and German (n = 175) HTA appraisals from 01 January 2019 to 31 December 2021. We also setup an advisory board of 11 ex-HTA leaders, payers, methodologists, healthcare providers and patient advocates, from France, Britain and Germany, to discuss opportunities to improve acceptance and adoption of HRQoL evidence in France. RESULTS: Our systematic review of HTA appraisals showed a higher HRQoL data rejection rate in France: in > 75% of cases the HRQoL evidence submitted was not accepted for the assessment (usually for methodological reasons, for example, data being considered exploratory; 16-75% of the appraisals mentioned HRQoL evidence, varying by therapeutic area). Overall, we found the French HTA to be more restrictive in its approach than IQWiG. CONCLUSIONS: Based on these findings we articulate collaborative proposals for industry and the HAS to improve acceptance of HRQoL evidence and create a positive feedback loop between HAS and industry along four dimensions (1) patient perception, (2) testing hierarchy, (3) trial design and (4) data collection.