Consolidated Health Economic Evaluation Reporting Standards for Interventions That Use Artificial Intelligence (CHEERS-AI).

OBJECTIVES: Economic evaluations (EEs) are commonly used by decision makers to understand the value of health interventions. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS 2022) provide reporting guidelines for EEs. Healthcare systems will increasingly see new interventions that use artificial intelligence (AI) to perform their function. We developed Consolidated Health Economic Evaluation Reporting Standards for Interventions that use AI (CHEERS-AI) to ensure EEs of AI-based health interventions are reported in a transparent and reproducible manner. METHODS: Potential CHEERS-AI reporting items were informed by 2 published systematic literature reviews of EEs and a contemporary update. A Delphi study was conducted using 3 survey rounds to elicit multidisciplinary expert views on 26 potential items, through a 9-point Likert rating scale and qualitative comments. An online consensus meeting was held to finalize outstanding reporting items. A digital health patient group reviewed the final checklist from a patient perspective. RESULTS: A total of 58 participants responded to survey round 1, 42, and 31 of whom responded to rounds 2 and 3, respectively. Nine participants joined the consensus meeting. Ultimately, 38 reporting items were included in CHEERS-AI. They comprised the 28 original CHEERS 2022 items, plus 10 new AI-specific reporting items. Additionally, 8 of the original CHEERS 2022 items were elaborated on to ensure AI-specific nuance is reported. CONCLUSIONS: CHEERS-AI should be used when reporting an EE of an intervention that uses AI to perform its function. CHEERS-AI will help decision makers and reviewers to understand important AI-specific details of an intervention, and any implications for the EE methods used and cost-effectiveness conclusions.

Health technology assessment of tests for SARS-CoV-2 and treatments for COVID-19: A proposed approach and best-practice recommendations.

OBJECTIVES: To develop best-practice guidance for health technology assessment (HTA) agencies when appraising diagnostic tests for SARS-CoV-2 and treatments for COVID-19. METHODS: We used a policy sandbox approach to develop best-practice guidance for HTA agencies to approach known challenges associated with assessing tests and treatments for COVID-19. The guidance was developed by a multi-stakeholder workshop of twenty-one participants representing HTA agencies, clinical and patient experts, academia, industry, and a payer, from across Europe and North America. The workshop was supported by extensive background work to identify the key challenges, including: targeted reviews of existing COVID-related methods guidance for assessing interventions and clinical guidelines, engagement with clinical experts, a survey and workshop of HTA agencies, a systematic review of published economic evaluations, and a workshop of health economic modelers. RESULTS: We suggest HTA agencies should consider using other types of evidence (e.g., real world) where high-quality randomized controlled trials may be lacking and healthcare systems would value timely HTA outputs. A "living" HTA approach may be useful, given the context of an evolving disease, scientific understanding and evidence base, allowing for decisions to be efficiently revisited in response to new information; particularly, if supported by a common "disease model" for COVID-19. Innovative ways of engaging with the public and clinicians, and early engagement with regulators and payers, are recommended. CONCLUSIONS: HTA agencies should consider the elements of this guidance that are most suited to their existing processes to enable them to assess the effectiveness and value of interventions for COVID-19.

Do Profit Margins of Pharmaceuticals Influence Reimbursement Decisions? A Discrete Choice Experiment Among Dutch Healthcare Decision Makers.

OBJECTIVES: This study aimed to investigate whether the profit margins of pharmaceuticals would influence the outcome of reimbursement decisions within the Dutch policy context. METHODS: We conducted a discrete choice experiment among 58 Dutch decision makers. In 20 choice sets, we asked respondents to indicate which of 2 pharmaceutical treatment options they would select for reimbursement. Options were described using 5 attributes (disease severity, incremental costs per quality-adjusted life-year, health gain, budget impact, and profit margin) with 3 levels each. Additionally, cognitive debriefing questions were presented, and for validation debriefing, interviews were conducted. Choice data were analyzed using mixed logit models, also to calculate marginal effects and choice probabilities. RESULTS: Results indicated that the specified levels of profit margins significantly influenced choices made. Decision makers were less likely to reimburse a product with a higher profit margin. The relative importance of profit margins was lower than that of the included traditional health technology assessment criteria, but not negligible. When asked directly, 61% of respondents indicated that profit margin should play a role in reimbursement decision making, although concerns about feasibility and the connection to price negotiations were voiced. CONCLUSIONS: Our results suggest that if available to decision makers the profit margin of pharmaceutical products would influence reimbursement decisions within the Dutch policy context. Higher profit margins would reduce the likelihood of reimbursement. Whether adding profit margin as an additional, explicit criterion to the health technology assessment decision framework would be feasible and desirable is open to further exploration.

Do Health Technology Assessment organisations consider manufacturers' costs in relation to drug price? A study of reimbursement reports.

INTRODUCTION: Drug reimbursement decisions are often made based on a price set by the manufacturer. In some cases, this price leads to public and scientific debates about whether its level can be justified in relation to its costs, including those related to research and development (R&D) and manufacturing. Such considerations could enter the decision process in collectively financed health care systems. This paper investigates whether manufacturers' costs in relation to drug prices, or profit margins, are explicitly mentioned and considered by health technology assessment (HTA) organisations. METHOD: An analysis of reimbursement reports for cancer drugs was performed. All relevant Dutch HTA-reports, published between 2017 and 2019, were selected and matched with HTA-reports from three other jurisdictions (England, Canada, Australia). Information was extracted. Additionally, reimbursement reports for three cases of expensive non-oncolytic orphan drugs prominent in pricing debates in the Netherlands were investigated in depth to examine consideration of profit margins. RESULTS: A total of 66 HTA-reports concerning 15 cancer drugs were included. None of these reports contained information on manufacturer's costs or profit margins. Some reports contained general considerations of the HTA organisation which related prices to manufacturers' costs: six contained a statement on the lack of price setting transparency, one mentioned recouping R&D costs as a potential argument to justify a high price. For the case studies, 21 HTA-reports were selected. One contained a cost-based price justification provided by the manufacturer. None of the other reports contained information on manufacturer's costs or profit margins. Six reports contained a discussion about lack of transparency. Reports from two jurisdictions contained invitations to justify high prices by demonstrating high costs. CONCLUSION: Despite the attention given to manufacturers' costs in relation to price in public debates and in the literature, this issue does not seem to get explicit systematic consideration in the reimbursement reports of expensive drugs.

Psychometric evaluation of the Mental Health Quality of Life (MHQoL) instrument in seven European countries.

INTRODUCTION: To make efficient use of available resources, decision-makers in healthcare may assess the costs and (health) benefits of health interventions. For interventions aimed at improving mental health capturing the full health benefits is an important challenge. The Mental Health Quality of Life (MHQoL) instrument was recently developed to meet this challenge. Evaluating the pyschometric properties of this instrument in different contexts remains important. METHODS: A psychometric evaluation of the MHQoL was performed using existing international, cross-sectional data with 7155 respondents from seven European countries (Denmark, France, Germany, Italy, Portugal, The Netherlands, Portugal and the United Kingdom). Reliability was examined by calculating Cronbach's alpha, a measure of internal consistency of the seven MHQoL dimensions, and by examining the association of the MHQoL sum scores with the MHQoL-VAS scores. Construct validity was examined by calculating Spearman's rank correlation coefficients between the MHQoL sum scores and EQ-5D index scores, EQ-VAS scores, EQ-5D anxiety/depression dimension scores, ICECAP-A index scores and PHQ-4 sum scores. RESULTS: The MHQoL was found to have good internal consistency for all seven countries. The MHQoL sum score and the MHQoL-VAS had a high correlation. Spearman's rank correlation coefficients were moderate to very high for all outcomes. CONCLUSION: Our results, based on data gathered in seven European countries, suggest that the MHQoL shows favourable psychometrical characteristics. While further validation remains important, the MHQoL may be a useful instrument in measuring mental health-related quality of life in the Western European context.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force.

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

The Impact of Decision Makers' Constraints on the Outcome of Value of Information Analysis.

BACKGROUND: When proven effective, decision making regarding reimbursement of new health technology typically involves ethical, social, legal, and health economic aspects and constraints. Nevertheless, when applying standard value of information (VOI) analysis, the value of collecting additional evidence is typically estimated assuming that only cost-effectiveness outcomes guide such decisions. OBJECTIVES: To illustrate how decision makers' constraints can be incorporated into VOI analyses and how these may influence VOI outcomes. METHODS: A simulation study was performed to estimate the cost-effectiveness of a new hypothetical technology compared with usual care. Constraints were defined for the new technology on 1) the maximum acceptable rate of complications and 2) the maximum acceptable additional budget. The expected value of perfect information (EVPI) for the new technology was estimated in various scenarios, both with and without incorporating these constraints. RESULTS: For a willingness-to-pay threshold of €20,000 per quality-adjusted life-year, the probability that the new technology was cost-effective equaled 57%, with an EVPI of €1868 per patient. Applying the complication rate constraint reduced the EVPI to €1137. Similarly, the EVPI reduced to €770 when applying the budget constraint. Applying both constraints simultaneously further reduced the EVPI to €318. CONCLUSIONS: When decision makers explicitly apply additional constraints, beyond a willingness-to-pay threshold, to reimbursement decisions, these constraints can and should be incorporated into VOI analysis as well, because they may influence VOI outcomes. This requires continuous interaction between VOI analysts and decision makers and is expected to improve both the relevance and the acceptance of VOI outcomes.

REVIEWING TRANSFERABILITY IN ECONOMIC EVALUATIONS ORIGINATING FROM EASTERN EUROPE.

OBJECTIVES: The aim of this study is to analyze the quality and transferability issues reported in published peer-reviewed English-language economic evaluations based in healthcare settings of the Central and Eastern European (CEE) and former Soviet countries. METHODS: A systematic search of economic evaluations of healthcare interventions was performed for Armenia, Azerbaijan, Belarus, Bulgaria, Estonia, Georgia, Turkmenistan, Kazakhstan, Lithuania, the former Yugoslav Republic of Macedonia, Republic of Moldova, Romania, the Russian Federation, Serbia, Slovenia, and Ukraine. The included studies were assessed according to their characteristics, quality (using Drummond's checklist), use of local data, and the transferability of inputs and results, if addressed. RESULTS: Most of the thirty-four economic evaluations identified were conducted from a healthcare or payer perspective (74 percent), with 47 percent of studies focusing on infectious diseases. The least frequently and transparently addressed parameters were the items' stated perspectives, relevant costs included, accurately measured costs in appropriate units, outcomes and costs credibly valued, and uncertainties addressed. Local data were often used to assess unit costs, baseline risk, and resource usage, while jurisdiction-specific utilities were included in only one study. Only 32 percent of relevant studies discussed the limitations of using foreign data, and 36 percent of studies discussed the transferability of their own study results to other jurisdictions. CONCLUSIONS: Transferability of the results is not sufficiently discussed in published economic evaluations. To simplify the transferability of studies to other jurisdictions, the following should be comprehensively addressed: uncertainty, impact of influential parameters, and data transferability. The transparency of reporting should be improved.

Do the Washington Panel recommendations hold for Europe: investigating the relation between quality of life versus work-status, absenteeism and presenteeism.

BACKGROUND: The question of how to value lost productivity in economic evaluations has been subject of debate in the past twenty years. According to the Washington panel, lost productivity influences health-related quality of life and should thus be considered a health effect instead of a cost to avoid double counting. Current empirical evidence on the inclusion of income loss when valuing health states is not decisive. We examined the relationship between three aspects of lost productivity (work-status, absenteeism and presenteeism) and patient or social valuation of health-related quality of life (HRQoL). METHODS: Cross-sectional survey data were collected from a total of 830 respondents with a rheumatic disorder from four West-European countries. Health-related quality of life was expressed in either the European societal utility using EQ-5D-3L or the patient valuation using EQ-VAS. The impact of work-status (four categories), absenteeism (absent from paid work during the past three months), and presenteeism (QQ method) on EQ-5D utilities and VAS scores was examined in linear regression analyses taking into account demographic characteristics and disease severity (duration, pain and restriction). RESULTS: The relationship between work-status, absenteeism or presenteeism and HRQoL was stronger for patient valuation than societal valuation. Compared to work-status and presenteeism the relationship between absenteeism and HRQoL was even less explicit. However, results for all measures of lost productivity are only marginally significant and negligible compared to the influence of disease-related restrictions. CONCLUSIONS: This survey study in patients with a rheumatic disorder in four European countries, does not fully support the Washington panel's claim that lost productivity is a significantly related with HRQoL, and this is even more apparent for absenteeism than for work-status and presenteeism. For West-European countries, there is no reason, to include absenteeism in the QALY. Findings need to be confirmed in other disease areas.

Examining the Effect of Depicting a Patient Affected by a Negative Reimbursement Decision in Healthcare on Public Disagreement with the Decision.

BACKGROUND: The availability of increasingly advanced and expensive new health technologies puts considerable pressure on publicly financed healthcare systems. Decisions to not-or no longer-reimburse a health technology from public funding may become inevitable. Nonetheless, policymakers are often pressured to amend or revoke negative reimbursement decisions due to the public disagreement that typically follows such decisions. Public disagreement may be reinforced by the publication of pictures of individual patients in the media. Our aim was to assess the effect of depicting a patient affected by a negative reimbursement decision on public disagreement with the decision. METHODS: We conducted a discrete choice experiment in a representative sample of the public (n = 1008) in the Netherlands and assessed the likelihood of respondents' disagreement with policymakers' decision to not reimburse a new pharmaceutical for one of two patient groups. We presented a picture of one of the patients affected by the decision for one patient group and "no picture available" for the other group. The groups were described on the basis of patients' age, health-related quality of life (HRQOL) and life expectancy (LE) before treatment, and HRQOL and LE gains from treatment. We applied random-intercept logit regression models to analyze the data. RESULTS: Our results indicate that respondents were more likely to disagree with the negative reimbursement decision when a picture of an affected patient was presented. Consistent with findings from other empirical studies, respondents were also more likely to disagree with the decision when patients were relatively young, had high levels of HRQOL and LE before treatment, and large LE gains from treatment. CONCLUSIONS: This study provides evidence for the effect of depicting individual, affected patients on public disagreement with negative reimbursement decisions in healthcare. Policymakers would do well to be aware of this effect so that they can anticipate it and implement policies to mitigate associated risks.

A Blueprint for Multi-use Disease Modeling in Health Economics: Results from Two Expert-Panel Consultations.

BACKGROUND: The current use of health economic decision models in HTA is mostly confined to single use cases, which may be inefficient and result in little consistency over different treatment comparisons, and consequently inconsistent health policy decisions, for the same disorder. Multi-use disease models (MUDMs) (other terms: generic models, whole disease models, disease models) may offer a solution. However, much is uncertain about their definition and application. The current research aimed to develop a blueprint for the application of MUDMs. METHODS: We elicited expert opinion using a two-round modified Delphi process. The panel consisted of experts and stakeholders in health economic modelling from various professional backgrounds. The first questionnaire concerned definition, terminology, potential applications, issues and recommendations for MUDMs and was based on an exploratory scoping review. In the second round, the panel members were asked to reconsider their input, based on feedback regarding first-round results, and to score issues and recommendations for priority. Finally, adding input from external advisors and policy makers in a structured way, an overview of issues and challenges was developed during two team consensus meetings. RESULTS: In total, 54 respondents contributed to the panel results. The term 'multi-use disease models' was proposed and agreed upon, and a definition was provided. The panel prioritized 10 potential applications (with comparing alternative policies and supporting resource allocation decisions as the top 2), while 20 issues (with model transparency and stakeholders' roles as the top 2) were identified as challenges. Opinions on potential features concerning operationalization of multi-use models were given, with 11 of these subsequently receiving high priority scores (regular updates and revalidation after updates were the top 2). CONCLUSIONS: MUDMs would improve on current decision support regarding cost-effectiveness information. Given feasibility challenges, this would be most relevant for diseases with multiple treatments, large burden of disease and requiring more complex models. The current overview offers policy makers a starting point to organize the development, use, and maintenance of MUDMs and to support choices concerning which diseases and policy decisions they will be helpful for.

Supporting decision making in cross-border regions: a health technology assessment tool for hospitals.

OBJECTIVES: The aim of this study was to develop an health technology assessment (HTA) decision tool to support the decision-making process on health technologies for hospital decision makers in cross-border regions. METHODS: Several methods were used to collect information necessary to develop the cross-border mini-HTA decision tool. The literature was inventoried on HTA in border regions and local settings and the use of HTA by local decision makers. Semi-structured interviews with hospital decision makers in cross-border regions were also performed. Based on group discussion of the resulting information, it was decided to use the Danish mini-HTA guideline as a starting point for development of the decision tool. After finishing the first version of the decision tool it was tested in two pilot studies. RESULTS: Some questions in the Danish mini-HTA guideline were not relevant. Other questions needed rephrasing and questions about cross-border situations were added. The pilots showed several missing topics, including legal questions and reimbursement issues. The final decision tool consists of three sections: a general section, a section for hospitals not cooperating cross-border and a section for hospitals that are cooperating with hospitals across a national or regional border. CONCLUSIONS: Based on our literature search, this may be the first cross-border mini-HTA decision tool. The decision tool will be of help for healthcare professionals and decision makers in border settings who would like to use HTA evidence to support their decision-making process.

Guidance on using real-world evidence from Western Europe in Central and Eastern European health policy decision making.

Aim: Real-world data and real-world evidence (RWE) are becoming more important for healthcare decision making and health technology assessment. We aimed to propose solutions to overcome barriers preventing Central and Eastern European (CEE) countries from using RWE generated in Western Europe. Materials & methods: To achieve this, following a scoping review and a webinar, the most important barriers were selected through a survey. A workshop was held with CEE experts to discuss proposed solutions. Results: Based on survey results, we selected the nine most important barriers. Multiple solutions were proposed, for example, the need for a European consensus, and building trust in using RWE. Conclusion: Through collaboration with regional stakeholders, we proposed a list of solutions to overcome barriers on transferring RWE from Western Europe to CEE countries.

Cost-Effectiveness of SGLT2 Inhibitors in a Real-World Population: A MICADO Model-Based Analysis Using Routine Data from a GP Registry.

OBJECTIVE: Sodium-glucose cotransporter 2 inhibitors (SGLT2i) have been shown to reduce the risk of cardiovascular complications, which largely drive diabetes' health and economic burdens. Trial results indicated that SGLT2i are cost effective. However, these findings may not be generalizable to the real-world target population. This study aims to evaluate the cost effectiveness of SGLT2i in a routine care type 2 diabetes population that meets Dutch reimbursement criteria using the MICADO model. METHODS: Individuals from the Hoorn Diabetes Care System cohort (N = 15,392) were filtered to satisfy trial inclusion criteria (including EMPA-REG, CANVAS, and DECLARE-TIMI58) or satisfy the current Dutch reimbursement criteria for SGLT2i. We validated a health economic model (MICADO) by comparing simulated and observed outcomes regarding the relative risks of events in the intervention and comparator arm from three trials, and used the validated model to evaluate the long-term health outcomes using the filtered cohorts' baseline characteristics and treatment effects from trials and a review of observational studies. The incremental cost-effectiveness ratio (ICER) of SGLT2i, compared with care-as-usual, was assessed from a third-party payer perspective, measured in euros (2021 price level), using a discount rate of 4% for costs and 1.5% for effects. RESULTS: From Dutch individuals with diabetes in routine care, 15.8% qualify for the current Dutch reimbursement criteria for SGLT2i. Their characteristics were significantly different (lower HbA1c, higher age, and generally more preexisting complications) than trial populations. After validating the MICADO model, we found that lifetime ICERs of SGLT2i, when compared with usual care, were favorable (< €20,000/QALY) for all filtered cohorts, resulting in an ICER of €5440/QALY using trial-based treatment effect estimates in reimbursed population. Several pragmatic scenarios were tested, the ICERs remained favorable. CONCLUSIONS: Although the Dutch reimbursement indications led to a target group that deviates from trial populations, SGLT2i are likely to be cost effective when compared with usual care.

Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment.

Background: Artificial intelligence (AI) has attracted much attention because of its enormous potential in healthcare, but uptake has been slow. There are substantial barriers that challenge health technology assessment (HTA) professionals to use AI-generated evidence for decision-making from large real-world databases (e.g., based on claims data). As part of the European Commission-funded HTx H2020 (Next Generation Health Technology Assessment) project, we aimed to put forward recommendations to support healthcare decision-makers in integrating AI into the HTA processes. The barriers, addressed by the paper, are particularly focusing on Central and Eastern European (CEE) countries, where the implementation of HTA and access to health databases lag behind Western European countries. Methods: We constructed a survey to rank the barriers to using AI for HTA purposes, completed by respondents from CEE jurisdictions with expertise in HTA. Using the results, two members of the HTx consortium from CEE developed recommendations on the most critical barriers. Then these recommendations were discussed in a workshop by a wider group of experts, including HTA and reimbursement decision-makers from both CEE countries and Western European countries, and summarized in a consensus report. Results: Recommendations have been developed to address the top 15 barriers in areas of (1) human factor-related barriers, focusing on educating HTA doers and users, establishing collaborations and best practice sharing; (2) regulatory and policy-related barriers, proposing increasing awareness and political commitment and improving the management of sensitive information for AI use; (3) data-related barriers, suggesting enhancing standardization and collaboration with data networks, managing missing and unstructured data, using analytical and statistical approaches to address bias, using quality assessment tools and quality standards, improving reporting, and developing better conditions for the use of data; and (4) technological barriers, suggesting sustainable development of AI infrastructure. Conclusion: In the field of HTA, the great potential of AI to support evidence generation and evaluation has not yet been sufficiently explored and realized. Raising awareness of the intended and unintended consequences of AI-based methods and encouraging political commitment from policymakers is necessary to upgrade the regulatory and infrastructural environment and knowledge base required to integrate AI into HTA-based decision-making processes better.

Recommendations for patient involvement in health technology assessment in Central and Eastern European countries.

Introduction: Meaningful patient involvement in health technology assessment (HTA) is essential in ensuring that the interests of the affected patient population, their families, and the general public are accurately reflected in coverage and reimbursement decisions. Central and Eastern European (CEE) countries are generally at less advanced stages of implementing HTA, which is particularly true for patient involvement activities. As part of the Horizon2020 HTx project, this research aimed to form recommendations for critical barriers to patient involvement in HTA in CEE countries. Methods: Built on previous research findings on potential barriers, a prioritisation survey was conducted online with CEE stakeholders. Recommendations for prioritised barriers were formed through a face-to-face workshop by CEE stakeholders and HTx experts. Results: A total of 105 stakeholders from 13 CEE countries completed the prioritisation survey and identified 12 of the 22 potential barriers as highly important. The workshop had 36 participants representing 9 CEE countries, and 5 Western European countries coming together to discuss solutions in order to form recommendations based on best practices, real-life experience, and transferability aspects. Stakeholder groups involved in both phases included HTA organisation representatives, payers, patients, caregivers, patient organisation representatives, patient experts, health care providers, academic and non-academic researchers, health care consultants and health technology manufacturers/providers. As a result, 12 recommendations were formed specified to the CEE region's context, but potentially useful for a broader geographic audience. Conclusion: In this paper, we present 12 recommendations for meaningful, systematic, and sustainable patient involvement in HTA in CEE countries. Our hope is that engaging more than a hundred CEE stakeholders in the study helped to spread awareness of the importance and potential of patient involvement and that the resulting recommendations provide tangible steps for the way forward. Future studies shall focus on country-specific case studies of the implemented recommendations.

Broadening the application of health technology assessment in the Netherlands: a worthwhile destination but not an easy ride?

Currently, reimbursement decisions based on health technology assessments (HTA) in the Netherlands mostly concern outpatient pharmaceuticals. The Dutch government aspires to broaden the systematic application of full HTA towards other types of health care in order to optimise the content of the basic benefit package. This paper identifies important challenges for broadening the scope of full HTA to other types of health care. Based on a description of the Dutch reimbursement decision-making process, five important characteristics of outpatient pharmaceuticals were identified, which are all relevant to the successful application of HTA: (i) closed reimbursement system, (ii) absence of alternative policy measures, (iii) existence of marketing authorisation, (iv) identifiable and accountable counterparty, and (v) product characteristics. For a selection of other types of health care, which may be subject to HTA more frequently in the future, deviations from these characteristics of outpatient pharmaceuticals are discussed. The implications of such deviations for performing HTA and the decision-making process are highlighted. It is concluded that broadening the application of HTA will require policy makers to meet both important policy-related and methodological challenges. These challenges differ per health care domain, which may inform policy makers which expansions of the current use of HTA are most feasible.

The Application and Implications of Novel Deterministic Sensitivity Analysis Methods.

Deterministic sensitivity analyses (DSA) remain important to interpret the effect of uncertainties in individual parameters on results of cost-effectiveness analyses. Classic DSA methodologies may lead to wrong conclusions due to a lack of or misleading information regarding marginal effects, non-linearity, likelihood and correlations. In addition, tornado diagrams are misleading in some situations. Recent advances in DSA methods have the potential to provide decision makers with more reliable information regarding the effects of uncertainties in individual parameters. This practical application discusses advances to classic DSA methods and their implications. Three methods are discussed: stepwise DSA, distributional DSA and probabilistic DSA. For each method, the technical specifications, options for presenting results, and its implications for decision making are discussed. Options for visualizing DSA results in incremental cost-effectiveness ratios and in incremental net benefits are presented. The use of stepwise DSA increases interpretability of marginal effects and non-linearities in the model, which is especially relevant when arbitrary ranges are implemented. Using the probability distribution of each parameter in distributional DSA provides insight on the likelihood of model outcomes while probabilistic DSA also includes the effects of correlations between parameters.Probabilistic DSA, preferably expressed in incremental net benefit, is the most appropriate method for providing insight on the effect of uncertainty in individual parameters on the estimate of cost effectiveness. However, the opportunities provided by probabilistic DSA may not always be needed for decision making. Other DSA methods, in particular distributional DSA, can sometimes be sufficient depending on model features. Decision makers must determine to which extent they will accept and implement these new and improved DSA methodologies and adjust guidelines accordingly.

Building a trusted framework for uncertainty assessment in rare diseases: suggestions for improvement (Response to "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases").

The aim of this letter to the editor is to provide a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. The authors of "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases" presented recommendations for reducing uncertainty in rare diseases. Their article is of great importance but unfortunately suffers from a lack of references to the wider uncertainty in Health Technology Assessment and research prioritisation literature and consequently fails to provide a trusted framework for decision-making in rare diseases. In this letter to the editor we critique the authors' tool and provide pointers as to how their proposal can be strengthened. We present references to the literature, including our own tool for uncertainty assessment (TRUST; unrelated to the authors' research), apply TRUST to two assessments of orphan drugs in rare diseases and provide a broader perspective on uncertainty and risk management in rare diseases, including a detailed research agenda.