Reply to Standaert, B. Comment on "Postma et al. Predicted Public Health and Economic Impact of Respiratory Syncytial Virus Vaccination with Variable Duration of Protection for Adults ≥60 Years in Belgium".

We have read the commentary from Baudouin Standaert [...].

Predicted Public Health and Economic Impact of Respiratory Syncytial Virus Vaccination with Variable Duration of Protection for Adults ≥60 Years in Belgium.

Respiratory syncytial virus (RSV) is a leading cause of acute respiratory infection (ARI) in older adults. This study used a static, cohort-based decision-tree model to estimate the public health and economic impact of vaccination against RSV in Belgians aged ≥60 years compared with no vaccination for different vaccine duration of protection profiles from a healthcare payer perspective. Three vaccine protection durations were compared (1, 3, and 5 years), and several sensitivity and scenario analyses were performed. Results showed that an RSV vaccine with a 3-year duration of protection would prevent 154,728 symptomatic RSV-ARI cases, 3688 hospitalizations, and 502 deaths over three years compared to no vaccination in older adults and would save EUR 35,982,857 in direct medical costs in Belgium. The number needed to vaccinate to prevent one RSV-ARI case was 11 for the 3-year duration profile, while it was 28 and 8 for the 1- and 5-year vaccine duration profiles, respectively. The model was generally robust in sensitivity analyses varying key input values. This study suggested that vaccination could substantially decrease the public health and economic burden of RSV in adults ≥60 years in Belgium, with benefits increasing with a longer duration of vaccine protection.

A MEA is a MEA is a MEA? Sequential decision making and the impact of different managed entry agreements at the manufacturer and payer level, using a case study for an oncology drug in England.

BACKGROUND: In a typical single-payer setting that uses an explicit cost-effectiveness (CE) threshold in its decision-making, the payer aims to maximize the net-monetary-benefit (NMB) given the CE threshold, whilst the manufacturer aims to maximize the expected discounted-cash-flow (DCF) resulting from the sales of that technology. Managed entry agreements (MEAs) are tools that are used to improve access to expensive technologies that would otherwise not be deemed to be cost-effective to payers. While simple discount on the list price is the most commonly applied MEA type, there are different forms, each having a different impact on the cost-effectiveness of the technology, on the lifetime DCF-per-patient and on the decision uncertainty. We aim to analyze the sequential decision-making (SDM) of different MEAs (i.e. simple discount, free treatment initiation, lifetime treatment acquisition cost-capping [LTTACC], performance-based money-back guarantee [MBG]) at the manufacturer and at the payer level, respectively. METHODS: We first model the SDM of the manufacturer and the payer as a sequential game and explain the challenges to find an equilibrium analytically. Then we propose a heuristic computational method to follow for each of the MEA types, based on practice. To demonstrate this SDM on a case study, a UK-based cost-utility analysis using a three-state, partitioned-survival-model was constructed to determine the cost-effectiveness of regorafenib versus best-supportive-care for the second-line treatment of hepatocellular carcinoma. The optimal agreement terms that would maximise the lifetime DCF-per-patient for each MEA, whilst remaining below the CE-threshold (£50,000/QALY gained) were obtained in the deterministic base-case. Robustness for each optimized MEA was then assessed using probabilistic sensitivity and scenario analyses, the value of information (VoI), and HTA-risk analyses. RESULTS: As expected, the introduction of all MEAs improved the probabilistic ICER and NMB values to (almost) acceptable levels, compared to the "no-MEA" case (ICER ~ £78,000/QALY-gained). The expected DCFs across the explored MEAs were all similar, whilst the payer strategy & uncertainty burden (PSUB) for regorafenib decreased in all MEAs explored. VoI analyses revealed that regorafenib mean-dose-intensity and time-on-treatment (ToT) parameters attributed most to the decision uncertainty. LTTACC provided the smallest PSUB and the most robust NMB estimates under parametric uncertainty. For scenarios assuming increased regorafenib ToT or mean-dose-intensity, LTACC again provided acceptable cost-effectiveness outcomes, whereas for scenarios assuming decreased regorafenib progression-free/overall survival effectiveness, only MBG resulted in plausible ICER values. In scenarios, where the source of uncertainty was not targeted by MEA parameters (e.g. the scenario assuming higher progressed disease resource utilization), all investigated MEA types resulted in unacceptable cost-effectiveness outcomes. CONCLUSION: Each MEA type has a different implication. The impact of different MEAs on the NMB is more noteworthy than on the DCF, in relative terms, hence payers will benefit from the early participation of the MEA design rather than leaving this up to the prerogative of the manufacturer. While simple discount might be practical for implementation purposes, other MEAs can provide additional benefits to the payer in terms of increased NMB, reduced decision risk and reduced uncertainty. MEA performance should be investigated not only under parametric uncertainty, but also under-identified structural uncertainty, and the barriers of implementation should be considered thoroughly before choosing the most appropriate MEA type.

TECH-VER: A Verification Checklist to Reduce Errors in Models and Improve Their Credibility.

BACKGROUND: In health economic literature, checklists or best practice recommendations on model validation/credibility always declare verification of the programmed model as a fundamental step, such as 'is the model implemented correctly and does the implementation accurately represent the conceptual model?' However, to date, little operational guidance for the model verification process has been given. In this study, we aimed to create an operational checklist for model users or reviewers to verify the technical implementation of health economic decision analytical models and document their verification efforts. METHODS: Literature on model validation, verification, programming errors and credibility was reviewed systematically from scientific databases. An initial beta version of the checklist was developed based on the checklists/tests identified from the literature and from authors' previous modeling/appraisal experience. Next, the first draft checklist was presented to a number of health economists on several occasions and was tested on different models (built in different software, developed by different stakeholders, including drug manufacturers, consultancies or academia), each time leading to an update of the checklist and culminating in the final version of the TECHnical VERification (TECH-VER) checklist, introduced in this paper. RESULTS: The TECH-VER necessitates a model reviewer (preferably independent), an executable and transparent model, its input sources, and detailed documentation (e.g. technical report/scientific paper) in which the conceptual model, its implementation, programmed model inputs, and results are reported. The TECH-VER checklist consists of five domains: (1) input calculations; (2) event-state (patient flow) calculations; (3) result calculations; (4) uncertainty analysis calculations; and (5) other overall checks (e.g. validity or interface). The first four domains reflect the verification of the components of a typical health economic model. For these domains, as a prerequisite of verification tests, the reviewer should identify the relevant calculations in the electronic model and assess the provided justifications for the methods used in the identified calculations. For this purpose, we recommend completeness/consistency checks. Afterwards, the verification tests can be conducted for the calculations in each of these stages by checking the correctness of the implementation of these calculations. For this purpose, the following type of tests are recommended in consecutive order: (i) black-box tests, i.e. checking if model calculations are in line with a priori expectations; (ii) white-box testing, i.e. going through the program code details line by line, or cell by cell (recommended for some crucial calculations and if there are some unexpected results from the black-box tests); and (iii) model replication/parallel programming (recommended only in certain situations, and if the issues related to the identified unexpected results from black-box tests could not be resolved through white-box testing). To reduce the time burden of model verification, we suggest a hierarchical order in tests i-iii, where going to the next step is necessary when the previous step fails. CONCLUSIONS: The TECH-VER checklist is a comprehensive checklist for the technical verification of decision analytical models, aiming to help identify model implementation errors and their root causes while improving the transparency and efficiency of the verification efforts. In addition to external reviews, we consider that the TECH-VER can be used as an internal training and quality control tool for new health economists, while developing their initial models. It is the authors' aim that the TECH-VER checklist transforms itself to an open-source living document, with possible future versions, or 'bolt-on' extensions for specific applications with additional 'fit-for-purpose' tests, as well as 'tips and tricks' and some demonstrative error examples. For this reason, the TECH-VER checklist and the list of black-box tests created in this paper and a few model verification examples is uploaded to an open access, online platform (github and the website of the institute), where other users will also be able to upload their original verification efforts and tests.

Severity-Adjusted Probability of Being Cost Effective.

BACKGROUND: In the context of priority setting, a differential cost-effectiveness threshold can be used to reflect a higher societal willingness to pay for quality-adjusted life-year gains in the worse off. However, uncertainty in the estimate of severity can lead to problems when evaluating the outcomes of cost-effectiveness analyses. OBJECTIVES: This study standardizes the assessment of severity, integrates its uncertainty with the uncertainty in cost-effectiveness results and provides decision makers with a new estimate: the severity-adjusted probability of being cost effective. METHODS: Severity is expressed in proportional and absolute shortfall and estimated using life tables and country-specific EQ-5D values. We use the three severity-based cost-effectiveness thresholds (€20.000, €50.000 and €80.000, per QALY) adopted in The Netherlands. We exemplify procedures of integrating uncertainty with a stylized example of a hypothetical oncology treatment. RESULTS: Applying our methods, taking into account the uncertainty in the cost-effectiveness results and in the estimation of severity identifies the likelihood of an intervention being cost effective when there is uncertainty about the appropriate severity-based cost-effectiveness threshold. CONCLUSIONS: Higher willingness-to-pay thresholds for severe diseases are implemented in countries to reflect societal concerns for an equitable distribution of resources. However, the estimates of severity are uncertain, patient populations are heterogeneous, and this can be accounted for with the severity-adjusted probability of being cost effective proposed in this study. The application to the Netherlands suggests that not adopting the new method could result in incorrect decisions in the reimbursement of new health technologies.

Ribociclib with an Aromatase Inhibitor for Previously Untreated, HR-Positive, HER2-Negative, Locally Advanced or Metastatic Breast Cancer: An Evidence Review Group Perspective of a NICE Single Technology Appraisal.

The National Institute for Health and Care Excellence, as part of the institute's single technology appraisal process, invited the manufacturer of ribociclib (Kisqali®, Novartis) to submit evidence regarding the clinical and cost effectiveness of the drug in combination with an aromatase inhibitor for the treatment of previously untreated, hormone receptor-positive, human epidermal growth factor receptor 2-negative, locally advanced or metastatic breast cancer. Kleijnen Systematic Reviews Ltd and Erasmus University Rotterdam were commissioned as the Evidence Review Group for this submission. The Evidence Review Group reviewed the evidence submitted by the manufacturer, corrected and validated the manufacturer's decision analytic model, and conducted exploratory analyses to assess the robustness and validity of the presented clinical and cost-effectiveness results. This article describes the company submission, the Evidence Review Group assessment and National Institute for Health and Care Excellence subsequent decisions. The main clinical effectiveness evidence was obtained from the MONALEESA-2 trial, a randomised controlled trial comparing ribociclib plus letrozole with placebo plus letrozole. Progression-free survival was significantly longer in the ribociclib group (95% confidence interval, 19.3-not reached) vs. 14.7 months (95% confidence interval 13.0-16.5) in the placebo group. To assess the cost effectiveness of ribociclib in combination with an aromatase inhibitor, the company developed an individual patient-level model using a discrete-event simulation approach in Microsoft® Excel. In the model, simulated patients move through a series of three health states until death, i.e. first-line progression-free survival, second-line progression-free survival and progressive disease. The length of progression-free survival during the first line was informed by the MONALEESA-2 trial. The benefit in progression-free survival in the first line was transferred to a benefit in overall survival assuming full progression-free survival to overall survival surrogacy (because of the immaturity of overall survival data from the MONALEESA-2 trial). Patient-level data from the BOLERO-2 trial, evaluating the addition of everolimus to exemestane in the second-line treatment of postmenopausal HR-positive advanced breast cancer, were used to inform the length of progression-free survival during the second line. Costs included in the model were treatment costs (e.g. technology acquisition costs of first, second, third and/or later line treatments), drug administration costs, monitoring costs and health state costs (including terminal care). Additionally, the costs of adverse events associated with the first-line treatment were incorporated. The Evidence Review Group recalculated the incremental cost-effectiveness ratio using data from a different data cut-off date from the MONALEESA-2 trial and by changing some assumptions (e.g. progression-free survival to overall survival surrogacy approach and post-progression third and/or later line treatment-related costs). After two appraisal committee meetings and a revised base case submitted by the company (including a second enhanced patient access scheme discount), the committee concluded that taking into account the uncertainties in the calculation of the cost effectiveness, there were plausible cost-effectiveness estimates broadly in the range that could be considered as a cost-effective use of National Health Service resources. Therefore, ribociclib was recommended as a treatment option for the first-line treatment of hormone receptor-positive, human epidermal growth factor receptor 2-negative breast cancer, contingent on the company providing ribociclib with the discount agreed in the second enhanced patient access scheme.

Obinutuzumab in Combination with Chemotherapy for the First-Line Treatment of Patients with Advanced Follicular Lymphoma : An Evidence Review Group Evaluation of the NICE Single Technology Appraisal.

The National Institute for Health and Care Excellence (NICE), as part of the institute's single technology appraisal (STA) process, invited the company that makes obinutuzumab (Roche Products Limited) to submit evidence of the clinical and cost effectiveness of the drug in combination with chemotherapy, with or without obinutuzumab as maintenance therapy for adult patients with untreated, advanced follicular lymphoma (FL) in the UK. Kleijnen Systematic Reviews Ltd (KSR), in collaboration with Erasmus University Rotterdam, was commissioned to act as the Evidence Review Group (ERG). This paper describes the company's submission, the ERG review, and NICE's subsequent decisions. The clinical evidence was derived from two phase III, company-sponsored, randomised, open-label studies. Most evidence on obinutuzumab was based on the GALLIUM trial that compared obinutuzumab in combination with chemotherapy as induction followed by obinutuzumab maintenance monotherapy with rituximab in combination with chemotherapy as induction followed by rituximab maintenance monotherapy in previously untreated patients with FL (grades 1-3a). Long-term clinical evidence was based on the PRIMA trial, studying the benefit of two years of rituximab maintenance after first-line treatment in patients with FL. The cost-effectiveness evidence submitted by the company relied on a partitioned survival cost-utility model, implemented in Microsoft® Excel. The base-case incremental cost-effectiveness ratio (ICER) presented in the company submission was <£20,000 per quality-adjusted life-year (QALY) gained. Although the ERG concluded that the economic model met the NICE reference case to a reasonable extent, some errors were identified and several assumptions made by the company were challenged. A new base-case scenario produced by the ERG suggested an ICER that was higher than the company base case, but still below £30,000 per QALY gained. However, some ERG scenario analyses were close to or even above the threshold. This was the case in particular for assuming a treatment effect that did not extend beyond trial follow-up. These results led to an initial negative recommendation by the appraisal committee. Subsequently, the company submitted a revised base case focusing on patients at intermediate or high risk of premature mortality. Simultaneously, a further price discount for obinutuzumab was granted. In addition to the company's revised base case, the ERG suggested a restriction of the treatment effect to 5 years and implemented biosimilar uptake and cheaper prices for rituximab. All of these adjustments did not exceed £30,000 per QALY gained and therefore the use of obinutuzumab for patients with advanced FL and a Follicular Lymphoma International Predictive Index (FLIPI) score of two or more could be recommended.

Pomalidomide with Dexamethasone for Treating Relapsed and Refractory Multiple Myeloma Previously Treated with Lenalidomide and Bortezomib: An Evidence Review Group Perspective of an NICE Single Technology Appraisal.

The National Institute for Health and Care Excellence (NICE), as part of the institute's single technology appraisal (STA) process, invited the manufacturer of pomalidomide (POM; Imnovid®, Celgene) to submit evidence regarding the clinical and cost effectiveness of the drug in combination with dexamethasone (POM + LoDEX) for the treatment of relapsed and refractory multiple myeloma (RRMM) after at least two regimens including lenalidomide (LEN) and bortezomib (BOR). Kleijnen Systematic Reviews Ltd (KSR) and Erasmus University Rotterdam were commissioned as the Evidence Review Group (ERG) for this submission. The ERG reviewed the evidence submitted by the manufacturer, validated the manufacturer's decision analytic model, and conducted exploratory analyses in order to assess the robustness and validity of the presented clinical and cost-effectiveness results. This paper describes the company submission, the ERG assessment, and NICE's subsequent decisions. The company conducted a systematic review to identify studies comparing POM with comparators outlined in the NICE scope: panobinostat with bortezomib and dexamethasone (PANO + BOR + DEX), bendamustine with thalidomide and dexamethasone (BTD) and conventional chemotherapy (CC). The main clinical effectiveness evidence was obtained from MM-003, a randomized controlled trial (RCT) comparing POM + LoDEX with high-dose dexamethasone (HiDEX; used as a proxy for CC). Additional data from other studies were also used as nonrandomized observational data sources for the indirect treatment comparison of POM + LoDEX with BTD and PANO + BOR + DEX. Covariate or treatment switching adjustment methods were used for each comparison. The model developed in Microsoft® Excel 2010 using a semi-Markov partitioned survival structure, submitted in the original submission to NICE for TA338, was adapted for the present assessment of the cost effectiveness of POM + LoDEX. Updated evidence from the clinical-effectiveness part was used for the survival modelling of progression-free survival and overall survival. For POM + LoDEX, the patient access scheme (PAS) discount was applied to the POM price. Three separate comparisons were conducted for each comparator, each comparison using a different dataset and adjustment methods. The ERG identified and corrected some errors, and the corrected incremental cost-effectiveness ratios (ICERs) for POM + LoDEX versus each comparator were presented: approximately £45,000 per quality-adjusted life-year (QALY) gained versus BTD, savings of approximately £143,000 per QALY lost versus PANO + BOR + DEX, and approximately £49,000 per QALY gained versus CC. The ERG also conducted full incremental analyses, which revealed that CC, POM + LoDEX and PANO + BOR + DEX were on the cost-effectiveness frontier. The committee's decision on the technology under analysis deemed that POM + LoDEX should be recommended as an option for treating multiple myeloma in adults at third or subsequent relapse of treatments including both LEN and BOR, contingent on the company providing POM with the discount agreed in the PAS.

Ramucirumab for Treating Advanced Gastric Cancer or Gastro-Oesophageal Junction Adenocarcinoma Previously Treated with Chemotherapy: An Evidence Review Group Perspective of a NICE Single Technology Appraisal.

The National Institute for Health and Care Excellence (NICE) invited the company that manufactures ramucirumab (Cyramza®, Eli Lilly and Company) to submit evidence of the clinical and cost effectiveness of the drug administered alone (monotherapy) or with paclitaxel (combination therapy) for treating adults with advanced gastric cancer or gastro-oesophageal junction (GC/GOJ) adenocarcinoma that were previously treated with chemotherapy, as part of the Institute's single technology appraisal (STA) process. Kleijnen Systematic Reviews Ltd (KSR), in collaboration with Erasmus University Rotterdam, was commissioned to act as the Evidence Review Group (ERG). This paper describes the company's submission, the ERG review, and NICE's subsequent decisions. Clinical effectiveness evidence for ramucirumab monotherapy (RAM), compared with best supportive care (BSC), was based on data from the REGARD trial. Clinical effectiveness evidence for ramucirumab combination therapy (RAM + PAC), compared with paclitaxel monotherapy (PAC), was based on data from the RAINBOW trial. In addition, the company undertook a network meta-analysis (NMA) to compare RAM + PAC with BSC and docetaxel. Cost-effectiveness evidence of monotherapy and combination therapy relied on partitioned survival, cost-utility models. The base-case incremental cost-effectiveness ratio (ICER) of the company was £188,640 (vs BSC) per quality-adjusted life-year (QALY) gained for monotherapy and £118,209 (vs BSC) per QALY gained for combination therapy. The ERG assessment indicated that the modelling structure represented the course of the disease; however, a few errors were identified and some of the input parameters were challenged. The ERG provided a new base case, with ICERs (vs BSC) of £188,100 (monotherapy) per QALY gained and £129,400 (combination therapy) per QALY gained and conducted additional exploratory analyses. The NICE Appraisal Committee (AC), considered the company's decision problem was in line with the NICE scope, with the exception of the choice of comparators for the combination therapy model. The most plausible ICER for ramucirumab monotherapy compared with BSC was £188,100 per QALY gained. The Committee considered that the ERG's exploratory analysis in which RAM + PAC was compared with PAC by using the direct head-to-head data (including utilities) from the RAINBOW trial, provided the most plausible ICER (i.e. £408,200 per QALY gained) for ramucirumab combination therapy. The Committee concluded that end-of-life considerations cannot be applied for either case, since neither failed to offer an extension to life of at least 3 months. The company did not submit a patient access scheme (PAS). After consideration of the evidence, the Committee concluded that ramucirumab alone or with paclitaxel could not be considered a cost-effective use of National Health Service resources for treating advanced GC/GOJ patients that were previously treated with chemotherapy, and therefore its use could not be recommended. We might wonder if a complete STA process is necessary for treatments without a PAS, which are, according to the company's submission, already associated with ICERs far above the currently accepted threshold in all (base-case, sensitivity and scenario) analyses.