A structured process for the validation of a decision-analytic model: application to a cost-effectiveness model for risk-stratified national breast screening.

BACKGROUND: Decision-makers require knowledge of the strengths and weaknesses of decision-analytic models used to evaluate healthcare interventions to be able to confidently use the results of such models to inform policy. A number of aspects of model validity have previously been described, but no systematic approach to assessing the validity of a model has been proposed. This study aimed to consolidate the different aspects of model validity into a step-by-step approach to assessing the strengths and weaknesses of a decision-analytic model. METHODS: A pre-defined set of steps were used to conduct the validation process of an exemplar early decision-analytic-model-based cost-effectiveness analysis of a risk-stratified national breast cancer screening programme [UK healthcare perspective; lifetime horizon; costs (£; 2021)]. Internal validation was assessed in terms of descriptive validity, technical validity and face validity. External validation was assessed in terms of operational validation, convergent validity (or corroboration) and predictive validity. RESULTS: The results outline the findings of each step of internal and external validation of the early decision-analytic-model and present the validated model (called 'MANC-RISK-SCREEN'). The positive aspects in terms of meeting internal validation requirements are shown together with the remaining limitations of MANC-RISK-SCREEN. CONCLUSION: Following a transparent and structured validation process, MANC-RISK-SCREEN has been shown to have satisfactory internal and external validity for use in informing resource allocation decision-making. We suggest that MANC-RISK-SCREEN can be used to assess the cost-effectiveness of exemplars of risk-stratified national breast cancer screening programmes (NBSP) from the UK perspective. IMPLICATIONS: A step-by-step process for conducting the validation of a decision-analytic model was developed for future use by health economists. Using this approach may help researchers to fully demonstrate the strengths and limitations of their model to decision-makers.

The implications of competing risks and direct treatment disutility in cardiovascular disease and osteoporotic fracture: risk prediction and cost effectiveness analysis.

Background: Clinical guidelines commonly recommend preventative treatments for people above a risk threshold. Therefore, decision-makers must have faith in risk prediction tools and model-based cost-effectiveness analyses for people at different levels of risk. Two problems that arise are inadequate handling of competing risks of death and failing to account for direct treatment disutility (i.e. the hassle of taking treatments). We explored these issues using two case studies: primary prevention of cardiovascular disease using statins and osteoporotic fracture using bisphosphonates. Objectives: Externally validate three risk prediction tools [QRISK®3, QRISK®-Lifetime, QFracture-2012 (ClinRisk Ltd, Leeds, UK)]; derive and internally validate new risk prediction tools for cardiovascular disease [competing mortality risk model with Charlson Comorbidity Index (CRISK-CCI)] and fracture (CFracture), accounting for competing-cause death; quantify direct treatment disutility for statins and bisphosphonates; and examine the effect of competing risks and direct treatment disutility on the cost-effectiveness of preventative treatments. Design, participants, main outcome measures, data sources: Discrimination and calibration of risk prediction models (Clinical Practice Research Datalink participants: aged 25-84 years for cardiovascular disease and aged 30-99 years for fractures); direct treatment disutility was elicited in online stated-preference surveys (people with/people without experience of statins/bisphosphonates); costs and quality-adjusted life-years were determined from decision-analytic modelling (updated models used in National Institute for Health and Care Excellence decision-making). Results: CRISK-CCI has excellent discrimination, similar to that of QRISK3 (Harrell's c = 0.864 vs. 0.865, respectively, for women; and 0.819 vs. 0.834, respectively, for men). CRISK-CCI has systematically better calibration, although both models overpredict in high-risk subgroups. People recommended for treatment (10-year risk of ≥ 10%) are younger when using QRISK-Lifetime than when using QRISK3, and have fewer observed events in a 10-year follow-up (4.0% vs. 11.9%, respectively, for women; and 4.3% vs. 10.8%, respectively, for men). QFracture-2012 underpredicts fractures, owing to under-ascertainment of events in its derivation. However, there is major overprediction among people aged 85-99 years and/or with multiple long-term conditions. CFracture is better calibrated, although it also overpredicts among older people. In a time trade-off exercise (n = 879), statins exhibited direct treatment disutility of 0.034; for bisphosphonates, it was greater, at 0.067. Inconvenience also influenced preferences in best-worst scaling (n = 631). Updated cost-effectiveness analysis generates more quality-adjusted life-years among people with below-average cardiovascular risk and fewer among people with above-average risk. If people experience disutility when taking statins, the cardiovascular risk threshold at which benefits outweigh harms rises with age (≥ 8% 10-year risk at 40 years of age; ≥ 38% 10-year risk at 80 years of age). Assuming that everyone experiences population-average direct treatment disutility with oral bisphosphonates, treatment is net harmful at all levels of risk. Limitations: Treating data as missing at random is a strong assumption in risk prediction model derivation. Disentangling the effect of statins from secular trends in cardiovascular disease in the previous two decades is challenging. Validating lifetime risk prediction is impossible without using very historical data. Respondents to our stated-preference survey may not be representative of the population. There is no consensus on which direct treatment disutilities should be used for cost-effectiveness analyses. Not all the inputs to the cost-effectiveness models could be updated. Conclusions: Ignoring competing mortality in risk prediction overestimates the risk of cardiovascular events and fracture, especially among older people and those with multimorbidity. Adjustment for competing risk does not meaningfully alter cost-effectiveness of these preventative interventions, but direct treatment disutility is measurable and has the potential to alter the balance of benefits and harms. We argue that this is best addressed in individual-level shared decision-making. Study registration: This study is registered as PROSPERO CRD42021249959. Funding: This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 15/12/22) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 4. See the NIHR Funding and Awards website for further award information.

Before offering a medicine to prevent disease, prescribers must expect it to do more good than harm. This balance depends on how likely it is that the person will develop the disease we want to prevent. But people might first die for other reasons. We call this a 'competing risk'. In most cases, the mathematical tools we use to estimate the chance of developing a disease do not account for competing risks. Another problem is that, when weighing up the benefits and harms of medicines, we ignore the hassle they cause patients, even when they do not cause side effects. We used two examples: statins to prevent heart disease and bisphosphonates to prevent fractures. First, we assessed if existing tools get predictions wrong by not accounting for competing risks. We found that they exaggerate the chance of heart attacks and strokes. However, the exaggeration is greatest among people who would clearly benefit from preventative treatment. So it may not change treatment decisions much. The fracture prediction tool we studied was very inaccurate, exaggerating risk among older people, but underestimating risk among younger people. We made a new fracture risk prediction tool. It gave better predictions, but it was still inaccurate for people aged > 85 years and those with several health problems. Next, we asked people questions designed to put a number on the hassle that statins and bisphosphonates cause. Most people thought that taking either is inconvenient, but the hassle factor for bisphosphonates is bigger. Finally, we updated the mathematical models that the National Institute for Health and Care Excellence used when recommending statins and bisphosphonates. We worked out if competing risks and the hassle of taking medicines make a difference to results. Statins remain a good idea for almost everyone, unless they really hate the idea of taking them. But bisphosphonates would do more harm than good for anyone who agrees with the hassle factor we found.

International Systematic Review of Utility Values Associated with Cardiovascular Disease and Reflections on Selecting Evidence for a UK Decision-Analytic Model.

PURPOSE: Evaluating interventions for cardiovascular disease (CVD) requires estimates of its effect on utility. We aimed to 1) systematically review utility estimates for CVDs published since 2013 and 2) critically appraise UK-relevant estimates and calculate corresponding baseline utility multipliers. METHODS: We searched MEDLINE and Embase (April 22, 2021) using CVD and utility terms. We screened results for primary studies reporting utility distributions for people with experience of heart failure, myocardial infarction, peripheral arterial disease, stable angina, stroke, transient ischemic attack, or unstable angina. We extracted characteristics from studies included. For UK estimates based on the EuroQoL 5-dimension (EQ-5D) measure, we assessed risk of bias and applicability to a decision-analytic model, pooled arms/time points as appropriate, and estimated baseline utility multipliers using predicted utility for age- and sex- matched populations without CVD. We sought utility sources from directly applicable studies with low risk of bias, prioritizing plausibility of severity ordering in our base-case model and highest population ascertainment in a sensitivity analysis. RESULTS: Most of the 403 studies identified used EQ-5D (n = 217) and most assessed Organisation for Economic Co-operation and Development populations (n = 262), although measures and countries varied widely. UK studies using EQ-5D (n = 29) produced very heterogeneous baseline utility multipliers for each type of CVD, precluding meta-analysis and implying different possible severity orderings. We could find sources that provided a plausible ordering of utilities while adequately representing health states. CONCLUSIONS: We cataloged international CVD utility estimates and calculated UK-relevant baseline utility multipliers. Modelers should consider unreported sources of heterogeneity, such as population differences, when selecting utility evidence from reviews. HIGHLIGHTS: Published systematic reviews have summarized estimates of utility associated with cardiovascular disease published up to 2013.We 1) reviewed utility estimates for 7 types of cardiovascular disease published since 2013, 2) critically appraised UK-relevant studies, and 3) estimated the effect of each cardiovascular disease on baseline utility.Our review 1) recommends a consistent and reliable set of baseline utility multipliers for 7 types of cardiovascular disease and 2) provides systematically identified reference information for researchers seeking utility evidence for their own context.

Estimating the cost-effectiveness of intermittently scanned continuous glucose monitoring in adults with type 1 diabetes in England.

OBJECTIVE: We previously showed that intermittently scanned continuous glucose monitoring (isCGM) reduces HbA1c at 24 weeks compared with self-monitoring of blood glucose with finger pricking (SMBG) in adults with type 1 diabetes and high HbA1c levels (58-97 mmol/mol [7.5%-11%]). We aim to assess the economic impact of isCGM compared with SMBG. METHODS: Participant-level baseline and follow-up health status (EQ-5D-5L) and within-trial healthcare resource-use data were collected. Quality-adjusted life-years (QALYs) were derived at 24 weeks, adjusting for baseline EQ-5D-5L. Participant-level costs were generated. Using the IQVIA CORE Diabetes Model, economic analysis was performed from the National Health Service perspective over a lifetime horizon, discounted at 3.5%. RESULTS: Within-trial EQ-5D-5L showed non-significant adjusted incremental QALY gain of 0.006 (95% CI: -0.007 to 0.019) for isCGM compared with SMBG and an adjusted cost increase of £548 (95% CI: 381-714) per participant. The lifetime projected incremental cost (95% CI) of isCGM was £1954 (-5108 to 8904) with an incremental QALY (95% CI) gain of 0.436 (0.195-0.652) resulting in an incremental cost-per-QALY of £4477. In all subgroups, isCGM had an incremental cost-per-QALY better than £20,000 compared with SMBG; for people with baseline HbA1c >75 mmol/mol (9.0%), it was cost-saving. Sensitivity analysis suggested that isCGM remains cost-effective if its effectiveness lasts for at least 7 years. CONCLUSION: While isCGM is associated with increased short-term costs, compared with SMBG, its benefits in lowering HbA1c will lead to sufficient long-term health-gains and cost-savings to justify costs, so long as the effect lasts into the medium term.

Development and validation of paired MEDLINE and Embase search filters for cost-utility studies.

BACKGROUND: Search filters are standardised sets of search terms, with validated performance, that are designed to retrieve studies with specific characteristics. A cost-utility analysis (CUA) is the preferred type of economic evaluation to underpin decision-making at the National Institute for Health and Care Excellence (NICE). Until now, when searching for economic evidence for NICE guidelines, we have used a broad set of health economic-related search terms, even when the reviewer's interest is confined to CUAs alone. METHODS: We developed search filters to retrieve CUAs from MEDLINE and Embase. Our aim was to achieve recall of 90% or better across both databases while reducing the overall yield compared with our existing broad economic filter. We used the relative recall method along with topic expert input to derive and validate 3 pairs of filters, assessed by their ability to identify a gold-standard set of CUAs that had been used in published NICE guidelines. We developed and validated MEDLINE and Embase filters in pairs (testing whether, when used together, they find target studies in at least 1 database), as this is how they are used in practice. We examined the proxy-precision of our new filters by comparing their overall yield with our previous approach using publications indexed in a randomly selected year (2010). RESULTS: All 3 filter-pairs exceeded our target recall and led to substantial improvements in search proxy-precision. Our paired 'sensitive' filters achieved 100% recall (95% CI 99.0 to 100%) in the validation set. Our paired 'precise' filters also had very good recall (97.6% [95%CI: 95.4 to 98.9%]). We estimate that, compared with our previous search strategy, using the paired 'sensitive' filters would reduce reviewer screening burden by a factor of 5 and the 'precise' versions would do so by a factor of more than 20. CONCLUSIONS: Each of the 3 paired cost-utility filters enable the identification of almost all CUAs from MEDLINE and Embase from the validation set, with substantial savings in screening workload compared to our previous search practice. We would encourage other researchers who regularly use multiple databases to consider validating search filters in combination as this will better reflect how they use databases in their everyday work.

Immediate birth for women between 34 and 37 weeks of gestation with prolonged preterm prelabour rupture of membranes and detection of vaginal or urine group B streptococcus: an economic evaluation.

OBJECTIVE: What are the costs, benefits and harms of immediate birth compared with expectant management in women with prolonged preterm prelabour rupture of membranes (PPROM) at 34+0 -36+6  weeks of gestation and detection of vaginal or urine group B streptococcus (GBS)? DESIGN: Mathematical decision model comprising three independent decision trees. SETTING: UK National Health Service (NHS) and personal social services perspective. POPULATION: Women testing positive for GBS with PPROM at 34+0 -36+6  weeks of gestation. METHODS: The model estimates lifetime costs and quality-adjusted life years (QALYs) using evidence from randomised trials, UK NHS data sources and further observational studies. Simulated events include neonatal infections, morbidity associated with preterm birth and consequences of caesarean birth. Deterministic and probabilistic sensitivity analyses (PSAs) were performed. MAIN OUTCOME MEASURES: QALYs, costs and incremental cost-effectiveness ratio (ICER). RESULTS: In this population, immediate birth dominates expectant management: it is more effective (average lifetime QALYs, 24.705 versus 24.371) and it is cheaper (average lifetime costs, £14,372 versus £19,311). In one-way sensitivity analysis, results are robust to all but the odds ratio estimating the relative effect on incidence of infections. Threshold analysis shows that the odds of infection only need to be >1.5% with expectant management for the benefit of avoiding infections to outweigh the disadvantages of immediate birth. In PSA, immediate birth is the preferred option in >80% of simulations. CONCLUSIONS: Neonatal GBS infections are expensive to treat and may result in substantial adverse health consequences. Therefore, immediate birth, which is associated with a reduced risk of neonatal infection compared with expectant management, is expected to generate better health outcomes and decreased lifetime costs. TWEETABLE ABSTRACT: For women with preterm prelabour rupture of membranes and group B streptococcus in vaginal or urine samples, immediate birth is associated with improved health in their babies and reduced costs, compared with expectant management.

Evolution of the evidence on the effectiveness and cost-effectiveness of acetylcholinesterase inhibitors and memantine for Alzheimer's disease: systematic review and economic model.

INTRODUCTION: in 2007 the National Institute of Health and Clinical Excellence (NICE) restricted the use of acetylcholinesterase inhibitors and memantine. METHODS: we conducted a health technology assessment (HTA) of the effectiveness and cost-effectiveness of donepezil, galantamine, rivastigmine and memantine for the treatment of AD to re-consider and up-date the evidence base used to inform the 2007 NICE decision. The systematic review of effectiveness targeted randomised controlled trials. A comprehensive search, including MEDLINE, Embase and the Cochrane Library, was conducted from January 2004 to March 2010. All key review steps were done by two reviewers. Random effects meta-analysis was conducted. The cost-effectiveness was assessed using a cohort-based model with three health states: pre-institutionalised, institutionalised and dead. The perspective was NHS and Personal Social Services and the cost year 2009. RESULTS: confidence about the size and statistical significance of the estimates of effect of galantamine, rivastigmine and memantine improved on function and global impact in particular. Cost-effectiveness also changed. For donepezil, galantamine and rivastigmine, the incremental cost per quality-adjusted life year (QALY) in 2004 was above £50,000; in 2010 the same drugs 'dominated' best supportive care (improved clinical outcome at reduced cost). This was primarily because of changes in the modelled costs of introducing the drugs. For memantine, the cost-effectiveness also improved from a range of £37-53,000 per QALY gained to a base-case of £32,000. CONCLUSION: there has been a change in the evidence base between 2004 and 2010 consistent with the change in NICE guidance. Further evolution in cost-effectiveness estimates is possible particularly if there are changes in drug prices.

Patient self-assessment of hospital pain, mood and health-related quality of life in adults with sickle cell disease.

INTRODUCTION: Acute pain is a hallmark of sickle cell disease (SCD) for which frequent hospital admissions may be required, affecting the quality of life of patients. OBJECTIVES: To characterise the relationship between adult patient self-reported sickle cell pain, mood and quality of life during and after hospital admissions. DESIGN: Longitudinal study across three time-points. SETTING: Secondary care, single specialist sickle cell centre. PARTICIPANTS: 510 adult patients with SCD admitted to hospital daycare or inpatient units. OUTCOME MEASURES: Self-assessments of pain, mood and health-related quality of life with health utility (measured on the EQ-5D) on admission, before discharge and at 1-week postdischarge. RESULTS: Mood, general health and quality of life showed significant steady improvements with reduction of pain in patients with SCD on admission to hospital, before discharge and at 1-week follow-up (p<0.01). Health utility scores derived from the EQ-5D showed a negative association with pain in regression analysis over the three time-points. CONCLUSION: Examining health-related quality of life and health utility in relation to pain during hospital admissions is valuable in terms of targeting appropriate psychological interventions within the context of a multidisciplinary approach to managing sickle cell pain. This has implications for healthcare costs.

Cost-effectiveness of dasatinib and nilotinib for imatinib-resistant or -intolerant chronic phase chronic myeloid leukemia.

OBJECTIVES: To estimate the cost-effectiveness of dasatinib and nilotinib compared with high-dose imatinib for people with chronic phase chronic myeloid leukemia, which are resistant to normal-dose imatinib and compared with interferon-α for people intolerant to imatinib, from the perspective of the UK National Health Service. METHODS: An an area under the curve partitioned survival model was developed to estimate the cost-effectiveness of dasatinib and nilotinib. Clinical effectiveness evidence was taken mostly from single-arm trials. RESULTS: Both progression-free survival and overall survival are highly uncertain. In the base case, patients take nilotinib for much less time than dasatinib. Nilotinib is expected to dominate high-dose imatinib, yielding slightly more (0.32) quality-adjusted life years (QALYs) at slightly less cost (£11,100 [pound sterling]) per person. Dasatinib is predicted to provide slightly more (0.53) QALYs at substantially greater cost (£48,900), yielding a very high incremental cost-effectiveness ratio of £91,500 QALY against high-dose imatinib. Cost-effectiveness, however, changes radically under the plausible assumption that the drugs are taken for the same time. For people intolerant to imatinib, nilotinib is expected to yield an incremental cost-effectiveness ratio of £104,700/QALY, and dasatinib £82,600/QALY compared with interferon-α. Further, both drugs represent poor value for money for a range of plausible structural assumptions. CONCLUSIONS: The model should be viewed as an exploratory analysis of the cost-effectiveness of dasatinib and nilotinib because it relies on many assumptions. Whilst clinical data remains immature, the cost-effectiveness of dasatinib and nilotinib for imatinib-resistant people is highly uncertain. Both nilotinib and dasatinib are highly unlikely to be cost-effective versus interferon-α for people intolerant to imatinib.

Carmustine implants for the treatment of newly diagnosed high-grade gliomas: a cost-utility analysis.

BACKGROUND: High-grade gliomas are aggressive brain tumours that are extremely challenging to treat effectively. The intracranial implantation of carmustine wafers (BCNU-W), which delivers chemotherapy directly to the affected area, may prolong survival in this population. However, no attention has yet been paid to the economic implications of BCNU-W in this setting. OBJECTIVE: To investigate the cost effectiveness of BCNU-W as an adjunct to surgery followed by radiotherapy, compared with surgery plus radiotherapy alone. Newly diagnosed, operable grade III and IV gliomas in a population with a mean age of 55 years were considered. METHODS: A Markov cost-utility model was developed in Microsoft Excel, adopting a UK NHS perspective. Transition probabilities and cost data (year 2004 values) were obtained from published literature or expert opinion. The model incorporated utility values, obtained from members of the public, reflecting the quality of life associated with high-grade glioma. The effects of uncertainty were explored through extensive one-way and probabilistic sensitivity analysis. RESULTS: Surgery with the implantation of BCNU-W followed by radiotherapy costs pound sterling 54 500 per additional QALY gained when compared with surgery plus radiotherapy alone. Probabilistic sensitivity analysis shows a <10% probability that BCNU-W would be considered cost effective at a willingness-to-pay threshold of pound sterling 30 000 per QALY. Although model outputs were sensitive to alterations in several key parameters, the incremental cost effectiveness of the intervention remained above pound sterling 30 000 per QALY in all analyses. CONCLUSION: Compared with usual care for the treatment of newly diagnosed high-grade gliomas, BCNU-W is unlikely to be considered a cost-effective use of healthcare resources when judged by the standards commonly adopted in England and Wales. However, the dreadful prognosis of the condition and the paucity of alternative therapies are additional issues that healthcare commissioners may choose to take into account when considering an adoption decision.