Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development.

BACKGROUND: The general issue of balancing the value of evidence about the performance of a technology and the value of access to a technology can be seen as central to a number of policy questions. Establishing the key principles of what assessments are needed, as well as how they should be made, will enable them to be addressed in an explicit and transparent manner. OBJECTIVES: The aims of this research are to (1) establish the key principles of what assessments are needed to inform an 'only in research' (OIR) or 'approval with research' (AWR) recommendation, (2) evaluate previous National Institute for Health and Clinical Evidence (NICE) guidance in which OIR or AWR recommendations were made or considered and (3) evaluate a range of alternative options to establish criteria, additional information and/or analysis that could be made available to inform the assessments needed. DATA SOURCES: All NICE draft and final guidance up to January 2010 was considered in the review of NICE technology appraisal guidance. Four case studies were used to evaluate the range of options of what information and analysis could be made available to inform the assessment required. These were based on a reanalysis of existing health technology appraisals for NICE or the Health Technology Assessment programme. REVIEW METHODS: A critical review of policies, practice and literature was undertaken using traditional systematic searching based on initial search terms informed by key publications. An iterative approach was adopted using 'pearl growing' evaluated through capture-recapture methods. In addition, grey literature, policy documents and other sources, such as special interest groups and the expertise of the Advisory Group for the project, were used to contribute to this process. RESULTS: A series of recommendations, or options, for NICE to consider were developed with the involvement of key stakeholders. These establish the key principles and associated criteria that might guide OIR and AWR recommendations and identify what, if any, additional information or analysis might be included in the technology appraisal process, including how such recommendations might be more likely to be implemented through publically funded and sponsored research. To meet these aims the research is broadly structured as follows. A critical review of policy, practice and literature in this area informs the development of a coherent conceptual framework to establish the key principles and the sequence of assessment and judgements required. This sequence of assessment and judgement is represented as an algorithm, which can also be summarised as a simple set of explicit criteria or a 7-point checklist of assessments. A review of previous NICE guidance in which OIR or AWR recommendations were either made or considered was undertaken to examine the extent to which the key principles are evident. The application of the checklist of assessment to a series of four case studies informs considerations of whether or not such assessments can be made based on existing information and analysis in current NICE appraisal and in what circumstances could additional information and/or analysis be useful. Finally, some of the implications that this more explicit assessment of OIR and AWR might have for policy (e.g. NICE guidance and drug pricing), the process of appraisal (e.g. greater involvement of research commissioners) and methods of appraisal (e.g. should additional information, evidence and analysis be required) are drawn together. At each stage this research has been informed by a diverse and international Advisory Group and the feedback from participants at two workshops involving a wide range of key stakeholders, which included members of NICE and its Advisory Committees (including lay members and other NICE programmes), patient advocates, manufacturers, and research and NHS commissioners, as well as relevant academics. LIMITATIONS: Further research is required to establish how these considerations could be integrated within a practical value-based pricing scheme. In addition, irrecoverable opportunity costs are commonly associated with many health technologies that offer future benefits following treatment. The significance of these types of irrecoverable costs is not widely recognised and further research to demonstrate their potential impact more generally is needed. CONCLUSIONS: The categories of guidance available to NICE have a wider application than is reflected in the review of previous guidance. Importantly, determining which category of guidance will be appropriate depends only partly on an assessment of expected cost-effectiveness. As well as AWR for technologies expected to be cost-effective and OIR for those not expected to be cost-effective, there are other important circumstances when OIR should be considered. In particular, for technologies expected to be cost-effective, OIR rather than approve may be appropriate when research is not possible with approval and OIR or even reject, rather than AWR or approve, may be appropriate even if research is possible with approval when there are significant irrecoverable costs. FUNDING: The National Institute for Health Research Health Technology Assessment programme.

What is the value of routinely testing full blood count, electrolytes and urea, and pulmonary function tests before elective surgery in patients with no apparent clinical indication and in subgroups of patients with common comorbidities: a systematic review of the clinical and cost-effective literature.

BACKGROUND: The evidence base which supported the National Institute for Health and Clinical Excellence (NICE) published Clinical Guideline 3 was limited and 50% was graded as amber. However, the use of tests as part of pre-operative work-up remains a low-cost but high-volume activity within the NHS, with substantial resource implications. The objective of this study was to identify, evaluate and synthesise the published evidence on the clinical effectiveness and cost-effectiveness of the routine use of three tests, full blood counts (FBCs), urea and electrolytes tests (U&Es) and pulmonary function tests, in the pre-operative work-up of otherwise healthy patients undergoing minor or intermediate surgery in the NHS. OBJECTIVE: The aims of this study were to estimate the clinical effectiveness and cost-effectiveness of routine pre-operative testing of FBC, electrolytes and renal function and pulmonary function in adult patients classified as American Society of Anaesthesiologists (ASA) grades 1 and 2 undergoing elective minor (grade 1) or intermediate (grade 2) surgical procedures; to compare NICE recommendations with current practice; to evaluate the cost-effectiveness of mandating or withdrawing each of these tests in this patient group; and to identify the expected value of information and whether or not it has value to the NHS in commissioning further primary research into the use of these tests in this group of patients. DATA SOURCES: The following electronic bibliographic databases were searched: (1) BIOSIS; (2) Cumulative Index to Nursing and Allied Health Literature; (3) Cochrane Database of Systematic Reviews; (4) Cochrane Central Register of Controlled Trials; (5) EMBASE; (6) MEDLINE; (7) MEDLINE In-Process & Other Non-Indexed Citations; (8) NHS Database of Abstracts of Reviews of Effects; (9) NBS Health Technology Assessment Database; and (10) Science Citation Index. To identify grey and unpublished literature, the Cochrane Register of Controlled Trials, National Research Register Archive, National Institute for Health Research Clinical Research Network Portfolio database and the Copernic Meta-search Engine were searched. A large routine data set which recorded the results of tests was obtained from Leeds Teaching Hospitals Trust. REVIEW METHODS: A systematic review of the literature was carried out. The searches were undertaken in March to April 2008 and June 2009. Searches were designed to retrieve studies that evaluated the clinical effectiveness and cost-effectiveness of routine pre-operative testing of FBC, electrolytes and renal function and pulmonary function in the above group of patients. A postal survey of current practice in testing patients in this group pre-operatively was undertaken in 2008. An exemplar cost-effectiveness model was constructed to demonstrate what form this would have taken had there been sufficient data. A large routine data set that recorded the results of tests was obtained from Leeds Teaching Hospitals Trust. This was linked to individual patient data with surgical outcomes, and regression models were estimated. RESULTS: A comprehensive and systematic search of both the clinical effectiveness and cost-effectiveness literature identified a large number of potentially relevant studies. However, when these studies were subjected to detailed review and quality assessment, it became clear that the literature provides no evidence on the clinical effectiveness and cost-effectiveness of these specific tests in the specific patient groups. The postal survey had a 17% response rate. Results reported that in ASA grade 1, patients aged < 40 years with no comorbidities undergoing minor surgery did not have routine tests for FBC, electrolytes and renal function and pulmonary function. The results from the regression model showed that the frequency of test use was not consistent with the hypothesis of their routine use. FBC tests were performed in only 58% of patients in the data set and U&E testing was carried out in only 57%. LIMITATIONS: Systematic searches of the clinical effectiveness and cost-effectiveness literature found that there is no evidence on the clinical effectiveness or cost-effectiveness of these tests in this specific clinical context for the NHS. A survey of NHS hospitals found that respondent trusts were implementing current NICE guidance in relation to pre-operative testing generally, and a de novo analysis of routine data on test utilisation and post-operative outcome found that the tests were not be used in routine practice; rather, use was related to an expectation of a more complex clinical case. The paucity of published evidence is a limitation of this study. The studies included relied on non-UK health-care systems data, which may not be transferable. The inclusion of non-randomised studies is associated with an increased risk of bias and confounding. Scoping work to establish the likely mechanism of action by which tests would impact upon outcomes and resource utilisation established that the cause of an abnormal test result is likely to be a pivotal determinant of the cost-effectiveness of a pre-operative test and therefore evaluations would need to consider tests in the context of the underlying risk of specific clinical problems (i.e. risk guided rather than routine use). CONCLUSIONS: The time of universal utilisation of pre-operative tests for all surgical patients is likely to have passed. The evidence we have identified, though weak, indicates that tests are increasingly utilised in patients in whom there is a reason to consider an underlying raised risk of a clinical abnormality that should be taken into account in their clinical management. It is likely that this strategy has led to substantial resource savings for the NHS, although there is not a published evidence base to establish that this is the case. The total expenditure on pre-operative tests across the NHS remains significant. Evidence on current practice indicates that clinical practice has changed to such a degree that the original research question is no longer relevant to UK practice. Future research on the value of these tests in pre-operative work-up should be couched in terms of the clinical effectiveness and cost-effectiveness in the identification of specific clinical abnormalities in patients with a known underlying risk. We suggest that undertaking a multicentre study making use of linked, routinely collected data sets would identify the extent and nature of pre-operative testing in this group of patients. FUNDING: The National Institute for Health Research Health Technology Assessment programme.

Enhanced external counterpulsation for the treatment of stable angina and heart failure: a systematic review and economic analysis.

OBJECTIVES: To determine the clinical effectiveness and cost-effectiveness of enhanced external counterpulsation (EECP) compared with usual care and placebo for refractory stable angina and heart failure, and to undertake analyses of the expected value of information to assess the potential value of future research on EECP. DATA SOURCES: Major electronic databases were searched between November 2007 and March 2008. REVIEW METHODS: A systematic review of the literature was undertaken and a decision model developed to compare EECP treatment with no treatment in adults with chronic stable angina. RESULTS: Five studies were included in the review. In the Multicenter Study of Enhanced External Counterpulsation (MUST-EECP), time to greater than or equal to 1-mm ST segment depression (exercise-induced ischaemia) was statistically significantly improved in the EECP group compared with the control group (sham EECP), mean difference (MD) 41 seconds [95% confidence interval (CI) 9.10-73.90]. However, there was no statistically significant difference between the EECP and control groups in the change in exercise duration from baseline to end of treatment, self-reported angina episodes or daily nitroglycerin use, and the clinical significance of the limited benefits was unclear. There was also a lack of data on long-term outcomes. There were more withdrawals due to adverse events in the EECP group than in the control group, as well as a greater proportion of patients with adverse events [relative risk (RR) 2.13, 95% CI 1.35-3.38]. The three non-randomised studies compared EECP with elective percutaneous coronary intervention (PCI) and usual care. There was a high risk of selection bias in all three studies and the results should be treated with considerable caution. The study comparing an EECP registry with a PCI registry reported similar 1-year all-cause mortality in both groups. In the Prospective Evaluation of EECP in Congestive Heart Failure (PEECH) trial, patients with heart failure were randomised to EECP or to usual care (pharmacotherapy only). At 6 months post treatment, the proportion of patients achieving at least a 60-second increase in exercise duration was higher in the EECP group (RR 1.39, 95% CI 0.89-2.16), but the proportion with an improvement in peak VO2 was similar in both groups. The clinical significance of this is unclear. The proportion of patients in the EECP group with an improvement in New York Heart Association classification was higher (RR 2.25, 95% CI 1.25-4.06) at 6 months, as was mean exercise duration, MD 34.6 (95% CI -4.86 to 74.06). There were more withdrawals in the EECP group than in the control group as a result of adverse events (RR 1.05, 95% CI 0.67-1.66). There were limitations in the generalisability of results of the trial and, again, a lack of data on long-term outcomes. The review of cost-effectiveness evidence found only one unpublished study but demonstrated that the long-term maintenance of quality of life benefits of EECP is central to the estimate of its cost-effectiveness. The incremental cost-effectiveness ratio of EECP was 18,643 pounds for each additional quality-adjusted life-year (QALY), with a probability of being cost-effective of 0.44 and 0.70 at cost-effectiveness thresholds of 20,000 pounds and 30,000 pounds per QALY gained respectively. Results were sensitive to the duration of health-related quality of life (HRQoL) benefits from treatment. CONCLUSIONS: The results from a single randomised controlled trial (MUST-EECP) do not provide firm evidence of the clinical effectiveness of EECP in refractory stable angina or in heart failure. High-quality studies are required to investigate the benefits of EECP, whether these outweigh the common adverse effects and its long-term cost-effectiveness in terms of quality of life benefits.

Not a NICE fallacy: a reply to Dr Quigley.

A repudiation of Muireann Quigley's argument that the National Institute for Health and Clinical Excellence (NICE) values and assesses the worth of people's lives; together with an alternative account of what it appears that NICE actually does, why these procedures are not unreasonable and some of the unresolved problems, especially when making interpersonal comparisons of health, which remain for NICE or, indeed, anyone seeking to determine the contents of the benefits bundles of a public health insurance programme such as the NHS. Some other ethically dubious propositions by Dr Quigley are also rejected.

The cost-effectiveness of etanercept and infliximab for the treatment of patients with psoriatic arthritis.

OBJECTIVE: Tumour necrosis factor (TNF) antagonists have been shown to improve the outcomes in patients with rheumatoid arthritis (RA) and psoriatic arthritis (PsA). We assess the cost-effectiveness of two TNF antagonists and so-called 'palliative care' for the treatment of active PsA from the perspective of the UK National Health Service (NHS). METHODS: Bayesian statistical methods were used to synthesize evidence from three Phase III trials, identified through a systematic review, and estimate the relative efficacy of etanercept, infliximab and palliative care. A probabilistic decision analytic model was then used to compare these treatments after the failure of at least two conventional disease-modifying anti-rheumatic drugs (DMARDs), following the British Society for Rheumatology (BSR) guidelines for use. The primary outcome measure, quality-adjusted life years (QALYs), was derived from utility values estimated as a function of disability measured by the Health Assessment Questionnaire (HAQ). The deterioration experienced in HAQ at treatment withdrawal (rebound) was incorporated using alternative scenarios to represent best- and worst-case assumptions. The model was extended beyond the trial duration to a 10-yr and lifetime horizon, using available evidence and expert opinion-based assumptions on disease progression. Resource utilization was based on literature, national databases and expert opinion. Prices were obtained from routine NHS sources and published literature. RESULTS: At a 10-yr time horizon, the incremental cost-effectiveness ratio (ICER) for etanercept compared with palliative care was pound sterling26 361 per QALY gained for the best-case rebound scenario, which increased to pound sterling30 628 for the worst-case. The ICERs for infliximab compared with etanercept were pound sterling165 363 and pound sterling205 345 per QALY, respectively. These findings are mainly explained by the fact that infliximab has higher acquisition and administration costs without substantially superior effectiveness compared with etanercept. Results were sensitive to estimates of rebound assumptions at withdrawal and the time horizon. CONCLUSIONS: Only results for etanercept remained within the range of cost-effectiveness estimates considered to represent value for money in the NHS by the National Institute for Health and Clinical Excellence. Further research appears most valuable in relation to the short-term effectiveness, utility parameters and assumptions regarding the effect of rebound.

Prenatal screening and treatment strategies to prevent group B streptococcal and other bacterial infections in early infancy: cost-effectiveness and expected value of information analyses.

OBJECTIVES: To determine the cost-effectiveness of prenatal strategies for preventing group B streptococci (GBS) and other serious bacterial infections in early infancy and to establish the expected value of further information. DATA SOURCES: Electronic databases were searched up to March 2006. Expert opinion was also sought. REVIEW METHODS: Twelve mutually exclusive maternal risk groups were defined at presentation in labour and the consequences considered of early-onset GBS and non-GBS bacterial infections and late onset GBS infection, measured in terms of lifetime NHS costs and quality-adjusted life-years (QALYs). These were for preterm delivery (<37 weeks): (1) planned Caesarean section, (2) previous baby with GBS disease, (3) positive urine or vaginal swab for GBS in current pregnancy, (4) fever >or=38 degrees C during labour, (5) membrane rupture >or=2 hours before labour onset, (6) membrane rupture <2 hours before labour onset. For term delivery (>or=37 weeks): (7) planned Caesarean section, (8) previous baby with GBS disease, (9) positive urine or vaginal swab for GBS in current pregnancy, (10) fever >or=38 degrees C during labour, (11) membrane rupture >or=18 hours, and (12) none of the above risk factors. Fourteen intervention strategies were applied to each maternal risk group. Data inputs were obtained from systematic reviews, primary data and expert opinion. The model parameters were simultaneously estimated from the data inputs using Bayesian evidence synthesis. The expected net benefit was calculated relative to no intervention for each intervention within each risk group for two scenarios, with and without vaccination. Interventions with more than a 1% probability of being cost-effective (i.e. maximising net benefit at a threshold of 25,000 pounds per QALY gained) in a specific risk group were combined to form strategies. To limit antibiotic exposure, women who were low risk at presentation could not be treated without a positive culture or polymerase chain reaction result. RESULTS: Current best practice, comprising intravenous treatment for pyrexia, previous GBS baby and previous GBS swab or urine culture, and oral treatment for preterm pre-labour membrane rupture (groups 2-5 and 8-10) was not cost-effective. All cost-effective options involved treatment of all preterm groups and high-risk term groups (groups 8-10). Testing high-risk women for maternal GBS colonisation would not be cost-effective, as even those with negative results would be better off treated to reduce the risk of early-onset non-GBS infection. In the absence of vaccination, culture-based testing of women in groups 11 and 12, combined with treatment for the rest, would be the most cost-effective strategy. If vaccination was available, vaccination for all and treatment for groups 1-10 would be marginally more cost-effective than treatment for groups 1-10 and culture for groups 11 and 12, but this is uncertain and is based on expert opinion on vaccine efficacy. The expected value of perfect information results suggest that moderate investment in research would be worthwhile. CONCLUSIONS: Based on our findings, immediate extension of current practice to treat all preterm and high-risk term groups would be beneficial. Further research aimed at the realisation of a GBS vaccine should be prioritized.

Etanercept and infliximab for the treatment of psoriatic arthritis: a systematic review and economic evaluation.

OBJECTIVES: To evaluate the clinical effectiveness, safety, tolerability and cost-effectiveness of etanercept and infliximab for the treatment of active and progressive psoriatic arthritis (PsA) in patients who have inadequate response to standard treatment, including disease-modifying antirheumatic drug (DMARD) therapy. DATA SOURCES: Electronic databases were searched up to July 2004. REVIEW METHODS: A systematic review evaluated the clinical efficacy and adverse effects of etanercept and infliximab. The efficacy of DMARDs in the treatment of PsA was also reviewed and treatments were compared using Bayesian evidence synthesis methods. Following evaluation of existing economic evaluations of etanercept and infliximab in PsA, a new economic model was developed (the York Model). This utilised the results from the evidence synthesis and data from a range of other sources. RESULTS: Across the two trials, at 12 weeks, around 65% of patients treated with etanercept achieved an American College of Rheumatology (ACR) 20 {pooled relative risk (RR) 4.19 [95% confidence interval (CI) 2.74 to 6.42]}, demonstrating a basic degree of efficacy in terms of arthritis-related symptoms. In addition, around 45% of patients treated with etanercept achieved an ACR 50 [pooled RR 10.84 (95% CI 4.47 to 26.28)] and around 12% achieved an ACR 70 [pooled RR 16.28 (95% CI 2.20 to 120.54)], demonstrating a good level of efficacy. The subgroup analyses conducted in one trial revealed that the effect of etanercept was not dependent upon patients' concomitant use of methotrexate. In addition, almost 85% of patients treated with etanercept achieved a Psoriatic Arthritis Response Criteria (PsARC) [pooled RR 2.60 (95% CI 1.96 to 3.45). The Psoriatic Area and Severity Index (PASI) results indicate some beneficial effect on psoriasis at 12 weeks; however, the data are sparse. The statistically significant reduction (improvement) in Health Assessment Questionnaire (HAQ) score with etanercept compared with placebo indicates a beneficial effect of etanercept on function. Similar results were seen at 24 weeks, except that the results for PASI 75 and PASI 50 now achieved statistical significance and data for Total Sharp Score annualised rate of progression were available; this was statistically significantly lower in etanercept-treated patients than in placebo-treated patients. Uncontrolled follow-up of patients indicates that treatment benefit may be maintained for at least 50 weeks. At 16 weeks, 65% of patients treated with infliximab achieved an ACR 20 [RR 6.80 (95% CI 2.89 to 16.01)], demonstrating a basic degree of efficacy in terms of arthritis-related symptoms. This level of efficacy was not dependent upon patients' concomitant use of methotrexate. Almost half the patients treated with infliximab achieved an ACR 50 [RR 49.00 (95% CI 3.06 to 785.06)] and over one-quarter achieved an ACR 70 [RR 31.00 (95% CI 1.90 to 504.86)] compared with none of the placebo group, demonstrating a good level of efficacy. In addition, 75% of patients treated with infliximab achieved a PsARC [RR 3.55 (95% CI 2.05 to 6.13)]. The beneficial treatment effect on psoriasis was also statistically significant with a mean difference in percentage change from baseline in PASI of -5 (95% CI -6.8 to -3.3), as was the percentage improvement from baseline in HAQ score with infliximab compared with placebo [mean difference 51.4 (95% CI 48.08 to 54.72)], indicating a beneficial effect of infliximab on functional status. Uncontrolled data from all measures of joint disease, psoriasis and HAQ collected up to 50 weeks of follow-up reflect those at 16 weeks. There were no radiographic assessments, so nothing can be determined about the potential or otherwise of infliximab to delay the progression of joint disease. Using the York cost-effectiveness model, infliximab was consistently dominated by etanercept because of its higher acquisition and administration costs without superior effectiveness. The incremental cost per quality-adjusted life-year (QALY) gained of etanercept compared with palliative care ranged from 14,818 pounds (females, 40-year time horizon) to 49,374 pounds (males, 1-year time horizon) if it is assumed that, when patients eventually fail on biological therapy, their disability (in terms of HAQ score) deteriorates by the same amount as it improved when they initially respond to treatment (rebound equal to gain). Results for etanercept ranged from 25,443 pounds (females, 40-year time horizon) to 49,441 pounds (males, 1-year time horizon) per QALY gained under the assumption that, when patients fail on therapy, their disability level returns to what it would have been had they never responded (rebound equal to natural history). CONCLUSIONS: The limited data available indicated that etanercept and infliximab are efficacious in the treatment of PsA with beneficial effects on both joint and psoriasis symptoms and on functional status. Short-term data indicated that etanercept can delay joint disease progression, but long-term data are needed. There are no controlled data as yet to indicate that infliximab can delay joint disease progression. Treatment with both etanercept and infliximab for 12 weeks demonstrated a significant degree of efficacy, with no statistically significant difference between them. For both drugs, adverse events were common with mild injection/infusion reactions being the main treatment-related effect. The York model indicated that etanercept is more cost-effective than infliximab as it has a lower cost with little difference in outcomes. The cost-effectiveness of etanercept is also sensitive to assumptions made about the extent of disease progression when patients are responding to therapy. The number of years for which a patient can be safely on biologicals is uncertain so these results should be considered with caution. Further research should include long-term controlled trials to confirm benefits, review adverse events and to explore further the implications of biologic therapy.

Wickedness or folly? The ethics of NICE's decisions.

A rebuttal is provided to each of the arguments adduced by John Harris, an Editor-in-Chief of the Journal of Medical Ethics, in two editorials in the journal in support of the view that National Institute for Health and Clinical Excellence's procedures and methods for making recommendations about healthcare procedures for use in the National Health Service in England and Wales are the product of "wickedness or folly or more likely both", "ethically illiterate as well as socially divisive", responsible for the "perversion of science as well as of morality" and are "contrary to basic morality and contrary to human rights".

A series of systematic reviews to inform a decision analysis for sampling and treating infected diabetic foot ulcers.

OBJECTIVES: To review systematically the evidence on the performance of diagnostic tests used to identify infection in diabetic foot ulcers (DFUs) and of interventions to treat infected DFUs. To use estimates derived from the systematic reviews to create a decision analytic model in order to identify the most effective method of diagnosing and treating infection and to identify areas of research that would lead to large reductions in clinical uncertainty. DATA SOURCES: Electronic databases covering period from inception of the database to November 2002. REVIEW METHODS: Selected studies were assessed against validated criteria and described in a narrative review. The structure of a decision analytic model was derived for two groups of patients in whom diagnostic tests were likely to be used. RESULTS: Three studies that investigated the performance of diagnostic tests for infection on populations including people with DFUs found that there was no evidence that single items on a clinical examination checklist were reliable in identifying infection in DFUs, that wound swabs perform poorly against wound biopsies, and that semi-quantitative analysis of wound swabs may be a useful alternative to quantitative analysis. However, few people with DFUs were included, so it was not possible to tell whether diagnostic performance differs for DFUs relative to wounds of other aetiologies. Twenty-three studies investigated the effectiveness (n = 23) or cost-effectiveness (n = 2) of antimicrobial agents for DFUs. Eight studied intravenous antibiotics, five oral antibiotics, four different topical agents such as dressings, four subcutaneous granulocyte colony stimulating factor (G-CSF), one evaluated oral and topical Ayurvedic preparations and one compared topical sugar versus antibiotics versus standard care. The majority of trials were underpowered and were too dissimilar to be pooled. There was no strong evidence for recommending any particular antimicrobial agent for the prevention of amputation, resolution of infection or ulcer healing. Topical pexiganan cream may be as effective as oral antibiotic treatment with ofloxacin for the resolution of local infection. Ampicillin and sulbactam were less costly than imipenem and cilastatin, a growth factor (G-CSF) was less costly than standard care and cadexomer iodine dressings may be less costly than daily dressings. A decision analytic model was derived for two groups of people, those for whom diagnostic testing would inform treatment--people with ulcers which do not appear infected but whose ulcer is not progressing despite optimal concurrent treatment--and those in whom a first course of antibiotics (prescribed empirically) have failed. There was insufficient information from the systematic reviews or interviews with experts to populate the model with transition probabilities for the sensitivity and specificity of diagnosis of infection in DFUs. Similarly, there was insufficient information on the probabilities of healing, amputation or death in the intervention studies for the two populations of interest. Therefore, we were unable to run the model to inform the most effective diagnostic and treatment strategy. CONCLUSIONS: The available evidence is too weak to be able to draw reliable implications for practice. This means that, in terms of diagnosis, infection in DFUs cannot be reliably identified using clinical assessment. This has implications for determining which patients need formal diagnostic testing for infection, on whether empirical treatment with antibiotics (before the results of diagnostic tests are available) leads to better outcomes, and on identifying the optimal methods of diagnostic testing. With respect to treatment, it is not known whether treatment with systemic or local antibiotics leads to better outcomes or whether any particular agent is more effective. Limited evidence suggests that both G-CSF and cadexomer iodine dressings may be less expensive than 'standard' care, that ampicillin/sulbactam may be less costly than imipenem/cilastatin, and that an unlicensed cream (pexiganan) may be as effective as oral ofloxacin. Further research is needed to ascertain the characteristics of infection in people with DFUs that influence healing and amputation outcomes, to determine whether detecting infection prior to treatment offers any benefit over empirical therapy, and to establish the most effective and cost-effective methods for detecting infection, as well as the relative effectiveness and cost-effectiveness of antimicrobial interventions for DFU infection.

Review of guidelines for good practice in decision-analytic modelling in health technology assessment.

OBJECTIVES: To identify existing guidelines and develop a synthesised guideline plus accompanying checklist. In addition to provide guidance on key theoretical, methodological and practical issues and consider the implications of this research for what might be expected of future decision-analytic models. DATA SOURCES: Electronic databases. REVIEW METHODS: A systematic review of existing good practice guidelines was undertaken to identify and summarise guidelines currently available for assessing the quality of decision-analytic models that have been undertaken for health technology assessment. A synthesised good practice guidance and accompanying checklist was developed. Two specific methods areas in decision modelling were considered. The first method's topic is the identification of parameter estimates from published literature. Parameter searches were developed and piloted using a case-study model. The second topic relates to bias in parameter estimates; that is, how to adjust estimates of treatment effect from observational studies where there are risks of selection bias. A systematic literature review was conducted to identify those studies looking at quantification of bias in parameter estimates and the implication of this bias. RESULTS: Fifteen studies met the inclusion criteria and were reviewed and consolidated into a single set of brief statements of good practice. From this, a checklist was developed and applied to three independent decision-analytic models. Although the checklist provided excellent guidance on some key issues for model evaluation, it was too general to pick up on the specific nuances of each model. The searches that were developed helped to identify important data for inclusion in the model. However, the quality of life searches proved to be problematic: the published search filters did not focus on those measures specific to cost-effectiveness analysis and although the strategies developed as part of this project were more successful few data were found. Of the 11 studies meeting the criteria on the effect of selection bias, five concluded that a non-randomised trial design is associated with bias and six studies found 'similar' estimates of treatment effects from observational studies or non-randomised clinical trials and randomised controlled trials (RCTs). One purpose of developing the synthesised guideline and checklist was to provide a framework for critical appraisal by the various parties involved in the health technology assessment process. First, the guideline and checklist can be used by groups that are reviewing other analysts' models and, secondly, the guideline and checklist could be used by the various analysts as they develop their models (to use it as a check on how they are developing and reporting their analyses). The Expert Advisory Group (EAG) that was convened to discuss the potential role of the guidance and checklist felt that, in general, the guidance and checklist would be a useful tool, although the checklist is not meant to be used exclusively to determine a model's quality, and so should not be used as a substitute for critical appraisal. CONCLUSIONS: The review of current guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, in other areas conflicting attributes are presented in different guidelines. In general, the checklist appears to perform well, in terms of identifying those aspects of the model that should be of particular concern to the reader. The checklist cannot, however, provide answers to the appropriateness of the model structure and structural assumptions, as these may be seen as a general problem with generic checklists and do not reflect any shortcoming with the synthesised guidance and checklist developed here. The assessment of the checklist, as well as feedback from the EAG, indicated the importance of its use in conjunction with a more general checklist or guidelines on economic evaluation. Further methods research into the following areas would be valuable: the quantification of selection bias in non-controlled studies and in controlled observational studies; the level of bias in the different non-RCT study designs; a comparison of results from RCTs with those from other non-randomised studies; assessment of the strengths and weaknesses of alternative ways to adjust for bias in a decision model; and how to prioritise searching for parameter estimates.

A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme.

OBJECTIVES: To demonstrate the benefits of using appropriate decision-analytic methods and value of information analysis (DA-VOI). Also to establish the feasibility and implications of applying these methods to inform the prioritisation process of the NHS Health Technology Assessment (HTA) programme, and possibly extending their use therein. DATA SOURCES: Three research topics that were considered by the HTA panels in the September 2002 and February 2003 prioritisation rounds. REVIEW METHODS: A brief and non-technical overview of DA-VOI methods was circulated to the panels and Prioritisation Strategy Group (PSG). For each case study the results were presented to the panels and the PSG in the form of brief case-study reports. Feedback on the DA-VOI analysis and its presentation was obtained in the form of completed questionnaires from panel members, and reports from panel senior lecturers and PSG members. RESULTS: Although none of the research topics identified met all of the original selection criteria for inclusion as case studies in the pilot, it was possible to construct appropriate decision-analytic models and conduct probabilistic analysis for each topic. In each case, the tasks were completed within the time-frame required by the existing HTA research prioritisation process. The brief case-study reports provided a description of the decision problem, a summary of the current evidence base and a characterisation of decision uncertainty in the form of cost-effectiveness acceptability curves. Estimates of value of information for the decision problem were presented for relevant patient groups and clinical settings, as well as the value of information associated with particular model inputs. The implications for the value of research in each of the areas were presented in general terms. Details were also provided on what the analysis suggested regarding the design of any future research in terms of features such as the relevant patient groups and comparators, and whether experimental design was likely to be required. CONCLUSIONS: The pilot study showed that, even with very short timelines, it is possible to undertake DA-VOI that can feed into the priority-setting process that has been developed for the HTA programme. There are however a number of areas that need to be established at the beginning of the process, such as clarification of the nature of the decision problem for which additional research is being considered, explicitness about which existing data should be used and how data that exhibit particular weaknesses should be down-weighted in the analysis. Other areas, including optimum application of researcher time, integrating the vignette (a summary of the clinical problem and existing evidence) and the use of DA-VOI, training, use of sensitivity analyses, and deployment of clinical expertise, are also considered in terms of the potential implementation of DA-VOI within the HTA programme. Recommendations for further research include how literature searching should focus on those variables to which the model's results are most sensitive and with the highest expected value of perfect information; methods of evidence synthesis (multiple parameter synthesis) to consider the evidence surrounding multiple comparators and networks of evidence; and ways in which the value of sample information can be used by the NHS HTA programme and other research funders to decide on the most efficient design of new evaluative research. There is also a need for an analytical framework to be developed that can jointly address the question of whether additional resources would better be devoted to additional research or interventions to change clinical practice.

Expected value of sample information calculations in medical decision modeling.

There has been an increasing interest in using expected value of information (EVI) theory in medical decision making, to identify the need for further research to reduce uncertainty in decision and as a tool for sensitivity analysis. Expected value of sample information (EVSI) has been proposed for determination of optimum sample size and allocation rates in randomized clinical trials. This article derives simple Monte Carlo, or nested Monte Carlo, methods that extend the use of EVSI calculations to medical decision applications with multiple sources of uncertainty, with particular attention to the form in which epidemiological data and research findings are structured. In particular, information on key decision parameters such as treatment efficacy are invariably available on measures of relative efficacy such as risk differences or odds ratios, but not on model parameters themselves. In addition, estimates of model parameters and of relative effect measures in the literature may be heterogeneous, reflecting additional sources of variation besides statistical sampling error. The authors describe Monte Carlo procedures for calculating EVSI for probability, rate, or continuous variable parameters in multi parameter decision models and approximate methods for relative measures such as risk differences, odds ratios, risk ratios, and hazard ratios. Where prior evidence is based on a random effects meta-analysis, the authors describe different ESVI calculations, one relevant for decisions concerning a specific patient group and the other for decisions concerning the entire population of patient groups. They also consider EVSI methods for new studies intended to update information on both baseline treatment efficacy and the relative efficacy of 2 treatments. Although there are restrictions regarding models with prior correlation between parameters, these methods can be applied to the majority of probabilistic decision models. Illustrative worked examples of EVSI calculations are given in an appendix.

Representing uncertainty: the role of cost-effectiveness acceptability curves.

Decision-making in health care is inevitably undertaken in a context of uncertainty concerning the effectiveness and costs of health care interventions and programmes. One method that has been suggested to represent this uncertainty is the cost-effectiveness acceptability curve. This technique, which directly addresses the decision-making problem, has advantages over confidence interval estimation for incremental cost-effectiveness ratios. However, despite these advantages, cost-effectiveness acceptability curves have yet to be widely adopted within the field of economic evaluation of health care technologies. In this paper we consider the relationship between cost-effectiveness acceptability curves and decision-making in health care, suggest the introduction of a new concept more relevant to decision-making, that of the cost-effectiveness frontier, and clarify the use of these techniques when considering decisions involving multiple interventions. We hope that as a result we can encourage the greater use of these techniques.

Bayesian value-of-information analysis. An application to a policy model of Alzheimer's disease.

A framework is presented that distinguishes the conceptually separate decisions of which treatment strategy is optimal from the question of whether more information is required to inform this choice in the future. The authors argue that the choice of treatment strategy should be based on expected utility, and the only valid reason to characterize the uncertainty surrounding outcomes of interest is to establish the value of acquiring additional information. A Bayesian decision theoretic approach is demonstrated through a probabilistic analysis of a published policy model of Alzheimer's disease. The expected value of perfect information is estimated for the decision to adopt a new pharmaceutical for the population of patients with Alzheimer's disease in the United States. This provides an upper bound on the value of additional research. The value of information is also estimated for each of the model inputs. This analysis can focus future research by identifying those parameters where more precise estimates would be most valuable and indicating whether an experimental design would be required. We also discuss how this type of analysis can also be used to design experimental research efficiently (identifying optimal sample size and optimal sample allocation) based on the marginal cost and marginal benefit of sample information. Value-of-information analysis can provide a measure of the expected payoff from proposed research, which can be used to set priorities in research and development. It can also inform an efficient regulatory framework for new healthcare technologies: an analysis of the value of information would define when a claim for a new technology should be deemed substantiated and when evidence should be considered competent and reliable when it is not cost-effective to gather any more information.

The FDA's regulation of health economic information.

Section 114 of the Food and Drug Administration Modernization Act of 1997 was intended to increase the flow of health economic information from pharmaceutical manufacturers to managed care decisionmakers. But the legislation raises a host of complex questions and has provoked diverse opinions from inside and outside the pharmaceutical industry. Moreover, the Food and Drug Administration (FDA) has yet to issue interpretative guidance on the subject. The challenge in implementing Section 114 lies in developing a policy that improves health economic information exchange while protecting consumers from misleading claims and preserving incentives for manufacturers to conduct rigorous studies.

Assessing quality in decision analytic cost-effectiveness models. A suggested framework and example of application.

Despite the growing use of decision analytic modelling in cost-effectiveness analysis, there is a relatively small literature on what constitutes good practice in decision analysis. The aim of this paper is to consider the concept of 'validity' and 'quality' in this area of evaluation, and to suggest a framework by which quality can be demonstrated on the part of the analyst and assessed by the reviewer and user. The paper begins by considering the purpose of cost-effectiveness models and argues that the their role is to identify optimum treatment decisions in the context of uncertainty about future states of the world. The issue of whether such models can be defined as 'scientific' is considered. The notion that decision analysis undertaken at time t can only be considered scientific if its outputs closely predict the results of a trial undertaken at time t + 1 is rejected as this ignores the need to make decisions on the basis of currently available evidence. Rather, the scientific characteristic of decision models is based on the fact that, in principle at least, such analyses can be falsified by comparison of two states of the world, one where resource allocation decisions are based on formal decision analysis and the other where such decisions are not. This section of the paper also rejects the idea of exact codification of scientific method in general, and of decision analysis in particular, as this risks rejecting potentially valuable models, may discourage the development of novel methods and can distort research priorities. However, the paper argues that it is both possible and necessary to develop a framework for assessing quality in decision models. Building on earlier work, various dimensions of quality in decision modelling are considered: model structure (disease states, options, time horizon and cycle length); data (identification, incorporation, handling uncertainty); and consistency (internal and external). Within this taxonomy a (nonexhaustive) list of questions about quality is suggested which are illustrated by their application to a specific published model. The paper argues that such a framework can never be prescriptive about every aspect of decision modelling. Rather, it should encourage the analyst to provide an explicit and comprehensive justification of their methods, and allow the user of the model to make an informed judgment about the relevance, coherence and usefulness of the analysis.

The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies.

The literature which considers the statistical properties of cost-effectiveness analysis has focused on estimating the sampling distribution of either an incremental cost-effectiveness ratio or incremental net benefit for classical inference. However, it is argued here that rules of inference are arbitrary and entirely irrelevant to the decisions which clinical and economic evaluations claim to inform. Decisions should be based only on the mean net benefits irrespective of whether differences are statistically significant or fall outside a Bayesian range of equivalence. Failure to make decisions in this way by accepting the arbitrary rules of inference will impose costs which can be measured in terms of resources or health benefits forgone. The distribution of net benefit is only relevant to deciding whether more information is required. A framework for decision making and establishing the value of additional information is presented which is consistent with the decision rules in CEA. This framework can distinguish the simultaneous but conceptually separate steps of deciding which alternatives should be chosen, given existing information, from the question of whether more information should be acquired. It also ensures that the type of information acquired is driven by the objectives of the health care system, is consistent with the budget constraint on service provision and that research is designed efficiently.