Twenty years of network meta-analysis: Continuing controversies and recent developments.

Network meta-analysis (NMA) is an extension of pairwise meta-analysis (PMA) which combines evidence from trials on multiple treatments in connected networks. NMA delivers internally consistent estimates of relative treatment efficacy, needed for rational decision making. Over its first 20 years NMA's use has grown exponentially, with applications in both health technology assessment (HTA), primarily re-imbursement decisions and clinical guideline development, and clinical research publications. This has been a period of transition in meta-analysis, first from its roots in educational and social psychology, where large heterogeneous datasets could be explored to find effect modifiers, to smaller pairwise meta-analyses in clinical medicine on average with less than six studies. This has been followed by narrowly-focused estimation of the effects of specific treatments at specific doses in specific populations in sparse networks, where direct comparisons are unavailable or informed by only one or two studies. NMA is a powerful and well-established technique but, in spite of the exponential increase in applications, doubts about the reliability and validity of NMA persist. Here we outline the continuing controversies, and review some recent developments. We suggest that heterogeneity should be minimized, as it poses a threat to the reliability of NMA which has not been fully appreciated, perhaps because it has not been seen as a problem in PMA. More research is needed on the extent of heterogeneity and inconsistency in datasets used for decision making, on formal methods for making recommendations based on NMA, and on the further development of multi-level network meta-regression.

Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals.

OBJECTIVES: Indirect comparisons via a common comparator (anchored comparisons) are commonly used in health technology assessment. However, common comparators may not be available, or the comparison may be biased due to differences in effect modifiers between the included studies. Recently proposed population adjustment methods aim to adjust for differences between study populations in the situation where individual patient data are available from at least one study, but not all studies. They can also be used when there is no common comparator or for single-arm studies (unanchored comparisons). We aim to characterise the use of population adjustment methods in technology appraisals (TAs) submitted to the United Kingdom National Institute for Health and Care Excellence (NICE). METHODS: We reviewed NICE TAs published between 01/01/2010 and 20/04/2018. RESULTS: Population adjustment methods were used in 7 percent (18/268) of TAs. Most applications used unanchored comparisons (89 percent, 16/18), and were in oncology (83 percent, 15/18). Methods used included matching-adjusted indirect comparisons (89 percent, 16/18) and simulated treatment comparisons (17 percent, 3/18). Covariates were included based on: availability, expert opinion, effective sample size, statistical significance, or cross-validation. Larger treatment networks were commonplace (56 percent, 10/18), but current methods cannot account for this. Appraisal committees received results of population-adjusted analyses with caution and typically looked for greater cost effectiveness to minimise decision risk. CONCLUSIONS: Population adjustment methods are becoming increasingly common in NICE TAs, although their impact on decisions has been limited to date. Further research is needed to improve upon current methods, and to investigate their properties in simulation studies.

A threshold analysis assessed the credibility of conclusions from network meta-analysis.

OBJECTIVE: To assess the reliability of treatment recommendations based on network meta-analysis (NMA). STUDY DESIGN AND SETTING: We consider evidence in an NMA to be potentially biased. Taking each pairwise contrast in turn, we use a structured series of threshold analyses to ask: (1) "How large would the bias in this evidence base have to be before it changed our decision?" and (2) "If the decision changed, what is the new recommendation?" We illustrate the method via two NMAs in which a Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessment for NMAs has been implemented: weight loss and osteoporosis. RESULTS: Four of the weight-loss NMA estimates were assessed as "low" and six as "moderate" quality by GRADE; for osteoporosis, six were "low," nine were "moderate," and 1 was "high." The threshold analysis suggests plausible bias in 3 of 10 estimates in the weight-loss network could have changed the treatment recommendation. For osteoporosis, plausible bias in 6 of 16 estimates could change the recommendation. There was no relation between plausible bias changing a treatment recommendation and the original GRADE assessments. CONCLUSIONS: Reliability judgments on individual NMA contrasts do not help decision makers understand whether a treatment recommendation is reliable. Threshold analysis reveals whether the final recommendation is robust against plausible degrees of bias in the data.

Automated generation of node-splitting models for assessment of inconsistency in network meta-analysis.

Network meta-analysis enables the simultaneous synthesis of a network of clinical trials comparing any number of treatments. Potential inconsistencies between estimates of relative treatment effects are an important concern, and several methods to detect inconsistency have been proposed. This paper is concerned with the node-splitting approach, which is particularly attractive because of its straightforward interpretation, contrasting estimates from both direct and indirect evidence. However, node-splitting analyses are labour-intensive because each comparison of interest requires a separate model. It would be advantageous if node-splitting models could be estimated automatically for all comparisons of interest. We present an unambiguous decision rule to choose which comparisons to split, and prove that it selects only comparisons in potentially inconsistent loops in the network, and that all potentially inconsistent loops in the network are investigated. Moreover, the decision rule circumvents problems with the parameterisation of multi-arm trials, ensuring that model generation is trivial in all cases. Thus, our methods eliminate most of the manual work involved in using the node-splitting approach, enabling the analyst to focus on interpreting the results.

The Cost Effectiveness of Psychological and Pharmacological Interventions for Social Anxiety Disorder: A Model-Based Economic Analysis.

BACKGROUND: Social anxiety disorder is one of the most persistent and common anxiety disorders. Individually delivered psychological therapies are the most effective treatment options for adults with social anxiety disorder, but they are associated with high intervention costs. Therefore, the objective of this study was to assess the relative cost effectiveness of a variety of psychological and pharmacological interventions for adults with social anxiety disorder. METHODS: A decision-analytic model was constructed to compare costs and quality adjusted life years (QALYs) of 28 interventions for social anxiety disorder from the perspective of the British National Health Service and personal social services. Efficacy data were derived from a systematic review and network meta-analysis. Other model input parameters were based on published literature and national sources, supplemented by expert opinion. RESULTS: Individual cognitive therapy was the most cost-effective intervention for adults with social anxiety disorder, followed by generic individual cognitive behavioural therapy (CBT), phenelzine and book-based self-help without support. Other drugs, group-based psychological interventions and other individually delivered psychological interventions were less cost-effective. Results were influenced by limited evidence suggesting superiority of psychological interventions over drugs in retaining long-term effects. The analysis did not take into account side effects of drugs. CONCLUSION: Various forms of individually delivered CBT appear to be the most cost-effective options for the treatment of adults with social anxiety disorder. Consideration of side effects of drugs would only strengthen this conclusion, as it would improve even further the cost effectiveness of individually delivered CBT relative to phenelzine, which was the next most cost-effective option, due to the serious side effects associated with phenelzine. Further research needs to determine more accurately the long-term comparative benefits and harms of psychological and pharmacological interventions for social anxiety disorder and establish their relative cost effectiveness with greater certainty.

Industry sponsorship bias in research findings: a network meta-analysis of LDL cholesterol reduction in randomised trials of statins.

OBJECTIVE: To explore the risk of industry sponsorship bias in a systematically identified set of placebo controlled and active comparator trials of statins. DESIGN: Systematic review and network meta-analysis. ELIGIBILITY: Open label and double blind randomised controlled trials comparing one statin with another at any dose or with control (placebo, diet, or usual care) for adults with, or at risk of developing, cardiovascular disease. Only trials that lasted longer than four weeks with more than 50 participants per trial arm were included. Two investigators assessed study eligibility. DATA SOURCES: Bibliographic databases and reference lists of relevant articles published between 1 January 1985 and 10 March 2013. DATA EXTRACTION: One investigator extracted data and another confirmed accuracy. MAIN OUTCOME MEASURE: Mean absolute change from baseline concentration of low density lipoprotein (LDL) cholesterol. DATA SYNTHESIS: Study level outcomes from randomised trials were combined using random effects network meta-analyses. RESULTS: We included 183 randomised controlled trials of statins, 103 of which were two-armed or multi-armed active comparator trials. When all of the existing randomised evidence was synthesised in network meta-analyses, there were clear differences in the LDL cholesterol lowering effects of individual statins at different doses. In general, higher doses resulted in higher reductions in baseline LDL cholesterol levels. Of a total of 146 industry sponsored trials, 64 were placebo controlled (43.8%). The corresponding number for the non-industry sponsored trials was 16 (43.2%). Of the 35 unique comparisons available in 37 non-industry sponsored trials, 31 were also available in industry sponsored trials. There were no systematic differences in magnitude between the LDL cholesterol lowering effects of individual statins observed in industry sponsored versus non-industry sponsored trials. In industry sponsored trials, the mean change from baseline LDL cholesterol level was on average 1.77 mg/dL (95% credible interval -11.12 to 7.66) lower than the change observed in non-industry sponsored trials. There was no detectable inconsistency in the evidence network. CONCLUSIONS: Our analysis shows that the findings obtained from industry sponsored statin trials seem similar in magnitude as those in non-industry sources. There are actual differences in the effectiveness of individual statins at various doses that explain previously observed discrepancies between industry and non-industry sponsored trials.

Simultaneous multioutcome synthesis and mapping of treatment effects to a common scale.

OBJECTIVES: A new method is presented for both synthesizing treatment effects on multiple outcomes subject to measurement error and estimating coherent mapping coefficients between all outcomes. It can be applied to sets of trials reporting different combinations of patient- or clinician-reported outcomes, including both disease-specific measures and generic health-related quality-of-life measures. It is underpinned by a structural equation model that includes measurement error and latent common treatment effect factor. Treatment effects can be expressed on any of the test instruments that have been used. METHODS: This is illustrated in a synthesis of eight placebo-controlled trials of TNF-α inhibitors in ankylosing spondylitis, each reporting treatment effects on between two and five of a total six test instruments. RESULTS: The method has advantages over other methods for synthesis of multiple outcome data, including standardization and multivariate normal synthesis. Unlike standardization, it allows synthesis of treatment effect information from test instruments sensitive to different underlying constructs. It represents a special case of previously proposed multivariate normal models for evidence synthesis, but unlike the former, it also estimates mappings. Combining synthesis and mapping as a single operation makes more efficient use of available data than do current mapping methods and generates treatment effects that are consistent with the mappings. A limitation, however, is that it can only generate mappings to and from those instruments on which some trial data exist. CONCLUSIONS: The method should be assessed in a wide range of data sets on different clinical conditions, before it can be used routinely in health technology assessment.

Illicit and pharmaceutical drug consumption estimated via wastewater analysis. Part B: placing back-calculations in a formal statistical framework.

Concentrations of metabolites of illicit drugs in sewage water can be measured with great accuracy and precision, thanks to the development of sensitive and robust analytical methods. Based on assumptions about factors including the excretion profile of the parent drug, routes of administration and the number of individuals using the wastewater system, the level of consumption of a drug can be estimated from such measured concentrations. When presenting results from these 'back-calculations', the multiple sources of uncertainty are often discussed, but are not usually explicitly taken into account in the estimation process. In this paper we demonstrate how these calculations can be placed in a more formal statistical framework by assuming a distribution for each parameter involved, based on a review of the evidence underpinning it. Using a Monte Carlo simulations approach, it is then straightforward to propagate uncertainty in each parameter through the back-calculations, producing a distribution for instead of a single estimate of daily or average consumption. This can be summarised for example by a median and credible interval. To demonstrate this approach, we estimate cocaine consumption in a large urban UK population, using measured concentrations of two of its metabolites, benzoylecgonine and norbenzoylecgonine. We also demonstrate a more sophisticated analysis, implemented within a Bayesian statistical framework using Markov chain Monte Carlo simulation. Our model allows the two metabolites to simultaneously inform estimates of daily cocaine consumption and explicitly allows for variability between days. After accounting for this variability, the resulting credible interval for average daily consumption is appropriately wider, representing additional uncertainty. We discuss possibilities for extensions to the model, and whether analysis of wastewater samples has potential to contribute to a prevalence model for illicit drug use.

Risk of pelvic inflammatory disease following Chlamydia trachomatis infection: analysis of prospective studies with a multistate model.

Our objective in this study was to estimate the probability that a Chlamydia trachomatis (CT) infection will cause an episode of clinical pelvic inflammatory disease (PID) and the reduction in such episodes among women with CT that could be achieved by annual screening. We reappraised evidence from randomized controlled trials of screening and controlled observational studies that followed untreated CT-infected and -uninfected women to measure the development of PID. Data from these studies were synthesized using a continuous-time Markov model which takes into account the competing risk of spontaneous clearance of CT. Using a 2-step piecewise homogenous Markov model that accounts for the distinction between prevalent and incident infections, we investigated the possibility that the rate of PID due to CT is greater during the period immediately following infection. The available data were compatible with both the homogenous and piecewise homogenous models. Given a homogenous model, the probability that a CT episode will cause clinical PID was 0.16 (95% credible interval (CrI): 0.06, 0.25), and annual screening would prevent 61% (95% CrI: 55, 67) of CT-related PID in women who became infected with CT. Assuming a piecewise homogenous model with a higher rate during the first 60 days, corresponding results were 0.16 (95% CrI: 0.07, 0.26) and 55% (95% CrI: 32, 72), respectively.

Evidence synthesis for decision making 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis.

When multiple parameters are estimated from the same synthesis model, it is likely that correlations will be induced between them. Network meta-analysis (mixed treatment comparisons) is one example where such correlations occur, along with meta-regression and syntheses involving multiple related outcomes. These correlations may affect the uncertainty in incremental net benefit when treatment options are compared in a probabilistic decision model, and it is therefore essential that methods are adopted that propagate the joint parameter uncertainty, including correlation structure, through the cost-effectiveness model. This tutorial paper sets out 4 generic approaches to evidence synthesis that are compatible with probabilistic cost-effectiveness analysis. The first is evidence synthesis by Bayesian posterior estimation and posterior sampling where other parameters of the cost-effectiveness model can be incorporated into the same software platform. Bayesian Markov chain Monte Carlo simulation methods with WinBUGS software are the most popular choice for this option. A second possibility is to conduct evidence synthesis by Bayesian posterior estimation and then export the posterior samples to another package where other parameters are generated and the cost-effectiveness model is evaluated. Frequentist methods of parameter estimation followed by forward Monte Carlo simulation from the maximum likelihood estimates and their variance-covariance matrix represent'a third approach. A fourth option is bootstrap resampling--a frequentist simulation approach to parameter uncertainty. This tutorial paper also provides guidance on how to identify situations in which no correlations exist and therefore simpler approaches can be adopted. Software suitable for transferring data between different packages, and software that provides a user-friendly interface for integrated software platforms, offering investigators a flexible way of examining alternative scenarios, are reviewed.

Evidence synthesis for decision making 7: a reviewer's checklist.

This checklist is for the review of evidence syntheses for treatment efficacy used in decision making based on either efficacy or cost-effectiveness. It is intended to be used for pairwise meta-analysis, indirect comparisons, and network meta-analysis, without distinction. It does not generate a quality rating and is not prescriptive. Instead, it focuses on a series of questions aimed at revealing the assumptions that the authors of the synthesis are expecting readers to accept, the adequacy of the arguments authors advance in support of their position, and the need for further analyses or sensitivity analyses. The checklist is intended primarily for those who review evidence syntheses, including indirect comparisons and network meta-analyses, in the context of decision making but will also be of value to those submitting syntheses for review, whether to decision-making bodies or journals. The checklist has 4 main headings: A) definition of the decision problem, B) methods of analysis and presentation of results, C) issues specific to network synthesis, and D) embedding the synthesis in a probabilistic cost-effectiveness model. The headings and implicit advice follow directly from the other tutorials in this series. A simple table is provided that could serve as a pro forma checklist.

Dose-comparative effects of different statins on serum lipid levels: a network meta-analysis of 256,827 individuals in 181 randomized controlled trials.

AIMS: The extent to which individual statins vary in terms of their impact on serum lipid levels has been studied mainly on the basis of placebo-controlled trials. Our objective was to review and quantify the dose-comparative effects of different statins on serum lipid levels using both placebo- and active-comparator trials. METHODS: We systematically reviewed randomized trials evaluating different statins in participants with, or at risk of developing, cardiovascular disease. We performed random-effects Bayesian network meta-analyses to quantify the the relative potency of individual statins across all possible dose combinations using both direct and indirect evidence. Dose-comparative effects were determined by estimating the mean change from baseline in serum lipids as compared to control treatment. (systematic review registration: PROSPERO 2011:CRD42011001470). RESULTS: We included 181 placebo-controlled and active-comparator trials including 256,827 individuals. There were 83 two-armed placebo-controlled trials and the remaining 98 were two- or multi-armed active-comparator trials. All statins reduced serum LDL and total cholesterol levels: higher doses resulted in higher reductions in pretreatment LDL and total cholesterol concentrations. In absolute terms, all statins significantly reduced LDL cholesterol levels as compared to control treatment from average baseline levels of approximately 150 mg/dl, except for fluvastatin at ≤20 mg/day and lovastatin at ≤10 mg/day. Atorvastatin, rosuvastatin, and simvastatin were broadly equivalent in terms of their LDL cholesterol-lowering effects. Dose-comparative effects of indivudual statins were not different between those with and without coronary heart disease at baseline. According to meta-regression analyses, LDL cholesterol-lowering effects of individual statins were not impacted by differences across trials in terms of baseline mean age and proportion of women as trial participants. Pretreatment LDL cholesterol concentrations had a marginally statistically significant effect on LDL cholesterol change from baseline. Mean differences from baseline in HDL cholesterol as compared to control treatment was not significant for any statin-dose combination. CONCLUSIONS: The findings of this comprehensive review provide supporting evidence for the dose-response relationship of statins in reducing LDL and total cholesterol. The LDL cholesterol-reducing effects of some statins appear less pronounced than the findings of previous meta-analyses, which is particularly the case for the high-dose formulations of atorvastatin and rosuvastatin. The most consistent evidence for a combined reduction in both LDL and total cholesterol was achieved with atorvastatin at >40 mg/day, rosuvastatin at >10 mg/day, and simvastatin at >40 mg/day, which appear equivalent in terms of their LDL and total cholesterol-reducing effects.

Which health-related quality-of-life outcome when planning randomized trials: disease-specific or generic, or both? A common factor model.

The primary outcomes in trials are usually disease-specific measures (DSMs) designed to be responsive to changes in the condition caused by treatment. For purposes of cost-effectiveness analysis, treatment effects on the DSM are often "mapped" into treatment effects on a generic health-related quality-of-life (QOL) scale, such as EuroQol five-dimensional questionnaire. Trialists have the option of including generic QOL measures as trial outcomes. We consider the relative efficiency (estimate divided by its standard error) of treatment effects derived from the DSM, the generic QOL, the generic QOL indirectly estimated from the mapped DSM, and a pooled estimate combining the direct and indirect information on the generic QOL. By using a "common factor" theory of the relationship between the DSM and the generic QOL, we define the circumstances under which indirectly estimated generic QOL is more efficient than the direct one and when a pooled QOL estimate is more efficient than the DSM estimate. As long as the DSM is more responsive, there is always a threshold sample size above which the indirect estimate has better precision than the direct estimate. This threshold, however, increases as the (1) relative responsiveness ratio of the DSM to the generic QOL increases, (2) precision of the estimated mapping coefficient increases, and (3) true effect becomes smaller. The pooled estimate on the generic QOL may be more efficient than the DSM itself unless the reliability of the DSM is particularly high. Trials powered on DSMs are likely to have sufficient power to detect treatment effect on the generic QOL if a pooled estimate is used. We conclude that generic QOL instruments should be routinely included in randomized controlled trials. Information on mapping coefficients and on relative responsiveness should be collected more systematically to facilitate both evidence synthesis and trial design.

Mixture-of-exponentials models to explain heterogeneity in studies of the duration of Chlamydia trachomatis infection.

Published studies of the duration of asymptomatic Chlamydia trachomatis infection in women have produced diverse estimates, and most reviewers have not attempted an evidence synthesis. We review the designs of duration studies, distinguishing between the incident cases presenting soon after infection in clinic-based studies and prevalent cases ascertained in population screening studies. We combine evidence from all studies under fixed-effect (single clearance rate), random-effect (study-specific clearance rate), and mixture-of-exponentials models, in which there are either two or three classes of infection that clear at different rates. We can identify classes as 'passive' infection and fast-clearing and slow-clearing infections. We estimate models by Bayesian MCMC and compared them using posterior mean residual deviance and the deviance information criterion. The single fixed-effect clearance rate model fitted very poorly. The random-effect model was adequate but inferior to the two-class and three-class mixture of exponentials. According to the two-class model, the proportion in the first class was 23% (95% CI: 16-31%), and the mean duration of C. trachomatis infection is 1.36 years (95% CI: 1.13-1.63 years). With the three-rate model, duration was similar, but identification of the proportions in each class (19%, 31%, and 49%) was poor. Although the random-effect model was descriptively adequate, the extreme degree of between-study variation in the clearance rate it predicted lacked biological plausibility. Differences in study recruitment and sampling mechanisms, acting through a mixture-of-exponentials model, better explains the apparent heterogeneity in duration.

An evaluation of the feasibility, cost and value of information of a multicentre randomised controlled trial of intravenous immunoglobulin for sepsis (severe sepsis and septic shock): incorporating a systematic review, meta-analysis and value of information analysis.

BACKGROUND: Sepsis is a syndrome characterised by a systemic inflammatory response to infection that leads to rapid acute organ failure and potentially rapid decline to death. Intravenous immunoglobulin (IVIG), a blood product derived from human donor blood, has been proposed as an adjuvant therapy for sepsis. OBJECTIVES: To describe current practice in the management of adult patients severely ill with sepsis (severe sepsis or septic shock) in the UK; to assess the clinical effectiveness of IVIG for severe sepsis and septic shock and to obtain the appropriate inputs for the relative efficacy parameters, and the key uncertainties associated with these parameters, required to populate the decision model; to develop a decision-analytic model structure and identify key parameter inputs consistent with the decision problem and relevant to an NHS setting; and to populate the decision model and determine the cost-effectiveness of IVIG and to estimate the value of additional primary research. DATA SOURCES: Existing literature on IVIG and severe sepsis. Existing case-mix and outcome data on critical care admissions. Survey data on management of admissions with severe sepsis. Databases searched for clinical effectiveness were Cochrane Infectious Diseases Group Specialized Trials Register, the Cochrane Trials Register, MEDLINE and EMBASE. Dates searched were 1 January 2002 to 2 October 2009 to update previous Cochrane review. Databases searched for cost-effectiveness were NHS Economic Evaluation Database (NHS EED) to 2 October 2009, MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations and EMBASE to 20 October 2009. REVIEW METHODS: Systematic literature searching with data extraction, descriptive analysis and clinical effectiveness and cost-effectiveness modelling of IVIG in severe sepsis. Additional primary data analysis. Expected value of information (EVI) analysis. RESULTS: Our meta-analysis, the first to simultaneously allow for type of IVIG (IVIG or immunoglobulin M-enriched polyclonal IVIG), choice of control (no treatment or albumin), study quality/publication bias and other potential covariates, indicated that the treatment effect of IVIG on mortality for patients with severe sepsis is borderline significant with a large degree of heterogeneity in treatment effect between individual studies. Modelling indicated that there were issues with bias associated with trial methodology, publication and small-study effects with the current evidence. The large degree of heterogeneity in treatment effects between studies, however, could be explained (best-fitting model) by a measure of study quality (i.e. use of albumin as control - as an indicator of proper blinding to treatment as a proxy for study quality - associated with decreased effect) and duration of IVIG therapy (longer duration associated with increased effect). In-depth discussion within the Expert Group on duration of IVIG therapy, with daily dose and total dose also clearly inter-related, indicated no clear clinical rationale for this association and exposed a lack of evidence on the understanding of the mechanism of action of IVIG in severe sepsis. Although the EVI analyses suggested substantial expected net benefit from a large, multicentre randomised controlled trial (RCT) evaluating the clinical effectiveness of IVIG, the remaining uncertainties around the design of such a study mean that we are unable to recommend it at this time. LIMITATIONS: As has been identified in previous meta-analyses, there are issues with the methodological quality of the available evidence. CONCLUSIONS: Although the results highlight the value for money obtained in conducting further primary research in this area, the biggest limitation for such research regards the uncertainties over the mechanism of action of IVIG and the heterogeneous nature of severe sepsis. Resolving these would allow for better definition of the plausibility of the effectiveness scenarios presented and, consequently, a better understanding of the cost-effectiveness of this treatment. This information would also inform the design of future, primary evaluative research. Our recommendations for future research focus on filling the knowledge gaps to inform a future multicentre RCT prior to recommending its immediate design and conduct. FUNDING: The National Institute for Health Research Health Technology Assessment programme.

Survival time outcomes in randomized, controlled trials and meta-analyses: the parallel universes of efficacy and cost-effectiveness.

OBJECTIVES: Many regulatory agencies require that manufacturers establish both efficacy and cost-effectiveness. The statistical analysis of the randomized, controlled trial (RCT) outcomes should be the same for both purposes. The question addressed by this article is the following: for survival outcomes, what is the relationship between the statistical analyses used to support inference and the statistical model used to support decision making based on cost-effectiveness analysis (CEA)? METHODS: We performed a review of CEAs alongside trials and CEAs based on a synthesis of RCT results, which were submitted to the National Institute for Health and Clinical Excellence (NICE) Technology Appraisal program and included survival outcomes. We recorded the summary statistics and the statistical models used in both efficacy and cost-effectiveness analyses as well as procedures for model diagnosis and selection. RESULTS: In no case was the statistical model for efficacy and CEA the same. For efficacy, relative risks or Cox regression was used. For CEA, the common practice was to fit a parametric model to the control arm, then to apply the hazard ratio from the efficacy analysis to predict the treatment arm. The proportional hazards assumption was seldom checked; the choice of model was seldom based on formal criteria, and uncertainty in model choice was seldom addressed and never propagated through the model. CONCLUSIONS: Both inference and decisions based on CEAs should be based on the same statistical model. This article shows that for survival outcomes, this is not the case. In the interests of transparency, trial protocols should specify a common procedure for model choice for both purposes. Further, the sufficient statistics and the life tables for each arm should be reported to improve transparency and to facilitate secondary analyses of results of RCTs.

Model averaging in the presence of structural uncertainty about treatment effects: influence on treatment decision and expected value of information.

BACKGROUND: Standard approaches to estimation of Markov models with data from randomized controlled trials tend either to make a judgment about which transition(s) treatments act on, or they assume that treatment has a separate effect on every transition. An alternative is to fit a series of models that assume that treatment acts on specific transitions. Investigators can then choose among alternative models using goodness-of-fit statistics. However, structural uncertainty about any chosen parameterization will remain and this may have implications for the resulting decision and the need for further research. METHODS: We describe a Bayesian approach to model estimation, and model selection. Structural uncertainty about which parameterization to use is accounted for using model averaging and we developed a formula for calculating the expected value of perfect information (EVPI) in averaged models. Marginal posterior distributions are generated for each of the cost-effectiveness parameters using Markov Chain Monte Carlo simulation in WinBUGS, or Monte-Carlo simulation in Excel (Microsoft Corp., Redmond, WA). We illustrate the approach with an example of treatments for asthma using aggregate-level data from a connected network of four treatments compared in three pair-wise randomized controlled trials. RESULTS: The standard errors of incremental net benefit using structured models is reduced by up to eight- or ninefold compared to the unstructured models, and the expected loss attaching to decision uncertainty by factors of several hundreds. Model averaging had considerable influence on the EVPI. CONCLUSIONS: Alternative structural assumptions can alter the treatment decision and have an overwhelming effect on model uncertainty and expected value of information. Structural uncertainty can be accounted for by model averaging, and the EVPI can be calculated for averaged models.

Parameterization of treatment effects for meta-analysis in multi-state Markov models.

Standard approaches to analysis of randomized controlled trials (RCTs) using Markov models make it difficult to generalize treatment effects to new patient groups and synthesize evidence across trials. This paper demonstrates how pair-wise and mixed treatment comparison meta-analysis can be applied to event history data for disease progression reported by RCTs. The data, in the form of aggregated discrete time transitions, have a multi-nomial likelihood. In order for evidence synthesis to be performed a structured approach to modelling the differences in the effects of the different treatments must be taken. A multi-state continuous-time Markov model similar to others used in published economic evaluations of asthma treatments is developed, with transition rates related to the likelihood via Kolmogorov's forward equations. The formulation in terms of rates allows a flexible characterization of summary treatment effects. These ideas are applied to an illustrative data set consisting of a set of five trials comparing eight different treatments for asthma. A range of models is developed in which the relative treatment effects act on forward, backward transitions, or both, and models are compared using the DIC. Bayesian inferential techniques are used and the parameters are estimated using MCMC simulation in WinBUGS. An intuitively appealing mechanism of action involving a single parameter acting on all backward transitions was identified for the relative effects of the treatments, which allowed the estimation of a pooled treatment effect, allowing us to rank the different treatment options within each connected evidence network to ascertain which were the most clinically effective.