To adjust or not to adjust in living systematic reviews? It's all about the context.

With each update of meta-analyses from living systematic reviews, treatment effects and their confidence intervals are recalculated. This often raises the question whether or not multiplicity is an issue and whether a method to adjust for multiplicity is needed. It seems that answering these questions is not that straightforward. We approach this matter by considering the context of systematic reviews and pointing out existing methods for handling multiplicity in meta-analysis. We conclude that multiplicity is not a relevant issue in living systematic reviews when they are planned with the aim to provide up-to-date evidence, without any direct control on the decision over future research. Multiplicity might be an issue, though, in living systematic reviews designed under a protocol involving a "stopping decision", which can be the case in living guideline development or in reimbursement decisions. Several appropriate methods exist for handling multiplicity in meta-analysis. Existing methods, however, are also associated with several technical and conceptual limitations, and could be improved in future methodological projects. To better decide whether an adjustment for multiplicity is necessary at all, authors and users of living systematic reviews should be aware of the context of the work and question whether there is a dependency between the effect estimates of the living systematic review and its stopping/updating or an influence on future research.

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study.

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was - 0.09 (95% CI - 0.26 to 0.09, PI - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

When does the placebo effect have an impact on network meta-analysis results?

The placebo effect is the 'effect of the simulation of treatment that occurs due to a participant's belief or expectation that a treatment is effective'. Although the effect might be of little importance for some conditions, it can have a great role in others, mostly when the evaluated symptoms are subjective. Several characteristics that include informed consent, number of arms in a study, the occurrence of adverse events and quality of blinding may influence response to placebo and possibly bias the results of randomised controlled trials. Such a bias is inherited in systematic reviews of evidence and their quantitative components, pairwise meta-analysis (when two treatments are compared) and network meta-analysis (when more than two treatments are compared). In this paper, we aim to provide red flags as to when a placebo effect is likely to bias pairwise and network meta-analysis treatment effects. The classic paradigm has been that placebo-controlled randomised trials are focused on estimating the treatment effect. However, the magnitude of placebo effect itself may also in some instances be of interest and has also lately received attention. We use component network meta-analysis to estimate placebo effects. We apply these methods to a published network meta-analysis, examining the relative effectiveness of four psychotherapies and four control treatments for depression in 123 studies.

Long-term efficacy of antipsychotic drugs in initially acutely ill adults with schizophrenia: systematic review and network meta-analysis.

Most acute phase antipsychotic drug trials in schizophrenia last only a few weeks, but patients must usually take these drugs much longer. We examined the long-term efficacy of antipsychotic drugs in acutely ill patients using network meta-analysis. We searched the Cochrane Schizophrenia Group register up to March 6, 2022 for randomized, blinded trials of at least 6-month duration on all second-generation and 18 first-generation antipsychotics. The primary outcome was change in overall symptoms of schizophrenia; secondary outcomes were all-cause discontinuation; change in positive, negative and depressive symptoms; quality of life, social functioning, weight gain, antiparkinson medication use, akathisia, serum prolactin level, QTc prolongation, and sedation. Confidence in the results was assessed by the CINeMA (Confidence in Network Meta-Analysis) framework. We included 45 studies with 11,238 participants. In terms of overall symptoms, olanzapine was on average more efficacious than ziprasidone (standardized mean difference, SMD=0.37, 95% CI: 0.26-0.49), asenapine (SMD=0.33, 95% CI: 0.21-0.45), iloperidone (SMD=0.32, 95% CI: 0.15-0.49), paliperidone (SMD=0.28, 95% CI: 0.11-0.44), haloperidol (SMD=0.27, 95% CI: 0.14-0.39), quetiapine (SMD=0.25, 95% CI: 0.12-0.38), aripiprazole (SMD=0.16, 95% CI: 0.04-0.28) and risperidone (SMD=0.12, 95% CI: 0.03-0.21). The 95% CIs for olanzapine versus aripiprazole and risperidone included the possibility of trivial effects. The differences between olanzapine and lurasidone, amisulpride, perphenazine, clozapine and zotepine were either small or uncertain. These results were robust in sensitivity analyses and in line with other efficacy outcomes and all-cause discontinuation. Concerning weight gain, the impact of olanzapine was higher than all other antipsychotics, with a mean difference ranging from -4.58 kg (95% CI: -5.33 to -3.83) compared to ziprasidone to -2.30 kg (95% CI: -3.35 to -1.25) compared to amisulpride. Our data suggest that olanzapine is more efficacious than a number of other antipsychotic drugs in the longer term, but its efficacy must be weighed against its side effect profile.

Considering multiple outcomes with different weights informed the hierarchy of interventions in network meta-analysis.

OBJECTIVES: Ranking metrics in network meta-analysis (NMA) are computed separately for each outcome. Our aim is to 1) present graphical ways to group competing interventions considering multiple outcomes and 2) use conjoint analysis for placing weights on the various outcomes based on the stakeholders' preferences. STUDY DESIGN AND SETTING: We used multidimensional scaling (MDS) and hierarchical tree clustering to visualize the extent of similarity of interventions in terms of the relative effects they produce through a random effect NMA. We reanalyzed a published network of 212 psychosis trials taking three outcomes into account as follows: reduction in symptoms of schizophrenia, all-cause treatment discontinuation, and weight gain. RESULTS: Conjoint analysis provides a mathematical method to transform judgements into weights that can be subsequently used to visually represent interventions on a two-dimensional plane or through a dendrogram. These plots provide insightful information about the clustering of interventions. CONCLUSION: Grouping interventions can help decision makers not only to identify the optimal ones in terms of benefit-risk balance but also choose one from the best cluster based on other grounds, such as cost, implementation etc. Placing weights on outcomes allows considering patient profile or preferences.

Estimating the sample size of sham-controlled randomized controlled trials using existing evidence.

Background: In randomized controlled trials (RCTs), the power is often 'reverse engineered' based on the number of participants that can realistically be achieved. An attractive alternative is planning a new trial conditional on the available evidence; a design of particular interest in RCTs that use a sham control arm (sham-RCTs). Methods: We explore the design of sham-RCTs, the role of sequential meta-analysis and  conditional planning in a systematic review of renal sympathetic denervation for patients with arterial hypertension. The main efficacy endpoint was mean change in 24-hour systolic blood pressure. We performed sequential meta-analysis to identify the time point where the null hypothesis would be rejected in a prospective scenario. Evidence-based conditional sample size calculations were performed based on fixed-effect meta-analysis. Results: In total, six sham-RCTs (981 participants) were identified. The first RCT was considerably larger (535 participants) than those subsequently published (median sample size of 80). All trial sample sizes were calculated assuming an unrealistically large intervention effect which resulted in low power when each study is considered as a stand-alone experiment. Sequential meta-analysis provided firm evidence against the null hypothesis with the synthesis of the first four trials (755 patients, cumulative mean difference -2.75 (95%CI -4.93 to -0.58) favoring the active intervention)). Conditional planning resulted in much larger sample sizes compared to those in the original trials, due to overoptimistic expected effects made by the investigators in individual trials, and potentially a time-effect association. Conclusions: Sequential meta-analysis of sham-RCTs can reach conclusive findings earlier and hence avoid exposing patients to sham-related risks. Conditional planning of new sham-RCTs poses important challenges as many surgical/minimally invasive procedures improve over time, the intervention effect is expected to increase in new studies and this violates the underlying assumptions. Unless this is accounted for, conditional planning will not improve the design of sham-RCTs.

Network meta-analysis and random walks.

Network meta-analysis (NMA) is a central tool for evidence synthesis in clinical research. The results of an NMA depend critically on the quality of evidence being pooled. In assessing the validity of an NMA, it is therefore important to know the proportion contributions of each direct treatment comparison to each network treatment effect. The construction of proportion contributions is based on the observation that each row of the hat matrix represents a so-called "evidence flow network" for each treatment comparison. However, the existing algorithm used to calculate these values is associated with ambiguity according to the selection of paths. In this article, we present a novel analogy between NMA and random walks. We use this analogy to derive closed-form expressions for the proportion contributions. A random walk on a graph is a stochastic process that describes a succession of random "hops" between vertices which are connected by an edge. The weight of an edge relates to the probability that the walker moves along that edge. We use the graph representation of NMA to construct the transition matrix for a random walk on the network of evidence. We show that the net number of times a walker crosses each edge of the network is related to the evidence flow network. By then defining a random walk on the directed evidence flow network, we derive analytically the matrix of proportion contributions. The random-walk approach has none of the associated ambiguity of the existing algorithm.

Introducing the Treatment Hierarchy Question in Network Meta-Analysis.

Comparative effectiveness research using network meta-analysis can present a hierarchy of competing treatments, from the most to the least preferable option. However, in published reviews, the research question associated with the hierarchy of multiple interventions is typically not clearly defined. Here we introduce the novel notion of a treatment hierarchy question that describes the criterion for choosing a specific treatment over one or more competing alternatives. For example, stakeholders might ask which treatment is most likely to improve mean survival by at least 2 years, or which treatment is associated with the longest mean survival. We discuss the most commonly used ranking metrics (quantities that compare the estimated treatment-specific effects), how the ranking metrics produce a treatment hierarchy, and the type of treatment hierarchy question that each ranking metric can answer. We show that the ranking metrics encompass the uncertainty in the estimation of the treatment effects in different ways, which results in different treatment hierarchies. When using network meta-analyses that aim to rank treatments, investigators should state the treatment hierarchy question they aim to address and employ the appropriate ranking metric to answer it. Following this new proposal will avoid some controversies that have arisen in comparative effectiveness research.

Answering complex hierarchy questions in network meta-analysis.

BACKGROUND: Network meta-analysis estimates all relative effects between competing treatments and can produce a treatment hierarchy from the most to the least desirable option according to a health outcome. While about half of the published network meta-analyses present such a hierarchy, it is rarely the case that it is related to a clinically relevant decision question. METHODS: We first define treatment hierarchy and treatment ranking in a network meta-analysis and suggest a simulation method to estimate the probability of each possible hierarchy to occur. We then propose a stepwise approach to express clinically relevant decision questions as hierarchy questions and quantify the uncertainty of the criteria that constitute them. The steps of the approach are summarized as follows: a) a question of clinical relevance is defined, b) the hierarchies that satisfy the defined question are collected and c) the frequencies of the respective hierarchies are added; the resulted sum expresses the certainty of the defined set of criteria to hold. We then show how the frequencies of all possible hierarchies relate to common ranking metrics. RESULTS: We exemplify the method and its implementation using two networks. The first is a network of four treatments for chronic obstructive pulmonary disease where the most probable hierarchy has a frequency of 28%. The second is a network of 18 antidepressants, among which Vortioxetine, Bupropion and Escitalopram occupy the first three ranks with frequency 19%. CONCLUSIONS: The developed method offers a generalised approach of producing treatment hierarchies in network meta-analysis, which moves towards attaching treatment ranking to a clear decision question, relevant to all or a subset of competing treatments.

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis.

BACKGROUND: Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN). METHODS: ROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results (within-study assessment of bias) and the potential for unpublished studies (across-study assessment of bias). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of "low risk", "some concerns", or "high risk" for the bias due to missing evidence is assigned to each estimate, which is our tool's final output. RESULTS: We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder. CONCLUSIONS: ROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.

Do reporting guidelines have an impact? Empirical assessment of changes in reporting before and after the PRISMA extension statement for network meta-analysis.

BACKGROUND: The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension statement for network meta-analysis (NMA) published in 2015 promotes comprehensive reporting in published systematic reviews with NMA. PRISMA-NMA includes 32 items: 27 core items as indicated in the 2009 PRISMA Statement and five items specific to the reporting of NMAs. Although NMA reporting is improving, it is unclear whether PRISMA-NMA has accelerated this improvement. We aimed to investigate the impact of PRISMA-NMA and highlight key items that require attention and improvement. METHODS: We updated our previous collection of NMAs with articles published between April 2015 and July 2018. We assessed the completeness of reporting for each NMA, including main manuscript and online supplements, using the PRISMA-NMA checklist. The PRISMA-NMA checklist originally includes 32 total items (i.e. a 32-point scale original PRISMA-NMA score). We also prepared a modified version of the PRISMA-NMA checklist with 49 items to evaluate separately at a more granular level all multiple-content items (i.e. a 49-point scale modified PRISMA-NMA score). We compared average reporting scores of articles published until and after 2015. RESULTS: In the 1144 included NMAs the mean modified PRISMA-NMA score was 32.1 (95% CI 31.8-32.4) of a possible 49-excellence-score. For 1-year increase, the mean modified score increased by 0.96 (95% CI 0.32 to 1.59) for 389 NMAs published until 2015 and by 0.53 (95% CI 0.02 to 1.04) for 755 NMAs published after 2015. The mean modified PRISMA-NMA score for NMAs published after 2015 was higher by 0.81 (95% CI 0.23 to 1.39) compared to before 2015 when adjusting for journal impact factor, type of review, funding, and treatment category. Description of summary effect sizes to be used, presentation of individual study data, sources of funding for the systematic review, and role of funders dropped in frequency after 2015 by 6-16%. CONCLUSIONS: NMAs published after 2015 more frequently reported the five items associated with NMA compared to those published until 2015. However, improvement in reporting after 2015 is compatible with that observed on a yearly basis until 2015, and hence, it could not be attributed solely to the publication of the PRISMA-NMA.

Meta-analysis as a system of springs.

Meta-analysis results are usually presented in forest plots, which show the individual study results and the summary effect along with their confidence intervals. In this paper, we propose a system of linear springs as a mechanical analogue of meta-analysis that enables visualization and enhances intuition. The length of a spring corresponds to a study treatment effect and the stiffness of the spring corresponds to its inverse variance. To synthesize study springs we use two main operations: connection in parallel and connection in series. We show the equivalence between meta-analysis and linear springs for fixed effect and random effects pairwise meta-analysis and we also derive indirect treatment effects. We use examples to illustrate the different meta-analytical schemes using the corresponding system of springs. The proposed visualization can serve as an educational tool, especially useful for researchers with no statistical background. The analogy between meta-analysis and springs facilitates intuition for notions such as heterogeneity and the differences between fixed and random effects meta-analysis.

Network meta-analysis results against a fictional treatment of average performance: Treatment effects and ranking metric.

BACKGROUND: Network meta-analysis (NMA) produces complex outputs as many comparisons between interventions are of interest. The estimated relative treatment effects are usually displayed in a forest plot or in a league table and several ranking metrics are calculated and presented. METHODS: In this article, we estimate relative treatment effects of each competing treatment against a fictional treatment of average performance using the "deviation from the means" coding that has been used to parametrize categorical covariates in regression models. We then use this alternative parametrization of the NMA model to present a ranking metric (PreTA: Preferable Than Average) interpreted as the probability that a treatment is better than a fictional treatment of average performance. RESULTS: We illustrate the alternative parametrization of the NMA model using two networks of interventions, a network of 18 antidepressants for acute depression and a network of four interventions for heavy menstrual bleeding. We also use these two networks to highlight differences among PreTA and existing ranking metrics. We further examine the agreement between PreTA and existing ranking metrics in 232 networks of interventions and conclude that their agreement depends on the precision with which relative effects are estimated. CONCLUSIONS: A forest plot with NMA relative treatment effects using "deviation from means" coding could complement presentation of NMA results in large networks and in absence of an obvious reference treatment. PreTA is a viable alternative to existing probabilistic ranking metrics that naturally incorporates uncertainty.

Comparative Efficacy and Tolerability of 32 Oral Antipsychotics for the Acute Treatment of Adults With Multi-Episode Schizophrenia: A Systematic Review and Network Meta-Analysis.

(Reprinted with permission from the Lancet 2019; 394: 939-51).

Antidepressant treatment in patients following acute coronary syndromes: a systematic review and Bayesian meta-analysis.

AIMS: The aim of this study is to investigate the effect of antidepressant therapy on mortality and cardiovascular outcomes in patients with acute coronary syndrome (ACS). METHODS AND RESULTS: We systematically searched PubMed, Embase, and the Cochrane Central Register of Controlled Trials and performed a Bayesian random-effects meta-analysis of randomized controlled trials that investigated antidepressant pharmacotherapy in patients following ACS. The primary outcome was all-cause mortality. Secondary outcomes were repeat hospitalizations and recurrent myocardial infarctions (MIs). Ten randomized controlled trials with a total of 1935 patients qualified for inclusion. Selective serotonin reuptake inhibitors were investigated in six, bupropion in three, and mirtazapine in one trial. Placebo was used as control in eight trials. There was no difference in all-cause mortality [odds ratio (OR) 0.97, 95% credible interval (CrI) 0.66-1.42] and recurrent MI (OR 0.64, 95% CrI 0.40-1.02) between patients receiving antidepressants compared with controls, whereas antidepressant therapy was associated with less repeat hospitalizations (OR 0.62, 95% CrI 0.40-0.94). In patients with ACS and concomitant depression, antidepressants reduced the odds of recurrent MI compared with usual care/placebo (OR 0.45, 95% CrI 0.25-0.81). Extended funnel plots suggest robustness of the observations. CONCLUSIONS: Antidepressants in patients following ACS have no effect on mortality but reduce repeat hospitalizations; in patients with depression, there is a reduced risk of recurrent MI with antidepressant therapy.

Agreement between ranking metrics in network meta-analysis: an empirical study.

OBJECTIVE: To empirically explore the level of agreement of the treatment hierarchies from different ranking metrics in network meta-analysis (NMA) and to investigate how network characteristics influence the agreement. DESIGN: Empirical evaluation from re-analysis of NMA. DATA: 232 networks of four or more interventions from randomised controlled trials, published between 1999 and 2015. METHODS: We calculated treatment hierarchies from several ranking metrics: relative treatment effects, probability of producing the best value [Formula: see text] and the surface under the cumulative ranking curve (SUCRA). We estimated the level of agreement between the treatment hierarchies using different measures: Kendall's τ and Spearman's ρ correlation; and the Yilmaz [Formula: see text] and Average Overlap, to give more weight to the top of the rankings. Finally, we assessed how the amount of the information present in a network affects the agreement between treatment hierarchies, using the average variance, the relative range of variance and the total sample size over the number of interventions of a network. RESULTS: Overall, the pairwise agreement was high for all treatment hierarchies obtained by the different ranking metrics. The highest agreement was observed between SUCRA and the relative treatment effect for both correlation and top-weighted measures whose medians were all equal to 1. The agreement between rankings decreased for networks with less precise estimates and the hierarchies obtained from [Formula: see text] appeared to be the most sensitive to large differences in the variance estimates. However, such large differences were rare. CONCLUSIONS: Different ranking metrics address different treatment hierarchy problems, however they produced similar rankings in the published networks. Researchers reporting NMA results can use the ranking metric they prefer, unless there are imprecise estimates or large imbalances in the variance estimates. In this case treatment hierarchies based on both probabilistic and non-probabilistic ranking metrics should be presented.