Transitivity, coherence, and reliability of network meta-analyses comparing proximal humerus fracture treatments: a meta-epidemiological study.

BACKGROUND: Network meta-analyses can be valuable for decision-makers in guiding clinical practice. However, for network meta-analysis results to be reliable, the assumptions of both transitivity and coherence must be met, and the methodology should adhere to current best practices. We aimed to assess whether network meta-analyses of randomized controlled trials (RCTs) comparing interventions for proximal humerus fractures provide reliable estimates of intervention effects. METHODS: We searched PubMed, EMBASE, The Cochrane Library, and Web of Science for network meta-analyses comparing interventions for proximal humerus fractures. We critically assessed the methodology regarding the development of a protocol, search strategy, trial inclusion, outcome extraction, and the methods used to conduct the network meta-analyses. We assessed the transitivity and coherence of the network graphs for the Constant score (CS), Disabilities of the Arm, Shoulder, and Hand score (DASH), and additional surgery. Transitivity was assessed by comparing probable effect modifiers (age, gender, fracture morphology, and comorbidities) across intervention comparisons. Coherence was assessed using Separating Indirect from Direct Evidence (SIDE) (Separating Indirect from Direct Evidence) and the design-by-treatment interaction test. We used CINeMA (Confidence in Network Meta-analyses) to assess the confidence in the results. RESULTS: None of the three included network meta-analyses had a publicly available protocol or data-analysis plan, and they all had methodological flaws that could threaten the validity of their results. Although we did not detect incoherence for most comparisons, the transitivity assumption was violated for CS, DASH, and additional surgery in all three network meta-analyses. Additionally, the confidence in the results was 'very low' primarily due to within-study bias, reporting bias, intransitivity, imprecision, and heterogeneity. CONCLUSIONS: Current network meta-analyses of RCTs comparing interventions for proximal humerus fractures do not provide reliable estimates of intervention effects. We advise caution in using these network meta-analyses to guide clinical practice. To improve the utility of network meta-analyses to guide clinical practice, journal editors should require that network meta-analyses are done according to a predefined analysis plan in a publicly available protocol and that both coherence and transitivity have been adequately assessed and reported.

A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results.

OBJECTIVES: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. STUDY DESIGN AND SETTING: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. RESULTS: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%-99%; quality of life 58%-76%; adverse events 87%-91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%-78%, quality of life; 10%-24%; adverse events 10%-13%). CONCLUSION: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.

The methodological quality was low and conclusions discordant for meta-analyses comparing proximal humerus fracture treatments: a meta-epidemiological study.

OBJECTIVE: To investigate the association between methodological quality and reported conclusions of meta-analyses comparing operative with non-operative treatments for proximal humerus fractures. STUDY DESIGN AND SETTING: Cross-sectional meta-epidemiological study. We searched EMBASE, PubMed, The Cochrane Library, and Web of Science for systematic reviews with meta-analyses comparing non-operative with operative treatments for proximal humerus fractures. Methodological quality was assessed using AMSTAR2 and the reported conclusions were scored for three outcome domains (functional outcome, quality of life, and harm) on a scale from 1 to 6. The Mann-Whitney and Kruskal-Wallis tests were used to investigate the association between methodological quality and reported conclusions. RESULTS: We included 21 systematic reviews: 19 pairwise meta-analyses and 2 network meta-analyses, although there are only 8 published randomized controlled trials. Most (n = 18) of the meta-analyses were rated as critically low quality, while the remaining 1 was rated as high quality. The conclusions were discordant for all three outcome domains, even for meta-analyses reporting similar inclusion criteria. We could not perform most of the statistical tests due to the predominantly critically low quality. CONCLUSION: The methodological quality was so predominantly critically low that it was not possible to evaluate the association between methodological quality and reported conclusions.

Low inter-observer agreement among experienced shoulder surgeons assessing overstuffing of glenohumeral resurfacing hemiarthroplasty based on plain radiographs.

BACKGROUND: In a clinical setting, a visual evaluation of post-implant radiographs is often used to assess the restoration of glenohumeral joint anatomy after resurfacing hemiarthroplasty and is a part of the decision-making process, in combination with other parameters, when evaluating patients with inferior clinical results. However, the reliability of this method of visual evaluation has not been reported. The aim of this study was to investigate the inter- and intra-observer agreement among experienced shoulder surgeons assessing overstuffing, implant positioning, and size following resurfacing hemiarthroplasty using plain standardized radiographs. METHODS: Six experienced shoulder surgeons independently classified implant inclination, size of the implant and if the joint seemed overstuffed, in 219 cases of post-implant radiographs. All cases were classified twice 3 weeks apart. Only radiographs with an anterior-posterior projection with a freely visible joint space were used. Non-weighted Cohen's kappa values were calculated for each coder pair and the mean used as an estimate of the overall inter-observer agreement. RESULTS: The overall inter-observer agreement for implant size (kappa, 0.48 and 0.41) and inclination angle was moderate in both rounds (kappa, 0.46 and 0.44), but only a fair agreement was found concerning the evaluation for stuffing of the joint (kappa, 0.24 and 0.28). Intra-observer agreement for implant size and stuffing ranged from fair to substantial while the agreement for inclination was moderate to substantial. CONCLUSIONS: Our results indicate that a visual evaluation of plain radiographs may be inadequate to evaluate overstuffing, implant positioning, and size following resurfacing hemiarthroplasty using plain standardized radiographs. Future studies may contribute to elucidate whether reliability increases if consensus on clear definitions and standardized methods of evaluation is made.