Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent.

ABSTRACT

OBJECTIVE: The objective of the study was to analyze methods of assessing "overall bias" in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews and sensitivity analyses related to overall risk of bias (RoB). STUDY DESIGN AND SETTING: From Cochrane reviews published within 3 years, from July 2015 to June 2018, we extracted data regarding methods of judging overall bias for a single trial, as well as details regarding methods used in frequency of RoB in sensitivity analyses. RESULTS: Of the 1,452 analyzed Cochrane reviews, 409 mentioned assessment of overall RoB on a study level. In 107 reviews, authors clearly specified key domains that determined the overall RoB, whereas in the remaining reviews, assessment of overall bias was not in line with the Cochrane Handbook. Among 268 Cochrane reviews that had any RoB-related sensitivity analysis, in 56 (21%) reviews, the authors reported a significant change for at least one outcome compared with the initial analysis. CONCLUSION: Highly heterogeneous approaches to summarizing overall RoB on a study level and using RoB for sensitivity analyses may yield inconsistent and incomparable results across Cochrane reviews.