Frequency of use and adequacy of Cochrane risk of bias tool 2 in non-Cochrane systematic reviews published in 2020: Meta-research study.

Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.

When is the evidence conclusive? Analysis of systematic reviews for which Cochrane declared that conclusions will not change with further studies.

Some Cochrane reviews were declared stable or closed, that is, not in need of updating. For some of them, it has been declared that conclusions will not (or it is unlikely they will) change with further studies. We explored whether there is a discernable decision-making pattern for decisions about the conclusiveness and stabilization of these reviews. We analyzed Cochrane reviews published until April 2020 labeled as stable or closed. We extracted the rationale leading to the decision declaring that the conclusion is not expected to change with further studies. Furthermore, we assessed whether the reviews used GRADE analysis. We extracted data from summary of findings (SoF) tables on the direction of effect, statistical significance, and I2 values for the first and primary outcomes in SoFs, conclusions in the abstract and review, and implications for practice and future research. We included 40 stable/closed Cochrane reviews. Rationales for their stabilization did not enable any insight into the Cochrane's decision-making algorithm for considering the evidence as conclusive. Among 191 outcomes presented in the SoFs, 70% were rated with either low or very low certainty evidence. None of the reviews mentioned in the text that the review should be stabilized or closed, or that there is sufficient evidence on the subject. Reasons for stabilizing/closing Cochrane reviews were unclear, and we could not discern any pattern of "conclusive review" traits. Definition of systematic review conclusiveness is still lacking, which may contribute to research waste.

How to decide whether a systematic review is stable and not in need of updating: Analysis of Cochrane reviews.

BACKGROUND: It is challenging to keep systematic reviews (SR) current and updated. Cochrane designated some of its SRs as "stable," that is, not in need of updating. The issue of stabilizing an SR is an important in research synthesis, because it could help reduce research waste. The aim of this study was to analyze publicly available justifications for stabilizing a Cochrane review, with the ultimate goal of helping to make decisions about whether the update of any SR is warranted. METHODS: We analyzed Cochrane reviews labeled as stable in Archie, Cochrane's system for managing the editorial/publishing process. From the "What's new" section of the reviews in the Cochrane Library, we extracted justification for stabilization. RESULTS: We included 545 Cochrane reviews labeled in Archie as stable on October 28, 2019. The most common of the five reasons for stabilization was that "last search did not identify any potentially relevant studies likely to change conclusions" (N = 99; 18%), followed by "research area no longer active" (N = 86; 16%), "review is or will be superseded" (N = 41; 7.5%), "evidence is conclusive" (N=35; 6.4%), and "intervention no longer in general use" (N = 34; 6.2%). For the 269 (49%) Cochrane reviews, we considered that the justification for stabilization was not clearly described, that is, sufficiently informative. CONCLUSIONS: Cochrane reviews would benefit from more transparency and consistency in publicly available justifications for stabilizing reviews. Further work in this field will help make decisions about the futility of further research and deciding on enough evidence in the field of research synthesis.

Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent.

OBJECTIVE: The objective of the study was to analyze methods of assessing "overall bias" in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews and sensitivity analyses related to overall risk of bias (RoB). STUDY DESIGN AND SETTING: From Cochrane reviews published within 3 years, from July 2015 to June 2018, we extracted data regarding methods of judging overall bias for a single trial, as well as details regarding methods used in frequency of RoB in sensitivity analyses. RESULTS: Of the 1,452 analyzed Cochrane reviews, 409 mentioned assessment of overall RoB on a study level. In 107 reviews, authors clearly specified key domains that determined the overall RoB, whereas in the remaining reviews, assessment of overall bias was not in line with the Cochrane Handbook. Among 268 Cochrane reviews that had any RoB-related sensitivity analysis, in 56 (21%) reviews, the authors reported a significant change for at least one outcome compared with the initial analysis. CONCLUSION: Highly heterogeneous approaches to summarizing overall RoB on a study level and using RoB for sensitivity analyses may yield inconsistent and incomparable results across Cochrane reviews.

The judgement of biases included in the category "other bias" in Cochrane systematic reviews of interventions: a systematic survey.

BACKGROUND: Clinical decisions are made based on Cochrane reviews, but the implementation of results of evidence syntheses such as Cochrane reviews is problematic if the evidence is not prepared consistently. All systematic reviews should assess the risk of bias (RoB) in included studies, and in Cochrane reviews, this is done by using Cochrane RoB tool. However, the tool is not necessarily applied according to the instructions. In this study, we aimed to determine the types of bias and their corresponding judgements noted in the 'other bias' domain of Cochrane RoB tool. METHODS: We analyzed Cochrane reviews that included randomized controlled trials (RCTs) and extracted data regarding 'other bias' from the RoB table and accompanying support for the judgment. We categorized different types of other bias. RESULTS: We analyzed 768 Cochrane reviews that included 11,369 RCTs. There were 602 (78%) Cochrane reviews that had 'other bias' domain in the RoB tool, and they included a total of 7811 RCTs. In the RoB table of 337 Cochrane reviews for at least one of the included trials it was indicated that no other bias was found and supporting explanations were inconsistently judged as low, unclear or high RoB. In the 524 Cochrane reviews that described various sources of other bias, there were 5762 individual types of explanations which we categorized into 31 groups. The judgments of the same supporting explanations were highly inconsistent. We found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. CONCLUSION: Cochrane authors mention a wide range of sources of other bias in the RoB tool and they inconsistently judge the same supporting explanations. Inconsistency in appraising risk of other bias hinders reliability and comparability of Cochrane systematic reviews. Discrepant and erroneous judgments of bias in evidence synthesis may hinder implementation of evidence in routine clinical practice and reduce confidence in otherwise trustworthy sources of information. These results can help authors of Cochrane and non-Cochrane reviews to gain insight into various sources of other bias that can be found in trials, and also to help them avoid mistakes that were recognized in published Cochrane reviews.

Assessments of attrition bias in Cochrane systematic reviews are highly inconsistent and thus hindering trial comparability.

BACKGROUND: An important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews are considered golden standard regarding systematic review methodology, but Cochrane's instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors' assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR). METHODS: We analyzed Cochrane reviews published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many Cochrane reviews had different judgments for the same supporting explanations. RESULTS: In the main analysis we included 10,292 judgments and supporting explanations for attrition bias from 729 Cochrane reviews. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of Cochrane review authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same Cochrane review. CONCLUSION: We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane's risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.

Revisiting established medicines: An overview of systematic reviews about ibuprofen and paracetamol for treating pain in children.

BACKGROUND AND OBJECTIVE: We explored how systematic reviews evaluated paracetamol and ibuprofen for treating pain in children, as these two non-opioid analgesics are well-established medicines included in most national essential medicines lists. DATABASES AND DATA TREATMENT: We carried out an overview of systematic reviews (SRs) of randomized controlled trials (RCTs) of interventions (PROSPERO registration: 42016045367). We searched MEDLINE, EMBASE, Cochrane Database of Systematic Reviews (CDSR) and Database of Reviews of Effects (DARE) up to 23 August 2017. We used AMSTAR checklist to analyse methodological quality of included SRs. RESULTS: We found 17 SRs with 72 unique RCTs; the majority of those trials included under 100 children. Positive conclusive evidence was found in only one SR, regarding safety of paracetamol. Conclusions of other SRs for efficacy and safety of ibuprofen and paracetamol were inconclusive, unclear, or there was no opinion. Only one SR analysed efficacy of ibuprofen and other non-steroidal anti-inflammatory drugs (NSAIDs) in chronic pain and the conclusion was that there was no evidence from RCTs that NSAIDs were effective for chronic non-cancer pain in children and adolescents. Most of the SRs addressed very narrow questions, included few trials, with few children and were of low or medium methodological quality. CONCLUSIONS: Most SRs on two relevant medicines have inconsistent conclusions and doubt upon their effectiveness. Instead of focusing on very narrow questions, SRs should examine more comprehensive research topics to obtain a general sense of consistency, particularly when analysing established medicines. SIGNIFICANCE: Evidence behind two analgesics-ibuprofen and paracetamol-that are well-established medicines for children in most countries appears limited, judging by the systematic reviews. The discrepancy between clinical use and the extensive evidence we reviewed may be a result of the selective criteria in the reviews examined. We need new, and better evidence syntheses supporting the use of these two medicines in wide indications regarding pain in children.

Pressures and interventions imposed on medical school teachers regarding students' examination grades.

OBJECTIVES: This study was designed to investigate the pressures and interventions experienced by medical school teachers in relation to grades awarded in assessments of students' knowledge. METHODS: Of 124 tenured teachers at the University of Split School of Medicine, 91 (73%) participated in the study through the anonymous completion of a questionnaire administered during the 2011-2012 academic year. RESULTS: A total of 42 (46%) teachers reported having experienced some kind of pressure or intervention imposed by students, students' parents or colleagues that was intended to ensure the student obtained a pass or a better grade on an examination. The most common forms of pressure imposed by students were persistent pleading, 'friendly' requests and crying. Students' parents and staff colleagues mostly used 'friendly' requests or indicated that they would like to discuss a student's results. Colleagues who contacted teachers about students' examination results included doctors who did not work at the school and other teachers at the School of Medicine. As a method of preventing the imposition of such pressures and interventions in relation to examination results, teachers suggested that students should be warned against these practices before their courses started and that appropriate policies should be initiated. Some of the teachers rated some specific pressures or interventions as fully or partially acceptable; these included the making of 'friendly' requests, persistent pleading, crying and yelling. Most of the teachers stated that they would report some instances of such interventions to the school management, but 15 (17%) teachers stated that they would not report any type of pressure. CONCLUSIONS: It is necessary to raise awareness of the unacceptability of such pressures and interventions in relation to examination results, and to change the current professional culture in which the imposition of such pressures by students, their parents and staff colleagues is considered acceptable and common.