RESUMO
The internal validity of conclusions about effectiveness or impact in systematic reviews, and of decisions based on them, depends on risk of bias assessments being conducted appropriately. However, a random sample of 50 recently-published articles claiming to be quantitative environmental systematic reviews found 64% did not include any risk of bias assessment, whilst nearly all that did omitted key sources of bias. Other limitations included lack of transparency, conflation of quality constructs, and incomplete application of risk of bias assessments to the data synthesis. This paper addresses deficiencies in risk of bias assessments by highlighting core principles that are required for risk of bias assessments to be fit-for-purpose, and presenting a framework based on these principles to guide review teams on conducting risk of bias assessments appropriately and consistently. The core principles require that risk of bias assessments be Focused, Extensive, Applied and Transparent (FEAT). These principles support risk of bias assessments, appraisal of risk of bias tools, and the development of new tools. The framework follows a Plan-Conduct-Apply-Report approach covering all stages of risk of bias assessment. The scope of this paper is comparative quantitative environmental systematic reviews which address PICO or PECO-type questions including, but not limited to, topic areas such as environmental management, conservation, ecosystem restoration, and analyses of environmental interventions, exposures, impacts and risks.
RESUMO
Systematic reviews are becoming a widely accepted gold standard in evidence synthesis for evidence-based and -informed policy and practice. Many organisations exist to coordinate the registration, conduct and publication of systematic reviews across a range of disciplines, including medicine, international development, and environmental management and biodiversity conservation. As the term 'systematic review' becomes more widely recognised, however, there is a risk that stakeholders may have only partial understanding of the rigorous methods required to produce a reliable systematic review. Here, we highlight one such example from the field of education and international development, where a World Bank report claimed to 'systematically review' six 'systematic reviews' that found divergent results. We critically appraise the six included reviews and the World Bank report itself using an a priori quality assessment tool. Our analysis shows that none of the six included reviews are classifiable as systematic reviews according to widely accepted criteria. We also find that the World Bank report failed to use true systematic review methods to synthesise the included reviews findings. Our study demonstrates the risks associated with partial understanding of the added value associated with systematic reviews and highlights a need for improved awareness of what systematic reviews are.
