The CEEDER database of evidence reviews: An open-access evidence service for researchers and decision-makers.

Evidence-informed decision-making aims to deliver effective actions informed by the best available evidence. Given the large quantity of primary literature, and time constraints faced by policy-makers and practitioners, well-conducted evidence reviews can provide a valuable resource to support decision-making. However, previous research suggests that some evidence reviews may not be sufficiently reliable to inform decisions in the environmental sector due to low standards of conduct and reporting. While some evidence reviews are of high reliability, there is currently no way for policy-makers and practitioners to quickly and easily find them among the many lower reliability ones. Alongside this lack of transparency, there is little incentive or support for review authors, editors and peer-reviewers to improve reliability. To address these issues, we introduce a new online, freely available and first-of-its-kind evidence service: the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER: www.environmentalevidence.org/ceeder). CEEDER aims to transform communication of evidence review reliability to researchers, policy-makers and practitioners through independent assessment of key aspects of the conduct, reporting and data limitations of available evidence reviews claiming to assess environmental impacts or the effectiveness of interventions relevant to policy and practice. At the same time, CEEDER will provide support to improve the standards of future evidence reviews and support evidence translation and knowledge mobilisation to help inform environmental decision-making.

Standardized reporting of the costs of management interventions for biodiversity conservation.

Effective conservation management interventions must combat threats and deliver benefits at costs that can be achieved within limited budgets. Considerable effort has focused on measuring the potential benefits of conservation interventions, but explicit quantification of the financial costs of implementation is rare. Even when costs have been quantified, haphazard and inconsistent reporting means published values are difficult to interpret. This reporting deficiency hinders progress toward a collective understanding of the financial costs of management interventions across projects and thus limits the ability to identify efficient solutions to conservation problems or attract adequate funding. We devised a standardized approach to describing financial costs reported for conservation interventions. The standards call for researchers and practitioners to describe the objective and outcome, context and methods, and scale of costed interventions, and to state which categories of costs are included and the currency and date for reported costs. These standards aim to provide enough contextual information that readers and future users can interpret the cost data appropriately. We suggest these standards be adopted by major conservation organizations, conservation science institutions, and journals so that cost reporting is comparable among studies. This would support shared learning and enhance the ability to identify and perform cost-effective conservation.

Potential types of bias when estimating causal effects in environmental research and how to interpret them.

To inform environmental policy and practice, researchers estimate effects of interventions/exposures by conducting primary research (e.g., impact evaluations) or secondary research (e.g., evidence reviews). If these estimates are derived from poorly conducted/reported research, then they could misinform policy and practice by providing biased estimates. Many types of bias have been described, especially in health and medical sciences. We aimed to map all types of bias from the literature that are relevant to estimating causal effects in the environmental sector. All the types of bias were initially identified by using the Catalogue of Bias (catalogofbias.org) and reviewing key publications (n = 11) that previously collated and described biases. We identified 121 (out of 206) types of bias that were relevant to estimating causal effects in the environmental sector. We provide a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research: risk of confounding biases; risk of post-intervention/exposure selection biases; risk of misclassified/mismeasured comparison biases; risk of performance biases; risk of detection biases; risk of outcome reporting biases; risk of outcome assessment biases, and four domains for secondary research: risk of searching biases; risk of screening biases; risk of study appraisal and data coding/extraction biases; risk of data synthesis biases. Our collation should help scientists and decision makers in the environmental sector be better aware of the nature of bias in estimation of causal effects. Future research is needed to formalise the definitions of the collated types of bias such as through decomposition using mathematical formulae.

Environmental evidence in action: on the science and practice of evidence synthesis and evidence-based decision-making.

In civil society we expect that policy and management decisions will be made using the best available evidence. Yet, it is widely known that there are many barriers that limit the extent to which that occurs. One way to overcome these barriers is via robust, comprehensive, transparent and repeatable evidence syntheses (such as systematic reviews) that attempt to minimize various forms of bias to present a summary of existing knowledge for decision-making purposes. Relative to other disciplines (e.g., health care, education), such evidence-based decision-making remains relatively nascent for environment management despite major threats to humanity, such as the climate, pollution and biodiversity crises demonstrating that human well-being is inextricably linked to the biophysical environment. Fortunately, there are a growing number of environmental evidence syntheses being produced that can be used by decision makers. It is therefore an opportune time to reflect on the science and practice of evidence-based decision-making in environment management to understand the extent to which evidence syntheses are embraced and applied in practice. Here we outline a number of key questions related to the use of environmental evidence that need to be explored in an effort to enhance evidence-based decision-making. There is an urgent need for research involving methods from social science, behavioural sciences, and public policy to understand the basis for patterns and trends in environmental evidence use (or misuse or ignorance). There is also a need for those who commission and produce evidence syntheses, as well as the end users of these syntheses to reflect on their experiences and share them with the broader evidence-based practice community to identify needs and opportunities for advancing the entire process of evidence-based practice. It is our hope that the ideas shared here will serve as a roadmap for additional scholarship that will collectively enhance evidence-based decision-making and ultimately benefit the environment and humanity.

Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions.

Accurate, unbiased and concise synthesis of available evidence following clear methodology and transparent reporting is necessary to support effective environmental policy and management decisions. Without this, less reliable and/or less objective reviews of evidence could inform decision making, leading to ineffective, resource wasteful interventions with potential for unintended consequences. We evaluated the reliability of over 1000 evidence syntheses (reviews and overviews) published between 2018 and 2020 that provide evidence on the impacts of human activities or effectiveness of interventions relevant to environmental management. The syntheses are drawn from the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER), an online, freely available evidence service for evidence users that assesses the reliability of evidence syntheses using a series of published criteria. We found that the majority of syntheses have problems with transparency, replicability and potential for bias. Overall, our results suggest that most recently published evidence syntheses are of low reliability to inform decision making. Reviews that followed guidance and reporting standards for evidence synthesis had improved assessment ratings, but there remains substantial variation in the standard of reviews amongst even these. Furthermore, the term 'systematic review', which implies conformity with a methodological standard, was frequently misused. A major objective of the CEEDER project is to improve the reliability of the global body of environmental evidence reviews. To this end we outline freely available online resources to help improve review conduct and reporting. We call on authors, editors and peer reviewers to use these resources to ensure more reliable syntheses in the future.

The why, what, and how of global biodiversity indicators beyond the 2010 target.

The 2010 biodiversity target agreed by signatories to the Convention on Biological Diversity directed the attention of conservation professionals toward the development of indicators with which to measure changes in biological diversity at the global scale. We considered why global biodiversity indicators are needed, what characteristics successful global indicators have, and how existing indicators perform. Because monitoring could absorb a large proportion of funds available for conservation, we believe indicators should be linked explicitly to monitoring objectives and decisions about which monitoring schemes deserve funding should be informed by predictions of the value of such schemes to decision making. We suggest that raising awareness among the public and policy makers, auditing management actions, and informing policy choices are the most important global monitoring objectives. Using four well-developed indicators of biological diversity (extent of forests, coverage of protected areas, Living Planet Index, Red List Index) as examples, we analyzed the characteristics needed for indicators to meet these objectives. We recommend that conservation professionals improve on existing indicators by eliminating spatial biases in data availability, fill gaps in information about ecosystems other than forests, and improve understanding of the way indicators respond to policy changes. Monitoring is not an end in itself, and we believe it is vital that the ultimate objectives of global monitoring of biological diversity inform development of new indicators.

Realizing an effectiveness revolution in environmental management.

The environmental movement of the 20(th) century has evolved into a large, diverse and well-financed global community that is increasingly required to prove its worth. Though the environmental sector collects and uses data to determine the status of ecological and social systems, the effectiveness of the programs and policies it uses to affect this status remains largely untested. As governments and donor institutions insist on greater transparency, accountability and evidence of what works and what does not, much is being learned from other fields (e.g. health services, education, international development) and increasingly sophisticated approaches are emerging to manage effectiveness. For example, program evaluation, adaptive management, and systematic review provide frameworks and methods to collect and use information to measure and improve performance. However, the critical data and collaborations necessary for an effectiveness revolution are marginalized by technical, cultural and political obstacles. Learning from other fields, the environmental sector must exploit key leverage points, such as flows of information and self-organization, to overcome impediments and create incentives to initiate and realize an era of effectiveness in environmental management.

Effectiveness of predator removal for enhancing bird populations.

Predation pressure on vulnerable bird species has made predator control an important issue for international nature conservation. Predator removal by culling or translocation is controversial, expensive, and time-consuming, and results are often temporary. Thus, it is important to assess its effectiveness from all available evidence. We used explicit systematic review methodology to determine the impact of predator removal on four measurable responses in birds: breeding performance (hatching success and fledging success) and population size (breeding and postbreeding). We used meta-analysis to summarize results from 83 predator removal studies from six continents. We also investigated whether characteristics of the prey, predator species, location, and study methodology explained heterogeneity in effect sizes. Removing predators increased hatching success, fledging success, and breeding populations. Removing all predator species achieved a significantly larger increase in breeding population than removing only a subset. Postbreeding population size was not improved on islands, or overall, but did increase on mainlands. Heterogeneity in effect sizes for the four population parameters was not explained by whether predators were native or introduced; prey were declining, migratory, or game species; or by the study methodology. Effect sizes for fledging success were smaller for ground-nesting birds than those that nest elsewhere, but the difference was not significant. We conclude that current evidence indicates that predator removal is an effective strategy for the conservation of vulnerable bird populations. Nevertheless, the ethical and practical problems associated with predator removal may lead managers to favor alternative, nonlethal solutions. Research is needed to provide and synthesize data to determine whether these are effective management practices for future policies on bird conservation.

A systematic review of evidence for the added benefits to health of exposure to natural environments.

BACKGROUND: There is increasing interest in the potential role of the natural environment in human health and well-being. However, the evidence-base for specific and direct health or well-being benefits of activity within natural compared to more synthetic environments has not been systematically assessed. METHODS: We conducted a systematic review to collate and synthesise the findings of studies that compare measurements of health or well-being in natural and synthetic environments. Effect sizes of the differences between environments were calculated and meta-analysis used to synthesise data from studies measuring similar outcomes. RESULTS: Twenty-five studies met the review inclusion criteria. Most of these studies were crossover or controlled trials that investigated the effects of short-term exposure to each environment during a walk or run. This included 'natural' environments, such as public parks and green university campuses, and synthetic environments, such as indoor and outdoor built environments. The most common outcome measures were scores of different self-reported emotions. Based on these data, a meta-analysis provided some evidence of a positive benefit of a walk or run in a natural environment in comparison to a synthetic environment. There was also some support for greater attention after exposure to a natural environment but not after adjusting effect sizes for pretest differences. Meta-analysis of data on blood pressure and cortisol concentrations found less evidence of a consistent difference between environments across studies. CONCLUSIONS: Overall, the studies are suggestive that natural environments may have direct and positive impacts on well-being, but support the need for investment in further research on this question to understand the general significance for public health.

Assessing the risk of bias in choice of search sources for environmental meta-analyses.

Results of meta-analyses are potentially valuable for informing environmental policy and practice decisions. However, selective sampling of primary studies through searches exclusively using widely used bibliographic platform(s) could bias estimates of effect sizes. Such search strategies are common in environmental evidence reviews, and if risk of bias can be detected, this would provide the first empirical evidence that comprehensiveness of searches needs to be improved. We compare the impact of using single and multiple bibliographic platform(s) searches vs more comprehensive searches on estimates of mean effect sizes. We used 137 published meta-analyses, based on multiple source searches, analyzing 9388 studies: 8095 sourced from commercially published articles; and 1293 from grey literature and unpublished data. Single-platform and multiple-platform searches missed studies in 100 and 80 of the meta-analyses, respectively: 52 and 46 meta-analyses provided larger-effect estimates; 32 and 28 meta-analyses provided smaller-effect estimates; eight and four meta-analyses provided opposite direction of estimates; and two each were unable to estimate effects due to missing all studies. Further, we found significant positive log-linear relationships between proportions of studies missed and the deviations of mean effect sizes, suggesting that as the number of studies missed increases, deviation of mean effect size is likely to expand. We also found significant differences in mean effect sizes between indexed and non-indexed studies for 35% of meta-analyses, indicating high risk of bias when the searches were restricted. We conclude that the restricted searches are likely to lead to unrepresentative samples of studies and biased estimates of true effects.

Strengthen causal models for better conservation outcomes for human well-being.

BACKGROUND: Understanding how the conservation of nature can lead to improvement in human conditions is a research area with significant growth and attention. Progress towards effective conservation requires understanding mechanisms for achieving impact within complex social-ecological systems. Causal models are useful tools for defining plausible pathways from conservation actions to impacts on nature and people. Evaluating the potential of different strategies for delivering co-benefits for nature and people will require the use and testing of clear causal models that explicitly define the logic and assumptions behind cause and effect relationships. OBJECTIVES AND METHODS: In this study, we outline criteria for credible causal models and systematically evaluated their use in a broad base of literature (~1,000 peer-reviewed and grey literature articles from a published systematic evidence map) on links between nature-based conservation actions and human well-being impacts. RESULTS: Out of 1,027 publications identified, only ~20% of articles used any type of causal models to guide their work, and only 14 total articles fulfilled all criteria for credibility. Articles rarely tested the validity of models with empirical data. IMPLICATIONS: Not using causal models risks poorly defined strategies, misunderstanding of potential mechanisms for affecting change, inefficient use of resources, and focusing on implausible efforts for achieving sustainability.

Eight problems with literature reviews and how to fix them.

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a 'critical friend' role in supporting would-be systematic reviews and avoiding individual responses to police use of the 'systematic review' label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

Author Correction: Eight problems with literature reviews and how to fix them.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Evidence Synthesis International (ESI): Position Statement.

This paper is the initial Position Statement of Evidence Synthesis International, a new partnership of organizations that produce, support and use evidence synthesis around the world. The paper (i) argues for the importance of synthesis as a research exercise to clarify what is known from research evidence to inform policy, practice and personal decision making; (ii) discusses core issues for research synthesis such as the role of research evidence in decision making, the role of perspectives, participation and democracy in research and synthesis as a core component of evidence ecosystems; (iii) argues for 9 core principles for ESI on the nature and role of research synthesis; and (iv) lists the 5 main goals of ESI as a coordinating partnership for promoting and enabling the production and use of research synthesis.

Effectiveness of engineered in-stream structure mitigation measures to increase salmonid abundance: a systematic review.

Engineered in-stream structures are often installed to increase salmonid abundance, either for commercial gain in fisheries or for conservation purposes in degraded habitats. Having been in widespread use for the last 80 years, at an estimated cost of hundreds of millions of U.S. dollars each year, the effectiveness of these structures is still widely debated in the literature. Many studies of varying quality have been undertaken that attempt to address this issue, but it has proved difficult for practitioners to develop a consensus regarding the utility of these structures, despite their continued use. Systematic review methodology was used to formally synthesize empirical evidence regarding the effectiveness of engineered in-stream structures as a management tool to increase salmonid abundance. Meta-analysis shows that evidence regarding the effectiveness of in-stream devices is equivocal. Heterogeneity is significant both for population size and local habitat preference. This heterogeneity is related to stream width, with in-stream devices being less effective in larger streams. Consequently, widespread use of in-stream structures for restoration, particularly in larger streams, is not supported by the current scientific evidence base.

Conservation focus on Europe: major conservation policy issues that need to be informed by conservation science.

Europe is one of the world's most densely populated continents and has a long history of human-dominated land- and seascapes. Europe is also at the forefront of developing and implementing multinational conservation efforts. In this contribution, we describe some top policy issues in Europe that need to be informed by high-quality conservation science. These include evaluation of the effectiveness of the Natura 2000 network of protected sites, implications of rapid economic and subsequent land-use change in Central and Eastern Europe, conservation of marine biodiversity and sustainability of fisheries, the effect of climate change on movement of species in highly fragmented landscapes, and attempts to assess the economic value of ecosystem services and biodiversity. Broad policy issues such as those identified are not easily amenable to scientific experiment. A key challenge at the science-policy interface is to identify the research questions underlying these problem areas so that conservation science can provide evidence to underpin future policy development.

Control of Pteridium aquilinum: meta-analysis of a multi-site study in the UK.

BACKGROUND AND AIMS: A great deal of money is spent controlling invasive weeds as part of international and national policies. It is essential that the funded treatments work across the region in which the policies operate. We argue that experiments across multiple sites are required to validate these programs as results from single sites may be misleading. Here, the control of Pteridium aquilinum (bracken) is used as a test example to address the following four questions. (1) Does the effectiveness of P. aquilinum-control treatments vary across sites? (2) Is the best treatment identified in previous research (cutting twice per year) consistent at all sites, and if not why not? (3) Is treatment performance related to P. aquilinum rhizome mass, litter cover or litter depth at the various sites? (4) Does successful P. aquilinum control influence species richness? METHODS: Pteridium aquilinum-control treatments were monitored for 10 years using six replicated experiments and analysed using meta-analysis. Meta-regressions were used to explore heterogeneity between sites. KEY RESULTS: The effectiveness of treatments varied between sites depending on the measure used to assess P. aquilinum performance. In general, cutting twice per year was the most successful treatment but on some sites other, less expensive treatments were as good. The effectiveness of treatments at different sites was not related to rhizome mass, but the effectiveness of most applied treatments were inversely related to post-control litter. Effective treatment was also associated with high species richness. CONCLUSIONS: It is concluded that successful development of national weed control programs requires multi-site experimental approaches. Here, meta-analyses demonstrate that variation in effectiveness between sites could be explained in part by pre-specified variables. Reliance on data from a single site for policy formulation is therefore clearly dangerous.