Potential types of bias when estimating causal effects in environmental research and how to interpret them.

To inform environmental policy and practice, researchers estimate effects of interventions/exposures by conducting primary research (e.g., impact evaluations) or secondary research (e.g., evidence reviews). If these estimates are derived from poorly conducted/reported research, then they could misinform policy and practice by providing biased estimates. Many types of bias have been described, especially in health and medical sciences. We aimed to map all types of bias from the literature that are relevant to estimating causal effects in the environmental sector. All the types of bias were initially identified by using the Catalogue of Bias (catalogofbias.org) and reviewing key publications (n = 11) that previously collated and described biases. We identified 121 (out of 206) types of bias that were relevant to estimating causal effects in the environmental sector. We provide a general interpretation of every relevant type of bias covered by seven risk-of-bias domains for primary research: risk of confounding biases; risk of post-intervention/exposure selection biases; risk of misclassified/mismeasured comparison biases; risk of performance biases; risk of detection biases; risk of outcome reporting biases; risk of outcome assessment biases, and four domains for secondary research: risk of searching biases; risk of screening biases; risk of study appraisal and data coding/extraction biases; risk of data synthesis biases. Our collation should help scientists and decision makers in the environmental sector be better aware of the nature of bias in estimation of causal effects. Future research is needed to formalise the definitions of the collated types of bias such as through decomposition using mathematical formulae.

Environmental evidence in action: on the science and practice of evidence synthesis and evidence-based decision-making.

In civil society we expect that policy and management decisions will be made using the best available evidence. Yet, it is widely known that there are many barriers that limit the extent to which that occurs. One way to overcome these barriers is via robust, comprehensive, transparent and repeatable evidence syntheses (such as systematic reviews) that attempt to minimize various forms of bias to present a summary of existing knowledge for decision-making purposes. Relative to other disciplines (e.g., health care, education), such evidence-based decision-making remains relatively nascent for environment management despite major threats to humanity, such as the climate, pollution and biodiversity crises demonstrating that human well-being is inextricably linked to the biophysical environment. Fortunately, there are a growing number of environmental evidence syntheses being produced that can be used by decision makers. It is therefore an opportune time to reflect on the science and practice of evidence-based decision-making in environment management to understand the extent to which evidence syntheses are embraced and applied in practice. Here we outline a number of key questions related to the use of environmental evidence that need to be explored in an effort to enhance evidence-based decision-making. There is an urgent need for research involving methods from social science, behavioural sciences, and public policy to understand the basis for patterns and trends in environmental evidence use (or misuse or ignorance). There is also a need for those who commission and produce evidence syntheses, as well as the end users of these syntheses to reflect on their experiences and share them with the broader evidence-based practice community to identify needs and opportunities for advancing the entire process of evidence-based practice. It is our hope that the ideas shared here will serve as a roadmap for additional scholarship that will collectively enhance evidence-based decision-making and ultimately benefit the environment and humanity.

Standards of conduct and reporting in evidence syntheses that could inform environmental policy and management decisions.

Accurate, unbiased and concise synthesis of available evidence following clear methodology and transparent reporting is necessary to support effective environmental policy and management decisions. Without this, less reliable and/or less objective reviews of evidence could inform decision making, leading to ineffective, resource wasteful interventions with potential for unintended consequences. We evaluated the reliability of over 1000 evidence syntheses (reviews and overviews) published between 2018 and 2020 that provide evidence on the impacts of human activities or effectiveness of interventions relevant to environmental management. The syntheses are drawn from the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER), an online, freely available evidence service for evidence users that assesses the reliability of evidence syntheses using a series of published criteria. We found that the majority of syntheses have problems with transparency, replicability and potential for bias. Overall, our results suggest that most recently published evidence syntheses are of low reliability to inform decision making. Reviews that followed guidance and reporting standards for evidence synthesis had improved assessment ratings, but there remains substantial variation in the standard of reviews amongst even these. Furthermore, the term 'systematic review', which implies conformity with a methodological standard, was frequently misused. A major objective of the CEEDER project is to improve the reliability of the global body of environmental evidence reviews. To this end we outline freely available online resources to help improve review conduct and reporting. We call on authors, editors and peer reviewers to use these resources to ensure more reliable syntheses in the future.

Eight problems with literature reviews and how to fix them.

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a 'critical friend' role in supporting would-be systematic reviews and avoiding individual responses to police use of the 'systematic review' label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

Author Correction: Eight problems with literature reviews and how to fix them.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

The CEEDER database of evidence reviews: An open-access evidence service for researchers and decision-makers.

Evidence-informed decision-making aims to deliver effective actions informed by the best available evidence. Given the large quantity of primary literature, and time constraints faced by policy-makers and practitioners, well-conducted evidence reviews can provide a valuable resource to support decision-making. However, previous research suggests that some evidence reviews may not be sufficiently reliable to inform decisions in the environmental sector due to low standards of conduct and reporting. While some evidence reviews are of high reliability, there is currently no way for policy-makers and practitioners to quickly and easily find them among the many lower reliability ones. Alongside this lack of transparency, there is little incentive or support for review authors, editors and peer-reviewers to improve reliability. To address these issues, we introduce a new online, freely available and first-of-its-kind evidence service: the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER: www.environmentalevidence.org/ceeder). CEEDER aims to transform communication of evidence review reliability to researchers, policy-makers and practitioners through independent assessment of key aspects of the conduct, reporting and data limitations of available evidence reviews claiming to assess environmental impacts or the effectiveness of interventions relevant to policy and practice. At the same time, CEEDER will provide support to improve the standards of future evidence reviews and support evidence translation and knowledge mobilisation to help inform environmental decision-making.

Evidence Synthesis International (ESI): Position Statement.

This paper is the initial Position Statement of Evidence Synthesis International, a new partnership of organizations that produce, support and use evidence synthesis around the world. The paper (i) argues for the importance of synthesis as a research exercise to clarify what is known from research evidence to inform policy, practice and personal decision making; (ii) discusses core issues for research synthesis such as the role of research evidence in decision making, the role of perspectives, participation and democracy in research and synthesis as a core component of evidence ecosystems; (iii) argues for 9 core principles for ESI on the nature and role of research synthesis; and (iv) lists the 5 main goals of ESI as a coordinating partnership for promoting and enabling the production and use of research synthesis.

Assessing the risk of bias in choice of search sources for environmental meta-analyses.

Results of meta-analyses are potentially valuable for informing environmental policy and practice decisions. However, selective sampling of primary studies through searches exclusively using widely used bibliographic platform(s) could bias estimates of effect sizes. Such search strategies are common in environmental evidence reviews, and if risk of bias can be detected, this would provide the first empirical evidence that comprehensiveness of searches needs to be improved. We compare the impact of using single and multiple bibliographic platform(s) searches vs more comprehensive searches on estimates of mean effect sizes. We used 137 published meta-analyses, based on multiple source searches, analyzing 9388 studies: 8095 sourced from commercially published articles; and 1293 from grey literature and unpublished data. Single-platform and multiple-platform searches missed studies in 100 and 80 of the meta-analyses, respectively: 52 and 46 meta-analyses provided larger-effect estimates; 32 and 28 meta-analyses provided smaller-effect estimates; eight and four meta-analyses provided opposite direction of estimates; and two each were unable to estimate effects due to missing all studies. Further, we found significant positive log-linear relationships between proportions of studies missed and the deviations of mean effect sizes, suggesting that as the number of studies missed increases, deviation of mean effect size is likely to expand. We also found significant differences in mean effect sizes between indexed and non-indexed studies for 35% of meta-analyses, indicating high risk of bias when the searches were restricted. We conclude that the restricted searches are likely to lead to unrepresentative samples of studies and biased estimates of true effects.

Strengthen causal models for better conservation outcomes for human well-being.

BACKGROUND: Understanding how the conservation of nature can lead to improvement in human conditions is a research area with significant growth and attention. Progress towards effective conservation requires understanding mechanisms for achieving impact within complex social-ecological systems. Causal models are useful tools for defining plausible pathways from conservation actions to impacts on nature and people. Evaluating the potential of different strategies for delivering co-benefits for nature and people will require the use and testing of clear causal models that explicitly define the logic and assumptions behind cause and effect relationships. OBJECTIVES AND METHODS: In this study, we outline criteria for credible causal models and systematically evaluated their use in a broad base of literature (~1,000 peer-reviewed and grey literature articles from a published systematic evidence map) on links between nature-based conservation actions and human well-being impacts. RESULTS: Out of 1,027 publications identified, only ~20% of articles used any type of causal models to guide their work, and only 14 total articles fulfilled all criteria for credibility. Articles rarely tested the validity of models with empirical data. IMPLICATIONS: Not using causal models risks poorly defined strategies, misunderstanding of potential mechanisms for affecting change, inefficient use of resources, and focusing on implausible efforts for achieving sustainability.

Standardized reporting of the costs of management interventions for biodiversity conservation.

Effective conservation management interventions must combat threats and deliver benefits at costs that can be achieved within limited budgets. Considerable effort has focused on measuring the potential benefits of conservation interventions, but explicit quantification of the financial costs of implementation is rare. Even when costs have been quantified, haphazard and inconsistent reporting means published values are difficult to interpret. This reporting deficiency hinders progress toward a collective understanding of the financial costs of management interventions across projects and thus limits the ability to identify efficient solutions to conservation problems or attract adequate funding. We devised a standardized approach to describing financial costs reported for conservation interventions. The standards call for researchers and practitioners to describe the objective and outcome, context and methods, and scale of costed interventions, and to state which categories of costs are included and the currency and date for reported costs. These standards aim to provide enough contextual information that readers and future users can interpret the cost data appropriately. We suggest these standards be adopted by major conservation organizations, conservation science institutions, and journals so that cost reporting is comparable among studies. This would support shared learning and enhance the ability to identify and perform cost-effective conservation.

A systematic review of the effectiveness of liming to mitigate impacts of river acidification on fish and macro-invertebrates.

The addition of calcium carbonate to catchments or watercourses--liming--has been used widely to mitigate freshwater acidification but the abatement of acidifying emissions has led to questions about its effectiveness and necessity. We conducted a systematic review and meta-analysis of the impact of liming streams and rivers on two key groups of river organisms: fish and invertebrates. On average, liming increased the abundance and richness of acid-sensitive invertebrates and increased overall fish abundance, but benefits were variable and not guaranteed in all rivers. Where B-A-C-I designs (before-after-control-impact) were used to reduce bias, there was evidence that liming decreased overall invertebrate abundance. This systematic review indicates that liming has the potential to mitigate the symptoms of acidification in some instances, but effects are mixed. Future studies should use robust designs to isolate recovery due to liming from decreasing acid deposition, and assess factors affecting liming outcomes.

A collaboratively-derived science-policy research agenda.

The need for policy makers to understand science and for scientists to understand policy processes is widely recognised. However, the science-policy relationship is sometimes difficult and occasionally dysfunctional; it is also increasingly visible, because it must deal with contentious issues, or itself becomes a matter of public controversy, or both. We suggest that identifying key unanswered questions on the relationship between science and policy will catalyse and focus research in this field. To identify these questions, a collaborative procedure was employed with 52 participants selected to cover a wide range of experience in both science and policy, including people from government, non-governmental organisations, academia and industry. These participants consulted with colleagues and submitted 239 questions. An initial round of voting was followed by a workshop in which 40 of the most important questions were identified by further discussion and voting. The resulting list includes questions about the effectiveness of science-based decision-making structures; the nature and legitimacy of expertise; the consequences of changes such as increasing transparency; choices among different sources of evidence; the implications of new means of characterising and representing uncertainties; and ways in which policy and political processes affect what counts as authoritative evidence. We expect this exercise to identify important theoretical questions and to help improve the mutual understanding and effectiveness of those working at the interface of science and policy.

Realizing an effectiveness revolution in environmental management.

The environmental movement of the 20(th) century has evolved into a large, diverse and well-financed global community that is increasingly required to prove its worth. Though the environmental sector collects and uses data to determine the status of ecological and social systems, the effectiveness of the programs and policies it uses to affect this status remains largely untested. As governments and donor institutions insist on greater transparency, accountability and evidence of what works and what does not, much is being learned from other fields (e.g. health services, education, international development) and increasingly sophisticated approaches are emerging to manage effectiveness. For example, program evaluation, adaptive management, and systematic review provide frameworks and methods to collect and use information to measure and improve performance. However, the critical data and collaborations necessary for an effectiveness revolution are marginalized by technical, cultural and political obstacles. Learning from other fields, the environmental sector must exploit key leverage points, such as flows of information and self-organization, to overcome impediments and create incentives to initiate and realize an era of effectiveness in environmental management.

The why, what, and how of global biodiversity indicators beyond the 2010 target.

The 2010 biodiversity target agreed by signatories to the Convention on Biological Diversity directed the attention of conservation professionals toward the development of indicators with which to measure changes in biological diversity at the global scale. We considered why global biodiversity indicators are needed, what characteristics successful global indicators have, and how existing indicators perform. Because monitoring could absorb a large proportion of funds available for conservation, we believe indicators should be linked explicitly to monitoring objectives and decisions about which monitoring schemes deserve funding should be informed by predictions of the value of such schemes to decision making. We suggest that raising awareness among the public and policy makers, auditing management actions, and informing policy choices are the most important global monitoring objectives. Using four well-developed indicators of biological diversity (extent of forests, coverage of protected areas, Living Planet Index, Red List Index) as examples, we analyzed the characteristics needed for indicators to meet these objectives. We recommend that conservation professionals improve on existing indicators by eliminating spatial biases in data availability, fill gaps in information about ecosystems other than forests, and improve understanding of the way indicators respond to policy changes. Monitoring is not an end in itself, and we believe it is vital that the ultimate objectives of global monitoring of biological diversity inform development of new indicators.

A systematic review of evidence for the added benefits to health of exposure to natural environments.

BACKGROUND: There is increasing interest in the potential role of the natural environment in human health and well-being. However, the evidence-base for specific and direct health or well-being benefits of activity within natural compared to more synthetic environments has not been systematically assessed. METHODS: We conducted a systematic review to collate and synthesise the findings of studies that compare measurements of health or well-being in natural and synthetic environments. Effect sizes of the differences between environments were calculated and meta-analysis used to synthesise data from studies measuring similar outcomes. RESULTS: Twenty-five studies met the review inclusion criteria. Most of these studies were crossover or controlled trials that investigated the effects of short-term exposure to each environment during a walk or run. This included 'natural' environments, such as public parks and green university campuses, and synthetic environments, such as indoor and outdoor built environments. The most common outcome measures were scores of different self-reported emotions. Based on these data, a meta-analysis provided some evidence of a positive benefit of a walk or run in a natural environment in comparison to a synthetic environment. There was also some support for greater attention after exposure to a natural environment but not after adjusting effect sizes for pretest differences. Meta-analysis of data on blood pressure and cortisol concentrations found less evidence of a consistent difference between environments across studies. CONCLUSIONS: Overall, the studies are suggestive that natural environments may have direct and positive impacts on well-being, but support the need for investment in further research on this question to understand the general significance for public health.

Effectiveness of predator removal for enhancing bird populations.

Predation pressure on vulnerable bird species has made predator control an important issue for international nature conservation. Predator removal by culling or translocation is controversial, expensive, and time-consuming, and results are often temporary. Thus, it is important to assess its effectiveness from all available evidence. We used explicit systematic review methodology to determine the impact of predator removal on four measurable responses in birds: breeding performance (hatching success and fledging success) and population size (breeding and postbreeding). We used meta-analysis to summarize results from 83 predator removal studies from six continents. We also investigated whether characteristics of the prey, predator species, location, and study methodology explained heterogeneity in effect sizes. Removing predators increased hatching success, fledging success, and breeding populations. Removing all predator species achieved a significantly larger increase in breeding population than removing only a subset. Postbreeding population size was not improved on islands, or overall, but did increase on mainlands. Heterogeneity in effect sizes for the four population parameters was not explained by whether predators were native or introduced; prey were declining, migratory, or game species; or by the study methodology. Effect sizes for fledging success were smaller for ground-nesting birds than those that nest elsewhere, but the difference was not significant. We conclude that current evidence indicates that predator removal is an effective strategy for the conservation of vulnerable bird populations. Nevertheless, the ethical and practical problems associated with predator removal may lead managers to favor alternative, nonlethal solutions. Research is needed to provide and synthesize data to determine whether these are effective management practices for future policies on bird conservation.

Conservation focus on Europe: major conservation policy issues that need to be informed by conservation science.

Europe is one of the world's most densely populated continents and has a long history of human-dominated land- and seascapes. Europe is also at the forefront of developing and implementing multinational conservation efforts. In this contribution, we describe some top policy issues in Europe that need to be informed by high-quality conservation science. These include evaluation of the effectiveness of the Natura 2000 network of protected sites, implications of rapid economic and subsequent land-use change in Central and Eastern Europe, conservation of marine biodiversity and sustainability of fisheries, the effect of climate change on movement of species in highly fragmented landscapes, and attempts to assess the economic value of ecosystem services and biodiversity. Broad policy issues such as those identified are not easily amenable to scientific experiment. A key challenge at the science-policy interface is to identify the research questions underlying these problem areas so that conservation science can provide evidence to underpin future policy development.

Effectiveness of engineered in-stream structure mitigation measures to increase salmonid abundance: a systematic review.

Engineered in-stream structures are often installed to increase salmonid abundance, either for commercial gain in fisheries or for conservation purposes in degraded habitats. Having been in widespread use for the last 80 years, at an estimated cost of hundreds of millions of U.S. dollars each year, the effectiveness of these structures is still widely debated in the literature. Many studies of varying quality have been undertaken that attempt to address this issue, but it has proved difficult for practitioners to develop a consensus regarding the utility of these structures, despite their continued use. Systematic review methodology was used to formally synthesize empirical evidence regarding the effectiveness of engineered in-stream structures as a management tool to increase salmonid abundance. Meta-analysis shows that evidence regarding the effectiveness of in-stream devices is equivocal. Heterogeneity is significant both for population size and local habitat preference. This heterogeneity is related to stream width, with in-stream devices being less effective in larger streams. Consequently, widespread use of in-stream structures for restoration, particularly in larger streams, is not supported by the current scientific evidence base.

The effectiveness of asulam for bracken (Pteridium aquilinum) control in the United Kingdom: a meta-analysis.

Bracken (Pteridium aquilinum) is a major problem for livestock-based extensive agriculture, conservation, recreation, and game management globally. It is an invasive species often achieving dominance to the detriment of other species. Control is essential to maintain plant communities such as grassland and lowland heath or if extensive grazing by domestic stock, particularly sheep, is to be viable on upland margins. Bracken is managed primarily by herbicide application or cutting but other techniques including rolling, burning, and grazing are also utilized. Here we evaluate the evidence regarding the effectiveness of asulam for the control of bracken. Thirteen studies provided data for meta-analyses which demonstrate that application of the herbicide asulam reduces bracken abundance. Subgroup analyses indicate that the number of treatments had an important impact, with multiple follow-up treatments more effective than one or two treatments. Management practices should reflect the requirement for repeated follow-up. There is insufficient available experimental evidence for quantitative analysis of the effectiveness of other management interventions, although this results from lack of reporting in papers where cutting and comparisons of cutting and asulam application are concerned. Systematic searching and meta-analytical synthesis have effectively demonstrated the limits of current knowledge, based on recorded empirical evidence, and increasing the call for more rigorous monitoring of bracken control techniques. Lack of experimental evidence on the effectiveness of management such as rolling or grazing with hardy cattle breeds contrasts with the widespread acceptance of their use through dissemination of experience.