Environmental evidence in action: on the science and practice of evidence synthesis and evidence-based decision-making.

In civil society we expect that policy and management decisions will be made using the best available evidence. Yet, it is widely known that there are many barriers that limit the extent to which that occurs. One way to overcome these barriers is via robust, comprehensive, transparent and repeatable evidence syntheses (such as systematic reviews) that attempt to minimize various forms of bias to present a summary of existing knowledge for decision-making purposes. Relative to other disciplines (e.g., health care, education), such evidence-based decision-making remains relatively nascent for environment management despite major threats to humanity, such as the climate, pollution and biodiversity crises demonstrating that human well-being is inextricably linked to the biophysical environment. Fortunately, there are a growing number of environmental evidence syntheses being produced that can be used by decision makers. It is therefore an opportune time to reflect on the science and practice of evidence-based decision-making in environment management to understand the extent to which evidence syntheses are embraced and applied in practice. Here we outline a number of key questions related to the use of environmental evidence that need to be explored in an effort to enhance evidence-based decision-making. There is an urgent need for research involving methods from social science, behavioural sciences, and public policy to understand the basis for patterns and trends in environmental evidence use (or misuse or ignorance). There is also a need for those who commission and produce evidence syntheses, as well as the end users of these syntheses to reflect on their experiences and share them with the broader evidence-based practice community to identify needs and opportunities for advancing the entire process of evidence-based practice. It is our hope that the ideas shared here will serve as a roadmap for additional scholarship that will collectively enhance evidence-based decision-making and ultimately benefit the environment and humanity.

Author Correction: Eight problems with literature reviews and how to fix them.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Eight problems with literature reviews and how to fix them.

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a 'critical friend' role in supporting would-be systematic reviews and avoiding individual responses to police use of the 'systematic review' label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

The CEEDER database of evidence reviews: An open-access evidence service for researchers and decision-makers.

Evidence-informed decision-making aims to deliver effective actions informed by the best available evidence. Given the large quantity of primary literature, and time constraints faced by policy-makers and practitioners, well-conducted evidence reviews can provide a valuable resource to support decision-making. However, previous research suggests that some evidence reviews may not be sufficiently reliable to inform decisions in the environmental sector due to low standards of conduct and reporting. While some evidence reviews are of high reliability, there is currently no way for policy-makers and practitioners to quickly and easily find them among the many lower reliability ones. Alongside this lack of transparency, there is little incentive or support for review authors, editors and peer-reviewers to improve reliability. To address these issues, we introduce a new online, freely available and first-of-its-kind evidence service: the Collaboration for Environmental Evidence Database of Evidence Reviews (CEEDER: www.environmentalevidence.org/ceeder). CEEDER aims to transform communication of evidence review reliability to researchers, policy-makers and practitioners through independent assessment of key aspects of the conduct, reporting and data limitations of available evidence reviews claiming to assess environmental impacts or the effectiveness of interventions relevant to policy and practice. At the same time, CEEDER will provide support to improve the standards of future evidence reviews and support evidence translation and knowledge mobilisation to help inform environmental decision-making.

Evidence Synthesis International (ESI): Position Statement.

This paper is the initial Position Statement of Evidence Synthesis International, a new partnership of organizations that produce, support and use evidence synthesis around the world. The paper (i) argues for the importance of synthesis as a research exercise to clarify what is known from research evidence to inform policy, practice and personal decision making; (ii) discusses core issues for research synthesis such as the role of research evidence in decision making, the role of perspectives, participation and democracy in research and synthesis as a core component of evidence ecosystems; (iii) argues for 9 core principles for ESI on the nature and role of research synthesis; and (iv) lists the 5 main goals of ESI as a coordinating partnership for promoting and enabling the production and use of research synthesis.

Assessing the risk of bias in choice of search sources for environmental meta-analyses.

Results of meta-analyses are potentially valuable for informing environmental policy and practice decisions. However, selective sampling of primary studies through searches exclusively using widely used bibliographic platform(s) could bias estimates of effect sizes. Such search strategies are common in environmental evidence reviews, and if risk of bias can be detected, this would provide the first empirical evidence that comprehensiveness of searches needs to be improved. We compare the impact of using single and multiple bibliographic platform(s) searches vs more comprehensive searches on estimates of mean effect sizes. We used 137 published meta-analyses, based on multiple source searches, analyzing 9388 studies: 8095 sourced from commercially published articles; and 1293 from grey literature and unpublished data. Single-platform and multiple-platform searches missed studies in 100 and 80 of the meta-analyses, respectively: 52 and 46 meta-analyses provided larger-effect estimates; 32 and 28 meta-analyses provided smaller-effect estimates; eight and four meta-analyses provided opposite direction of estimates; and two each were unable to estimate effects due to missing all studies. Further, we found significant positive log-linear relationships between proportions of studies missed and the deviations of mean effect sizes, suggesting that as the number of studies missed increases, deviation of mean effect size is likely to expand. We also found significant differences in mean effect sizes between indexed and non-indexed studies for 35% of meta-analyses, indicating high risk of bias when the searches were restricted. We conclude that the restricted searches are likely to lead to unrepresentative samples of studies and biased estimates of true effects.

Strengthen causal models for better conservation outcomes for human well-being.

BACKGROUND: Understanding how the conservation of nature can lead to improvement in human conditions is a research area with significant growth and attention. Progress towards effective conservation requires understanding mechanisms for achieving impact within complex social-ecological systems. Causal models are useful tools for defining plausible pathways from conservation actions to impacts on nature and people. Evaluating the potential of different strategies for delivering co-benefits for nature and people will require the use and testing of clear causal models that explicitly define the logic and assumptions behind cause and effect relationships. OBJECTIVES AND METHODS: In this study, we outline criteria for credible causal models and systematically evaluated their use in a broad base of literature (~1,000 peer-reviewed and grey literature articles from a published systematic evidence map) on links between nature-based conservation actions and human well-being impacts. RESULTS: Out of 1,027 publications identified, only ~20% of articles used any type of causal models to guide their work, and only 14 total articles fulfilled all criteria for credibility. Articles rarely tested the validity of models with empirical data. IMPLICATIONS: Not using causal models risks poorly defined strategies, misunderstanding of potential mechanisms for affecting change, inefficient use of resources, and focusing on implausible efforts for achieving sustainability.

Standardized reporting of the costs of management interventions for biodiversity conservation.

Effective conservation management interventions must combat threats and deliver benefits at costs that can be achieved within limited budgets. Considerable effort has focused on measuring the potential benefits of conservation interventions, but explicit quantification of the financial costs of implementation is rare. Even when costs have been quantified, haphazard and inconsistent reporting means published values are difficult to interpret. This reporting deficiency hinders progress toward a collective understanding of the financial costs of management interventions across projects and thus limits the ability to identify efficient solutions to conservation problems or attract adequate funding. We devised a standardized approach to describing financial costs reported for conservation interventions. The standards call for researchers and practitioners to describe the objective and outcome, context and methods, and scale of costed interventions, and to state which categories of costs are included and the currency and date for reported costs. These standards aim to provide enough contextual information that readers and future users can interpret the cost data appropriately. We suggest these standards be adopted by major conservation organizations, conservation science institutions, and journals so that cost reporting is comparable among studies. This would support shared learning and enhance the ability to identify and perform cost-effective conservation.

A systematic review of the effectiveness of liming to mitigate impacts of river acidification on fish and macro-invertebrates.

The addition of calcium carbonate to catchments or watercourses--liming--has been used widely to mitigate freshwater acidification but the abatement of acidifying emissions has led to questions about its effectiveness and necessity. We conducted a systematic review and meta-analysis of the impact of liming streams and rivers on two key groups of river organisms: fish and invertebrates. On average, liming increased the abundance and richness of acid-sensitive invertebrates and increased overall fish abundance, but benefits were variable and not guaranteed in all rivers. Where B-A-C-I designs (before-after-control-impact) were used to reduce bias, there was evidence that liming decreased overall invertebrate abundance. This systematic review indicates that liming has the potential to mitigate the symptoms of acidification in some instances, but effects are mixed. Future studies should use robust designs to isolate recovery due to liming from decreasing acid deposition, and assess factors affecting liming outcomes.

A collaboratively-derived science-policy research agenda.

The need for policy makers to understand science and for scientists to understand policy processes is widely recognised. However, the science-policy relationship is sometimes difficult and occasionally dysfunctional; it is also increasingly visible, because it must deal with contentious issues, or itself becomes a matter of public controversy, or both. We suggest that identifying key unanswered questions on the relationship between science and policy will catalyse and focus research in this field. To identify these questions, a collaborative procedure was employed with 52 participants selected to cover a wide range of experience in both science and policy, including people from government, non-governmental organisations, academia and industry. These participants consulted with colleagues and submitted 239 questions. An initial round of voting was followed by a workshop in which 40 of the most important questions were identified by further discussion and voting. The resulting list includes questions about the effectiveness of science-based decision-making structures; the nature and legitimacy of expertise; the consequences of changes such as increasing transparency; choices among different sources of evidence; the implications of new means of characterising and representing uncertainties; and ways in which policy and political processes affect what counts as authoritative evidence. We expect this exercise to identify important theoretical questions and to help improve the mutual understanding and effectiveness of those working at the interface of science and policy.

Realizing an effectiveness revolution in environmental management.

The environmental movement of the 20(th) century has evolved into a large, diverse and well-financed global community that is increasingly required to prove its worth. Though the environmental sector collects and uses data to determine the status of ecological and social systems, the effectiveness of the programs and policies it uses to affect this status remains largely untested. As governments and donor institutions insist on greater transparency, accountability and evidence of what works and what does not, much is being learned from other fields (e.g. health services, education, international development) and increasingly sophisticated approaches are emerging to manage effectiveness. For example, program evaluation, adaptive management, and systematic review provide frameworks and methods to collect and use information to measure and improve performance. However, the critical data and collaborations necessary for an effectiveness revolution are marginalized by technical, cultural and political obstacles. Learning from other fields, the environmental sector must exploit key leverage points, such as flows of information and self-organization, to overcome impediments and create incentives to initiate and realize an era of effectiveness in environmental management.

The why, what, and how of global biodiversity indicators beyond the 2010 target.

The 2010 biodiversity target agreed by signatories to the Convention on Biological Diversity directed the attention of conservation professionals toward the development of indicators with which to measure changes in biological diversity at the global scale. We considered why global biodiversity indicators are needed, what characteristics successful global indicators have, and how existing indicators perform. Because monitoring could absorb a large proportion of funds available for conservation, we believe indicators should be linked explicitly to monitoring objectives and decisions about which monitoring schemes deserve funding should be informed by predictions of the value of such schemes to decision making. We suggest that raising awareness among the public and policy makers, auditing management actions, and informing policy choices are the most important global monitoring objectives. Using four well-developed indicators of biological diversity (extent of forests, coverage of protected areas, Living Planet Index, Red List Index) as examples, we analyzed the characteristics needed for indicators to meet these objectives. We recommend that conservation professionals improve on existing indicators by eliminating spatial biases in data availability, fill gaps in information about ecosystems other than forests, and improve understanding of the way indicators respond to policy changes. Monitoring is not an end in itself, and we believe it is vital that the ultimate objectives of global monitoring of biological diversity inform development of new indicators.

A systematic review of evidence for the added benefits to health of exposure to natural environments.

BACKGROUND: There is increasing interest in the potential role of the natural environment in human health and well-being. However, the evidence-base for specific and direct health or well-being benefits of activity within natural compared to more synthetic environments has not been systematically assessed. METHODS: We conducted a systematic review to collate and synthesise the findings of studies that compare measurements of health or well-being in natural and synthetic environments. Effect sizes of the differences between environments were calculated and meta-analysis used to synthesise data from studies measuring similar outcomes. RESULTS: Twenty-five studies met the review inclusion criteria. Most of these studies were crossover or controlled trials that investigated the effects of short-term exposure to each environment during a walk or run. This included 'natural' environments, such as public parks and green university campuses, and synthetic environments, such as indoor and outdoor built environments. The most common outcome measures were scores of different self-reported emotions. Based on these data, a meta-analysis provided some evidence of a positive benefit of a walk or run in a natural environment in comparison to a synthetic environment. There was also some support for greater attention after exposure to a natural environment but not after adjusting effect sizes for pretest differences. Meta-analysis of data on blood pressure and cortisol concentrations found less evidence of a consistent difference between environments across studies. CONCLUSIONS: Overall, the studies are suggestive that natural environments may have direct and positive impacts on well-being, but support the need for investment in further research on this question to understand the general significance for public health.

Effectiveness of predator removal for enhancing bird populations.

Predation pressure on vulnerable bird species has made predator control an important issue for international nature conservation. Predator removal by culling or translocation is controversial, expensive, and time-consuming, and results are often temporary. Thus, it is important to assess its effectiveness from all available evidence. We used explicit systematic review methodology to determine the impact of predator removal on four measurable responses in birds: breeding performance (hatching success and fledging success) and population size (breeding and postbreeding). We used meta-analysis to summarize results from 83 predator removal studies from six continents. We also investigated whether characteristics of the prey, predator species, location, and study methodology explained heterogeneity in effect sizes. Removing predators increased hatching success, fledging success, and breeding populations. Removing all predator species achieved a significantly larger increase in breeding population than removing only a subset. Postbreeding population size was not improved on islands, or overall, but did increase on mainlands. Heterogeneity in effect sizes for the four population parameters was not explained by whether predators were native or introduced; prey were declining, migratory, or game species; or by the study methodology. Effect sizes for fledging success were smaller for ground-nesting birds than those that nest elsewhere, but the difference was not significant. We conclude that current evidence indicates that predator removal is an effective strategy for the conservation of vulnerable bird populations. Nevertheless, the ethical and practical problems associated with predator removal may lead managers to favor alternative, nonlethal solutions. Research is needed to provide and synthesize data to determine whether these are effective management practices for future policies on bird conservation.

Conservation focus on Europe: major conservation policy issues that need to be informed by conservation science.

Europe is one of the world's most densely populated continents and has a long history of human-dominated land- and seascapes. Europe is also at the forefront of developing and implementing multinational conservation efforts. In this contribution, we describe some top policy issues in Europe that need to be informed by high-quality conservation science. These include evaluation of the effectiveness of the Natura 2000 network of protected sites, implications of rapid economic and subsequent land-use change in Central and Eastern Europe, conservation of marine biodiversity and sustainability of fisheries, the effect of climate change on movement of species in highly fragmented landscapes, and attempts to assess the economic value of ecosystem services and biodiversity. Broad policy issues such as those identified are not easily amenable to scientific experiment. A key challenge at the science-policy interface is to identify the research questions underlying these problem areas so that conservation science can provide evidence to underpin future policy development.

Effectiveness of engineered in-stream structure mitigation measures to increase salmonid abundance: a systematic review.

Engineered in-stream structures are often installed to increase salmonid abundance, either for commercial gain in fisheries or for conservation purposes in degraded habitats. Having been in widespread use for the last 80 years, at an estimated cost of hundreds of millions of U.S. dollars each year, the effectiveness of these structures is still widely debated in the literature. Many studies of varying quality have been undertaken that attempt to address this issue, but it has proved difficult for practitioners to develop a consensus regarding the utility of these structures, despite their continued use. Systematic review methodology was used to formally synthesize empirical evidence regarding the effectiveness of engineered in-stream structures as a management tool to increase salmonid abundance. Meta-analysis shows that evidence regarding the effectiveness of in-stream devices is equivocal. Heterogeneity is significant both for population size and local habitat preference. This heterogeneity is related to stream width, with in-stream devices being less effective in larger streams. Consequently, widespread use of in-stream structures for restoration, particularly in larger streams, is not supported by the current scientific evidence base.

The effectiveness of asulam for bracken (Pteridium aquilinum) control in the United Kingdom: a meta-analysis.

Bracken (Pteridium aquilinum) is a major problem for livestock-based extensive agriculture, conservation, recreation, and game management globally. It is an invasive species often achieving dominance to the detriment of other species. Control is essential to maintain plant communities such as grassland and lowland heath or if extensive grazing by domestic stock, particularly sheep, is to be viable on upland margins. Bracken is managed primarily by herbicide application or cutting but other techniques including rolling, burning, and grazing are also utilized. Here we evaluate the evidence regarding the effectiveness of asulam for the control of bracken. Thirteen studies provided data for meta-analyses which demonstrate that application of the herbicide asulam reduces bracken abundance. Subgroup analyses indicate that the number of treatments had an important impact, with multiple follow-up treatments more effective than one or two treatments. Management practices should reflect the requirement for repeated follow-up. There is insufficient available experimental evidence for quantitative analysis of the effectiveness of other management interventions, although this results from lack of reporting in papers where cutting and comparisons of cutting and asulam application are concerned. Systematic searching and meta-analytical synthesis have effectively demonstrated the limits of current knowledge, based on recorded empirical evidence, and increasing the call for more rigorous monitoring of bracken control techniques. Lack of experimental evidence on the effectiveness of management such as rolling or grazing with hardy cattle breeds contrasts with the widespread acceptance of their use through dissemination of experience.

Guidelines for systematic review in conservation and environmental management.

An increasing number of applied disciplines are utilizing evidence-based frameworks to review and disseminate the effectiveness of management and policy interventions. The rationale is that increased accessibility of the best available evidence will provide a more efficient and less biased platform for decision making. We argue that there are significant benefits for conservation in using such a framework, but the scientific community needs to undertake and disseminate more systematic reviews before the full benefit can be realized. We devised a set of guidelines for undertaking formalized systematic review, based on a health services model. The guideline stages include planning and conducting a review, including protocol formation, search strategy, data inclusion, data extraction, and analysis. Review dissemination is addressed in terms of current developments and future plans for a Web-based open-access library. By the use of case studies we highlight critical modifications to guidelines for protocol formulation, data-quality assessment, data extraction, and data synthesis for conservation and environmental management. Ecological data presented significant but soluble challenges for the systematic review process, particularly in terms of the quantity, accessibility, and diverse quality of available data. In the field of conservation and environmental management there needs to be further engagement of scientists and practitioners to develop and take ownership of an evidence-based framework.

Effectiveness of management interventions to control invasion by Rhododendron ponticum.

Rhododendron ponticum is an invasive species in many countries, including the United Kingdom, Ireland, Belgium, and France. It poses a serious threat to native flora and fauna, as it is capable of altering entire seminatural communities through its vigorous spread. Control is essential if the conservation value of some communities, such as oak woodland and lowland heath, are to be successfully maintained. Commonly used interventions are herbicide application, herbicide application postcut, and cutting (manual or mechanical) alone. Various techniques have been developed to apply these interventions, but often retreatment of the area is required, increasing the cost of control. Here, we evaluate the effectiveness of some commonly used interventions for R. ponticum control using a systematic review methodology. Eleven studies provided data for statistical analysis. Meta-analyses of captured data show that postcut application of the herbicide Glyphosate or applying the herbicides Metsulfuron-methyl or Imazapyr (no cut) can effectively reduce a R. ponticum stand. There is insufficient available experimental evidence for effectiveness of any other intervention. The systematic review process has demonstrated the lack of replicated studies with controls or long-term monitoring and increases the call for more rigorous monitoring of all conservation management interventions. The quality of experimental evidence of the effectiveness of some interventions contrasts with the acceptance of their use through dissemination of experience. The collection and objective review of experience will require active collaboration of organizations concerned with R. ponticum control.