Predicting and reasoning about replicability using structured groups.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

What is a Good Calibration Question?

Weighted aggregation of expert judgments based on their performance on calibration questions may improve mathematically aggregated judgments relative to equal weights. However, obtaining validated, relevant calibration questions can be difficult. If so, should analysts settle for equal weights? Or should they use calibration questions that are easier to obtain but less relevant? In this article, we examine what happens to the out-of-sample performance of weighted aggregations of the classical model (CM) compared to equal weighted aggregations when the set of calibration questions includes many so-called "irrelevant" questions, those that might ordinarily be considered to be outside the domain of the questions of interest. We find that performance weighted aggregations outperform equal weights on the combined CM score, but not on statistical accuracy (i.e., calibration). Importantly, there was no appreciable difference in performance when weights were developed on relevant versus irrelevant questions. Experts were unable to adapt their knowledge across vastly different domains, and in-sample validation did not accurately predict out-of-sample performance on irrelevant questions. We suggest that if relevant calibration questions cannot be found, then analysts should use equal weights, and draw on alternative techniques to improve judgments. Our study also indicates limits to the predictive accuracy of performance weighted aggregation, and the degree to which expertise can be adapted across domains. We note limitations in our study and urge further research into the effect of question type on the reliability of performance weighted aggregations.

Strengthening conservation science as a crisis discipline by addressing challenges of precaution, privilege, and individualism.

Conservation science deals with crises and supports policy interventions devised to mitigate highly uncertain threats that pose irreversible harm. When conventional policy tools, such as quantitative risk assessments, are insufficient, the precautionary principle provides a practical framework and range of robust heuristics. Yet, precaution is often resisted in many policy arenas, especially those involving powerful self-interests, and this resistance is compounded by structures of privilege and competitive individualism in science. We describe key drivers and effects of such resistance in conservation science. These include a loss of rigor under uncertainty, an erosion of crisis response capabilities, and a further reinforcement of privileged interests in conservation politics. We recommend open acknowledgement of the pressures exerted by power inside science; greater recognition for the value of the precautionary principle under uncertainty; deliberate measures to resist competitive individualism; support for blind review, open science, and data sharing; and a shift from hierarchical multidisciplinarity toward more egalitarian transdisciplinarity to accelerate advances in conservation science. Article impact statement: Precautionary principle, privilege structures among disciplines, and culture of individualism link to effective conservation policy making.

Fortalecimiento de las Ciencias de la Conservación como Disciplinas de Crisis Resumen Las ciencias de la conservación tratan con crisis y respaldan a muchas intervenciones políticas para mitigar las amenazas altamente inciertas que representan un daño irreversible. El principio de precaución proporciona un marco práctico y una gama de heurística sólida cuando son insuficientes las herramientas convencionales de políticas como las evaluaciones cuantitativas de riesgo. Aun así, con frecuencia se niega el uso de la precaución en muchas arenas políticas, especialmente en aquellas que involucran intereses propios de mucho poder, y esta negación se agrava con las estructuras de privilegio y el individualismo competitivo presentes en la ciencia. Describimos los factores y efectos clave de dicha resistencia en las ciencias de la conservación. Estos incluyen la pérdida del rigor bajo la incertidumbre, un desgaste de las capacidades de respuesta a la crisis y un reforzamiento más profundo de los intereses privilegiados en las políticas de conservación. Recomendamos que se realice una aceptación abierta de las presiones ejercidas por el poder dentro de la ciencia; un mayor reconocimiento del valor del principio de precaución bajo la incertidumbre; que se lleven a cabo medidas deliberadas para oponerse al individualismo competitivo; que se apoye a las revisiones a ciegas, la ciencia abierta y la difusión de datos; y que se realice un cambio de la multidisciplinariedad jerárquica a una transdisciplinariedad más igualitaria para acelerar los avances dentro de las ciencias de la conservación.

Traits explain invasion of alien plants into tropical rainforests.

1. The establishment of new botanic gardens in tropical regions highlights a need for weed risk assessment tools suitable for tropical ecosystems. The relevance of plant traits for invasion into tropical rainforests has not been well studied.2. Working in and around four botanic gardens in Indonesia where 590 alien species have been planted, we estimated the effect of four plant traits, plus time since species introduction, on: (a) the naturalization probability and (b) abundance (density) of naturalized species in adjacent native tropical rainforests; and (c) the distance that naturalized alien plants have spread from the botanic gardens.3. We found that specific leaf area (SLA) strongly differentiated 23 naturalized from 78 non-naturalized alien species (randomly selected from 577 non-naturalized species) in our study. These trends may indicate that aliens with high SLA, which had a higher probability of naturalization, benefit from at least two factors when establishing in tropical forests: high growth rates and occupation of forest gaps. Naturalized aliens had high SLA and tended to be short. However, plant height was not significantly related to species' naturalization probability when considered alongside other traits.4. Alien species that were present in the gardens for over 30 years and those with small seeds also had higher probabilities of becoming naturalized, indicating that garden plants can invade the understorey of closed canopy tropical rainforests, especially when invading species are shade tolerant and have sufficient time to establish.5. On average, alien species that were not animal dispersed spread 78 m further into the forests and were more likely to naturalize than animal-dispersed species. We did not detect relationships between the measured traits and estimated density of naturalized aliens in the adjacent forests.6. Synthesis: Traits were able to differentiate alien species from botanic gardens that naturalized in native forest from those that did not; this is promising for developing trait-based risk assessment in the tropics. To limit the risk of invasion and spread into adjacent native forests, we suggest tropical botanic gardens avoid planting alien species with fast carbon capture strategies and those that are shade tolerant.

Protecting biodiversity and economic returns in resource-rich tropical forests.

In pursuit of socioeconomic development, many countries are expanding oil and mineral extraction into tropical forests. These activities seed access to remote, biologically rich areas, thereby endangering global biodiversity. Here we demonstrate that conservation solutions that effectively balance the protection of biodiversity and economic revenues are possible in biologically valuable regions. Using spatial data on oil profits and predicted species and ecosystem extents, we optimise the protection of 741 terrestrial species and 20 ecosystems of the Ecuadorian Amazon, across a range of opportunity costs (i.e. sacrifices of extractive profit). For such an optimisation, giving up 5% of a year's oil profits (US$ 221 million) allows for a protected area network that retains of an average of 65% of the extent of each species/ecosystem. This performance far exceeds that of the network produced by simple land area optimisation which requires a sacrifice of approximately 40% of annual oil profits (US$ 1.7 billion), and uses only marginally less land, to achieve equivalent levels of ecological protection. Applying spatial statistics to remotely sensed, historic deforestation data, we further focus the optimisation to areas most threatened by imminent forest loss. We identify Emergency Conservation Targets: areas that are essential to a cost-effective conservation reserve network and at imminent risk of destruction, thus requiring urgent and effective protection. Governments should employ the methods presented here when considering extractive led development options, to responsibly manage the associated ecological-economic trade-offs and protect natural capital. Article Impact Statement: Governments controlling resource extraction from tropical forests can arrange production and conservation to retain biodiversity and profits. This article is protected by copyright. All rights reserved.

Weighting and aggregating expert ecological judgments.

Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.

Traits influence detection of exotic plant species in tropical forests.

Detecting exotic plant species is essential for invasive species management. By accounting for factors likely to affect species' detection rates (e.g. survey conditions, observer experience), detectability models can help choose search methods and allocate search effort. Integrating information on species' traits can refine detectability models, and might be particularly valuable if these traits can help improve estimates of detectability where data on particular species are rare. Analysing data collected during line transect distance sampling surveys in Indonesia, we used a multi-species hierarchical distance sampling model to evaluate how plant height, leaf size, leaf shape, and survey location influenced plant species detectability in secondary tropical rainforests. Detectability of the exotic plant species increased with plant height and leaf size. Detectability varied among the different survey locations. We failed to detect a clear effect of leaf shape on detectability. This study indicates that information on traits might improve predictions about exotic species detection, which can then be used to optimise the allocation of search effort for efficient species management. The innovation of the study lies in the multi-species distance sampling model, where the distance-detection function depends on leaf traits and height. The method can be applied elsewhere, including for different traits that may be relevant in other contexts. Trait-based multispecies distance sampling can be a practical approach for sampling exotic shrubs, herbs, or grasses species in the understorey of tropical forests.

Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management.

INTRODUCTION: Natural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements. AIMS AND METHODS: In this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol ('Investigate', 'Discuss', 'Estimate', 'Aggregate') through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness. RESULTS AND CONCLUSIONS: The results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

The Value of Performance Weights and Discussion in Aggregated Expert Judgments.

In risky situations characterized by imminent decisions, scarce resources, and insufficient data, policymakers rely on experts to estimate model parameters and their associated uncertainties. Different elicitation and aggregation methods can vary substantially in their efficacy and robustness. While it is generally agreed that biases in expert judgments can be mitigated using structured elicitations involving groups rather than individuals, there is still some disagreement about how to best elicit and aggregate judgments. This mostly concerns the merits of using performance-based weighting schemes to combine judgments of different individuals (rather than assigning equal weights to individual experts), and the way that interaction between experts should be handled. This article aims to contribute to, and complement, the ongoing discussion on these topics.

Metaresearch for Evaluating Reproducibility in Ecology and Evolution.

Recent replication projects in other disciplines have uncovered disturbingly low levels of reproducibility, suggesting that those research literatures may contain unverifiable claims. The conditions contributing to irreproducibility in other disciplines are also present in ecology. These include a large discrepancy between the proportion of "positive" or "significant" results and the average statistical power of empirical research, incomplete reporting of sampling stopping rules and results, journal policies that discourage replication studies, and a prevailing publish-or-perish research culture that encourages questionable research practices. We argue that these conditions constitute sufficient reason to systematically evaluate the reproducibility of the evidence base in ecology and evolution. In some cases, the direct replication of ecological research is difficult because of strong temporal and spatial dependencies, so here, we propose metaresearch projects that will provide proxy measures of reproducibility.

Clarifying values, risk perceptions, and attitudes to resolve or avoid social conflicts in invasive species management.

Decision makers and researchers recognize the need to effectively confront the social dimensions and conflicts inherent to invasive species research and management. Yet, despite numerous contentious situations that have arisen, no systematic evaluation of the literature has examined the commonalities in the patterns and types of these emergent social issues. Using social and ecological keywords, we reviewed trends in the social dimensions of invasive species research and management and the sources and potential solutions to problems and conflicts that arise around invasive species. We integrated components of cognitive hierarchy theory and risk perceptions theory to provide a conceptual framework to identify, distinguish, and provide understanding of the driving factors underlying disputes associated with invasive species. In the ISI Web of Science database, we found 15,915 peer-reviewed publications on biological invasions, 124 of which included social dimensions of this phenomenon. Of these 124, 28 studies described specific contentious situations. Social approaches to biological invasions have emerged largely in the last decade and have focused on both environmental social sciences and resource management. Despite being distributed in a range of journals, these 124 articles were concentrated mostly in ecology and conservation-oriented outlets. We found that conflicts surrounding invasive species arose based largely on differences in value systems and to a lesser extent stakeholder and decision maker's risk perceptions. To confront or avoid such situations, we suggest integrating the plurality of environmental values into invasive species research and management via structured decision making techniques, which enhance effective risk communication that promotes trust and confidence between stakeholders and decision makers.

Strategic foresight: how planning for the unpredictable can improve environmental decision-making.

Advanced warning of potential new opportunities and threats related to biodiversity allows decision-makers to act strategically to maximize benefits or minimize costs. Strategic foresight explores possible futures, their consequences for decisions, and the actions that promote more desirable futures. Foresight tools, such as horizon scanning and scenario planning, are increasingly used by governments and business for long-term strategic planning and capacity building. These tools are now being applied in ecology, although generally not as part of a comprehensive foresight strategy. We highlight several ways foresight could play a more significant role in environmental decisions by: monitoring existing problems, highlighting emerging threats, identifying promising new opportunities, testing the resilience of policies, and defining a research agenda.

Voting systems for environmental decisions.

Voting systems aggregate preferences efficiently and are often used for deciding conservation priorities. Desirable characteristics of voting systems include transitivity, completeness, and Pareto optimality, among others. Voting systems that are common and potentially useful for environmental decision making include simple majority, approval, and preferential voting. Unfortunately, no voting system can guarantee an outcome, while also satisfying a range of very reasonable performance criteria. Furthermore, voting methods may be manipulated by decision makers and strategic voters if they have knowledge of the voting patterns and alliances of others in the voting populations. The difficult properties of voting systems arise in routine decision making when there are multiple criteria and management alternatives. Because each method has flaws, we do not endorse one method. Instead, we urge organizers to be transparent about the properties of proposed voting systems and to offer participants the opportunity to approve the voting system as part of the ground rules for operation of a group.

Addressing a critique of the TEASI framework for invasive species risk assessment.

We address criticism that the Transport, Establishment, Abundance, Spread, Impact (TEASI) framework does not facilitate objective mapping of risk assessment methods nor defines best practice. We explain why TEASI is appropriate for mapping, despite inherent challenges, and how TEASI offers considerations for best practices, rather than suggesting one best practice.

Scientific foundations for an IUCN Red List of ecosystems.

An understanding of risks to biodiversity is needed for planning action to slow current rates of decline and secure ecosystem services for future human use. Although the IUCN Red List criteria provide an effective assessment protocol for species, a standard global assessment of risks to higher levels of biodiversity is currently limited. In 2008, IUCN initiated development of risk assessment criteria to support a global Red List of ecosystems. We present a new conceptual model for ecosystem risk assessment founded on a synthesis of relevant ecological theories. To support the model, we review key elements of ecosystem definition and introduce the concept of ecosystem collapse, an analogue of species extinction. The model identifies four distributional and functional symptoms of ecosystem risk as a basis for assessment criteria: A) rates of decline in ecosystem distribution; B) restricted distributions with continuing declines or threats; C) rates of environmental (abiotic) degradation; and D) rates of disruption to biotic processes. A fifth criterion, E) quantitative estimates of the risk of ecosystem collapse, enables integrated assessment of multiple processes and provides a conceptual anchor for the other criteria. We present the theoretical rationale for the construction and interpretation of each criterion. The assessment protocol and threat categories mirror those of the IUCN Red List of species. A trial of the protocol on terrestrial, subterranean, freshwater and marine ecosystems from around the world shows that its concepts are workable and its outcomes are robust, that required data are available, and that results are consistent with assessments carried out by local experts and authorities. The new protocol provides a consistent, practical and theoretically grounded framework for establishing a systematic Red List of the world's ecosystems. This will complement the Red List of species and strengthen global capacity to report on and monitor the status of biodiversity.