Predicting and reasoning about replicability using structured groups.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Reimagining peer review as an expert elicitation process.

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the 'best' experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Challenges in estimation, uncertainty quantification and elicitation for pandemic modelling.

The estimation of parameters and model structure for informing infectious disease response has become a focal point of the recent pandemic. However, it has also highlighted a plethora of challenges remaining in the fast and robust extraction of information using data and models to help inform policy. In this paper, we identify and discuss four broad challenges in the estimation paradigm relating to infectious disease modelling, namely the Uncertainty Quantification framework, data challenges in estimation, model-based inference and prediction, and expert judgement. We also postulate priorities in estimation methodology to facilitate preparation for future pandemics.

What is a Good Calibration Question?

Weighted aggregation of expert judgments based on their performance on calibration questions may improve mathematically aggregated judgments relative to equal weights. However, obtaining validated, relevant calibration questions can be difficult. If so, should analysts settle for equal weights? Or should they use calibration questions that are easier to obtain but less relevant? In this article, we examine what happens to the out-of-sample performance of weighted aggregations of the classical model (CM) compared to equal weighted aggregations when the set of calibration questions includes many so-called "irrelevant" questions, those that might ordinarily be considered to be outside the domain of the questions of interest. We find that performance weighted aggregations outperform equal weights on the combined CM score, but not on statistical accuracy (i.e., calibration). Importantly, there was no appreciable difference in performance when weights were developed on relevant versus irrelevant questions. Experts were unable to adapt their knowledge across vastly different domains, and in-sample validation did not accurately predict out-of-sample performance on irrelevant questions. We suggest that if relevant calibration questions cannot be found, then analysts should use equal weights, and draw on alternative techniques to improve judgments. Our study also indicates limits to the predictive accuracy of performance weighted aggregation, and the degree to which expertise can be adapted across domains. We note limitations in our study and urge further research into the effect of question type on the reliability of performance weighted aggregations.

An introduction to decision science for conservation.

Biodiversity conservation decisions are difficult, especially when they involve differing values, complex multidimensional objectives, scarce resources, urgency, and considerable uncertainty. Decision science embodies a theory about how to make difficult decisions and an extensive array of frameworks and tools that make that theory practical. We sought to improve conceptual clarity and practical application of decision science to help decision makers apply decision science to conservation problems. We addressed barriers to the uptake of decision science, including a lack of training and awareness of decision science; confusion over common terminology and which tools and frameworks to apply; and the mistaken impression that applying decision science must be time consuming, expensive, and complex. To aid in navigating the extensive and disparate decision science literature, we clarify meaning of common terms: decision science, decision theory, decision analysis, structured decision-making, and decision-support tools. Applying decision science does not have to be complex or time consuming; rather, it begins with knowing how to think through the components of a decision utilizing decision analysis (i.e., define the problem, elicit objectives, develop alternatives, estimate consequences, and perform trade-offs). This is best achieved by applying a rapid-prototyping approach. At each step, decision-support tools can provide additional insight and clarity, whereas decision-support frameworks (e.g., priority threat management and systematic conservation planning) can aid navigation of multiple steps of a decision analysis for particular contexts. We summarize key decision-support frameworks and tools and describe to which step of a decision analysis, and to which contexts, each is most useful to apply. Our introduction to decision science will aid in contextualizing current approaches and new developments, and help decision makers begin to apply decision science to conservation problems.

Las decisiones sobre la conservación de la biodiversidad son difíciles de tomar, especialmente cuando involucran diferentes valores, objetivos multidimensionales complejos, recursos limitados, urgencia y una incertidumbre considerable. Las ciencias de la decisión incorporan una teoría sobre cómo tomar decisiones difíciles y una variedad extensa de marcos de trabajo y herramientas que transforman esa teoría en práctica. Buscamos mejorar la claridad conceptual y la aplicación práctica de las ciencias de la decisión para ayudar al órgano decisorio a aplicar estas ciencias a los problemas de conservación. Nos enfocamos en las barreras para la aceptación de las ciencias de la decisión, incluyendo la falta de capacitación y de conciencia por estas ciencias; la confusión por la terminología común y cuáles herramientas y marcos de trabajo aplicar; y la impresión errónea de que la aplicación de estas ciencias consume tiempo y debe ser costosa y compleja. Para asistir en la navegación de la literatura extensa y dispar de las ciencias de la decisión, aclaramos el significado de varios términos comunes: ciencias de la decisión, teoría de la decisión, análisis de decisiones, toma estructurada de decisiones y herramientas de apoyo para las decisiones. La aplicación de las ciencias de la decisión no tiene que ser compleja ni debe llevar mucho tiempo; de hecho, todo comienza con saber cómo pensar detenidamente en los componentes de una decisión mediante el análisis de decisiones (es decir, definir el problema, producir objetivos, desarrollar alternativas, estimar consecuencias y realizar compensaciones). Lo anterior se logra de mejor manera mediante la aplicación de una estrategia prototipos rápidos. En cada paso, las herramientas de apoyo para las decisiones pueden proporcionar visión y claridad adicionales, mientras que los marcos de apoyo para las decisiones (p.ej.: gestión de amenazas prioritarias y planeación sistemática de la conservación) pueden asistir en la navegación de los diferentes pasos de un análisis de decisiones para contextos particulares. Resumimos los marcos de trabajo y las herramientas más importantes de apoyo para las decisiones y describimos el paso, y el contexto, del análisis de decisiones para el que es más útil aplicarlos. Nuestra introducción a las ciencias de la decisión apoyará en la contextualización de las estrategias actuales y los nuevos desarrollos, y ayudarán al órgano decisorio a comenzar a aplicar estas ciencias en los problemas de conservación.

Strengthening conservation science as a crisis discipline by addressing challenges of precaution, privilege, and individualism.

Conservation science deals with crises and supports policy interventions devised to mitigate highly uncertain threats that pose irreversible harm. When conventional policy tools, such as quantitative risk assessments, are insufficient, the precautionary principle provides a practical framework and range of robust heuristics. Yet, precaution is often resisted in many policy arenas, especially those involving powerful self-interests, and this resistance is compounded by structures of privilege and competitive individualism in science. We describe key drivers and effects of such resistance in conservation science. These include a loss of rigor under uncertainty, an erosion of crisis response capabilities, and a further reinforcement of privileged interests in conservation politics. We recommend open acknowledgement of the pressures exerted by power inside science; greater recognition for the value of the precautionary principle under uncertainty; deliberate measures to resist competitive individualism; support for blind review, open science, and data sharing; and a shift from hierarchical multidisciplinarity toward more egalitarian transdisciplinarity to accelerate advances in conservation science. Article impact statement: Precautionary principle, privilege structures among disciplines, and culture of individualism link to effective conservation policy making.

Fortalecimiento de las Ciencias de la Conservación como Disciplinas de Crisis Resumen Las ciencias de la conservación tratan con crisis y respaldan a muchas intervenciones políticas para mitigar las amenazas altamente inciertas que representan un daño irreversible. El principio de precaución proporciona un marco práctico y una gama de heurística sólida cuando son insuficientes las herramientas convencionales de políticas como las evaluaciones cuantitativas de riesgo. Aun así, con frecuencia se niega el uso de la precaución en muchas arenas políticas, especialmente en aquellas que involucran intereses propios de mucho poder, y esta negación se agrava con las estructuras de privilegio y el individualismo competitivo presentes en la ciencia. Describimos los factores y efectos clave de dicha resistencia en las ciencias de la conservación. Estos incluyen la pérdida del rigor bajo la incertidumbre, un desgaste de las capacidades de respuesta a la crisis y un reforzamiento más profundo de los intereses privilegiados en las políticas de conservación. Recomendamos que se realice una aceptación abierta de las presiones ejercidas por el poder dentro de la ciencia; un mayor reconocimiento del valor del principio de precaución bajo la incertidumbre; que se lleven a cabo medidas deliberadas para oponerse al individualismo competitivo; que se apoye a las revisiones a ciegas, la ciencia abierta y la difusión de datos; y que se realice un cambio de la multidisciplinariedad jerárquica a una transdisciplinariedad más igualitaria para acelerar los avances dentro de las ciencias de la conservación.

Traits explain invasion of alien plants into tropical rainforests.

1. The establishment of new botanic gardens in tropical regions highlights a need for weed risk assessment tools suitable for tropical ecosystems. The relevance of plant traits for invasion into tropical rainforests has not been well studied.2. Working in and around four botanic gardens in Indonesia where 590 alien species have been planted, we estimated the effect of four plant traits, plus time since species introduction, on: (a) the naturalization probability and (b) abundance (density) of naturalized species in adjacent native tropical rainforests; and (c) the distance that naturalized alien plants have spread from the botanic gardens.3. We found that specific leaf area (SLA) strongly differentiated 23 naturalized from 78 non-naturalized alien species (randomly selected from 577 non-naturalized species) in our study. These trends may indicate that aliens with high SLA, which had a higher probability of naturalization, benefit from at least two factors when establishing in tropical forests: high growth rates and occupation of forest gaps. Naturalized aliens had high SLA and tended to be short. However, plant height was not significantly related to species' naturalization probability when considered alongside other traits.4. Alien species that were present in the gardens for over 30 years and those with small seeds also had higher probabilities of becoming naturalized, indicating that garden plants can invade the understorey of closed canopy tropical rainforests, especially when invading species are shade tolerant and have sufficient time to establish.5. On average, alien species that were not animal dispersed spread 78 m further into the forests and were more likely to naturalize than animal-dispersed species. We did not detect relationships between the measured traits and estimated density of naturalized aliens in the adjacent forests.6. Synthesis: Traits were able to differentiate alien species from botanic gardens that naturalized in native forest from those that did not; this is promising for developing trait-based risk assessment in the tropics. To limit the risk of invasion and spread into adjacent native forests, we suggest tropical botanic gardens avoid planting alien species with fast carbon capture strategies and those that are shade tolerant.

Biodiversity conservation as a promising frontier for behavioural science.

Human activities are degrading ecosystems worldwide, posing existential threats for biodiversity and humankind. Slowing and reversing this degradation will require profound and widespread changes to human behaviour. Behavioural scientists are therefore well placed to contribute intellectual leadership in this area. This Perspective aims to stimulate a marked increase in the amount and breadth of behavioural research addressing this challenge. First, we describe the importance of the biodiversity crisis for human and non-human prosperity and the central role of human behaviour in reversing this decline. Next, we discuss key gaps in our understanding of how to achieve behaviour change for biodiversity conservation and suggest how to identify key behaviour changes and actors capable of improving biodiversity outcomes. Finally, we outline the core components for building a robust evidence base and suggest priority research questions for behavioural scientists to explore in opening a new frontier of behavioural science for the benefit of nature and human wellbeing.

Pre-screening workers to overcome bias amplification in online labour markets.

Groups have access to more diverse information and typically outperform individuals on problem solving tasks. Crowdsolving utilises this principle to generate novel and/or superior solutions to intellective tasks by pooling the inputs from a distributed online crowd. However, it is unclear whether this particular instance of "wisdom of the crowd" can overcome the influence of potent cognitive biases that habitually lead individuals to commit reasoning errors. We empirically test the prevalence of cognitive bias on a popular crowdsourcing platform, examining susceptibility to bias of online panels at the individual and aggregate levels. We then investigate the use of the Cognitive Reflection Test, notable for its predictive validity for both susceptibility to cognitive biases in test settings and real-life reasoning, as a screening tool to improve collective performance. We find that systematic biases in crowdsourced answers are not as prevalent as anticipated, but when they occur, biases are amplified with increasing group size, as predicted by the Condorcet Jury Theorem. The results further suggest that pre-screening individuals with the Cognitive Reflection Test can substantially enhance collective judgement and improve crowdsolving performance.

Protecting biodiversity and economic returns in resource-rich tropical forests.

In pursuit of socioeconomic development, many countries are expanding oil and mineral extraction into tropical forests. These activities seed access to remote, biologically rich areas, thereby endangering global biodiversity. Here we demonstrate that conservation solutions that effectively balance the protection of biodiversity and economic revenues are possible in biologically valuable regions. Using spatial data on oil profits and predicted species and ecosystem extents, we optimise the protection of 741 terrestrial species and 20 ecosystems of the Ecuadorian Amazon, across a range of opportunity costs (i.e. sacrifices of extractive profit). For such an optimisation, giving up 5% of a year's oil profits (US$ 221 million) allows for a protected area network that retains of an average of 65% of the extent of each species/ecosystem. This performance far exceeds that of the network produced by simple land area optimisation which requires a sacrifice of approximately 40% of annual oil profits (US$ 1.7 billion), and uses only marginally less land, to achieve equivalent levels of ecological protection. Applying spatial statistics to remotely sensed, historic deforestation data, we further focus the optimisation to areas most threatened by imminent forest loss. We identify Emergency Conservation Targets: areas that are essential to a cost-effective conservation reserve network and at imminent risk of destruction, thus requiring urgent and effective protection. Governments should employ the methods presented here when considering extractive led development options, to responsibly manage the associated ecological-economic trade-offs and protect natural capital. Article Impact Statement: Governments controlling resource extraction from tropical forests can arrange production and conservation to retain biodiversity and profits. This article is protected by copyright. All rights reserved.

Weighting and aggregating expert ecological judgments.

Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.

Untapped potential of collective intelligence in conservation and environmental decision making.

Environmental decisions are often deferred to groups of experts, committees, or panels to develop climate policy, plan protected areas, or negotiate trade-offs for biodiversity conservation. There is, however, surprisingly little empirical research on the performance of group decision making related to the environment. We examined examples from a range of different disciplines, demonstrating the emergence of collective intelligence (CI) in the elicitation of quantitative estimates, crowdsourcing applications, and small-group problem solving. We explored the extent to which similar tools are used in environmental decision making. This revealed important gaps (e.g., a lack of integration of fundamental research in decision-making practice, absence of systematic evaluation frameworks) that obstruct mainstreaming of CI. By making judicious use of interdisciplinary learning opportunities, CI can be harnessed effectively to improve decision making in conservation and environmental management. To elicit reliable quantitative estimates an understanding of cognitive psychology and to optimize crowdsourcing artificial intelligence tools may need to be incorporated. The business literature offers insights into the importance of soft skills and diversity in team effectiveness. Environmental problems set a challenging and rich testing ground for collective-intelligence tools and frameworks. We argue this creates an opportunity for significant advancement in decision-making research and practice.

Potencial No Explotado de la Inteligencia Colectiva en la Toma de Decisiones Ambientales y de Conservación Resumen Las decisiones ambientales comúnmente se difieren a grupos de expertos, comités, o paneles para desarrollar la política climática, planear las áreas protegidas o negociar compensaciones por la conservación de la biodiversidad. Aun así, sorprendentemente, existen pocas investigaciones empíricas sobre el desempeño de la toma grupal de decisiones en relación con el ambiente. Examinamos los ejemplos de una gama de disciplinas diferentes, demostrando el surgimiento de la inteligencia colectiva en la obtención de estimaciones cuantitativas, las aplicaciones de la colaboración masiva y la resolución de problemas en grupos pequeños. Exploramos el alcance que tienen las herramientas similares que se usan en la toma de decisiones ambientales. Esto último reveló vacíos importantes (p. ej.: la falta de integración de investigaciones fundamentales en la práctica de la toma de decisiones, la ausencia de marcos de trabajo de evaluación sistemática) que obstruyen la popularización de la inteligencia colectiva. Si hacemos un uso juicioso de las oportunidades de aprendizaje interdisciplinario, la inteligencia colectiva puede aprovecharse efectivamente para mejorar la toma de decisiones en el manejo ambiental y de conservación. La incorporación de un entendimiento de la psicología cognitiva y la optimización de las herramientas de IA para la colaboración masiva pueden ser necesarias para obtener estimados cuantitativos confiables. La literatura de los negocios ofrece conocimientos sobre la importancia de las habilidades blandas y la diversidad en la efectividad del equipo. Los problemas ambientales plantean un campo de pruebas rico y desafiante para las herramientas y los marcos de trabajo de inteligencia colectiva. Argumentamos que esto crea una oportunidad para el avance significativo en la investigación y la práctica de la toma de decisiones.

Traits influence detection of exotic plant species in tropical forests.

Detecting exotic plant species is essential for invasive species management. By accounting for factors likely to affect species' detection rates (e.g. survey conditions, observer experience), detectability models can help choose search methods and allocate search effort. Integrating information on species' traits can refine detectability models, and might be particularly valuable if these traits can help improve estimates of detectability where data on particular species are rare. Analysing data collected during line transect distance sampling surveys in Indonesia, we used a multi-species hierarchical distance sampling model to evaluate how plant height, leaf size, leaf shape, and survey location influenced plant species detectability in secondary tropical rainforests. Detectability of the exotic plant species increased with plant height and leaf size. Detectability varied among the different survey locations. We failed to detect a clear effect of leaf shape on detectability. This study indicates that information on traits might improve predictions about exotic species detection, which can then be used to optimise the allocation of search effort for efficient species management. The innovation of the study lies in the multi-species distance sampling model, where the distance-detection function depends on leaf traits and height. The method can be applied elsewhere, including for different traits that may be relevant in other contexts. Trait-based multispecies distance sampling can be a practical approach for sampling exotic shrubs, herbs, or grasses species in the understorey of tropical forests.

Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management.

INTRODUCTION: Natural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements. AIMS AND METHODS: In this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol ('Investigate', 'Discuss', 'Estimate', 'Aggregate') through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness. RESULTS AND CONCLUSIONS: The results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

Does Size Matter to Models? Exploring the Effect of Herd Size on Outputs of a Herd-Level Disease Spread Simulator.

Disease spread modeling is widely used by veterinary authorities to predict the impact of emergency animal disease outbreaks in livestock and to evaluate the cost-effectiveness of different management interventions. Such models require knowledge of basic disease epidemiology as well as information about the population of animals at risk. Essential demographic information includes the production system, animal numbers, and their spatial locations yet many countries with significant livestock industries do not have publically available and accurate animal population information at the farm level that can be used in these models. The impact of inaccuracies in data on model outputs and the decisions based on these outputs is seldom discussed. In this analysis, we used the Australian Animal Disease model to simulate the spread of foot-and-mouth disease seeded into high-risk herds in six different farming regions in New Zealand. We used three different susceptible animal population datasets: (1) a gold standard dataset comprising known herd sizes, (2) a dataset where herd size was simulated from a beta-pert distribution for each herd production type, and (3) a dataset where herd size was simplified to the median herd size for each herd production type. We analyzed the model outputs to compare (i) the extent of disease spread, (ii) the length of the outbreaks, and (iii) the possible impacts on decisions made for simulated outbreaks in different regions. Model outputs using the different datasets showed statistically significant differences, which could have serious implications for decision making by a competent authority. Outbreak duration, number of infected properties, and vaccine doses used during the outbreak were all significantly smaller for the gold standard dataset when compared with the median herd size dataset. Initial outbreak location and disease control strategy also significantly influenced the duration of the outbreak and number of infected premises. The study findings demonstrate the importance of having accurate national-level population datasets to ensure effective decisions are made before and during disease outbreaks, reducing the damage and cost.