Predicting and reasoning about replicability using structured groups.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

What's on the horizon for community-based conservation? Emerging threats and opportunities.

Community-based conservation can support livelihoods and biodiversity, while reinforcing local and Indigenous values, cultures, and institutions. Its delivery can help address cross-cutting global challenges, such as climate change, conservation, poverty, and food security. Therefore, understanding trends in community-based conservation is pertinent to setting and implementing global goals. We undertook a horizon scan to prioritize 15 emerging threats and opportunities expected to impact the future effectiveness of community-based conservation. Topics relate to global biodiversity policy; human rights; shifting human geography; inclusion, diversity, equity, and access; conservation finance and income; and economic reforms. Our findings offer guidance on strengthening community-based conservation to achieve global environmental and development goals.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Reimagining peer review as an expert elicitation process.

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the 'best' experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Towards open, reliable, and transparent ecology and evolutionary biology.

Unreliable research programmes waste funds, time, and even the lives of the organisms we seek to help and understand. Reducing this waste and increasing the value of scientific evidence require changing the actions of both individual researchers and the institutions they depend on for employment and promotion. While ecologists and evolutionary biologists have somewhat improved research transparency over the past decade (e.g. more data sharing), major obstacles remain. In this commentary, we lift our gaze to the horizon to imagine how researchers and institutions can clear the path towards more credible and effective research programmes.

Bioengineering horizon scan 2020.

Horizon scanning is intended to identify the opportunities and threats associated with technological, regulatory and social change. In 2017 some of the present authors conducted a horizon scan for bioengineering (Wintle et al., 2017). Here we report the results of a new horizon scan that is based on inputs from a larger and more international group of 38 participants. The final list of 20 issues includes topics spanning from the political (the regulation of genomic data, increased philanthropic funding and malicious uses of neurochemicals) to the environmental (crops for changing climates and agricultural gene drives). The early identification of such issues is relevant to researchers, policy-makers and the wider public.

Verbal probabilities: Very likely to be somewhat more confusing than numbers.

People interpret verbal expressions of probabilities (e.g. 'very likely') in different ways, yet words are commonly preferred to numbers when communicating uncertainty. Simply providing numerical translations alongside reports or text containing verbal probabilities should encourage consistency, but these guidelines are often ignored. In an online experiment with 924 participants, we compared four different formats for presenting verbal probabilities with the numerical guidelines used in the US Intelligence Community Directive (ICD) 203 to see whether any could improve the correspondence between the intended meaning and participants' interpretation ('in-context'). This extends previous work in the domain of climate science. The four experimental conditions we tested were: 1. numerical guidelines bracketed in text, e.g. X is very unlikely (05-20%), 2. click to see the full guidelines table in a new window, 3. numerical guidelines appear in a mouse over tool tip, and 4. no guidelines provided (control). Results indicate that correspondence with the ICD 203 standard is substantially improved only when numerical guidelines are bracketed in text. For this condition, average correspondence was 66%, compared with 32% in the control. We also elicited 'context-free' numerical judgements from participants for each of the seven verbal probability expressions contained in ICD 203 (i.e., we asked participants what range of numbers they, personally, would assign to those expressions), and constructed 'evidence-based lexicons' based on two methods from similar research, 'membership functions' and 'peak values', that reflect our large sample's intuitive translations of the terms. Better aligning the intended and assumed meaning of fuzzy words like 'unlikely' can reduce communication problems between the reporter and receiver of probabilistic information. In turn, this can improve decision making under uncertainty.

Ten-year assessment of the 100 priority questions for global biodiversity conservation.

In 2008, a group of conservation scientists compiled a list of 100 priority questions for the conservation of the world's biodiversity. However, now almost a decade later, no one has yet published a study gauging how much progress has been made in addressing these 100 high-priority questions in the peer-reviewed literature. We took a first step toward reexamining the 100 questions to identify key knowledge gaps that remain. Through a combination of a questionnaire and a literature review, we evaluated each question on the basis of 2 criteria: relevance and effort. We defined highly relevant questions as those that - if answered - would have the greatest impact on global biodiversity conservation and quantified effort based on the number of review publications addressing a particular question, which we used as a proxy for research effort. Using this approach, we identified a set of questions that, despite being perceived as highly relevant, have been the focus of relatively few review publications over the past 10 years. These questions covered a broad range of topics but predominantly tackled 3 major themes: conservation and management of freshwater ecosystems, role of societal structures in shaping interactions between people and the environment, and impacts of conservation interventions. We believe these questions represent important knowledge gaps that have received insufficient attention and may need to be prioritized in future research.

The Value of Performance Weights and Discussion in Aggregated Expert Judgments.

In risky situations characterized by imminent decisions, scarce resources, and insufficient data, policymakers rely on experts to estimate model parameters and their associated uncertainties. Different elicitation and aggregation methods can vary substantially in their efficacy and robustness. While it is generally agreed that biases in expert judgments can be mitigated using structured elicitations involving groups rather than individuals, there is still some disagreement about how to best elicit and aggregate judgments. This mostly concerns the merits of using performance-based weighting schemes to combine judgments of different individuals (rather than assigning equal weights to individual experts), and the way that interaction between experts should be handled. This article aims to contribute to, and complement, the ongoing discussion on these topics.

Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis.

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document - Intelligence Community Directive (ICD) 203 - and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.

A transatlantic perspective on 20 emerging issues in biological engineering.

Advances in biological engineering are likely to have substantial impacts on global society. To explore these potential impacts we ran a horizon scanning exercise to capture a range of perspectives on the opportunities and risks presented by biological engineering. We first identified 70 potential issues, and then used an iterative process to prioritise 20 issues that we considered to be emerging, to have potential global impact, and to be relatively unknown outside the field of biological engineering. The issues identified may be of interest to researchers, businesses and policy makers in sectors such as health, energy, agriculture and the environment.

Metaresearch for Evaluating Reproducibility in Ecology and Evolution.

Recent replication projects in other disciplines have uncovered disturbingly low levels of reproducibility, suggesting that those research literatures may contain unverifiable claims. The conditions contributing to irreproducibility in other disciplines are also present in ecology. These include a large discrepancy between the proportion of "positive" or "significant" results and the average statistical power of empirical research, incomplete reporting of sampling stopping rules and results, journal policies that discourage replication studies, and a prevailing publish-or-perish research culture that encourages questionable research practices. We argue that these conditions constitute sufficient reason to systematically evaluate the reproducibility of the evidence base in ecology and evolution. In some cases, the direct replication of ecological research is difficult because of strong temporal and spatial dependencies, so here, we propose metaresearch projects that will provide proxy measures of reproducibility.

A 2017 Horizon Scan of Emerging Issues for Global Conservation and Biological Diversity.

We present the results of our eighth annual horizon scan of emerging issues likely to affect global biological diversity, the environment, and conservation efforts in the future. The potential effects of these novel issues might not yet be fully recognized or understood by the global conservation community, and the issues can be regarded as both opportunities and risks. A diverse international team with collective expertise in horizon scanning, science communication, and conservation research, practice, and policy reviewed 100 potential issues and identified 15 that qualified as emerging, with potential substantial global effects. These issues include new developments in energy storage and fuel production, sand extraction, potential solutions to combat coral bleaching and invasive marine species, and blockchain technology.

A Horizon Scan of Global Conservation Issues for 2016.

This paper presents the results of our seventh annual horizon scan, in which we aimed to identify issues that could have substantial effects on global biological diversity in the future, but are not currently widely well known or understood within the conservation community. Fifteen issues were identified by a team that included researchers, practitioners, professional horizon scanners, and journalists. The topics include use of managed bees as transporters of biological control agents, artificial superintelligence, electric pulse trawling, testosterone in the aquatic environment, building artificial oceanic islands, and the incorporation of ecological civilization principles into government policies in China.

A horizon scan of global conservation issues for 2015.

This paper presents the results of our sixth annual horizon scan, which aims to identify phenomena that may have substantial effects on the global environment, but are not widely known or well understood. A group of professional horizon scanners, researchers, practitioners, and a journalist identified 15 topics via an iterative, Delphi-like process. The topics include a novel class of insecticide compounds, legalisation of recreational drugs, and the emergence of a new ecosystem associated with ice retreat in the Antarctic.

Using strategic foresight to assess conservation opportunity.

The nature of conservation challenges can foster a reactive, rather than proactive approach to decision making. Failure to anticipate problems before they escalate results in the need for more costly and time-consuming solutions. Proactive conservation requires forward-looking approaches to decision making that consider possible futures without being overly constrained by the past. Strategic foresight provides a structured process for considering the most desirable future and for mapping the most efficient and effective approaches to promoting that future with tools that facilitate creative thinking. The process involves 6 steps: setting the scope, collecting inputs, analyzing signals, interpreting the information, determining how to act, and implementing the outcomes. Strategic foresight is ideal for seeking, recognizing, and realizing conservation opportunities because it explicitly encourages a broad-minded, forward-looking perspective on an issue. Despite its potential value, the foresight process is rarely used to address conservation issues, and previous attempts have generally failed to influence policy. We present the strategic foresight process as it can be used for proactive conservation planning, describing some of the key tools in the foresight tool kit and how they can be used to identify and exploit different types of conservation opportunities. Scanning is an important tool for collecting and organizing diverse streams of information and can be used to recognize new opportunities and those that could be created. Scenario planning explores how current trends, drivers of change, and key uncertainties might influence the future and can be used to identify barriers to opportunities. Backcasting is used to map out a path to a goal and can determine how to remove barriers to opportunities. We highlight how the foresight process was used to identify conservation opportunities during the development of a strategic plan to address climate change in New York State. The plan identified solutions that should be effective across a range of possible futures. Illustrating the application of strategic foresight to identify conservation opportunities should provide the impetus for decision makers to explore strategic foresight as a way to support more proactive conservation policy, planning, and management.

Exploring risk judgments in a trade dispute using Bayesian networks.

Bayesian networks (BNs) are graphical modeling tools that are generally recommended for exploring what-if scenarios, visualizing systems and problems, and for communication between stakeholders during decision making. In this article, we investigate their potential for exploring different perspectives in trade disputes. To do so, we draw on a specific case study that was arbitrated by the World Trade Organization (WTO): the Australia-New Zealand apples dispute. The dispute centered on disagreement about judgments contained within Australia's 2006 import risk analysis (IRA). We built a range of BNs of increasing complexity that modeled various approaches to undertaking IRAs, from the basic qualitative and semi-quantitative risk analyses routinely performed in government agencies, to the more complex quantitative simulation undertaken by Australia in the apples dispute. We found the BNs useful for exploring disagreements under uncertainty because they are probabilistic and transparently represent steps in the analysis. Different scenarios and evidence can easily be entered. Specifically, we explore the sensitivity of the risk output to different judgments (particularly volume of trade). Thus, we explore how BNs could usefully aid WTO dispute settlement. We conclude that BNs are preferable to basic qualitative and semi-quantitative risk analyses because they offer an accessible interface and are mathematically sound. However, most current BN modeling tools are limited compared with complex simulations, as was used in the 2006 apples IRA. Although complex simulations may be more accurate, they are a black box for stakeholders. BNs have the potential to be a transparent aid to complex decision making, but they are currently computationally limited. Recent technological software developments are promising.

Expert status and performance.

Expert judgements are essential when time and resources are stretched or we face novel dilemmas requiring fast solutions. Good advice can save lives and large sums of money. Typically, experts are defined by their qualifications, track record and experience. The social expectation hypothesis argues that more highly regarded and more experienced experts will give better advice. We asked experts to predict how they will perform, and how their peers will perform, on sets of questions. The results indicate that the way experts regard each other is consistent, but unfortunately, ranks are a poor guide to actual performance. Expert advice will be more accurate if technical decisions routinely use broadly-defined expert groups, structured question protocols and feedback.