The Value of Performance Weights and Discussion in Aggregated Expert Judgments.

In risky situations characterized by imminent decisions, scarce resources, and insufficient data, policymakers rely on experts to estimate model parameters and their associated uncertainties. Different elicitation and aggregation methods can vary substantially in their efficacy and robustness. While it is generally agreed that biases in expert judgments can be mitigated using structured elicitations involving groups rather than individuals, there is still some disagreement about how to best elicit and aggregate judgments. This mostly concerns the merits of using performance-based weighting schemes to combine judgments of different individuals (rather than assigning equal weights to individual experts), and the way that interaction between experts should be handled. This article aims to contribute to, and complement, the ongoing discussion on these topics.

Predicting and reasoning about replicability using structured groups.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Eliciting expert knowledge in conservation science.

Expert knowledge is used widely in the science and practice of conservation because of the complexity of problems, relative lack of data, and the imminent nature of many conservation decisions. Expert knowledge is substantive information on a particular topic that is not widely known by others. An expert is someone who holds this knowledge and who is often deferred to in its interpretation. We refer to predictions by experts of what may happen in a particular context as expert judgments. In general, an expert-elicitation approach consists of five steps: deciding how information will be used, determining what to elicit, designing the elicitation process, performing the elicitation, and translating the elicited information into quantitative statements that can be used in a model or directly to make decisions. This last step is known as encoding. Some of the considerations in eliciting expert knowledge include determining how to work with multiple experts and how to combine multiple judgments, minimizing bias in the elicited information, and verifying the accuracy of expert information. We highlight structured elicitation techniques that, if adopted, will improve the accuracy and information content of expert judgment and ensure uncertainty is captured accurately. We suggest four aspects of an expert elicitation exercise be examined to determine its comprehensiveness and effectiveness: study design and context, elicitation design, elicitation method, and elicitation output. Just as the reliability of empirical data depends on the rigor with which it was acquired so too does that of expert knowledge.

Prioritizing global conservation efforts.

One of the most pressing issues facing the global conservation community is how to distribute limited resources between regions identified as priorities for biodiversity conservation. Approaches such as biodiversity hotspots, endemic bird areas and ecoregions are used by international organizations to prioritize conservation efforts globally. Although identifying priority regions is an important first step in solving this problem, it does not indicate how limited resources should be allocated between regions. Here we formulate how to allocate optimally conservation resources between regions identified as priorities for conservation--the 'conservation resource allocation problem'. Stochastic dynamic programming is used to find the optimal schedule of resource allocation for small problems but is intractable for large problems owing to the "curse of dimensionality". We identify two easy-to-use and easy-to-interpret heuristics that closely approximate the optimal solution. We also show the importance of both correctly formulating the problem and using information on how investment returns change through time. Our conservation resource allocation approach can be applied at any spatial scale. We demonstrate the approach with an example of optimal resource allocation among five priority regions in Wallacea and Sundaland, the transition zone between Asia and Australasia.

Cost-effective global conservation spending is robust to taxonomic group.

Priorities for conservation investment at a global scale that are based on a single taxon have been criticized because geographic richness patterns vary taxonomically. However, these concerns focused only on biodiversity patterns and did not consider the importance of socioeconomic factors, which must also be included if conservation funding is to be allocated efficiently. In this article, we create efficient global funding schedules that use information about conservation costs, predicted habitat loss rates, and the endemicity of seven different taxonomic groups. We discover that these funding allocation schedules are less sensitive to variation in taxon assessed than to variation in cost and threat. Two-thirds of funding is allocated to the same regions regardless of the taxon, compared with only one-fifth if threat and cost are not included in allocation decisions. Hence, if socioeconomic factors are considered, we can be more confident about global-scale decisions guided by single taxonomic groups.

Conserving biodiversity efficiently: what to do, where, and when.

Conservation priority-setting schemes have not yet combined geographic priorities with a framework that can guide the allocation of funds among alternate conservation actions that address specific threats. We develop such a framework, and apply it to 17 of the world's 39 Mediterranean ecoregions. This framework offers an improvement over approaches that only focus on land purchase or species richness and do not account for threats. We discover that one could protect many more plant and vertebrate species by investing in a sequence of conservation actions targeted towards specific threats, such as invasive species control, land acquisition, and off-reserve management, than by relying solely on acquiring land for protected areas. Applying this new framework will ensure investment in actions that provide the most cost-effective outcomes for biodiversity conservation. This will help to minimise the misallocation of scarce conservation resources.

Reducing overconfidence in the interval judgments of experts.

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18]).

Optimal dynamic allocation of conservation funding among priority regions.

The optimal allocation of conservation resources between biodiverse conservation regions has generally been calculated using stochastic dynamic programming, or using myopic heuristics. These solutions are hard to interpret and may not be optimal. To overcome these two limitations, this paper approaches the optimal conservation resource allocation problem using optimal control theory. A solution using Pontryagin's maximum principle provides novel insight into the general properties of efficient conservation resource allocation strategies, and allows more extensive testing of the performance of myopic heuristics. We confirmed that a proposed heuristic (minimize short-term loss) yields near-optimal results in complex allocation situations, and found that a qualitative allocation feature observed in previous analyses (bang-bang allocation) is a general property of the optimal allocation strategy.

Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis.

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq's Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document - Intelligence Community Directive (ICD) 203 - and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.

Incorporating the effects of socioeconomic uncertainty into priority setting for conservation investment.

Uncertainty in the implementation and outcomes of conservation actions that is not accounted for leaves conservation plans vulnerable to potential changes in future conditions. We used a decision-theoretic approach to investigate the effects of two types of investment uncertainty on the optimal allocation of global conservation resources for land acquisition in the Mediterranean Basin. We considered uncertainty about (1) whether investment will continue and (2) whether the acquired biodiversity assets are secure, which we termed transaction uncertainty and performance uncertainty, respectively. We also developed and tested the robustness of different rules of thumb for guiding the allocation of conservation resources when these sources of uncertainty exist. In the presence of uncertainty in future investment ability (transaction uncertainty), the optimal strategy was opportunistic, meaning the investment priority should be to act where uncertainty is highest while investment remains possible. When there was a probability that investments would fail (performance uncertainty), the optimal solution became a complex trade-off between the immediate biodiversity benefits of acting in a region and the perceived longevity of the investment. In general, regions were prioritized for investment when they had the greatest performance certainty, even if an alternative region was highly threatened or had higher biodiversity value. The improved performance of rules of thumb when accounting for uncertainty highlights the importance of explicitly incorporating sources of investment uncertainty and evaluating potential conservation investments in the context of their likely long-term success.

Links between media communication and local perceptions of climate change in an indigenous society.

Indigenous societies hold a great deal of ethnoclimatological knowledge that could potentially be of key importance for both climate change science and local adaptation; yet, we lack studies examining how such knowledge might be shaped by media communication. This study systematically investigates the interplay between local observations of climate change and the reception of media information amongst the Tsimane', an indigenous society of Bolivian Amazonia where the scientific discourse of anthropogenic climate change has barely reached. Specifically, we conducted a Randomized Evaluation with a sample of 424 household heads in 12 villages to test to what degree local accounts of climate change are influenced by externally influenced awareness. We randomly assigned villages to a treatment and control group, conducted workshops on climate change with villages in the treatment group, and evaluated the effects of information dissemination on individual climate change perceptions. Results of this work suggest that providing climate change information through participatory workshops does not noticeably influence individual perceptions of climate change. Such findings stress the challenges involved in translating between local and scientific framings of climate change, and gives cause for concern about how to integrate indigenous peoples and local knowledge with global climate change policy debates.

Expert status and performance.

Expert judgements are essential when time and resources are stretched or we face novel dilemmas requiring fast solutions. Good advice can save lives and large sums of money. Typically, experts are defined by their qualifications, track record and experience. The social expectation hypothesis argues that more highly regarded and more experienced experts will give better advice. We asked experts to predict how they will perform, and how their peers will perform, on sets of questions. The results indicate that the way experts regard each other is consistent, but unfortunately, ranks are a poor guide to actual performance. Expert advice will be more accurate if technical decisions routinely use broadly-defined expert groups, structured question protocols and feedback.

Protecting biodiversity when money matters: maximizing return on investment.

BACKGROUND: Conventional wisdom identifies biodiversity hotspots as priorities for conservation investment because they capture dense concentrations of species. However, density of species does not necessarily imply conservation 'efficiency'. Here we explicitly consider conservation efficiency in terms of species protected per dollar invested. METHODOLOGY/PRINCIPAL FINDINGS: We apply a dynamic return on investment approach to a global biome and compare it with three alternate priority setting approaches and a random allocation of funding. After twenty years of acquiring habitat, the return on investment approach protects between 32% and 69% more species compared to the other priority setting approaches. To correct for potential inefficiencies of protecting the same species multiple times we account for the complementarity of species, protecting up to three times more distinct vertebrate species than alternate approaches. CONCLUSIONS/SIGNIFICANCE: Incorporating costs in a return on investment framework expands priorities to include areas not traditionally highlighted as priorities based on conventional irreplaceability and vulnerability approaches.