Managing offshore multi-use settings: Use of conceptual mapping to reduce uncertainty of co-locating seaweed aquaculture and wind farms.

The offshore Multi-use Setting (MUS) is a concept that aims to co-locate marine industrial activities, including wind farms and aquaculture. MUS is considered an innovative approach to promoting efficiency in space and resource use whilst contributing global policy priorities. However, the impacts of MUS development across social, economic, and environmental domains are uncertain, hindering the commercialisation of the concept. In this study, we initially demonstrate the potential consequences of co-locating seaweed aquaculture and a wind farm as a step towards MUS. Using a hypothetical case study and modified Delphi methodology, 14 subject matter experts predicted potential outcomes across social and environmental objectives. Five Cognitive maps and impact tables of 58 potential consequences were generated based on experts' perspective on co-locating seaweed aquaculture and a wind farm. The findings highlight the potential to exasperate pressures in the area, including those already attributed to wind farm operations, such as species mortality and stakeholder conflict. However, it may also enhance social-ecological conditions, such as resource provisioning and promoting habitat functionality in the region, through the addition of seaweed aquaculture. The cognitive maps demonstrate the complexity of managing MUS implementation, where high degree of variability and uncertainty about the outcomes is present. The findings of this study provide the vital entry point to performing further integrative assessment and modelling approaches, such as probabilistic analysis and simulations, in support of MUS decision-making. The research also strongly recommends alternative strategies in the pursuit of combining seaweed production and wind farms to avoid significant financial (among many other) trade-offs and risks. More broadly, we have found that our approach's ability to visually represent a complex situation while considering multiple objectives could be immensely valuable for other bioeconomy innovations or nature-based solutions. It helps mitigate the potential for expensive investments without a comprehensive evaluation of the associated risks and negative impacts, as necessitated by the principles of sustainability in decision-making.

Predicting and reasoning about replicability using structured groups.

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

Comparing wildlife habitat suitability models based on expert opinion with camera trap detections.

Expert knowledge is used in the development of wildlife habitat suitability models (HSMs) for management and conservation decisions. However, the consistency of such models has been questioned. Focusing on 1 method for elicitation, the analytic hierarchy process, we generated expert-based HSMs for 4 felid species: 2 forest specialists (ocelot [Leopardus pardalis] and margay [Leopardus wiedii]) and 2 habitat generalist species (Pampas cat [Leopardus colocola] and puma [Puma concolor]). Using these HSMs, species detections from camera-trap surveys, and generalized linear models, we assessed the effect of study species and expert attributes on the correspondence between expert models and camera-trap detections. We also examined whether aggregation of participant responses and iterative feedback improved model performance. We ran 160 HSMs and found that models for specialist species showed higher correspondence with camera-trap detections (AUC [area under the receiver operating characteristic curve] >0.7) than those for generalists (AUC < 0.7). Model correspondence increased as participant years of experience in the study area increased, but only for the understudied generalist species, Pampas cat (ß = 0.024 [SE 0.007]). No other participant attribute was associated with model correspondence. Feedback and revision of models improved model correspondence, and aggregating judgments across multiple participants improved correspondence only for specialist species. The average correspondence of aggregated judgments increased as group size increased but leveled off after 5 experts for all species. Our results suggest that correspondence between expert models and empirical surveys increases as habitat specialization increases. We encourage inclusion of participants knowledgeable of the study area and model validation for expert-based modeling of understudied and generalist species.

Comparación entre los modelos de idoneidad de hábitat basados en la opinión de expertos y la detecciones con cámaras trampa Resumen El conocimiento de expertos se usa en el desarrollo de modelos de idoneidad de hábitat (MIH) para la gestión y la toma de decisiones en conservación. Sin embargo, se ha cuestionado la coherencia de dichos modelos. Utilizamos un solo método, el proceso de jerarquización analítica, para generar MIH para cuatro felinos: dos especies especialistas de bosque (Leopardus pardalis y L. wiedii)) y dos generalistas de hábitat (Leopardus colocola y Puma concolor). Usamos estos MIH, la detección de las especies mediante censos de cámaras trampa y modelos lineales generalizados, para analizar el efecto de dichas especies y las características de los expertos sobre la correspondencia entre los modelos expertos y las detecciones con cámaras trampa. También analizamos si la agregación de las respuestas de los participantes y la retroalimentación iterativa mejoran el desempeño del modelo. Analizamos 160 MIH y encontramos que los modelos para las especies especialistas mostraron una correspondencia mayor con las detecciones de cámarastrampa (ABC [área bajo la curva de la característica operante receptora] >0.7) para las especies generalistas (ABC < 0.7). La correspondencia del modelo incrementó conforme incrementaron los años de experiencia de los participantes en el área de estudio, pero sólo para Leopardus colocola, una especie generalista y poco estudiada (ß = 0.024 [SE 0.007]). Ninguna otra característica de los participantes se asoció con la correspondencia del modelo. La retroalimentación y la revisión de los modelos aumentaron la correspondencia y la agregación de opiniones de múltiples participantes aumentó la correspondencia sólo para las especies especialistas. La correspondencia promedio de las opiniones agregadas incrementó conforme creció el tamaño grupal, aunque se niveló después de cinco expertos para todas las especies. Nuestros resultados sugieren que la correspondencia entre los modelos de expertos y las evaluaciones empíricos incrementan conforme aumenta la especialización del hábitat. Promovemos la inclusión de participantes conocedores del área de estudio y la validación del modelo para el modelado de expertos de especies generalistas y poco estudiadas.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Reimagining peer review as an expert elicitation process.

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the 'best' experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Challenges in estimation, uncertainty quantification and elicitation for pandemic modelling.

The estimation of parameters and model structure for informing infectious disease response has become a focal point of the recent pandemic. However, it has also highlighted a plethora of challenges remaining in the fast and robust extraction of information using data and models to help inform policy. In this paper, we identify and discuss four broad challenges in the estimation paradigm relating to infectious disease modelling, namely the Uncertainty Quantification framework, data challenges in estimation, model-based inference and prediction, and expert judgement. We also postulate priorities in estimation methodology to facilitate preparation for future pandemics.

What is a Good Calibration Question?

Weighted aggregation of expert judgments based on their performance on calibration questions may improve mathematically aggregated judgments relative to equal weights. However, obtaining validated, relevant calibration questions can be difficult. If so, should analysts settle for equal weights? Or should they use calibration questions that are easier to obtain but less relevant? In this article, we examine what happens to the out-of-sample performance of weighted aggregations of the classical model (CM) compared to equal weighted aggregations when the set of calibration questions includes many so-called "irrelevant" questions, those that might ordinarily be considered to be outside the domain of the questions of interest. We find that performance weighted aggregations outperform equal weights on the combined CM score, but not on statistical accuracy (i.e., calibration). Importantly, there was no appreciable difference in performance when weights were developed on relevant versus irrelevant questions. Experts were unable to adapt their knowledge across vastly different domains, and in-sample validation did not accurately predict out-of-sample performance on irrelevant questions. We suggest that if relevant calibration questions cannot be found, then analysts should use equal weights, and draw on alternative techniques to improve judgments. Our study also indicates limits to the predictive accuracy of performance weighted aggregation, and the degree to which expertise can be adapted across domains. We note limitations in our study and urge further research into the effect of question type on the reliability of performance weighted aggregations.

Uncertainty Quantification with Experts: Present Status and Research Needs.

Expert elicitation is deployed when data are absent or uninformative and critical decisions must be made. In designing an expert elicitation, most practitioners seek to achieve best practice while balancing practical constraints. The choices made influence the required time and effort investment, the quality of the elicited data, experts' engagement, the defensibility of results, and the acceptability of resulting decisions. This piece outlines some of the common choices practitioners encounter when designing and conducting an elicitation. We discuss the evidence supporting these decisions and identify research gaps. This will hopefully allow practitioners to better navigate the literature, and will inspire the expert judgment research community to conduct well powered, replicable experiments that properly address the research gaps identified.

An introduction to decision science for conservation.

Biodiversity conservation decisions are difficult, especially when they involve differing values, complex multidimensional objectives, scarce resources, urgency, and considerable uncertainty. Decision science embodies a theory about how to make difficult decisions and an extensive array of frameworks and tools that make that theory practical. We sought to improve conceptual clarity and practical application of decision science to help decision makers apply decision science to conservation problems. We addressed barriers to the uptake of decision science, including a lack of training and awareness of decision science; confusion over common terminology and which tools and frameworks to apply; and the mistaken impression that applying decision science must be time consuming, expensive, and complex. To aid in navigating the extensive and disparate decision science literature, we clarify meaning of common terms: decision science, decision theory, decision analysis, structured decision-making, and decision-support tools. Applying decision science does not have to be complex or time consuming; rather, it begins with knowing how to think through the components of a decision utilizing decision analysis (i.e., define the problem, elicit objectives, develop alternatives, estimate consequences, and perform trade-offs). This is best achieved by applying a rapid-prototyping approach. At each step, decision-support tools can provide additional insight and clarity, whereas decision-support frameworks (e.g., priority threat management and systematic conservation planning) can aid navigation of multiple steps of a decision analysis for particular contexts. We summarize key decision-support frameworks and tools and describe to which step of a decision analysis, and to which contexts, each is most useful to apply. Our introduction to decision science will aid in contextualizing current approaches and new developments, and help decision makers begin to apply decision science to conservation problems.

Las decisiones sobre la conservación de la biodiversidad son difíciles de tomar, especialmente cuando involucran diferentes valores, objetivos multidimensionales complejos, recursos limitados, urgencia y una incertidumbre considerable. Las ciencias de la decisión incorporan una teoría sobre cómo tomar decisiones difíciles y una variedad extensa de marcos de trabajo y herramientas que transforman esa teoría en práctica. Buscamos mejorar la claridad conceptual y la aplicación práctica de las ciencias de la decisión para ayudar al órgano decisorio a aplicar estas ciencias a los problemas de conservación. Nos enfocamos en las barreras para la aceptación de las ciencias de la decisión, incluyendo la falta de capacitación y de conciencia por estas ciencias; la confusión por la terminología común y cuáles herramientas y marcos de trabajo aplicar; y la impresión errónea de que la aplicación de estas ciencias consume tiempo y debe ser costosa y compleja. Para asistir en la navegación de la literatura extensa y dispar de las ciencias de la decisión, aclaramos el significado de varios términos comunes: ciencias de la decisión, teoría de la decisión, análisis de decisiones, toma estructurada de decisiones y herramientas de apoyo para las decisiones. La aplicación de las ciencias de la decisión no tiene que ser compleja ni debe llevar mucho tiempo; de hecho, todo comienza con saber cómo pensar detenidamente en los componentes de una decisión mediante el análisis de decisiones (es decir, definir el problema, producir objetivos, desarrollar alternativas, estimar consecuencias y realizar compensaciones). Lo anterior se logra de mejor manera mediante la aplicación de una estrategia prototipos rápidos. En cada paso, las herramientas de apoyo para las decisiones pueden proporcionar visión y claridad adicionales, mientras que los marcos de apoyo para las decisiones (p.ej.: gestión de amenazas prioritarias y planeación sistemática de la conservación) pueden asistir en la navegación de los diferentes pasos de un análisis de decisiones para contextos particulares. Resumimos los marcos de trabajo y las herramientas más importantes de apoyo para las decisiones y describimos el paso, y el contexto, del análisis de decisiones para el que es más útil aplicarlos. Nuestra introducción a las ciencias de la decisión apoyará en la contextualización de las estrategias actuales y los nuevos desarrollos, y ayudarán al órgano decisorio a comenzar a aplicar estas ciencias en los problemas de conservación.

Weighting and aggregating expert ecological judgments.

Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.

Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management.

INTRODUCTION: Natural resource management uses expert judgement to estimate facts that inform important decisions. Unfortunately, expert judgement is often derived by informal and largely untested protocols, despite evidence that the quality of judgements can be improved with structured approaches. We attribute the lack of uptake of structured protocols to the dearth of illustrative examples that demonstrate how they can be applied within pressing time and resource constraints, while also improving judgements. AIMS AND METHODS: In this paper, we demonstrate how the IDEA protocol for structured expert elicitation may be deployed to overcome operational challenges while improving the quality of judgements. The protocol was applied to the estimation of 14 future abiotic and biotic events on the Great Barrier Reef, Australia. Seventy-six participants with varying levels of expertise related to the Great Barrier Reef were recruited and allocated randomly to eight groups. Each participant provided their judgements using the four-step question format of the IDEA protocol ('Investigate', 'Discuss', 'Estimate', 'Aggregate') through remote elicitation. When the events were realised, the participant judgements were scored in terms of accuracy, calibration and informativeness. RESULTS AND CONCLUSIONS: The results demonstrate that the IDEA protocol provides a practical, cost-effective, and repeatable approach to the elicitation of quantitative estimates and uncertainty via remote elicitation. We emphasise that i) the aggregation of diverse individual judgements into pooled group judgments almost always outperformed individuals, and ii) use of a modified Delphi approach helped to remove linguistic ambiguity, and further improved individual and group judgements. Importantly, the protocol encourages review, critical appraisal and replication, each of which is required if judgements are to be used in place of data in a scientific context. The results add to the growing body of literature that demonstrates the merit of using structured elicitation protocols. We urge decision-makers and analysts to use insights and examples to improve the evidence base of expert judgement in natural resource management.

Prospective cohort study of procalcitonin levels in children with cancer presenting with febrile neutropenia.

BACKGROUND: Febrile neutropenia (FNP) causes significant morbidity and mortality in children undergoing treatment for cancer. The development of clinical decision rules to help stratify risks in paediatric FNP patients and the use of inflammatory biomarkers to identify high risk patients is an area of recent research. This study aimed to assess if procalcitonin (PCT) levels could be used to help diagnose or exclude severe infection in children with cancer who present with febrile neutropenia, both as a single measurement and in addition to previously developed clinical decision rules. METHODS: This prospective cohort study of a diagnostic test included patients between birth and 18 years old admitted with febrile neutropenia to the Paediatric Oncology and Haematology Ward in Leeds between 1st October 2012 and 30th September 2013. Each admission with FNP was treated as a separate episode. Blood was taken for a procalcitonin level at admission with routine investigations. 'R' was used for statistical analysis. Likelihood ratios were calculated and multivariable logistic regression. RESULTS: Forty-eight episodes from 27 patients were included. PCT >2 ng/dL was strongly associated with increased risk of severe infection (likelihood ratio of 26 [95% CI 3.5, 190]). The data suggests that the clinical decision rules are largely ineffective at risk stratification, frequently over-stating the risk of individual episodes. High procalcitonin levels on admission are correlated with a greatly increased risk of severe infection. CONCLUSIONS: This study does not show a definitive benefit in using PCT in FNP though it supports further research on its use. The benefit of novel biomarkers has not been proven and before introducing new tests for patients it is important their benefit above existing features is proven, particularly due to the increasing importance of health economics.

Small intestinal atresia in a defined population: occurrence, prenatal diagnosis and survival.

OBJECTIVE: To describe the prevalence, associated anomalies, prenatal diagnosis, and survival of cases of congenital small intestinal atresia (SIA). METHOD: Data were extracted from the Northern Congenital Abnormality Survey (NorCAS) for cases delivered during 1991 to 2001. RESULTS: A total of 99 cases of SIA were notified from 372 717 registered births. Twenty-five (25.8%) SIA cases were associated with a chromosomal anomaly. Of the 72 SIA cases with normal karyotype, 18 (25%) were associated with other structural anomalies. The total prevalence for all SIA cases over the 11 years was 2.66 per 10 000 registered births (95% CI 2.13, 3.18), and the livebirth prevalence was 2.37 per 10 000 livebirths (95% CI 1.88, 2.87). For jejunoileal atresia, the total prevalence increased from 0.54 per 10 000 births during 1991-1995 to 1.11 per 10 000 births during 1996-2001, and livebirth prevalence increased from 0.49 to 1.06 per 10 000 livebirths. The more proximal the lesion the prenatal diagnosis sensitivity was slightly better with 19 (44.2%) cases of duodenal atresia and 3 (42.8%) cases of jejunal atresia diagnosed prenatally by routine ultrasonography. CONCLUSION: This population-based study of SIA has confirmed several previous findings but, by considering subtypes separately, it also suggests a trend towards an increase in the prevalence of jejunoileal atresia (JIA).