Evidence and uncertainty: An info-gap analysis of uncertainty-augmenting evidence.

Decisions in many disciplines are based on understanding and evidence. More evidence is better than less when it enhances the decision-maker's understanding. This is achieved by reducing uncertainty confronting the decision-maker and reducing the potential for misunderstanding and failure. However, some evidence may actually augment uncertainty by revealing prior error or ignorance. True evidence that augments uncertainty is important because it identifies inadequacies of current understanding and may suggest directions for rectifying this. True evidence that reduces uncertainty may simply reconfirm or strengthen prior understanding. Uncertainty-augmenting evidence, when it is true, can support the expansion of one's previously incomplete understanding. A dilemma arises because both reduction and enhancement of uncertainty can be beneficial, and both are not simultaneously possible on the same issue. That is, uncertainty can be either pernicious or propitious. Info-gap theory provides a response. The info-gap robustness function enables protection against pernicious uncertainty by inhibiting failure. The info-gap opportuneness function enables exploitation of propitious uncertainty by facilitating wonderful windfall outcomes. The dilemma of uncertainty-augmenting evidence is that robustness and opportuneness are in conflict; a decision that enhances one, worsens the other. This antagonism between robustness and opportuneness-between protecting against pernicious uncertainty and exploiting propitious uncertainty-is characterized in a generic proposition and corollary. These results are illustrated in an example of allocation of limited resources.

Managing uncertainty in decision-making for conservation science.

Science-based decision-making is the ideal. However, scientific knowledge is incomplete, and sometimes wrong. Responsible science-based policy, planning, and action must exploit knowledge while managing uncertainty. I considered the info-gap method to manage deep uncertainty surrounding knowledge that is used for decision-making in conservation. A central concept is satisficing, which means satisfying a critical requirement. Alternative decisions are prioritized based on their robustness to uncertainty, and critical outcome requirements are satisficed. Robustness is optimized; outcome is satisficed. This is called robust satisficing. A decision with a suboptimal outcome may be preferred over a decision with a putatively optimal outcome if the former can more robustly achieve an acceptable outcome. Many biodiversity conservation applications employ info-gap theory, under which parameter uncertainty but not uncertainty in functional relations is considered. I considered info-gap models of functional uncertainty, widely used outside of conservation science, as applied to conservation of a generic endangered species by translocation to a new region. I focused on 2 uncertainties: the future temperature is uncertain due to climate change, and the shape of the reproductive output function is uncertain due to translocation to an unfamiliar region. The value of new information is demonstrated based on the robustness function, and the info-gap opportuneness function demonstrates the potential for better-than-anticipated outcomes.

Gestión de la incertidumbre en las decisiones para las ciencias de la conservación Resumen Lo ideal es tomar decisiones con base en la ciencia. Sin embargo, el conocimiento científico está incompleto y a veces es incorrecto. Las políticas, planeaciones y acciones responsables basadas en la ciencia deben explotar el conocimiento mientras que gestionan la incertidumbre. Consideré el método de vacío de información para gestionar la incertidumbre profunda en torno al conocimiento usado para las decisiones de conservación. Un concepto central es satisfacción que significa cumplir con un requerimiento crítico. Las decisiones alternativas se priorizan con base en su solidez respecto a la incertidumbre y se cumplen los requerimientos críticos de los resultados. La solidez mejora, el resultado se cumple. A esto se le llama satisfacción sólida. Puede que se prefiera una decisión con un resultado subóptimo por encima de una decisión con un resultado óptimo posible si la primera puede lograr con mayor solidez un resultado aceptable. Muchas aplicaciones de conservación de la biodiversidad usan la teoría del vacío de información, la cual considera la incertidumbre del parámetro, pero no la incertidumbre en las relaciones funcionales. Consideré los modelos de vacío de información en la incertidumbre funcional, usados de forma extensa fuera de las ciencias de la conservación, aplicados a la conservación de una especie genérica amenazada mediante la reubicación a una nueva región. Me enfoqué en dos incertidumbres: la temperatura en el futuro es incierta debido al cambio climático y la forma de la función del rendimiento reproductivo es incierta debido a la reubicación a una región desconocida. El valor de la nueva información queda demostrado con base en la función de la solidez y la función de la conveniencia demuestra el potencial para resultados mejores a lo esperado.

Robust Policy in Times of Pandemic.

The pandemic exposes policymakers to fundamental uncertainties about future economic scenarios. While policymakers have to act forcefully to mitigate the impact on the economy, these conditions call for policy strategies that are also robust to uncertainty. This article compares two concepts of robust strategies: robust control and robust satisficing. It argues that a robust satisficing strategy is preferred and shows that the crisis responses of governments and central banks in Europe share features of robust satisficing in several dimensions.

Reasoned Decision Making Without Math? Adaptability and Robustness in Response to Surprise.

Many real-world planning and decision problems are far too uncertain, too variable, and too complicated to support realistic mathematical models. Nonetheless, we explain the usefulness, in these situations, of qualitative insights from mathematical decision theory. We demonstrate the integration of info-gap robustness in decision problems in which surprise and ignorance are predominant and where personal and collective psychological factors are critical. We present practical guidelines for employing adaptable-choice strategies as a proxy for robustness against uncertainty. These guidelines include being prepared for more surprises than we intuitively expect, retaining sufficiently many options to avoid premature closure and conflicts among preferences, and prioritizing outcomes that are steerable, whose consequences are observable, and that do not entail sunk costs, resource depletion, or high transition costs. We illustrate these concepts and guidelines with the example of the medical management of the 2003 SARS outbreak in Vietnam.

Interpreting PPV and NPV of Diagnostic Tests with Uncertain Prevalence.

OBJECTIVE: Medical decision-making is often uncertain. The positive predictive value (PPV) and negative predictive value (NPV) are conditional probabilities characterizing diagnostic tests and assessing diagnostic interventions in clinical medicine and epidemiology. The PPV is the probability that a patient has a specified disease, given a positive test result for that disease. The NPV is the probability that a patient does not have the disease, given a negative test result for that disease. Both values depend on disease incidence or prevalence, which may be highly uncertain for unfamiliar diseases, epidemics, etc. Probability distributions for this uncertainty are usually unavailable. We develop a non-probabilistic method for interpreting PPV and NPV with uncertain prevalence. METHODS: Uncertainty in PPV and NPV is managed with the non-probabilistic concept of robustness in info-gap theory. Robustness of PPV or NPV estimates is the greatest uncertainty (in prevalence) at which the estimate's error is acceptable. RESULTS: Four properties are demonstrated. Zeroing: best estimates of PPV or NPV have no robustness to uncertain prevalence; best estimates are unreliable for interpreting diagnostic tests. Trade-off: robustness increases as error increases; this trade-off identifies robustly reliable error in PPV or NPV. Preference reversal: sometimes sub-optimal PPV or NPV estimates are more robust to uncertain incidence or prevalence than optimal estimates, motivating reversal of preference from the putative optimum to the sub-optimal estimate. Trade-off between specificity and robustness to uncertainty: the robustness increases as test-specificity decreases. These four properties underlie the interpretation of PPV and NPV. CONCLUSIONS: The PPV and NPV assess diagnostic tests, but are sensitive to lack of knowledge that generates non-probabilistic uncertain prevalence and must be supplemented with robustness analysis. When uncertainties abound, as with unfamiliar diseases, assessing robustness is critical to avoiding erroneous decisions.

A new multicriteria risk mapping approach based on a multiattribute frontier concept.

Invasive species risk maps provide broad guidance on where to allocate resources for pest monitoring and regulation, but they often present individual risk components (such as climatic suitability, host abundance, or introduction potential) as independent entities. These independent risk components are integrated using various multicriteria analysis techniques that typically require prior knowledge of the risk components' importance. Such information is often nonexistent for many invasive pests. This study proposes a new approach for building integrated risk maps using the principle of a multiattribute efficient frontier and analyzing the partial order of elements of a risk map as distributed in multidimensional criteria space. The integrated risks are estimated as subsequent multiattribute frontiers in dimensions of individual risk criteria. We demonstrate the approach with the example of Agrilus biguttatus Fabricius, a high-risk pest that may threaten North American oak forests in the near future. Drawing on U.S. and Canadian data, we compare the performance of the multiattribute ranking against a multicriteria linear weighted averaging technique in the presence of uncertainties, using the concept of robustness from info-gap decision theory. The results show major geographic hotspots where the consideration of tradeoffs between multiple risk components changes integrated risk rankings. Both methods delineate similar geographical regions of high and low risks. Overall, aggregation based on a delineation of multiattribute efficient frontiers can be a useful tool to prioritize risks for anticipated invasive pests, which usually have an extremely poor prior knowledge base.

Uncertainty in the association between socio-demographic characteristics and mental health.

Questionnaires are among the most basic and widespread tools to assess the mental health of a population in epidemiological and public health studies. Their most obvious advantage (firsthand self-report) is also the source of their main problems: the raw data requires interpretation, and are a snapshot of the specific sample's status at a given time. Efforts to deal with both issues created a bi-dimensional space defined by two orthogonal axes, in which most of the quantitative mental health research can be located. Methods aimed to assure that mental health diagnoses are solidly grounded on existing raw data are part of the individual validity axis. Tools allowing the generalization of the results across the entire population compose the collective validity axis. This paper raises a different question. Since one goal of mental health assessments is to obtain results that can be generalized to some extent, an important question is how robust is a questionnaire result when applied to a different population or to the same population at a different time. In this case, there is deep uncertainty, without any a priori probabilistic information. The main claim of this paper is that this task requires the development of a new robustness to deep uncertainty axis, defining a three-dimensional research space. We demonstrate the analysis of deep uncertainty using the concept of robustness in info-gap decision theory. Based on data from questionnaires collected before and during the Covid-19 pandemic, we first locate a mental health assessment in the space defined by the individual validity axis and the collective validity axis. Then we develop a model of info-gap robustness to uncertainty in mental health assessment, showing how the robustness to deep uncertainty axis interacts with the other two axes, highlighting the contributions and the limitations of this approach. The ability to measure robustness to deep uncertainty in the mental health realm is important particularly in troubled and changing times. In this paper, we provide the basic methodological building blocks of the suggested approach using the outbreak of Covid-19 as a recent example.

Info-gap management of public health Policy for TB with HIV-prevalence and epidemiological uncertainty.

BACKGROUND: Formulation and evaluation of public health policy commonly employs science-based mathematical models. For instance, epidemiological dynamics of TB is dominated, in general, by flow between actively and latently infected populations. Thus modelling is central in planning public health intervention. However, models are highly uncertain because they are based on observations that are geographically and temporally distinct from the population to which they are applied. AIMS: We aim to demonstrate the advantages of info-gap theory, a non-probabilistic approach to severe uncertainty when worst cases cannot be reliably identified and probability distributions are unreliable or unavailable. Info-gap is applied here to mathematical modelling of epidemics and analysis of public health decision-making. METHODS: Applying info-gap robustness analysis to tuberculosis/HIV (TB/HIV) epidemics, we illustrate the critical role of incorporating uncertainty in formulating recommendations for interventions. Robustness is assessed as the magnitude of uncertainty that can be tolerated by a given intervention. We illustrate the methodology by exploring interventions that alter the rates of diagnosis, cure, relapse and HIV infection. RESULTS: We demonstrate several policy implications. Equivalence among alternative rates of diagnosis and relapse are identified. The impact of initial TB and HIV prevalence on the robustness to uncertainty is quantified. In some configurations, increased aggressiveness of intervention improves the predicted outcome but also reduces the robustness to uncertainty. Similarly, predicted outcomes may be better at larger target times, but may also be more vulnerable to model error. CONCLUSIONS: The info-gap framework is useful for managing model uncertainty and is attractive when uncertainties on model parameters are extreme. When a public health model underlies guidelines, info-gap decision theory provides valuable insight into the confidence of achieving agreed-upon goals.

Why risk analysis is difficult, and some thoughts on how to proceed.

Risk analysis is challenged in three ways by uncertainty. Our understanding of the world and its uncertainties is evolving; indeterminism is an inherent part of the open universe in which we live; and learning from experience involves untestable assumptions. We discuss several concepts of robustness as tools for responding to these epistemological challenges. The use of models is justified, even though they are known to err. A concept of robustness is illustrated in choosing between a conventional technology and an innovative, promising, but more uncertain technology. We explain that nonprobabilistic robust decisions are sometimes good probabilistic bets. Info-gap and worst-case concepts of robustness are compared. Finally, we examine the exploitation of favorable but uncertain opportunities and its relation to robust decision making.

Doing our best: optimization and the management of risk.

Tools and concepts of optimization are widespread in decision-making, design, and planning. There is a moral imperative to "do our best." Optimization underlies theories in physics and biology, and economic theories often presume that economic agents are optimizers. We argue that in decisions under uncertainty, what should be optimized is robustness rather than performance. We discuss the equity premium puzzle from financial economics, and explain that the puzzle can be resolved by using the strategy of satisficing rather than optimizing. We discuss design of critical technological infrastructure, showing that satisficing of performance requirements--rather than optimizing them--is a preferable design concept. We explore the need for disaster recovery capability and its methodological dilemma. The disparate domains--economics and engineering--illuminate different aspects of the challenge of uncertainty and of the significance of robust-satisficing.

Interpreting null results from measurements with uncertain correlations: an info-gap approach.

Null events­not detecting a pernicious agent­are the basis for declaring the agent is absent. Repeated nulls strengthen confidence in the declaration. However, correlations between observations are difficult to assess in many situations and introduce uncertainty in interpreting repeated nulls. We quantify uncertain correlations using an info-gap model, which is an unbounded family of nested sets of possible probabilities. An info-gap model is nonprobabilistic and entails no assumption about a worst case. We then evaluate the robustness, to uncertain correlations, of estimates of the probability of a null event. This is then the basis for evaluating a nonprobabilistic robustness-based confidence interval for the probability of a null.

Robustness of risk maps and survey networks to knowledge gaps about a new invasive pest.

In pest risk assessment it is frequently necessary to make management decisions regarding emerging threats under severe uncertainty. Although risk maps provide useful decision support for invasive alien species, they rarely address knowledge gaps associated with the underlying risk model or how they may change the risk estimates. Failure to recognize uncertainty leads to risk-ignorant decisions and miscalculation of expected impacts as well as the costs required to minimize these impacts. Here we use the information gap concept to evaluate the robustness of risk maps to uncertainties in key assumptions about an invading organism. We generate risk maps with a spatial model of invasion that simulates potential entries of an invasive pest via international marine shipments, their spread through a landscape, and establishment on a susceptible host. In particular, we focus on the question of how much uncertainty in risk model assumptions can be tolerated before the risk map loses its value. We outline this approach with an example of a forest pest recently detected in North America, Sirex noctilio Fabricius. The results provide a spatial representation of the robustness of predictions of S. noctilio invasion risk to uncertainty and show major geographic hotspots where the consideration of uncertainty in model parameters may change management decisions about a new invasive pest. We then illustrate how the dependency between the extent of uncertainties and the degree of robustness of a risk map can be used to select a surveillance network design that is most robust to knowledge gaps about the pest.

Reconciling uncertain costs and benefits in Bayes nets for invasive species management.

Bayes nets are used increasingly to characterize environmental systems and formalize probabilistic reasoning to support decision making. These networks treat probabilities as exact quantities. Sensitivity analysis can be used to evaluate the importance of assumptions and parameter estimates. Here, we outline an application of info-gap theory to Bayes nets that evaluates the sensitivity of decisions to possibly large errors in the underlying probability estimates and utilities. We apply it to an example of management and eradication of Red Imported Fire Ants in Southern Queensland, Australia and show how changes in management decisions can be justified when uncertainty is considered.

Detection capacity, information gaps and the design of surveillance programs for invasive forest pests.

Integrated pest risk maps and their underlying assessments provide broad guidance for establishing surveillance programs for invasive species, but they rarely account for knowledge gaps regarding the pest of interest or how these can be reduced. In this study we demonstrate how the somewhat competing notions of robustness to uncertainty and potential knowledge gains could be used in prioritizing large-scale surveillance activities. We illustrate this approach with the example of an invasive pest recently detected in North America, Sirex noctilio Fabricius. First, we formulate existing knowledge about the pest into a stochastic model and use the model to estimate the expected utility of surveillance efforts across the landscape. The expected utility accounts for the distribution, abundance and susceptibility of the host resource as well as the value of timely S. noctilio detections. Next, we make use of the info-gap decision theory framework to explore two alternative pest surveillance strategies. The first strategy aims for timely, certain detections and attempts to maximize the robustness to uncertainty about S. noctilio behavior; the second strategy aims to maximize the potential knowledge gain about the pest via unanticipated (i.e., opportune) detections. The results include a set of spatial outputs for each strategy that can be used independently to prioritize surveillance efforts. However, we demonstrate an alternative approach in which these outputs are combined via the Pareto ranking technique into a single priority map that outlines the survey regions with the best trade-offs between both surveillance strategies.

Info-gap theory and robust design of surveillance for invasive species: the case study of Barrow Island.

Surveillance for invasive non-indigenous species (NIS) is an integral part of a quarantine system. Estimating the efficiency of a surveillance strategy relies on many uncertain parameters estimated by experts, such as the efficiency of its components in face of the specific NIS, the ability of the NIS to inhabit different environments, and so on. Due to the importance of detecting an invasive NIS within a critical period of time, it is crucial that these uncertainties be accounted for in the design of the surveillance system. We formulate a detection model that takes into account, in addition to structured sampling for incursive NIS, incidental detection by untrained workers. We use info-gap theory for satisficing (not minimizing) the probability of detection, while at the same time maximizing the robustness to uncertainty. We demonstrate the trade-off between robustness to uncertainty, and an increase in the required probability of detection. An empirical example based on the detection of Pheidole megacephala on Barrow Island demonstrates the use of info-gap analysis to select a surveillance strategy.

Evaluation of neural network robust reliability using information-gap theory.

A novel technique for the evaluation of neural network robustness against uncertainty using a nonprobabilistic approach is presented. Conventional optimization techniques were employed to train multilayer perceptron (MLP) networks, which were then probed with an uncertainty analysis using an information-gap model to quantify the network response to uncertainty in the input data. It is demonstrated that the best performing network on data with low uncertainty is not in general the optimal network on data with a higher degree of input uncertainty. Using the concepts of information-gap theory, this paper develops a theoretical framework for information-gap uncertainty applied to neural networks, and explores the practical application of the procedure to three sample cases. The first consists of a simple two-dimensional (2-D) classification network operating on a known Gaussian distribution, the second a nine-lass vibration classification problem from an aircraft wing, and the third a two-class example from a database of breast cancer incidence.

Info-gap robust-satisficing model of foraging behavior: do foragers optimize or satisfice?

In this note we compare two mathematical models of foraging that reflect two competing theories of animal behavior: optimizing and robust satisficing. The optimal-foraging model is based on the marginal value theorem (MVT). The robust-satisficing model developed here is an application of info-gap decision theory. The info-gap robust-satisficing model relates to the same circumstances described by the MVT. We show how these two alternatives translate into specific predictions that at some points are quite disparate. We test these alternative predictions against available data collected in numerous field studies with a large number of species from diverse taxonomic groups. We show that a large majority of studies appear to support the robust-satisficing model and reject the optimal-foraging model.

Price-based vs. quantity-based environmental regulation under Knightian uncertainty: an info-gap robust satisficing perspective.

Conventional wisdom among environmental economists is that the relative slopes of the marginal social benefit and marginal social cost functions determine whether a price-based or quantity-based environmental regulation leads to higher expected social welfare. We revisit the choice between price-based vs. quantity-based environmental regulation under Knightian uncertainty; that is, when uncertainty cannot be modeled with known moments of probability distributions. Under these circumstances, the policy objective cannot be to maximize the expected net benefits of emissions control. Instead, we evaluate an emissions tax and an aggregate abatement standard in terms of maximizing the range of uncertainty under which the welfare loss from error in the estimates of the marginal benefits and costs of emissions control can be limited. The main result of our work is that the same criterion involving the relative slopes of the marginal benefit and cost functions determines whether price-based or quantity-based control is more robust to unstructured uncertainty. Hence, not only does the relative slopes criterion lead to the policy that maximizes the expected net benefits of control under structured uncertainty, it also leads to the policy that maximizes robustness to unstructured uncertainty.