Freezing revisited: coordinated autonomic and central optimization of threat coping.

Animals have sophisticated mechanisms for coping with danger. Freezing is a unique state that, upon threat detection, allows evidence to be gathered, response possibilities to be previsioned and preparations to be made for worst-case fight or flight. We propose that - rather than reflecting a passive fear state - the particular somatic and cognitive characteristics of freezing help to conceal overt responses, while optimizing sensory processing and action preparation. Critical for these functions are the neurotransmitters noradrenaline and acetylcholine, which modulate neural information processing and also control the sympathetic and parasympathetic branches of the autonomic nervous system. However, the interactions between autonomic systems and the brain during freezing, and the way in which they jointly coordinate responses, remain incompletely explored. We review the joint actions of these systems and offer a novel computational framework to describe their temporally harmonized integration. This reconceptualization of freezing has implications for its role in decision-making under threat and for psychopathology.

"Liking" as an early and editable draft of long-run affective value.

Psychological and neural distinctions between the technical concepts of "liking" and "wanting" pose important problems for motivated choice for goods. Why could we "want" something that we do not "like," or "like" something but be unwilling to exert effort to acquire it? Here, we suggest a framework for answering these questions through the medium of reinforcement learning. We consider "liking" to provide immediate, but preliminary and ultimately cancellable, information about the true, long-run worth of a good. Such initial estimates, viewed through the lens of what is known as potential-based shaping, help solve the temporally complex learning problems faced by animals.

Wagers for work: Decomposing the costs of cognitive effort.

Some aspects of cognition are more taxing than others. Accordingly, many people will avoid cognitively demanding tasks in favor of simpler alternatives. Which components of these tasks are costly, and how much, remains unknown. Here, we use a novel task design in which subjects request wages for completing cognitive tasks and a computational modeling procedure that decomposes their wages into the costs driving them. Using working memory as a test case, our approach revealed that gating new information into memory and protecting against interference are costly. Critically, other factors, like memory load, appeared less costly. Other key factors which may drive effort costs, such as error avoidance, had minimal influence on wage requests. Our approach is sensitive to individual differences, and could be used in psychiatric populations to understand the true underlying nature of apparent cognitive deficits.

Bayesian reinforcement learning: A basic overview.

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Reframing dopamine: A controlled controller at the limbic-motor interface.

Pavlovian influences notoriously interfere with operant behaviour. Evidence suggests this interference sometimes coincides with the release of the neuromodulator dopamine in the nucleus accumbens. Suppressing such interference is one of the targets of cognitive control. Here, using the examples of active avoidance and omission behaviour, we examine the possibility that direct manipulation of the dopamine signal is an instrument of control itself. In particular, when instrumental and Pavlovian influences come into conflict, dopamine levels might be affected by the controlled deployment of a reframing mechanism that recasts the prospect of possible punishment as an opportunity to approach safety, and the prospect of future reward in terms of a possible loss of that reward. We operationalize this reframing mechanism and fit the resulting model to rodent behaviour from two paradigmatic experiments in which accumbens dopamine release was also measured. We show that in addition to matching animals' behaviour, the model predicts dopamine transients that capture some key features of observed dopamine release at the time of discriminative cues, supporting the idea that modulation of this neuromodulator is amongst the repertoire of cognitive control strategies.

Delphi Process for Validation of Fluid Treatment Algorithm for Critically Ill Pediatric Trauma Patients.

INTRODUCTION: While intravenous fluid therapy is essential to re-establishing volume status in children who have experienced trauma, aggressive resuscitation can lead to various complications. There remains a lack of consensus on whether pediatric trauma patients will benefit from a liberal or restrictive crystalloid resuscitation approach and how to optimally identify and transition between fluid phases. METHODS: A panel was comprised of physicians with expertise in pediatric trauma, critical care, and emergency medicine. A three-round Delphi process was conducted via an online survey, with each round being followed by a live video conference. Experts agreed or disagreed with each aspect of the proposed fluid management algorithm on a five-level Likert scale. The group opinion level defined an algorithm parameter's acceptance or rejection with greater than 75% agreement resulting in acceptance and greater than 50% disagreement resulting in rejection. The remaining were discussed and re-presented in the next round. RESULTS: Fourteen experts from five Level 1 pediatric trauma centers representing three subspecialties were included. Responses were received from 13/14 participants (93%). In round 1, 64% of the parameters were accepted, while the remaining 36% were discussed and re-presented. In round 2, 90% of the parameters were accepted. Following round 3, there was 100% acceptance by all the experts on the revised and final version of the algorithm. CONCLUSIONS: We present a validated algorithm for intavenous fluid management in pediatric trauma patients that focuses on the de-escalation of fluids. Focusing on this time point of fluid therapy will help minimize iatrogenic complications of crystalloid fluids within this patient population.

Human subjects exploit a cognitive map for credit assignment.

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

Depression: a decision-theoretic analysis.

The manifold symptoms of depression are common and often transient features of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioral implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.

A primer on the use of computational modelling to investigate affective states, affective disorders and animal welfare in non-human animals.

Objective measures of animal emotion-like and mood-like states are essential for preclinical studies of affective disorders and for assessing the welfare of laboratory and other animals. However, the development and validation of measures of these affective states poses a challenge partly because the relationships between affect and its behavioural, physiological and cognitive signatures are complex. Here, we suggest that the crisp characterisations offered by computational modelling of the underlying, but unobservable, processes that mediate these signatures should provide better insights. Although this computational psychiatry approach has been widely used in human research in both health and disease, translational computational psychiatry studies remain few and far between. We explain how building computational models with data from animal studies could play a pivotal role in furthering our understanding of the aetiology of affective disorders, associated affective states and the likely underlying cognitive processes involved. We end by outlining the basic steps involved in a simple computational analysis.

Accuracy of NGAL as a Biomarker for Urinary Tract Infection in Young Febrile Children: An Individual Patient Data Meta-Analysis.

OBJECTIVE: To compare the accuracy of urine neutrophil gelatinase-associated lipocalin (NGAL) and leukocyte esterase (LE) for the diagnosis of urinary tract infection (UTI) in children. STUDY DESIGN: We performed a systematic review and individual patient data meta-analysis of studies that examined urine NGAL as a marker of UTI in children <18 years of age. We created a standardized definition of UTI and applied it to all included children. We compared sensitivity, specificity, and the area under the receiver operating characteristic curve (AUC) of NGAL with LE. RESULTS: We included individual patient data from 3 studies for a total of 845 children. Included children had a mean age of 0.9 years (SD, 0.6 years). Using a cutoff of 32.7 ng/mL, NGAL had a sensitivity of 90.3% (95% CI: 83.2%-95.0%) and specificity of 93.7% (95% CI: 91.7%-95.4%) for the diagnosis of UTI. LE, using a cutoff of ≧ trace had a sensitivity of 81.1% (95% CI: 72.5%-87.9%) and specificity of 97.0% (95% CI: 95.4%-98.1%). The AUC for NGAL was 0.95 (95% CI: 0.92-0.98). The AUC for LE was 0.90 (95% CI: 0.86-0.93). CONCLUSION: In young, febrile children, urinary NGAL is more sensitive for the diagnosis of UTI than LE but is slightly less specific.

Foraging for foundations in decision neuroscience: insights from ethology.

Modern decision neuroscience offers a powerful and broad account of human behaviour using computational techniques that link psychological and neuroscientific approaches to the ways that individuals can generate near-optimal choices in complex controlled environments. However, until recently, relatively little attention has been paid to the extent to which the structure of experimental environments relates to natural scenarios, and the survival problems that individuals have evolved to solve. This situation not only risks leaving decision-theoretic accounts ungrounded but also makes various aspects of the solutions, such as hard-wired or Pavlovian policies, difficult to interpret in the natural world. Here, we suggest importing concepts, paradigms and approaches from the fields of ethology and behavioural ecology, which concentrate on the contextual and functional correlates of decisions made about foraging and escape and address these lacunae.

Interactions between attributions and beliefs at trial-by-trial level: Evidence from a novel computer game task.

Inferring causes of the good and bad events that we experience is part of the process of building models of our own capabilities and of the world around us. Making such inferences can be difficult because of complex reciprocal relationships between attributions of the causes of particular events, and beliefs about the capabilities and skills that influence our role in bringing them about. Abnormal causal attributions have long been studied in connection with psychiatric disorders, notably depression and paranoia; however, the mechanisms behind attributional inferences and the way they can go awry are not fully understood. We administered a novel, challenging, game of skill to a substantial population of healthy online participants, and collected trial-by-trial time series of both their beliefs about skill and attributions about the causes of the success and failure of real experienced outcomes. We found reciprocal relationships that provide empirical confirmation of the attribution-self representation cycle theory. This highlights the dynamic nature of the processes involved in attribution, and validates a framework for developing and testing computational accounts of attribution-belief interactions.

Vigilance, arousal, and acetylcholine: Optimal control of attention in a simple detection task.

Paying attention to particular aspects of the world or being more vigilant in general can be interpreted as forms of 'internal' action. Such arousal-related choices come with the benefit of increasing the quality and situational appropriateness of information acquisition and processing, but incur potentially expensive energetic and opportunity costs. One implementational route for these choices is widespread ascending neuromodulation, including by acetylcholine (ACh). The key computational question that elective attention poses for sensory processing is when it is worthwhile paying these costs, and this includes consideration of whether sufficient information has yet been collected to justify the higher signal-to-noise ratio afforded by greater attention and, particularly if a change in attentional state is more expensive than its maintenance, when states of heightened attention ought to persist. We offer a partially observable Markov decision-process treatment of optional attention in a detection task, and use it to provide a qualitative model of the results of studies using modern techniques to measure and manipulate ACh in rodents performing a similar task.

Optimism and pessimism in optimised replay.

The replay of task-relevant trajectories is known to contribute to memory consolidation and improved task performance. A wide variety of experimental data show that the content of replayed sequences is highly specific and can be modulated by reward as well as other prominent task variables. However, the rules governing the choice of sequences to be replayed still remain poorly understood. One recent theoretical suggestion is that the prioritization of replay experiences in decision-making problems is based on their effect on the choice of action. We show that this implies that subjects should replay sub-optimal actions that they dysfunctionally choose rather than optimal ones, when, by being forgetful, they experience large amounts of uncertainty in their internal models of the world. We use this to account for recent experimental data demonstrating exactly pessimal replay, fitting model parameters to the individual subjects' choices.

Tracking human skill learning with a hierarchical Bayesian sequence model.

Humans can implicitly learn complex perceptuo-motor skills over the course of large numbers of trials. This likely depends on our becoming better able to take advantage of ever richer and temporally deeper predictive relationships in the environment. Here, we offer a novel characterization of this process, fitting a non-parametric, hierarchical Bayesian sequence model to the reaction times of human participants' responses over ten sessions, each comprising thousands of trials, in a serial reaction time task involving higher-order dependencies. The model, adapted from the domain of language, forgetfully updates trial-by-trial, and seamlessly combines predictive information from shorter and longer windows onto past events, weighing the windows proportionally to their predictive power. As the model implies a posterior over window depths, we were able to determine how, and how many, previous sequence elements influenced individual participants' internal predictions, and how this changed with practice. Already in the first session, the model showed that participants had begun to rely on two previous elements (i.e., trigrams), thereby successfully adapting to the most prominent higher-order structure in the task. The extent to which local statistical fluctuations in trigram frequency influenced participants' responses waned over subsequent sessions, as participants forgot the trigrams less and evidenced skilled performance. By the eighth session, a subset of participants shifted their prior further to consider a context deeper than two previous elements. Finally, participants showed resistance to interference and slow forgetting of the old sequence when it was changed in the final sessions. Model parameters for individual participants covaried appropriately with independent measures of working memory and error characteristics. In sum, the model offers the first principled account of the adaptive complexity and nuanced dynamics of humans' internal sequence representations during long-term implicit skill learning.

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons.

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

Biased belief priors versus biased belief updating: Differential correlates of depression and anxiety.

Individuals prone to anxiety and depression often report beliefs and make judgements about themselves that are more negative than those reported by others. We use computational modeling of a richly naturalistic task to disentangle the role of negative priors versus negatively biased belief updating and to investigate their association with different dimensions of Internalizing psychopathology. Undergraduate participants first provided profiles for a hypothetical tech internship. They then viewed pairs of other profiles and selected the individual they would prefer to work alongside out of each pair. In a subsequent phase of the experiment, participants made judgments about their relative popularity as hypothetical internship partners both before any feedback and after each of 20 items of feedback revealing whether or not they had been selected as the preferred teammate from a given pairing. Scores on latent factors of general negative affect, anxiety-specific affect and depression-specific affect were estimated using participants' self-report scores on standardized measures of anxiety and depression together with factor loadings from a bifactor analysis conducted previously. Higher scores on the depression-specific factor were linked to more negative prior beliefs but were not associated with differences in belief updating. In contrast, higher scores on the anxiety-specific factor were associated with a negative bias in belief updating but no difference in prior beliefs. These findings indicate that, to at least some extent, distinct processes may impact the formation of belief priors and in-the-moment belief updating and that these processes may be differentially disrupted in depression and anxiety. Future directions for enquiry include examination of the possibility that prior beliefs biases in depression might reflect generalization from prior experiences or global schema whereas belief updating biases in anxiety might be more situationally specific.

Adversarial vulnerabilities of human decision-making.

Adversarial examples are carefully crafted input patterns that are surprisingly poorly classified by artificial and/or natural neural networks. Here we examine adversarial vulnerabilities in the processes responsible for learning and choice in humans. Building upon recent recurrent neural network models of choice processes, we propose a general framework for generating adversarial opponents that can shape the choices of individuals in particular decision-making tasks toward the behavioral patterns desired by the adversary. We show the efficacy of the framework through three experiments involving action selection, response inhibition, and social decision-making. We further investigate the strategy used by the adversary in order to gain insights into the vulnerabilities of human choice. The framework may find applications across behavioral sciences in helping detect and avoid flawed choice.

Uncertainty in learning, choice, and visual fixation.

Uncertainty plays a critical role in reinforcement learning and decision making. However, exactly how it influences behavior remains unclear. Multiarmed-bandit tasks offer an ideal test bed, since computational tools such as approximate Kalman filters can closely characterize the interplay between trial-by-trial values, uncertainty, learning, and choice. To gain additional insight into learning and choice processes, we obtained data from subjects' overt allocation of gaze. The estimated value and estimation uncertainty of options influenced what subjects looked at before choosing; these same quantities also influenced choice, as additionally did fixation itself. A momentary measure of uncertainty in the form of absolute prediction errors determined how long participants looked at the obtained outcomes. These findings affirm the importance of uncertainty in multiple facets of behavior and help delineate its effects on decision making.