Learning attentional templates for value-based decision-making.

Attention filters sensory inputs to enhance task-relevant information. It is guided by an "attentional template" that represents the stimulus features that are currently relevant. To understand how the brain learns and uses templates, we trained monkeys to perform a visual search task that required them to repeatedly learn new attentional templates. Neural recordings found that templates were represented across the prefrontal and parietal cortex in a structured manner, such that perceptually neighboring templates had similar neural representations. When the task changed, a new attentional template was learned by incrementally shifting the template toward rewarded features. Finally, we found that attentional templates transformed stimulus features into a common value representation that allowed the same decision-making mechanisms to deploy attention, regardless of the identity of the template. Altogether, our results provide insight into the neural mechanisms by which the brain learns to control attention and how attention can be flexibly deployed across tasks.

Effort Foraging Task reveals positive correlation between individual differences in the cost of cognitive and physical effort in humans.

Effort-based decisions, in which people weigh potential future rewards against effort costs required to achieve those rewards involve both cognitive and physical effort, though the mechanistic relationship between them is not yet understood. Here, we use an individual differences approach to isolate and measure the computational processes underlying effort-based decisions and test the association between cognitive and physical domains. Patch foraging is an ecologically valid reward rate maximization problem with well-developed theoretical tools. We developed the Effort Foraging Task, which embedded cognitive or physical effort into patch foraging, to quantify the cost of both cognitive and physical effort indirectly, by their effects on foraging choices. Participants chose between harvesting a depleting patch, or traveling to a new patch that was costly in time and effort. Participants' exit thresholds (reflecting the reward they expected to receive by harvesting when they chose to travel to a new patch) were sensitive to cognitive and physical effort demands, allowing us to quantify the perceived effort cost in monetary terms. The indirect sequential choice style revealed effort-seeking behavior in a minority of participants (preferring high over low effort) that has apparently been missed by many previous approaches. Individual differences in cognitive and physical effort costs were positively correlated, suggesting that these are perceived and processed in common. We used canonical correlation analysis to probe the relationship of task measures to self-reported affect and motivation, and found correlations of cognitive effort with anxiety, cognitive function, behavioral activation, and self-efficacy, but no similar correlations with physical effort.

Specialized coding of sensory, motor and cognitive variables in VTA dopamine neurons.

There is increased appreciation that dopamine neurons in the midbrain respond not only to reward1 and reward-predicting cues1,2, but also to other variables such as the distance to reward3, movements4-9 and behavioural choices10,11. An important question is how the responses to these diverse variables are organized across the population of dopamine neurons. Whether individual dopamine neurons multiplex several variables, or whether there are subsets of neurons that are specialized in encoding specific behavioural variables remains unclear. This fundamental question has been difficult to resolve because recordings from large populations of individual dopamine neurons have not been performed in a behavioural task with sufficient complexity to examine these diverse variables simultaneously. Here, to address this gap, we used two-photon calcium imaging through an implanted lens to record the activity of more than 300 dopamine neurons from the ventral tegmental area of the mouse midbrain during a complex decision-making task. As mice navigated in a virtual-reality environment, dopamine neurons encoded an array of sensory, motor and cognitive variables. These responses were functionally clustered, such that subpopulations of neurons transmitted information about a subset of behavioural variables, in addition to encoding reward. These functional clusters were spatially organized, with neighbouring neurons more likely to be part of the same cluster. Together with the topography between dopamine neurons and their projections, this specialization and anatomical organization may aid downstream circuits in correctly interpreting the wide range of signals transmitted by dopamine neurons.

Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning.

The ability to acquire abstract knowledge is a hallmark of human intelligence and is believed by many to be one of the core differences between humans and neural network models. Agents can be endowed with an inductive bias towards abstraction through meta-learning, where they are trained on a distribution of tasks that share some abstract structure that can be learned and applied. However, because neural networks are hard to interpret, it can be difficult to tell whether agents have learned the underlying abstraction, or alternatively statistical patterns that are characteristic of that abstraction. In this work, we compare the performance of humans and agents in a meta-reinforcement learning paradigm in which tasks are generated from abstract rules. We define a novel methodology for building "task metamers" that closely match the statistics of the abstract tasks but use a different underlying generative process, and evaluate performance on both abstract and metamer tasks. We find that humans perform better at abstract tasks than metamer tasks whereas common neural network architectures typically perform worse on the abstract tasks than the matched metamers. This work provides a foundation for characterizing differences between humans and machine learning that can be used in future work towards developing machines with more human-like behavior.

Humans decompose tasks by trading off utility and computational cost.

Human behavior emerges from planning over elaborate decompositions of tasks into goals, subgoals, and low-level actions. How are these decompositions created and used? Here, we propose and evaluate a normative framework for task decomposition based on the simple idea that people decompose tasks to reduce the overall cost of planning while maintaining task performance. Analyzing 11,117 distinct graph-structured planning tasks, we find that our framework justifies several existing heuristics for task decomposition and makes predictions that can be distinguished from two alternative normative accounts. We report a behavioral study of task decomposition (N = 806) that uses 30 randomly sampled graphs, a larger and more diverse set than that of any previous behavioral study on this topic. We find that human responses are more consistent with our framework for task decomposition than alternative normative accounts and are most consistent with a heuristic-betweenness centrality-that is justified by our approach. Taken together, our results suggest the computational cost of planning is a key principle guiding the intelligent structuring of goal-directed behavior.

An item response theory analysis of the matrix reasoning item bank (MaRs-IB).

Matrix reasoning tasks are among the most widely used measures of cognitive ability in the behavioral sciences, but the lack of matrix reasoning tests in the public domain complicates their use. Here, we present an extensive investigation and psychometric validation of the matrix reasoning item bank (MaRs-IB), an open-access set of matrix reasoning items. In a first study, we calibrate the psychometric functioning of the items in the MaRs-IB in a large sample of adult participants (N = 1501). Using additive multilevel item structure models, we establish that the MaRs-IB has many desirable psychometric properties: its items span a wide range of difficulty, possess medium-to-large levels of discrimination, and exhibit robust associations between item complexity and difficulty. However, we also find that item clones are not always psychometrically equivalent and cannot be assumed to be exchangeable. In a second study, we demonstrate how experimenters can use the estimated item parameters to design new matrix reasoning tests using optimal item assembly. Specifically, we design and validate two new sets of test forms in an independent sample of adults (N = 600). We find these new tests possess good reliability and convergent validity with an established measure of matrix reasoning. We hope that the materials and results made available here will encourage experimenters to use the MaRs-IB in their research.

Rat Anterior Cingulate Cortex Continuously Signals Decision Variables in a Patch Foraging Task.

In patch foraging tasks, animals must decide whether to remain with a depleting resource or to leave it in search of a potentially better source of reward. In such tasks, animals consistently follow the general predictions of optimal foraging theory (the marginal value theorem; MVT): to leave a patch when the reward rate in the current patch depletes to the average reward rate across patches. Prior studies implicate an important role for the anterior cingulate cortex (ACC) in foraging decisions based on MVT: within single trials, ACC activity increases immediately preceding foraging decisions, and across trials, these dynamics are modulated as the value of staying in the patch depletes to the average reward rate. Here, we test whether these activity patterns reflect dynamic encoding of decision-variables and whether these signals are directly involved in decision-making. We developed a leaky accumulator model based on the MVT that generates estimates of decision variables within and across trials, and tested model predictions against ACC activity recorded from male rats performing a patch foraging task. Model predicted changes in MVT decision variables closely matched rat ACC activity. Next, we pharmacologically inactivated ACC in male rats to test the contribution of these signals to decision-making. ACC inactivation had a profound effect on rats' foraging decisions and response times (RTs) yet rats still followed the MVT decision rule. These findings indicate that the ACC encodes foraging-related variables for reasons unrelated to patch-leaving decisions.SIGNIFICANCE STATEMENT The ability to make adaptive patch-foraging decisions, to remain with a depleting resource or search for better alternatives, is critical to animal well-being. Previous studies have found that anterior cingulate cortex (ACC) activity is modulated at different points in the foraging decision process, raising questions about whether the ACC guides ongoing decisions or serves a more general purpose of regulating cognitive control. To investigate the function of the ACC in foraging, the present study developed a dynamic model of behavior and neural activity, and tested model predictions using recordings and inactivation of ACC. Findings revealed that ACC continuously signals decision variables but that these signals are more likely used to monitor and regulate ongoing processes than to guide foraging decisions.

Depression: a decision-theoretic analysis.

The manifold symptoms of depression are common and often transient features of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioral implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.

Surviving threats: neural circuit and computational implications of a new taxonomy of defensive behaviour.

Research on defensive behaviour in mammals has in recent years focused on elicited reactions; however, organisms also make active choices when responding to danger. We propose a hierarchical taxonomy of defensive behaviour on the basis of known psychological processes. Included are three categories of reactions (reflexes, fixed reactions and habits) and three categories of goal-directed actions (direct action-outcome behaviours and actions based on implicit or explicit forecasting of outcomes). We then use this taxonomy to guide a summary of findings regarding the underlying neural circuits.

In for a penny, in for a pound: examining motivated memory through the lens of retrieved context models.

When people encounter items that they believe will help them gain reward, they later remember them better than others. A recent model of emotional memory, the emotional context maintenance and retrieval model (eCMR), predicts that these effects would be stronger when stimuli that predict high and low reward can compete with each other during both encoding and retrieval. We tested this prediction in two experiments. Participants were promised £1 for remembering some pictures, but only a few pence for remembering others. Their recall of the content of the pictures they saw was tested after 1 min and, in experiment 2, also after 24 h. Memory at the immediate test showed effects of list composition. Recall of stimuli that predicted high reward was greater than of stimuli that predicted lower reward, but only when high- and low-reward items were studied and recalled together, not when they were studied and recalled separately. More high-reward items in mixed lists were forgotten over a 24-h retention interval compared with items studied in other conditions, but reward did not modulate the forgetting rate, a null effect that should be replicated in a larger sample. These results confirm eCMR's predictions, although further research is required to compare that model against alternatives.

Deficient Goal-Directed Control in a Population Characterized by Extreme Goal Pursuit.

Research in computational psychiatry has sought to understand the basis of compulsive behavior by relating it to basic psychological and neural mechanisms: specifically, goal-directed versus habitual control. These psychological categories have been further identified with formal computational algorithms, model-based and model-free learning, which helps to provide quantitative tools to distinguish them. Computational psychiatry may be particularly useful for examining phenomena in individuals with anorexia nervosa (AN), whose self-starvation appears both excessively goal directed and habitual. However, these laboratory-based studies have not aimed to examine complex behavior, as seen outside the laboratory, in contexts that extend beyond monetary rewards. We therefore assessed (1) whether behavior in AN was characterized by enhanced or diminished model-based behavior, (2) the domain specificity of any abnormalities by comparing learning in a food-specific (i.e., illness-relevant) context as well as in a monetary context, and (3) whether impairments were secondary to starvation by comparing learning before and after initial treatment. Across all conditions, individuals with AN, relative to healthy controls, showed an impairment in model-based, but not model-free, learning, suggesting a general and persistent contribution of habitual over goal-directed control, across domains and time points. Thus, eating behavior in individuals with AN that appears very goal-directed may be under more habitual than goal-directed control, and this is not remediated by achieving weight restoration.

A simple model for learning in volatile environments.

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Role of Human Ventromedial Prefrontal Cortex in Learning and Recall of Enhanced Extinction.

Standard fear extinction relies on the ventromedial prefrontal cortex (vmPFC) to form a new memory given the omission of threat. Using fMRI in humans, we investigated whether replacing threat with novel neutral outcomes (instead of just omitting threat) facilitates extinction by engaging the vmPFC more effectively than standard extinction. Computational modeling of associability (indexing surprise strength and dynamically modulating learning rates) characterized skin conductance responses and vmPFC activity during novelty-facilitated but not standard extinction. Subjects who showed faster within-session updating of associability during novelty-facilitated extinction also expressed better extinction retention the next day, as expressed through skin conductance responses. Finally, separable patterns of connectivity between the amygdala and ventral versus dorsal mPFC characterized retrieval of novelty-facilitated versus standard extinction memories, respectively. These results indicate that replacing threat with novel outcomes stimulates vmPFC involvement on extinction trials, leading to a more durable long-term extinction memory.SIGNIFICANCE STATEMENT Psychiatric disorders characterized be excessive fear are a major public health concern. Popular clinical treatments, such as exposure therapy, are informed by principles of Pavlovian extinction. Thus, there is motivation to optimize extinction strategies in the laboratory so as to ultimately develop more effective clinical treatments. Here, we used functional neuroimaging in humans and found that replacing (rather than just omitting) expected aversive events with novel and neutral outcomes engages the ventromedial prefrontal cortex during extinction learning. Enhanced extinction also diminished activity in threat-related networks (e.g., the insula, thalamus) during immediate extinction and a 24 h extinction retention test. This is new evidence for how behavioral protocols designed to enhance extinction affects neurocircuitry underlying the learning and retention of extinction memories.

A Computational Role for Top-Down Modulation from Frontal Cortex in Infancy.

Recent findings have shown that full-term infants engage in top-down sensory prediction, and these predictions are impaired as a result of premature birth. Here, we use an associative learning model to uncover the neuroanatomical origins and computational nature of this top-down signal. Infants were exposed to a probabilistic audiovisual association. We find that both groups (full term, preterm) have a comparable stimulus-related response in sensory and frontal lobes and track prediction error in their frontal lobes. However, preterm infants differ from their full-term peers in weaker tracking of prediction error in sensory regions. We infer that top-down signals from the frontal lobe to the sensory regions carry information about prediction error. Using computational learning models and comparing neuroimaging results from full-term and preterm infants, we have uncovered the computational content of top-down signals in young infants when they are engaged in a probabilistic associative learning.

Sympathetic involvement in time-constrained sequential foraging.

Appraising sequential offers relative to an unknown future opportunity and a time cost requires an optimization policy that draws on a learned estimate of an environment's richness. Converging evidence points to a learning asymmetry, whereby estimates of this richness update with a bias toward integrating positive information. We replicate this bias in a sequential foraging (prey selection) task and probe associated activation within the sympathetic branch of the autonomic system, using trial-by-trial measures of simultaneously recorded cardiac autonomic physiology. We reveal a unique adaptive role for the sympathetic branch in learning. It was specifically associated with adaptation to a deteriorating environment: it correlated with both the rate of negative information integration in belief estimates and downward changes in moment-to-moment environmental richness, and was predictive of optimal performance on the task. The findings are consistent with a framework whereby autonomic function supports the learning demands of prey selection.

Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies.

Computational modeling plays an important role in modern neuroscience research. Much previous research has relied on statistical methods, separately, to address two problems that are actually interdependent. First, given a particular computational model, Bayesian hierarchical techniques have been used to estimate individual variation in parameters over a population of subjects, leveraging their population-level distributions. Second, candidate models are themselves compared, and individual variation in the expressed model estimated, according to the fits of the models to each subject. The interdependence between these two problems arises because the relevant population for estimating parameters of a model depends on which other subjects express the model. Here, we propose a hierarchical Bayesian inference (HBI) framework for concurrent model comparison, parameter estimation and inference at the population level, combining previous approaches. We show that this framework has important advantages for both parameter estimation and model comparison theoretically and experimentally. The parameters estimated by the HBI show smaller errors compared to other methods. Model comparison by HBI is robust against outliers and is not biased towards overly simplistic models. Furthermore, the fully Bayesian approach of our theory enables researchers to make inference on group-level parameters by performing HBI t-test.

A distinct inferential mechanism for delusions in schizophrenia.

Delusions, a core symptom of psychosis, are false beliefs that are rigidly held with strong conviction despite contradictory evidence. Alterations in inferential processes have long been proposed to underlie delusional pathology, but previous attempts to show this have failed to yield compelling evidence for a specific relationship between inferential abnormalities and delusional severity in schizophrenia. Using a novel, incentivized information-sampling task (a modified version of the beads task), alongside well-characterized decision-making tasks, we sought a mechanistic understanding of delusions in a sample of medicated and unmedicated patients with schizophrenia who exhibited a wide range of delusion severity. In this novel task, participants chose whether to draw beads from one of two hidden jars or to guess the identity of the hidden jar, in order to minimize financial loss from a monetary endowment, and concurrently reported their probability estimates for the hidden jar. We found that patients with higher delusion severity exhibited increased information seeking (i.e. increased draws-to-decision behaviour). This increase was highly specific to delusion severity as compared to the severity of other psychotic symptoms, working-memory capacity, and other clinical and socio-demographic characteristics. Delusion-related increases in information seeking were present in unmedicated patients, indicating that they were unlikely due to antipsychotic medication. In addition, after adjusting for delusion severity, patients as a whole exhibited decreased information seeking relative to healthy individuals, a decrease that correlated with lower socioeconomic status. Computational analyses of reported probability estimates further showed that more delusional patients exhibited abnormal belief updating characterized by stronger reliance on prior beliefs formed early in the inferential process, a feature that correlated with increased information seeking in patients. Other decision-making parameters that could have theoretically explained the delusion effects, such as those related to subjective valuation, were uncorrelated with both delusional severity and information seeking among the patients. In turn, we found some preliminary evidence that subjective valuation (rather than belief updating) may explain group differences in information seeking unrelated to delusions. Together, these results suggest that abnormalities in belief updating, characterized by stronger reliance on prior beliefs formed by incorporating information presented earlier in the inferential process, may be a core computational mechanism of delusional ideation in psychosis. Our results thus provide direct empirical support for an inferential mechanism that naturally captures the characteristic rigidity associated with delusional beliefs.

The importance of mixed selectivity in complex cognitive tasks.

Single-neuron activity in the prefrontal cortex (PFC) is tuned to mixtures of multiple task-related aspects. Such mixed selectivity is highly heterogeneous, seemingly disordered and therefore difficult to interpret. We analysed the neural activity recorded in monkeys during an object sequence memory task to identify a role of mixed selectivity in subserving the cognitive functions ascribed to the PFC. We show that mixed selectivity neurons encode distributed information about all task-relevant aspects. Each aspect can be decoded from the population of neurons even when single-cell selectivity to that aspect is eliminated. Moreover, mixed selectivity offers a significant computational advantage over specialized responses in terms of the repertoire of input-output functions implementable by readout neurons. This advantage originates from the highly diverse nonlinear selectivity to mixtures of task-relevant variables, a signature of high-dimensional neural representations. Crucially, this dimensionality is predictive of animal behaviour as it collapses in error trials. Our findings recommend a shift of focus for future studies from neurons that have easily interpretable response tuning to the widely observed, but rarely analysed, mixed selectivity neurons.

Independent Neural Computation of Value from Other People's Confidence.

Expectation of reward can be shaped by the observation of actions and expressions of other people in one's environment. A person's apparent confidence in the likely reward of an action, for instance, makes qualities of their evidence, not observed directly, socially accessible. This strategy is computationally distinguished from associative learning methods that rely on direct observation, by its use of inference from indirect evidence. In twenty-three healthy human subjects, we isolated effects of first-hand experience, other people's choices, and the mediating effect of their confidence, on decision-making and neural correlates of value within ventromedial prefrontal cortex (vmPFC). Value derived from first-hand experience and other people's choices (regardless of confidence) were indiscriminately represented across vmPFC. However, value computed from agent choices weighted by their associated confidence was represented with specificity for ventromedial area 10. This pattern corresponds to shifts of connectivity and overlapping cognitive processes along a posterior-anterior vmPFC axis. Task behavior and self-reported self-reliance for decision-making in other social contexts correlated. The tendency to conform in other social contexts corresponded to increased activation in cortical regions previously shown to respond to social conflict in proportion to subsequent conformity (Campbell-Meiklejohn et al., 2010). The tendency to self-monitor predicted a selectively enhanced response to accordance with others in the right temporoparietal junction (rTPJ). The findings anatomically decompose vmPFC value representations according to computational requirements and provide biological insight into the social transmission of preference and reassurance gained from the confidence of others. SIGNIFICANCE STATEMENT: Decades of research have provided evidence that the ventromedial prefrontal cortex (vmPFC) signals the satisfaction we expect from imminent actions. However, we have a surprisingly modest understanding of the organization of value across this substantial and varied region. This study finds that using cues of the reliability of other peoples' knowledge to enhance expectation of personal success generates value correlates that are anatomically distinct from those concurrently computed from direct, personal experience. This suggests that representation of decision values in vmPFC is suborganized according to the underlying computation, consistent with what we know about the anatomical heterogeneity of the region. These results also provide insight into the observational learning process by which someone else's confidence can sway and reassure our choices.

Chronic and Acute Stress Promote Overexploitation in Serial Decision Making.

Many decisions that humans make resemble foraging problems in which a currently available, known option must be weighed against an unknown alternative option. In such foraging decisions, the quality of the overall environment can be used as a proxy for estimating the value of future unknown options against which current prospects are compared. We hypothesized that such foraging-like decisions would be characteristically sensitive to stress, a physiological response that tracks biologically relevant changes in environmental context. Specifically, we hypothesized that stress would lead to more exploitative foraging behavior. To test this, we investigated how acute and chronic stress, as measured by changes in cortisol in response to an acute stress manipulation and subjective scores on a questionnaire assessing recent chronic stress, relate to performance in a virtual sequential foraging task. We found that both types of stress bias human decision makers toward overexploiting current options relative to an optimal policy. These findings suggest a possible computational role of stress in decision making in which stress biases judgments of environmental quality.SIGNIFICANCE STATEMENT Many of the most biologically relevant decisions that we make are foraging-like decisions about whether to stay with a current option or search the environment for a potentially better one. In the current study, we found that both acute physiological and chronic subjective stress are associated with greater overexploitation or staying at current options for longer than is optimal. These results suggest a domain-general way in which stress might bias foraging decisions through changing one's appraisal of the overall quality of the environment. These novel findings not only have implications for understanding how this important class of foraging decisions might be biologically implemented, but also for understanding the computational role of stress in behavior and cognition more broadly.