Goals, usefulness and abstraction in value-based choice.

Colombian drug lord Pablo Escobar, while on the run, purportedly burned two million dollars in banknotes to keep his daughter warm. A stark reminder that, in life, circumstances and goals can quickly change, forcing us to reassess and modify our values on-the-fly. Studies in decision-making and neuroeconomics have often implicitly equated value to reward, emphasising the hedonic and automatic aspect of the value computation, while overlooking its functional (concept-like) nature. Here we outline the computational and biological principles that enable the brain to compute the usefulness of an option or action by creating abstractions that flexibly adapt to changing goals. We present different algorithmic architectures, comparing ideas from artificial intelligence (AI) and cognitive neuroscience with psychological theories and, when possible, drawing parallels.

Humans actively sample evidence to support prior beliefs.

No one likes to be wrong. Previous research has shown that participants may underweight information incompatible with previous choices, a phenomenon called confirmation bias. In this paper, we argue that a similar bias exists in the way information is actively sought. We investigate how choice influences information gathering using a perceptual choice task and find that participants sample more information from a previously chosen alternative. Furthermore, the higher the confidence in the initial choice, the more biased information sampling becomes. As a consequence, when faced with the possibility of revising an earlier decision, participants are more likely to stick with their original choice, even when incorrect. Critically, we show that agency controls this phenomenon. The effect disappears in a fixed sampling condition where presentation of evidence is controlled by the experimenter, suggesting that the way in which confirmatory evidence is acquired critically impacts the decision process. These results suggest active information acquisition plays a critical role in the propagation of strongly held beliefs over time.

Value signals guide abstraction during learning.

The human brain excels at constructing and using abstractions, such as rules, or concepts. Here, in two fMRI experiments, we demonstrate a mechanism of abstraction built upon the valuation of sensory features. Human volunteers learned novel association rules based on simple visual features. Reinforcement-learning algorithms revealed that, with learning, high-value abstract representations increasingly guided participant behaviour, resulting in better choices and higher subjective confidence. We also found that the brain area computing value signals - the ventromedial prefrontal cortex - prioritised and selected latent task elements during abstraction, both locally and through its connection to the visual cortex. Such a coding scheme predicts a causal role for valuation. Hence, in a second experiment, we used multivoxel neural reinforcement to test for the causality of feature valuation in the sensory cortex, as a mechanism of abstraction. Tagging the neural representation of a task feature with rewards evoked abstraction-based decisions. Together, these findings provide a novel interpretation of value as a goal-dependent, key factor in forging abstract representations.

Visual attention modulates the integration of goal-relevant evidence and not value.

When choosing between options, such as food items presented in plain view, people tend to choose the option they spend longer looking at. The prevailing interpretation is that visual attention increases value. However, in previous studies, 'value' was coupled to a behavioural goal, since subjects had to choose the item they preferred. This makes it impossible to discern if visual attention has an effect on value, or, instead, if attention modulates the information most relevant for the goal of the decision-maker. Here, we present the results of two independent studies-a perceptual and a value-based task-that allow us to decouple value from goal-relevant information using specific task-framing. Combining psychophysics with computational modelling, we show that, contrary to the current interpretation, attention does not boost value, but instead it modulates goal-relevant information. This work provides a novel and more general mechanism by which attention interacts with choice.

Confidence modulates exploration and exploitation in value-based learning.

Uncertainty is ubiquitous in cognitive processing. In this study, we aim to investigate the ability agents possess to track and report the noise inherent in their mental operations, often in the form of confidence judgments. Here, we argue that humans can use uncertainty inherent in their representations of value beliefs to arbitrate between exploration and exploitation. Such uncertainty is reflected in explicit confidence judgments. Using a novel variant of a multi-armed bandit paradigm, we studied how beliefs were formed and how uncertainty in the encoding of these value beliefs (belief confidence) evolved over time. We found that people used uncertainty to arbitrate between exploration and exploitation, reflected in a higher tendency toward exploration when their confidence in their value representations was low. We furthermore found that value uncertainty can be linked to frameworks of metacognition in decision making in two ways. First, belief confidence drives decision confidence, i.e. people's evaluation of their own choices. Second, individuals with higher metacognitive insight into their choices were also better at tracing the uncertainty in their environment. Together, these findings argue that such uncertainty representations play a key role in the context of cognitive control.

The neural and cognitive architecture for learning from a small sample.

Artificial intelligence algorithms are capable of fantastic exploits, yet they are still grossly inefficient compared with the brain's ability to learn from few exemplars or solve problems that have not been explicitly defined. What is the secret that the evolution of human intelligence has unlocked? Generalization is one answer, but there is more to it. The brain does not directly solve difficult problems, it is able to recast them into new and more tractable problems. Here, we propose a model whereby higher cognitive functions profoundly interact with reinforcement learning to drastically reduce the degrees of freedom of the search space, simplifying complex problems, and fostering more efficient learning.

Compulsivity Reveals a Novel Dissociation between Action and Confidence.

Confidence and actions are normally tightly interwoven-if I am sure that it is going to rain, I will take an umbrella-therefore, it is difficult to understand their interplay. Stimulated by the ego-dystonic nature of obsessive-compulsive disorder (OCD), where compulsive actions are recognized as disproportionate, we hypothesized that action and confidence might be independently updated during learning. Participants completed a predictive-inference task designed to identify how action and confidence evolve in response to surprising changes in the environment. While OCD patients (like controls) correctly updated their confidence according to changes in the environment, their actions (unlike those of controls) mostly disregarded this knowledge. Therefore, OCD patients develop an accurate, internal model of the environment but fail to use it to guide behavior. Results demonstrated a novel dissociation between confidence and action, suggesting a cognitive architecture whereby confidence estimates can accurately track the statistic of the environment independently from performance.

Prior preferences beneficially influence social and non-social learning.

Our personal preferences affect a broad array of social behaviors. This includes the way we learn the preferences of others, an ability that often relies on limited or ambiguous information. Here we report an egocentric influence on this type of social learning that is reflected in both performance and response times. Using computational models that combine inter-trial learning and intra-trial choice, we find transient effects of participants' preferences on the learning process, through the influence of priors, and persistent effects on the choice process. A second experiment shows that these effects generalize to non-social learning, though participants in the social learning experiment appeared to additionally benefit by using their knowledge about the popularity of certain preferences. We further find that the domain-general egocentric influences we identify can yield performance advantages in uncertain environments.People often assume that other people share their preferences, but how exactly this bias manifests itself in learning and decision-making is unclear. Here, authors show that a person's own preferences influence learning in both social and non-social situations, and that this bias improves performance.

Social Information Is Integrated into Value and Confidence Judgments According to Its Reliability.

How much we like something, whether it be a bottle of wine or a new film, is affected by the opinions of others. However, the social information that we receive can be contradictory and vary in its reliability. Here, we tested whether the brain incorporates these statistics when judging value and confidence. Participants provided value judgments about consumer goods in the presence of online reviews. We found that participants updated their initial value and confidence judgments in a Bayesian fashion, taking into account both the uncertainty of their initial beliefs and the reliability of the social information. Activity in dorsomedial prefrontal cortex tracked the degree of belief update. Analogous to how lower-level perceptual information is integrated, we found that the human brain integrates social information according to its reliability when judging value and confidence.SIGNIFICANCE STATEMENT The field of perceptual decision making has shown that the sensory system integrates different sources of information according to their respective reliability, as predicted by a Bayesian inference scheme. In this work, we hypothesized that a similar coding scheme is implemented by the human brain to process social signals and guide complex, value-based decisions. We provide experimental evidence that the human prefrontal cortex's activity is consistent with a Bayesian computation that integrates social information that differs in reliability and that this integration affects the neural representation of value and confidence.

A bilingual disadvantage in metacognitive processing.

Recent research indicating that bilingualism is associated with enhanced executive function suggests that this enhancement may operate within a broader spectrum of cognitive abilities than previously thought (e.g., Stocco & Prat, 2014). In this study, we focus on metacognition or the ability to evaluate one's own cognitive performance (Flavell, 1979). Over the course of two experiments, we presented young healthy adult monolinguals and bilinguals with a perceptual two-alternative-forced-choice task followed by confidence judgements. Results from both experiments indicated that bilingual participants showed a disadvantage in metacognitive efficiency, determined through the calculation of Mratio (Maniscalco & Lau, 2014). Our findings provide novel insight into the potential differences in bilingual and monolingual cognition, which may indicate a bilingual disadvantage. Results are discussed with reference to the balance of advantages versus disadvantages associated with multilanguage learning.

Emotion-induced loss aversion and striatal-amygdala coupling in low-anxious individuals.

Adapting behavior to changes in the environment is a crucial ability for survival but such adaptation varies widely across individuals. Here, we asked how humans alter their economic decision-making in response to emotional cues, and whether this is related to trait anxiety. Developing an emotional decision-making task for functional magnetic resonance imaging, in which gambling decisions were preceded by emotional and non-emotional primes, we assessed emotional influences on loss aversion, the tendency to overweigh potential monetary losses relative to gains. Our behavioral results revealed that only low-anxious individuals exhibited increased loss aversion under emotional conditions. This emotional modulation of decision-making was accompanied by a corresponding emotion-elicited increase in amygdala-striatal functional connectivity, which correlated with the behavioral effect across participants. Consistent with prior reports of 'neural loss aversion', both amygdala and ventral striatum tracked losses more strongly than gains, and amygdala loss aversion signals were exaggerated by emotion, suggesting a potential role for this structure in integrating value and emotion cues. Increased loss aversion and striatal-amygdala coupling induced by emotional cues may reflect the engagement of adaptive harm-avoidance mechanisms in low-anxious individuals, possibly promoting resilience to psychopathology.

How beliefs about self-creation inflate value in the human brain.

Humans have a tendency to overvalue their own ideas and creations. Understanding how these errors in judgement emerge is important for explaining suboptimal decisions, as when individuals and groups choose self-created alternatives over superior or equal ones. We show that such overvaluation is a reconstructive process that emerges when participants believe they have created an item, regardless of whether this belief is true or false. This overvaluation is observed both when false beliefs of self-creation are elicited (Experiment 1) or implanted (Experiment 2). Using brain imaging data we highlight the brain processes mediating an interaction between value and belief of self-creation. Specifically, following the creation manipulation there is an increased functional connectivity during valuation between the right caudate nucleus, where we show BOLD activity correlated with subjective value, and the left amygdala, where we show BOLD activity is linked to subjective belief. Our study highlights psychological and neurobiological processes through which false beliefs alter human valuation and in doing so throw light on a common source of error in judgements of value.

In the mind of the market: theory of mind biases value computation during financial bubbles.

The ability to infer intentions of other agents, called theory of mind (ToM), confers strong advantages for individuals in social situations. Here, we show that ToM can also be maladaptive when people interact with complex modern institutions like financial markets. We tested participants who were investing in an experimental bubble market, a situation in which the price of an asset is much higher than its underlying fundamental value. We describe a mechanism by which social signals computed in the dorsomedial prefrontal cortex affect value computations in ventromedial prefrontal cortex, thereby increasing an individual's propensity to 'ride' financial bubbles and lose money. These regions compute a financial metric that signals variations in order flow intensity, prompting inference about other traders' intentions. Our results suggest that incorporating inferences about the intentions of others when making value judgments in a complex financial market could lead to the formation of market bubbles.

Confidence in value-based choice.

Decisions are never perfect, with confidence in one's choices fluctuating over time. How subjective confidence and valuation of choice options interact at the level of brain and behavior is unknown. Using a dynamic model of the decision process, we show that confidence reflects the evolution of a decision variable over time, explaining the observed relation between confidence, value, accuracy and reaction time. As predicted by our dynamic model, we show that a functional magnetic resonance imaging signal in human ventromedial prefrontal cortex (vmPFC) reflects both value comparison and confidence in the value comparison process. Crucially, individuals varied in how they related confidence to accuracy, allowing us to show that this introspective ability is predicted by a measure of functional connectivity between vmPFC and rostrolateral prefrontal cortex. Our findings provide a mechanistic link between noise in value comparison and metacognitive awareness of choice, enabling us both to want and to express knowledge of what we want.

A range-normalization model of context-dependent choice: a new model and evidence.

Most utility theories of choice assume that the introduction of an irrelevant option (called the decoy) to a choice set does not change the preference between existing options. On the contrary, a wealth of behavioral data demonstrates the dependence of preference on the decoy and on the context in which the options are presented. Nevertheless, neural mechanisms underlying context-dependent preference are poorly understood. In order to shed light on these mechanisms, we design and perform a novel experiment to measure within-subject decoy effects. We find within-subject decoy effects similar to what have been shown previously with between-subject designs. More importantly, we find that not only are the decoy effects correlated, pointing to similar underlying mechanisms, but also these effects increase with the distance of the decoy from the original options. To explain these observations, we construct a plausible neuronal model that can account for decoy effects based on the trial-by-trial adjustment of neural representations to the set of available options. This adjustment mechanism, which we call range normalization, occurs when the nervous system is required to represent different stimuli distinguishably, while being limited to using bounded neural activity. The proposed model captures our experimental observations and makes new predictions about the influence of the choice set size on the decoy effects, which are in contrast to previous models of context-dependent choice preference. Critically, unlike previous psychological models, the computational resource required by our range-normalization model does not increase exponentially as the set size increases. Our results show that context-dependent choice behavior, which is commonly perceived as an irrational response to the presence of irrelevant options, could be a natural consequence of the biophysical limits of neural representation in the brain.

Neural mechanisms underlying paradoxical performance for monetary incentives are driven by loss aversion.

Employers often make payment contingent on performance in order to motivate workers. We used fMRI with a novel incentivized skill task to examine the neural processes underlying behavioral responses to performance-based pay. We found that individuals' performance increased with increasing incentives; however, very high incentive levels led to the paradoxical consequence of worse performance. Between initial incentive presentation and task execution, striatal activity rapidly switched between activation and deactivation in response to increasing incentives. Critically, decrements in performance and striatal deactivations were directly predicted by an independent measure of behavioral loss aversion. These results suggest that incentives associated with successful task performance are initially encoded as a potential gain; however, when actually performing a task, individuals encode the potential loss that would arise from failure.

Amygdala damage eliminates monetary loss aversion.

Losses are a possibility in many risky decisions, and organisms have evolved mechanisms to evaluate and avoid them. Laboratory and field evidence suggests that people often avoid risks with losses even when they might earn a substantially larger gain, a behavioral preference termed "loss aversion." The cautionary brake on behavior known to rely on the amygdala is a plausible candidate mechanism for loss aversion, yet evidence for this idea has so far not been found. We studied two rare individuals with focal bilateral amygdala lesions using a series of experimental economics tasks. To measure individual sensitivity to financial losses we asked participants to play a variety of monetary gambles with possible gains and losses. Although both participants retained a normal ability to respond to changes in the gambles' expected value and risk, they showed a dramatic reduction in loss aversion compared to matched controls. The findings suggest that the amygdala plays a key role in generating loss aversion by inhibiting actions with potentially deleterious outcomes.

A genetically mediated bias in decision making driven by failure of amygdala control.

Genetic variation at the serotonin transporter-linked polymorphic region (5-HTTLPR) is associated with altered amygdala reactivity and lack of prefrontal regulatory control. Similar regions mediate decision-making biases driven by contextual cues and ambiguity, for example the "framing effect." We hypothesized that individuals hemozygous for the short (s) allele at the 5-HTTLPR would be more susceptible to framing. Participants, selected as homozygous for either the long (la) or s allele, performed a decision-making task where they made choices between receiving an amount of money for certain and taking a gamble. A strong bias was evident toward choosing the certain option when the option was phrased in terms of gains and toward gambling when the decision was phrased in terms of losses (the frame effect). Critically, this bias was significantly greater in the ss group compared with the lala group. In simultaneously acquired functional magnetic resonance imaging data, the ss group showed greater amygdala during choices made in accord, compared with those made counter to the frame, an effect not seen in the lala group. These differences were also mirrored by differences in anterior cingulate-amygdala coupling between the genotype groups during decision making. Specifically, lala participants showed increased coupling during choices made counter to, relative to those made in accord with, the frame, with no such effect evident in ss participants. These data suggest that genetically mediated differences in prefrontal-amygdala interactions underpin interindividual differences in economic decision making.

How choice reveals and shapes expected hedonic outcome.

Humans tend to modify their attitudes to align with past action. For example, after choosing between similarly valued alternatives, people rate the selected option as better than they originally did, and the rejected option as worse. However, it is unknown whether these modifications in evaluation reflect an underlying change in the physiological representation of a stimulus' expected hedonic value and our emotional response to it. Here, we addressed this question by combining participants' estimations of the pleasure they will derive from future events, with brain imaging data recorded while they imagined those events, both before, and after, choosing between them. Participants rated the selected alternatives as better after the decision stage relative to before, whereas discarded alternatives were valued less. Our functional magnetic resonance imaging findings reveal that postchoice changes in preference are tracked in caudate nucleus activity. Specifically, the difference in blood oxygenation level-dependent (BOLD) signal associated with the selected and rejected stimuli was enhanced after a decision was taken, reflecting the choice that had just been made. This finding suggests that the physiological representation of a stimulus' expected hedonic value is altered by a commitment to it. Furthermore, before any revaluation induced by the decision process, our data show that BOLD signal in this same region reflects the choices we are likely to make at a later time.