Learning rules of engagement for social exchange within and between groups.

Globalizing economies and long-distance trade rely on individuals from different cultural groups to negotiate agreement on what to give and take. In such settings, individuals often lack insight into what interaction partners deem fair and appropriate, potentially seeding misunderstandings, frustration, and conflict. Here, we examine how individuals decipher distinct rules of engagement and adapt their behavior to reach agreements with partners from other cultural groups. Modeling individuals as Bayesian learners with inequality aversion reveals that individuals, in repeated ultimatum bargaining with responders sampled from different groups, can be more generous than needed. While this allows them to reach agreements, it also gives rise to biased beliefs about what is required to reach agreement with members from distinct groups. Preregistered behavioral (N = 420) and neuroimaging experiments (N = 49) support model predictions: Seeking equitable agreements can lead to overly generous behavior toward partners from different groups alongside incorrect beliefs about prevailing norms of what is appropriate in groups and cultures other than one's own.

A shared brain system forming confidence judgment across cognitive domains.

Confidence is typically defined as a subjective judgment about whether a decision is right. Decisions are based on sources of information that come from various cognitive domains and are processed in different brain systems. An unsettled question is whether the brain computes confidence in a similar manner whatever the domain or in a manner that would be idiosyncratic to each domain. To address this issue, human participants performed two tasks probing confidence in decisions made about the same material (history and geography statements), but based on different cognitive processes: semantic memory for deciding whether the statement was true or false, and duration perception for deciding whether the statement display was long or short. At the behavioral level, we found that the same factors (difficulty, accuracy, response time, and confidence in the preceding decision) predicted confidence judgments in both tasks. At the neural level, we observed using functional magnetic resonance imaging that confidence judgments in both tasks were associated to activity in the same brain regions: positively in the ventromedial prefrontal cortex and negatively in a prefronto-parietal network. Together, these findings suggest the existence of a shared brain system that generates confidence judgments in a similar manner across cognitive domains.

Decision-making under risk and ambiguity in adults with Tourette syndrome.

BACKGROUND: Tourette syndrome (TS) as well as its most common comorbidities are associated with a higher propensity for risky behaviour in everyday life. However, it is unclear whether this increased risk propensity in real-life contexts translates into a generally increased attitude towards risk. We aimed to assess decision-making under risk and ambiguity based on prospect theory by considering the effects of comorbidities and medication. METHODS: Fifty-four individuals with TS and 32 healthy controls performed risk and ambiguity decision-making tasks under both gains and losses conditions. Behavioural and computational parameters were evaluated using (i) univariate analysis to determine parameters difference taking independently; (ii) supervised multivariate analysis to evaluate whether our parameters could jointly account for between-group differences (iii) unsupervised multivariate analysis to explore the potential presence of sub-groups. RESULTS: Except for general 'noisier' (less consistent) decisions in TS, we showed no specific risk-taking behaviour in TS or any relation with tics severity or antipsychotic medication. However, the presence of comorbidities was associated with distortion of decision-making. Specifically, TS with obsessive-compulsive disorder comorbidity was associated with a higher risk-taking profile to increase gain and a higher risk-averse profile to decrease loss. TS with attention-deficit hyperactivity disorder comorbidity was associated with risk-seeking in the ambiguity context to reduce a potential loss. CONCLUSIONS: Impaired valuation of risk and ambiguity was not related to TS per se. Our findings are important for clinical practice: the involvement of individuals with TS in real-life risky situations may actually rather result from other factors such as psychiatric comorbidities.

Specificity and sensitivity of the fixed-point test for binary mixture distributions.

When two cognitive processes contribute to a behavioral output-each process producing a specific distribution of the behavioral variable of interest-and when the mixture proportion of these two processes varies as a function of an experimental condition, a common density point should be present in the observed distributions of the data across said conditions. In principle, one can statistically test for the presence (or absence) of a fixed point in experimental data to provide evidence in favor of (or against) the presence of a mixture of processes, whose proportions are affected by an experimental manipulation. In this paper, we provide an empirical diagnostic of this test to detect a mixture of processes. We do so using resampling of real experimental data under different scenarios, which mimic variations in the experimental design suspected to affect the sensitivity and specificity of the fixed-point test (i.e., mixture proportion, time on task, and sample size). Resampling such scenarios with real data allows us to preserve important features of data which are typically observed in real experiments while maintaining tight control over the properties of the resampled scenarios. This is of particular relevance considering such stringent assumptions underlying the fixed-point test. With this paper, we ultimately aim at validating the fixed-point property of binary mixture data and at providing some performance metrics to researchers aiming at testing the fixed-point property on their experimental data.

Metacognition and the effect of incentive motivation in two compulsive disorders: Gambling disorder and obsessive-compulsive disorder.

AIMS: Compulsivity is a common phenotype among psychiatric disorders, such as obsessive-compulsive disorder (OCD) and gambling disorder (GD). Deficiencies in metacognition, such as the inability to estimate one's performance via confidence judgments could contribute to pathological decision-making. Earlier research has shown that patients with OCD exhibit underconfidence, while patients with GD exhibit overconfidence. Moreover, it is known that motivational states (e.g. monetary incentives) influence metacognition, with gain (respectively loss) prospects increasing (respectively decreasing) confidence. Here, we reasoned that OCD and GD symptoms might correspond to an exacerbation of this interaction between metacognition and motivation. METHODS: We hypothesized GD's overconfidence to be exaggerated during gain prospects, while OCD's underconfidence to be worsened in loss context, which we expected to see represented in ventromedial prefrontal cortex (VMPFC) blood-oxygen-level-dependent activity. We tested those hypotheses in a task-based functional magnetic resonance imaging (fMRI) design (27 patients with GD, 28 patients with OCD, 55 controls). The trial is registered in the Dutch Trial Register (NL6171). RESULTS: We showed increased confidence for patients with GD versus patients with OCD, which could partly be explained by sex and IQ. Although our primary analyses did not support the hypothesized interaction between incentives and groups, exploratory analyses did show increased confidence in patients with GD specifically in gain context. fMRI analyses confirmed a central role for VMPFC in the processing of confidence and incentives, but no differences between the groups. CONCLUSION: Patients with OCD and those with GD reside at opposite ends of the confidence spectrum, while no interaction with incentives was found, nor group differences in neuronal processing of confidence.

Neurocognitive Underpinnings of Aggressive Predation in Economic Contests.

Competitions are part and parcel of daily life and require people to invest time and energy to gain advantage over others and to avoid (the risk of) falling behind. Whereas the behavioral mechanisms underlying competition are well documented, its neurocognitive underpinnings remain poorly understood. We addressed this using neuroimaging and computational modeling of individual investment decisions aimed at exploiting one's counterpart ("attack") or at protecting against exploitation by one's counterpart ("defense"). Analyses revealed that during attack relative to defense (i) individuals invest less and are less successful; (ii) computations of expected reward are strategically more sophisticated (reasoning level k = 4 vs. k = 3 during defense); (iii) ventral striatum activity tracks reward prediction errors; (iv) risk prediction errors were not correlated with neural activity in either ROI or whole-brain analyses; and (v) successful exploitation correlated with neural activity in the bilateral ventral striatum, left OFC, left anterior insula, left TPJ, and lateral occipital cortex. We conclude that, in economic contests, coming out ahead (vs. not falling behind) involves sophisticated strategic reasoning that engages both reward and value computation areas and areas associated with theory of mind.

Robust valence-induced biases on motor response and confidence in human reinforcement learning.

In simple instrumental-learning tasks, humans learn to seek gains and to avoid losses equally well. Yet, two effects of valence are observed. First, decisions in loss-contexts are slower. Second, loss contexts decrease individuals' confidence in their choices. Whether these two effects are two manifestations of a single mechanism or whether they can be partially dissociated is unknown. Across six experiments, we attempted to disrupt the valence-induced motor bias effects by manipulating the mapping between decisions and actions and imposing constraints on response times (RTs). Our goal was to assess the presence of the valence-induced confidence bias in the absence of the RT bias. We observed both motor and confidence biases despite our disruption attempts, establishing that the effects of valence on motor and metacognitive responses are very robust and replicable. Nonetheless, within- and between-individual inferences reveal that the confidence bias resists the disruption of the RT bias. Therefore, although concomitant in most cases, valence-induced motor and confidence biases seem to be partly dissociable. These results highlight new important mechanistic constraints that should be incorporated in learning models to jointly explain choice, reaction times and confidence.

Contextual influence on confidence judgments in human reinforcement learning.

The ability to correctly estimate the probability of one's choices being correct is fundamental to optimally re-evaluate previous choices or to arbitrate between different decision strategies. Experimental evidence nonetheless suggests that this metacognitive process-confidence judgment- is susceptible to numerous biases. Here, we investigate the effect of outcome valence (gains or losses) on confidence while participants learned stimulus-outcome associations by trial-and-error. In two experiments, participants were more confident in their choices when learning to seek gains compared to avoiding losses, despite equal difficulty and performance between those two contexts. Computational modelling revealed that this bias is driven by the context-value, a dynamically updated estimate of the average expected-value of choice options, necessary to explain equal performance in the gain and loss domain. The biasing effect of context-value on confidence, revealed here for the first time in a reinforcement-learning context, is therefore domain-general, with likely important functional consequences. We show that one such consequence emerges in volatile environments, where the (in)flexibility of individuals' learning strategies differs when outcomes are framed as gains or losses. Despite apparent similar behavior- profound asymmetries might therefore exist between learning to avoid losses and learning to seek gains.

How the Level of Reward Awareness Changes the Computational and Electrophysiological Signatures of Reinforcement Learning.

The extent to which subjective awareness influences reward processing, and thereby affects future decisions, is currently largely unknown. In the present report, we investigated this question in a reinforcement learning framework, combining perceptual masking, computational modeling, and electroencephalographic recordings (human male and female participants). Our results indicate that degrading the visibility of the reward decreased, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increased their tendency to repeat previous choices. We dissociated electrophysiological signatures evoked by the reward-based learning processes from those elicited by the reward-independent repetition of previous choices and showed that these neural activities were significantly modulated by reward visibility. Overall, this report sheds new light on the neural computations underlying reward-based learning and decision-making and highlights that awareness is beneficial for the trial-by-trial adjustment of decision-making strategies.SIGNIFICANCE STATEMENT The notion of reward is strongly associated with subjective evaluation, related to conscious processes such as "pleasure," "liking," and "wanting." Here we show that degrading reward visibility in a reinforcement learning task decreases, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increases subjects' tendency to repeat previous choices. Electrophysiological recordings, in combination with computational modeling, show that neural activities were significantly modulated by reward visibility. Overall, we dissociate different neural computations underlying reward-based learning and decision-making, which highlights a beneficial role of reward awareness in adjusting decision-making strategies.

Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: a meta-analytical approach using diffusion decision modeling.

Reinforcement learning (RL) models describe how humans and animals learn by trial-and-error to select actions that maximize rewards and minimize punishments. Traditional RL models focus exclusively on choices, thereby ignoring the interactions between choice preference and response time (RT), or how these interactions are influenced by contextual factors. However, in the field of perceptual decision-making, such interactions have proven to be important to dissociate between different underlying cognitive processes. Here, we investigated such interactions to shed new light on overlooked differences between learning to seek rewards and learning to avoid losses. We leveraged behavioral data from four RL experiments, which feature manipulations of two factors: outcome valence (gains vs. losses) and feedback information (partial vs. complete feedback). A Bayesian meta-analysis revealed that these contextual factors differently affect RTs and accuracy: While valence only affects RTs, feedback information affects both RTs and accuracy. To dissociate between the latent cognitive processes, we jointly fitted choices and RTs across all experiments with a Bayesian, hierarchical diffusion decision model (DDM). We found that the feedback manipulation affected drift rate, threshold, and non-decision time, suggesting that it was not a mere difficulty effect. Moreover, valence affected non-decision time and threshold, suggesting a motor inhibition in punishing contexts. To better understand the learning dynamics, we finally fitted a combination of RL and DDM (RLDDM). We found that while the threshold was modulated by trial-specific decision conflict, the non-decision time was modulated by the learned context valence. Overall, our results illustrate the benefits of jointly modeling RTs and choice data during RL, to reveal subtle mechanistic differences underlying decisions in different learning contexts.

Neural mechanisms underlying contextual dependency of subjective values: converging evidence from monkeys and humans.

A major challenge for decision theory is to account for the instability of expressed preferences across time and context. Such variability could arise from specific properties of the brain system used to assign subjective values. Growing evidence has identified the ventromedial prefrontal cortex (VMPFC) as a key node of the human brain valuation system. Here, we first replicate this observation with an fMRI study in humans showing that subjective values of painting pictures, as expressed in explicit pleasantness ratings, are specifically encoded in the VMPFC. We then establish a bridge with monkey electrophysiology, by comparing single-unit activity evoked by visual cues between the VMPFC and the orbitofrontal cortex. At the neural population level, expected reward magnitude was only encoded in the VMPFC, which also reflected subjective cue values, as expressed in Pavlovian appetitive responses. In addition, we demonstrate in both species that the additive effect of prestimulus activity on evoked activity has a significant impact on subjective values. In monkeys, the factor dominating prestimulus VMPFC activity was trial number, which likely indexed variations in internal dispositions related to fatigue or satiety. In humans, prestimulus VMPFC activity was externally manipulated through changes in the musical context, which induced a systematic bias in subjective values. Thus, the apparent stochasticity of preferences might relate to the VMPFC automatically aggregating the values of contextual features, which would bias subsequent valuation because of temporal autocorrelation in neural activity.

A critical role for the hippocampus in the valuation of imagined outcomes.

Many choice situations require imagining potential outcomes, a capacity that was shown to involve memory brain regions such as the hippocampus. We reasoned that the quality of hippocampus-mediated simulation might therefore condition the subjective value assigned to imagined outcomes. We developed a novel paradigm to assess the impact of hippocampus structure and function on the propensity to favor imagined outcomes in the context of intertemporal choices. The ecological condition opposed immediate options presented as pictures (hence directly observable) to delayed options presented as texts (hence requiring mental stimulation). To avoid confounding simulation process with delay discounting, we compared this ecological condition to control conditions using the same temporal labels while keeping constant the presentation mode. Behavioral data showed that participants who imagined future options with greater details rated them as more likeable. Functional MRI data confirmed that hippocampus activity could account for subjects assigning higher values to simulated options. Structural MRI data suggested that grey matter density was a significant predictor of hippocampus activation, and therefore of the propensity to favor simulated options. Conversely, patients with hippocampus atrophy due to Alzheimer's disease, but not patients with Fronto-Temporal Dementia, were less inclined to favor options that required mental simulation. We conclude that hippocampus-mediated simulation plays a critical role in providing the motivation to pursue goals that are not present to our senses.

Neural mechanisms underlying motivation of mental versus physical effort.

Mental and physical efforts, such as paying attention and lifting weights, have been shown to involve different brain systems. These cognitive and motor systems, respectively, include cortical networks (prefronto-parietal and precentral regions) as well as subregions of the dorsal basal ganglia (caudate and putamen). Both systems appeared sensitive to incentive motivation: their activity increases when we work for higher rewards. Another brain system, including the ventral prefrontal cortex and the ventral basal ganglia, has been implicated in encoding expected rewards. How this motivational system drives the cognitive and motor systems remains poorly understood. More specifically, it is unclear whether cognitive and motor systems can be driven by a common motivational center or if they are driven by distinct, dedicated motivational modules. To address this issue, we used functional MRI to scan healthy participants while performing a task in which incentive motivation, cognitive, and motor demands were varied independently. We reasoned that a common motivational node should (1) represent the reward expected from effort exertion, (2) correlate with the performance attained, and (3) switch effective connectivity between cognitive and motor regions depending on task demand. The ventral striatum fulfilled all three criteria and therefore qualified as a common motivational node capable of driving both cognitive and motor regions of the dorsal striatum. Thus, we suggest that the interaction between a common motivational system and the different task-specific systems underpinning behavioral performance might occur within the basal ganglia.

Your goal is mine: unraveling mimetic desires in the human brain.

The spread of desires among individuals is widely believed to shape motivational drives in human populations. However, objective evidence for this phenomenon and insights into the underlying brain mechanisms are still lacking. Here we show that participants rated objects as more desirable once perceived as the goals of another agent's action. We then unravel the mechanisms underpinning such goal contagion, using functional neuroimaging. As expected, observing goal-directed actions activated a parietofrontal network known as the mirror neuron system (MNS), whereas subjective desirability ratings were represented in a ventral striatoprefrontal network known as the brain valuation system (BVS). Crucially, the induction of mimetic desires through action observation involved the modulation of BVS activity through MNS activity. Furthermore, MNS-BVS effective connectivity predicted individual susceptibility toward mimetic desires. We therefore suggest that MNS-BVS interaction represents a fundamental mechanism explaining how nonverbal behavior propagates desires without the need for explicit, intentional communication.

Experiential values are underweighted in decisions involving symbolic options.

Standard models of decision-making assume each option is associated with subjective value, regardless of whether this value is inferred from experience (experiential) or explicitly instructed probabilistic outcomes (symbolic). In this study, we present results that challenge the assumption of unified representation of experiential and symbolic value. Across nine experiments, we presented participants with hybrid decisions between experiential and symbolic options. Participants' choices exhibited a pattern consistent with a systematic neglect of the experiential values. This normatively irrational decision strategy held after accounting for alternative explanations, and persisted even when it bore an economic cost. Overall, our results demonstrate that experiential and symbolic values are not symmetrically considered in hybrid decisions, suggesting they recruit different representational systems that may be assigned different priority levels in the decision process. These findings challenge the dominant models commonly used in value-based decision-making research.

Linking confidence biases to reinforcement-learning processes.

We systematically misjudge our own performance in simple economic tasks. First, we generally overestimate our ability to make correct choices-a bias called overconfidence. Second, we are more confident in our choices when we seek gains than when we try to avoid losses-a bias we refer to as the valence-induced confidence bias. Strikingly, these two biases are also present in reinforcement-learning (RL) contexts, despite the fact that outcomes are provided trial-by-trial and could, in principle, be used to recalibrate confidence judgments online. How confidence biases emerge and are maintained in reinforcement-learning contexts is thus puzzling and still unaccounted for. To explain this paradox, we propose that confidence biases stem from learning biases, and test this hypothesis using data from multiple experiments, where we concomitantly assessed instrumental choices and confidence judgments, during learning and transfer phases. Our results first show that participants' choices in both tasks are best accounted for by a reinforcement-learning model featuring context-dependent learning and confirmatory updating. We then demonstrate that the complex, biased pattern of confidence judgments elicited during both tasks can be explained by an overweighting of the learned value of the chosen option in the computation of confidence judgments. We finally show that, consequently, the individual learning model parameters responsible for the learning biases-confirmatory updating and outcome context-dependency-are predictive of the individual metacognitive biases. We conclude suggesting that the metacognitive biases originate from fundamentally biased learning computations. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Neural and computational underpinnings of biased confidence in human reinforcement learning.

While navigating a fundamentally uncertain world, humans and animals constantly evaluate the probability of their decisions, actions or statements being correct. When explicitly elicited, these confidence estimates typically correlates positively with neural activity in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

Dopamine-dependent reinforcement of motor skill learning: evidence from Gilles de la Tourette syndrome.

Reinforcement learning theory has been extensively used to understand the neural underpinnings of instrumental behaviour. A central assumption surrounds dopamine signalling reward prediction errors, so as to update action values and ensure better choices in the future. However, educators may share the intuitive idea that reinforcements not only affect choices but also motor skills such as typing. Here, we employed a novel paradigm to demonstrate that monetary rewards can improve motor skill learning in humans. Indeed, healthy participants progressively got faster in executing sequences of key presses that were repeatedly rewarded with 10 euro compared with 1 cent. Control tests revealed that the effect of reinforcement on motor skill learning was independent of subjects being aware of sequence-reward associations. To account for this implicit effect, we developed an actor-critic model, in which reward prediction errors are used by the critic to update state values and by the actor to facilitate action execution. To assess the role of dopamine in such computations, we applied the same paradigm in patients with Gilles de la Tourette syndrome, who were either unmedicated or treated with neuroleptics. We also included patients with focal dystonia, as an example of hyperkinetic motor disorder unrelated to dopamine. Model fit showed the following dissociation: while motor skills were affected in all patient groups, reinforcement learning was selectively enhanced in unmedicated patients with Gilles de la Tourette syndrome and impaired by neuroleptics. These results support the hypothesis that overactive dopamine transmission leads to excessive reinforcement of motor sequences, which might explain the formation of tics in Gilles de la Tourette syndrome.

Pharmacological modulation of subliminal learning in Parkinson's and Tourette's syndromes.

Theories of instrumental learning aim to elucidate the mechanisms that integrate success and failure to improve future decisions. One computational solution consists of updating the value of choices in proportion to reward prediction errors, which are potentially encoded in dopamine signals. Accordingly, drugs that modulate dopamine transmission were shown to impact instrumental learning performance. However, whether these drugs act on conscious or subconscious learning processes remains unclear. To address this issue, we examined the effects of dopamine-related medications in a subliminal instrumental learning paradigm. To assess generality of dopamine implication, we tested both dopamine enhancers in Parkinson's disease (PD) and dopamine blockers in Tourette's syndrome (TS). During the task, patients had to learn from monetary outcomes the expected value of a risky choice. The different outcomes (rewards and punishments) were announced by visual cues, which were masked such that patients could not consciously perceive them. Boosting dopamine transmission in PD patients improved reward learning but worsened punishment avoidance. Conversely, blocking dopamine transmission in TS patients favored punishment avoidance but impaired reward seeking. These results thus extend previous findings in PD to subliminal situations and to another pathological condition, TS. More generally, they suggest that pharmacological manipulation of dopamine transmission can subconsciously drive us to either get more rewards or avoid more punishments.

The computational roots of positivity and confirmation biases in reinforcement learning.

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.