Common intentional binding effects across diverse sensory modalities in touch-free voluntary actions.

The intentional binding effect refers to the phenomenon where the perceived temporal interval between a voluntary action and its sensory consequence is subjectively compressed. Prior research revealed the importance of tactile feedback from the keyboard on this effect. Here we examined the necessity of such tactile feedback by utilizing a touch-free key-press device without haptic feedback, and explored how initial/outcome sensory modalities (visual/auditory/tactile) and their consistency influence the intentional binding effect. Participants estimated three delay lengths (250, 550, or 850 ms) between the initial and outcome stimuli. Results showed that regardless of the combinations of sensory modalities between the initial and the outcome stimuli (i.e., modal consistency), the intentional binding effect was only observed in the 250 ms delay condition. This findings indicate a stable intentional binding effect both within and across sensory modalities, supporting the existence of a shared mechanism underlying the binding effect in touch-free voluntary actions.

Sex Differences and Exogenous Estrogen Influence Learning and Brain Responses to Prediction Errors.

Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug addiction. In this computational pharmacological fMRI study, we investigate the effects of both factors, sex and estrogen, on reinforcement learning and the dopaminergic system in humans; 67 male and 64 naturally cycling female volunteers, the latter in their low-hormone phase, were randomly assigned, double-blind, to take E2 or placebo. They completed a reinforcement learning task in the MRI scanner for which we have previously shown reward prediction error (RPE)-related activity to be dopaminergic. We found RPE-related brain activity to be enhanced in women compared with men and to a greater extent when E2 levels were elevated in both sexes. However, both factors, female sex and E2, slowed adaptation to RPEs (smaller learning rate). This discrepancy of larger RPE-related activity yet smaller learning rates can be explained by organizational sex differences and activational effects of circulating E2, which both affect DA release differently to DA receptor binding capacities.

Stress reduces both model-based and model-free neural computations during flexible learning.

Stressful events are thought to impair the flexible adaptation to changing environments, yet the underlying mechanisms are largely unknown. Here, we combined computational modeling and functional magnetic resonance imaging (fMRI) to elucidate the neurocomputational mechanisms underlying stress-induced deficits in flexible learning. Healthy participants underwent a stress or control manipulation before they completed, in the MRI scanner, a Markov decision task, frequently used to dissociate model-based and model-free contributions to choice, with repeated reversals of reward contingencies. Our results showed that stress attenuated the behavioral sensitivity to reversals in reward contingencies. Computational modeling further indicated that stress specifically affected the use of value computations for subsequent action selection. This reduced application of learned information on subsequent behavior was paralleled by a stress-induced reduction in inferolateral prefrontal cortex activity during model-free computations. For model-based learning, stress decreased specifically posterior, but not anterior, hippocampal activity, pointing to a functional segregation of model-based processing and its modulation by stress along the hippocampal longitudinal axis. Our findings shed light on the mechanisms underlying deficits in flexible learning under stress and indicate that, in highly dynamic environments, stress may hamper both model-based and model-free contributions to adaptive behavior.

Congruence of Inherent and Acquired Values Facilitates Reward-Based Decision-Making.

UNLABELLED: Most real-life cues exhibit certain inherent values that may interfere with or facilitate the acquisition of new expected values during associative learning. In particular, when inherent and acquired values are congruent, learning may progress more rapidly. Here we investigated such an influence through a 2 × 2 factorial design, using attractiveness (high/low) of the facial picture as a proxy for the inherent value of the cue and its reward probability (high/low) as a surrogate for the acquired value. Each picture was paired with a monetary win or loss either congruently or incongruently. Behavioral results from 32 human participants indicated both faster response time and faster learning rate for value-congruent cue-outcome pairings. Model-based fMRI analysis revealed a fractionation of reinforcement learning (RL) signals in the ventral striatum, including a strong and novel correlation between the cue-specific decaying learning rate and BOLD activity in the ventral caudate. Additionally, we detected a functional link between neural signals of both learning rate and reward prediction error in the ventral striatum, and the signal of expected value in the ventromedial prefrontal cortex, showing a novel confirmation of the mathematical RL model via functional connectivity. SIGNIFICANCE STATEMENT: Most real-world decisions require the integration of inherent value and sensitivity to outcomes to facilitate adaptive learning. Inherent value is drawing increasing interest from decision scientists because it influences decisions in contexts ranging from advertising to investing. This study provides novel insight into how inherent value influences the acquisition of new expected value during associative learning. Specifically, we find that the congruence between the inherent value and the acquired reward influences the neural coding of learning rate. We also show for the first time that neuroimaging signals coding the learning rate, prediction error, and acquired value follow the multiplicative Rescorla-Wagner learning rule, a finding predicted by reinforcement learning theory.

Interaction of Instrumental and Goal-Directed Learning Modulates Prediction Error Representations in the Ventral Striatum.

Goal-directed and instrumental learning are both important controllers of human behavior. Learning about which stimulus event occurs in the environment and the reward associated with them allows humans to seek out the most valuable stimulus and move through the environment in a goal-directed manner. Stimulus-response associations are characteristic of instrumental learning, whereas response-outcome associations are the hallmark of goal-directed learning. Here we provide behavioral, computational, and neuroimaging results from a novel task in which stimulus-response and response-outcome associations are learned simultaneously but dominate behavior at different stages of the experiment. We found that prediction error representations in the ventral striatum depend on which type of learning dominates. Furthermore, the amygdala tracks the time-dependent weighting of stimulus-response versus response-outcome learning. Our findings suggest that the goal-directed and instrumental controllers dynamically engage the ventral striatum in representing prediction errors whenever one of them is dominating choice behavior. SIGNIFICANCE STATEMENT: Converging evidence in human neuroimaging studies has shown that the reward prediction errors are correlated with activity in the ventral striatum. Our results demonstrate that this region is simultaneously correlated with a stimulus prediction error. Furthermore, the learning system that is currently dominating behavioral choice dynamically engages the ventral striatum for computing its prediction errors. This demonstrates that the prediction error representations are highly dynamic and influenced by various experimental context. This finding points to a general role of the ventral striatum in detecting expectancy violations and encoding error signals regardless of the specific nature of the reinforcer itself.

Altered behavioral and neural responsiveness to counterfactual gains in the elderly.

Counterfactual information processing refers to the consideration of events that did not occur in comparison to those actually experienced, in order to determine optimal actions, and can be formulated as computational learning signals, referred to as fictive prediction errors. Decision making and the neural circuitry for counterfactual processing are altered in healthy elderly adults. This experiment investigated age differences in neural systems for decision making with knowledge of counterfactual outcomes. Two groups of healthy adult participants, young (N = 30; ages 19-30 years) and elderly (N = 19; ages 65-80 years), were scanned with fMRI during 240 trials of a strategic sequential investment task in which a particular strategy of differentially weighting counterfactual gains and losses during valuation is associated with more optimal performance. Elderly participants earned significantly less than young adults, differently weighted counterfactual consequences and exploited task knowledge, and exhibited altered activity in a fronto-striatal circuit while making choices, compared to young adults. The degree to which task knowledge was exploited was positively correlated with modulation of neural activity by expected value in the vmPFC for young adults, but not in the elderly. These findings demonstrate that elderly participants' poor task performance may be related to different counterfactual processing.

Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism.

For making decisions in everyday life we often have first to infer the set of environmental features that are relevant for the current task. Here we investigated the computational mechanisms underlying the evolution of beliefs about the relevance of environmental features in a dynamical and noisy environment. For this purpose we designed a probabilistic Wisconsin card sorting task (WCST) with belief solicitation, in which subjects were presented with stimuli composed of multiple visual features. At each moment in time a particular feature was relevant for obtaining reward, and participants had to infer which feature was relevant and report their beliefs accordingly. To test the hypothesis that attentional focus modulates the belief update process, we derived and fitted several probabilistic and non-probabilistic behavioral models, which either incorporate a dynamical model of attentional focus, in the form of a hierarchical winner-take-all neuronal network, or a diffusive model, without attention-like features. We used Bayesian model selection to identify the most likely generative model of subjects' behavior and found that attention-like features in the behavioral model are essential for explaining subjects' responses. Furthermore, we demonstrate a method for integrating both connectionist and Bayesian models of decision making within a single framework that allowed us to infer hidden belief processes of human subjects.

Lesion mapping of cognitive control and value-based decision making in the prefrontal cortex.

A considerable body of previous research on the prefrontal cortex (PFC) has helped characterize the regional specificity of various cognitive functions, such as cognitive control and decision making. Here we provide definitive findings on this topic, using a neuropsychological approach that takes advantage of a unique dataset accrued over several decades. We applied voxel-based lesion-symptom mapping in 344 individuals with focal lesions (165 involving the PFC) who had been tested on a comprehensive battery of neuropsychological tasks. Two distinct functional-anatomical networks were revealed within the PFC: one associated with cognitive control (response inhibition, conflict monitoring, and switching), which included the dorsolateral prefrontal cortex and anterior cingulate cortex and a second associated with value-based decision-making, which included the orbitofrontal, ventromedial, and frontopolar cortex. Furthermore, cognitive control tasks shared a common performance factor related to set shifting that was linked to the rostral anterior cingulate cortex. By contrast, regions in the ventral PFC were required for decision-making. These findings provide detailed causal evidence for a remarkable functional-anatomical specificity in the human PFC.

Neuroanatomical correlates of executive functions: a neuropsychological approach using the EXAMINER battery.

Executive functions (EF) encompass a variety of higher-order capacities such as judgment, planning, decision-making, response monitoring, insight, and self-regulation. Measuring such abilities quantitatively and establishing their neural correlates has proven to be challenging. Here, using a lesion-deficit approach, we report the neural correlates of a variety of EF tests that were developed under the auspices of the NINDS-supported EXAMINER project (Kramer, 2011; www.examiner.ucsf.edu). We administered a diverse set of EF tasks that tap three general domains--cognitive, social/emotional, and insight--to 37 patients with focal lesions to the frontal lobes, and 25 patients with lesions outside the frontal lobes. Using voxel-based lesion-symptom mapping (VLSM), we found that damage to the ventromedial prefrontal cortex (vmPFC) was predominately associated with deficits in social/emotional aspects of EF, while damage to dorsolateral prefrontal cortex (dlPFC) and anterior cingulate was predominately associated with deficits in cognitive aspects of EF. Evidence for an important role of some non-frontal regions (e.g., the temporal poles) in some aspects of EF was also found. The results provide further evidence for the neural basis of EF, and extend previous findings of the dissociation between the roles of the ventromedial and dorsolateral prefrontal sectors in organizing, implementing, and monitoring goal-directed behavior.

Protocol for predicting multivariate change of brain patterns using model-informed fMRI activations.

Investigating the spatially distributed information contained in fMRI data is essential for understanding brain functions. Here, we present a protocol to dynamically predict short-term changes in neural patterns using trial-by-trial blood-oxygen-level-dependent (BOLD) activity of a seed region. We describe steps for setting fMRI data acquisition parameters and quantification of changes in multivariate patterns. We then detail procedures for defining seed regions and identifying brain areas in which changes in multivariate patterns can be predicted by BOLD activity of the seed region. For complete details on the use and execution of this protocol, please refer to Möhring et al.1.

The neural network of sensory attenuation: A neuroimaging meta-analysis.

Sensory attenuation refers to the reduction in sensory intensity resulting from self-initiated actions compared to stimuli initiated externally. A classic example is scratching oneself without feeling itchy. This phenomenon extends across various sensory modalities, including visual, auditory, somatosensory, and nociceptive stimuli. The internal forward model proposes that during voluntary actions, an efferent copy of the action command is sent out to predict sensory feedback. This predicted sensory feedback is then compared with the actual sensory feedback, leading to the suppression or reduction of sensory stimuli originating from self-initiated actions. To further elucidate the neural mechanisms underlying sensory attenuation effect, we conducted an extensive meta-analysis of functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) studies. Utilizing activation likelihood estimation (ALE) analysis, our results revealed significant activations in a prominent cluster encompassing the right superior temporal gyrus (rSTG), right middle temporal gyrus (rMTG), and right insula when comparing external-generated with self-generated conditions. Additionally, significant activation was observed in the right anterior cerebellum when comparing self-generated to external-generated conditions. Further analysis using meta-analytic connectivity modeling (MACM) unveiled distinct brain networks co-activated with the rMTG and right cerebellum, respectively. Based on these findings, we propose that sensory attenuation arises from the suppression of reflexive inputs elicited by self-initiated actions through the internal forward modeling of a cerebellum-centered action prediction network, enabling the "sensory conflict detection" regions to effectively discriminate between inputs resulting from self-induced actions and those originating externally.

Prediction errors drive dynamic changes in neural patterns that guide behavior.

Learning describes the process by which our internal expectation models of the world are updated by surprising outcomes (prediction errors [PEs]) to improve predictions of future events. However, the mechanisms through which error signals dynamically influence existing neural representations are unknown. Here, we use functional magnetic resonance imaging (fMRI) in humans solving a two-step Markov decision task to investigate changes in neural activation patterns following PEs. Using a dynamic multivariate pattern analysis, we can show that PE-related fMRI responses in error-coding regions predict trial-by-trial changes in multivariate neural patterns in the orbitofrontal cortex, the precuneus, and the ventromedial prefrontal cortex (vmPFC). Importantly, the dynamics of these pattern changes in the vmPFC also predicted upcoming changes in choice strategies and thus highlight the importance of these pattern changes for behavior.

Humans depart from optimal computational models of interactive decision-making during competition under partial information.

Decision making under uncertainty in multiagent settings is of increasing interest in decision science. The degree to which human agents depart from computationally optimal solutions in socially interactive settings is generally unknown. Such understanding provides insight into how social contexts affect human interaction and the underlying contributions of Theory of Mind. In this paper, we adapt the well-known 'Tiger Problem' from artificial-agent research to human participants in solo and interactive settings. Compared to computationally optimal solutions, participants gathered less information before outcome-related decisions when competing than cooperating with others. These departures from optimality were not haphazard but showed evidence of improved performance through learning. Costly errors emerged under conditions of competition, yielding both lower rates of rewarding actions and accuracy in predicting others. Taken together, this work provides a novel approach and insights into studying human social interaction when shared information is partial.

fMRI reveals how pain modulates visual object processing in the ventral visual stream.

It is well known that pain attracts attention and interferes with cognition. Given that the mechanisms behind this phenomenon are largely unknown, we used functional magnetic resonance imaging and presented visual objects with or without concomitant pain stimuli. To test for the specificity of pain, we compared this modulatory effect with a previously established modulatory effect of working memory on visual object processing. Our data showed a comparable behavioral effect of both types of modulation and identified the lateral occipital complex (LOC) as the site of modulation in the ventral visual stream, for both pain and working memory. However, the sources of these modulatory effects differed for the two processes. Whereas the source of modulation for working memory could be attributed to the parietal cortex, the modulatory effect of pain was observed in the rostral anterior cingulate cortex (rACC), an area ideally suited to link pain perception and attentional control.

Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making.

Considerable evidence has emerged to implicate ventromedial prefrontal cortex in encoding expectations of future reward during value-based decision making. However, the nature of the learned associations upon which such representations depend is much less clear. Here, we aimed to determine whether expected reward representations in this region could be driven by action-outcome associations, rather than being dependent on the associative value assigned to particular discriminative stimuli. Subjects were scanned with functional magnetic resonance imaging while performing 2 variants of a simple reward-related decision task. In one version, subjects made choices between 2 different physical motor responses in the absence of discriminative stimuli, whereas in the other version, subjects chose between 2 different stimuli that were randomly assigned to different responses on a trial-by-trial basis. Using an extension of a reinforcement learning algorithm, we found activity in ventromedial prefrontal cortex tracked expected future reward during the action-based task as well as during the stimulus-based task, indicating that value representations in this region can be driven by action-outcome associations. These findings suggest that ventromedial prefrontal cortex may play a role in encoding the value of chosen actions irrespective of whether those actions denote physical motor responses or more abstract decision options.

A brain network supporting social influences in human decision-making.

Humans learn from their own trial-and-error experience and observing others. However, it remains unknown how brain circuits compute expected values when direct learning and social learning coexist in uncertain environments. Using a multiplayer reward learning paradigm with 185 participants (39 being scanned) in real time, we observed that individuals succumbed to the group when confronted with dissenting information but observing confirming information increased their confidence. Leveraging computational modeling and functional magnetic resonance imaging, we tracked direct valuation through experience and vicarious valuation through observation and their dissociable, but interacting neural representations in the ventromedial prefrontal cortex and the anterior cingulate cortex, respectively. Their functional coupling with the right temporoparietal junction representing instantaneous social information instantiated a hitherto uncharacterized social prediction error, rather than a reward prediction error, in the putamen. These findings suggest that an integrated network involving the brain's reward hub and social hub supports social influence in human decision-making.

Dose-dependent effects of estrogen on prediction error related neural activity in the nucleus accumbens of healthy young women.

RATIONALE: Whereas the effect of the sex steroid 17-beta-estradiol (E2) on dopaminergic (DA) transmission in the nucleus accumbens (NAc) is well evidenced in female rats, studies in humans are inconsistent. Moreover, linear and inverted u-shaped dose response curves have been observed for E2's effects on hippocampal plasticity, but the shape of dose response curves for E2's effects on the NAc is much less characterized. OBJECTIVES: Investigation of dose response curves for E2's effects on DA-related neural activity in the human NAc. METHODS: Placebo or E2 valerate in doses of 2, 4, 6 or 12 mg was orally administered to 125 naturally cycling young women during the low-hormone menstruation phase on two consecutive days using a randomized, double-blinded design. The E2 treatment regimen induced a wide range of E2 levels, from physiological (2- and 4-mg groups; equivalent to cycle peak) to supraphysiological levels (6- and 12-mg groups; equivalent to early pregnancy). This made it possible to study different dose response functions for E2's effects on NAc activity. During E2 peak, participants performed a well-established reversal learning paradigm. We used trial-wise prediction errors (PE) estimated via a computational reinforcement learning model as a proxy for dopaminergic activity. Linear and quadratic regression analyses predicting PE-related NAc activity from salivary E2 levels were calculated. RESULTS: There was a positive linear relationship between PE-associated NAc activity and salivary E2 increases. CONCLUSIONS: The randomized, placebo-controlled elevation of E2 levels stimulates NAc activity in the human brain, likely mediated by dopaminergic processes.

Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices.

The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Theory of mind and decision science: Towards a typology of tasks and computational models.

The ability to form a Theory of Mind (ToM), i.e., to theorize about others' mental states to explain and predict behavior in relation to attributed intentional states, constitutes a hallmark of human cognition. These abilities are multi-faceted and include a variety of different cognitive sub-functions. Here, we focus on decision processes in social contexts and review a number of experimental and computational modeling approaches in this field. We provide an overview of experimental accounts and formal computational models with respect to two dimensions: interactivity and uncertainty. Thereby, we aim at capturing the nuances of ToM functions in the context of social decision processes. We suggest there to be an increase in ToM engagement and multiplexing as social cognitive decision-making tasks become more interactive and uncertain. We propose that representing others as intentional and goal directed agents who perform consequential actions is elicited only at the edges of these two dimensions. Further, we argue that computational models of valuation and beliefs follow these dimensions to best allow researchers to effectively model sophisticated ToM-processes. Finally, we relate this typology to neuroimaging findings in neurotypical (NT) humans, studies of persons with autism spectrum (AS), and studies of nonhuman primates.