Inhibitory mechanisms in the prefrontal-cortex differentially mediate Putamen activity during valence-based learning.

Learning from appetitive and aversive stimuli is important for survival. It involves interactions between the prefrontal cortex and subcortical structures, with inhibition playing a crucial role. However, direct evidence for this in humans is limited. Here, we overcome the difficulty of measuring inhibition in the human brain and find that GABA, the main inhibitory neurotransmitter, affects how the dACC interacts with subcortical structures during appetitive and aversive learning differently. We used 7T magnetic resonance spectroscopy (MRS) to track GABA levels in the dACC alongside whole-brain fMRI scans while participants engaged in appetitive and aversive learning tasks. During appetitive learning, dACC GABA levels were negatively correlated with learning performance and BOLD activity measured from the dACC and the Putamen. While under aversive learning, dACC GABA concentration negatively correlated with the functional connectivity between the dACC and the Putamen. Our results show that inhibition in the dACC mediates appetitive and aversive learning in humans through distinct mechanisms.

The neurobehavioral correlates of exploration without learning: Trading off value for explicit, prospective, and variable information gains.

Exploration is typically motivated by gaining information, with previous research showing that potential information gains drive a "directed" type of exploration. Yet, this research usually studies exploration in the context of learning paradigms and does not directly manipulate multiple levels of information gain. Here, we present a task that isolates learning from decision-making and controls the magnitude of prospective information gains. As predicted, participants explore more with larger future information gains. Both value gains and information gains, at a trial-by-trial level, engage the ventromedial prefrontal cortex (vmPFC), the ventral striatum (VStr), the amygdala, the dorsal anterior cingulate cortex (dACC), and the anterior insula (aINS). Moreover, individual sensitivities to value gains and information gains modulate the vmPFC, dACC, and aINS, but the amygdala and VStr are modulated only by individual sensitivities to information gains. Overall, we identify the neural circuitry of information-based exploration and its relationship with inter-individual exploration biases.

Irrelevant Threats Linger and Affect Behavior in High Anxiety.

Threat-related information attracts attention and disrupts ongoing behavior, and particularly so for more anxious individuals. Yet, it is unknown how and to what extent threat-related information leave lingering influences on behavior (e.g., by impeding ongoing learning processes). Here, human male and female participants (N = 47) performed probabilistic reinforcement learning tasks where irrelevant distracting faces (neutral, happy, or fearful) were presented together with relevant monetary feedback. Behavioral modeling was combined with fMRI data (N = 27) to explore the neurocomputational bases of learning relevant and irrelevant information. In two separate studies, individuals with high trait anxiety showed increased avoidance of objects previously paired with the combination of neutral monetary feedback and fearful faces (but not neutral or happy faces). Behavioral modeling revealed that high anxiety increased the integration of fearful faces during feedback learning, and fMRI results (regarded as provisional, because of a relatively small sample size) further showed that variance in the prediction error signal, uniquely accounted for by fearful faces, correlated more strongly with activity in the right DLPFC for more anxious individuals. Behavioral and neuronal dissociations indicated that the threat-related distractors did not simply disrupt learning processes. By showing that irrelevant threats exert long-lasting influences on behavior, our results extend previous research that separately showed that anxiety increases learning from aversive feedbacks and distractibility by threat-related information. Our behavioral results, combined with the proposed neurocomputational mechanism, may help explain how increased exposure to irrelevant affective information contributes to the acquisition of maladaptive behaviors in more anxious individuals.SIGNIFICANCE STATEMENT In modern-day society, people are increasingly exposed to various types of irrelevant information (e.g., intruding social media announcements). Yet, the neurocomputational mechanisms influenced by irrelevant information during learning, and their interactions with increasingly distracted personality types are largely unknown. Using a reinforcement learning task, where relevant feedback is presented together with irrelevant distractors (emotional faces), we reveal an interaction between irrelevant threat-related information (fearful faces) and interindividual anxiety levels. fMRI shows provisional evidence for an interaction between anxiety levels and the coupling between activity in the DLPFC and learning signals specifically elicited by fearful faces. Our study reveals how irrelevant threat-related information may become entrenched in the anxious psyche and contribute to long-lasting abnormal behaviors.

Average reward rates enable motivational transfer across independent reinforcement learning tasks.

Outcomes and feedbacks on performance may influence behavior beyond the context in which it was received, yet it remains unclear what neurobehavioral mechanisms may account for such lingering influences on behavior. The average reward rate (ARR) has been suggested to regulate motivated behavior, and was found to interact with dopamine-sensitive cognitive processes, such as vigilance and associative memory encoding. The ARR could therefore provide a bridge between independent tasks when these are performed in temporal proximity, such that the reward rate obtained in one task could influence performance in a second subsequent task. Reinforcement learning depends on the coding of prediction error signals by dopamine neurons and their downstream targets, in particular the nucleus accumbens. Because these brain regions also respond to changes in ARR, reinforcement learning may be vulnerable to changes in ARR. To test this hypothesis, we designed a novel paradigm in which participants (n = 245) performed two probabilistic reinforcement learning tasks presented in interleaved trials. The ARR was controlled by an "induction" task which provided feedback with a low (p = 0.58), a medium (p = 0.75), or a high probability of reward (p = 0.92), while the impact of ARR on reinforcement learning was tested by a second "reference" task with a constant reward probability (p = 0.75). We find that performance was significantly lower in the reference task when the induction task provided low reward probabilities (i.e., during low levels of ARR), as compared to the medium and high ARR conditions. Behavioral modeling further revealed that the influence of ARR is best described by models which accumulates average rewards (rather than average prediction errors), and where the ARR directly modulates the prediction error signal (rather than affecting learning rates or exploration). Our results demonstrate how affective information in one domain may transfer and affect motivated behavior in other domains. These findings are particularly relevant for understanding mood disorders, but may also inform abnormal behaviors attributed to dopamine dysfunction.

A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety.

Exploration reduces uncertainty about the environment and improves the quality of future decisions, but at the cost of provisional uncertain and suboptimal outcomes. Although anxiety promotes intolerance to uncertainty, it remains unclear whether and by which mechanisms anxiety relates to exploratory decision-making. We use a dynamic three-armed-bandit task and find that higher trait-anxiety is associated with increased exploration, which in turn harms overall performance. We identify two distinct behavioral sources: first, decisions made by anxious individuals are guided toward reduction of uncertainty; and second, decisions are less guided by immediate value gains. These findings are similar in both loss and gain domains, and further demonstrate that an affective trait relates to exploration and results in an inverse-U-shaped relationship between anxiety and overall performance. Additional imaging data (fMRI) suggests that normative anxiety correlates negatively with the representation of expected-value in the dorsal-anterior-cingulate-cortex, and in contrast, positively with the representation of uncertainty in the anterior-insula. We conclude that a trade-off between value-gains and uncertainty-reduction entails maladaptive decision-making in individuals with higher normal-range anxiety.

Prior Reward Conditioning Dampens Hippocampal and Striatal Responses during an Associative Memory Task.

Offering reward during encoding typically leads to better memory [Adcock, R. A., Thangavel, A., Whitfield-Gabrieli, S.,Knutson, B., & Gabrieli, J. D. E. Reward-motivated learning: Mesolimbic activation precedes memory formation. Neuron, 50, 507-517, 2006]. Whether such memory benefit persists when tested in a different task context remains, however, largely understudied [Wimmer, G. E., & Buechel, C. Reactivation of reward-related patterns from single past episodes supports memory-based decision making. Journal of Neuroscience, 36, 2868-2880, 2016]. Here, we ask whether reward at encoding leads to a generalized advantage across learning episodes, a question of high importance for any everyday life applications, from education to patient rehabilitation. Although we confirmed that offering monetary reward increased responses in the ventral striatum and pleasantness judgments for pictures used as stimuli, this immediate beneficial effect of reward did not carry over to a subsequent and different picture-location association memory task during which no reward was delivered. If anything, a trend for impaired memory accuracy was observed for the initially high-rewarded pictures as compared to low-rewarded ones. In line with this trend in behavioral performance, fMRI activity in reward (i.e., ventral striatum) and in memory (i.e., hippocampus) circuits was reduced during the encoding of new associations using previously highly rewarded pictures (compared to low-reward pictures). These neural effects extended to new pictures from same, previously highly rewarded semantic category. Twenty-four hours later, delayed recall of associations involving originally highly rewarded items was accompanied by decreased functional connectivity between the hippocampus and two brain regions implicated in value-based learning, the ventral striatum and the ventromedial PFC. We conclude that acquired reward value elicits a downward value-adjustment signal in the human reward circuit when reactivated in a novel nonrewarded context, with a parallel disengagement of memory-reward (hippocampal-striatal) networks, likely to undermine new associative learning. Although reward is known to promote learning, here we show how it may subsequently hinder hippocampal and striatal responses during new associative memory formation.

Prediction errors bidirectionally bias time perception.

Time perception and prediction errors are essential for everyday life. We hypothesized that their putative shared circuitry in the striatum might enable these two functions to interact. We show that positive and negative prediction errors bias time perception by increasing and decreasing perceived time, respectively. Imaging and behavioral modeling identify this interaction to occur in the putamen. Depending on context, this interaction may have beneficial or adverse effects.

Interplay between midbrain and dorsal anterior cingulate regions arbitrates lingering reward effects on memory encoding.

Rewarding events enhance memory encoding via dopaminergic influences on hippocampal plasticity. Phasic dopamine release depends on immediate reward magnitude, but presumably also on tonic dopamine levels, which may vary as a function of the average accumulation of reward over time. Using model-based fMRI in combination with a novel associative memory task, we show that immediate reward magnitude exerts a monotonically increasing influence on the nucleus accumbens, ventral tegmental area (VTA), and hippocampal activity during encoding, and enhances memory. By contrast, average reward levels modulate feedback-related responses in the VTA and hippocampus in a non-linear (inverted U-shape) fashion, with similar effects on memory performance. Additionally, the dorsal anterior cingulate cortex (dACC) monotonically tracks average reward levels, while VTA-dACC functional connectivity is non-linearly modulated (inverted U-shape) by average reward. We propose that the dACC computes the net behavioral impact of average reward and relays this information to memory circuitry via the VTA.

Neurocomputational correlates of learned irrelevance in humans.

Inappropriate behaviors may result from acquiring maladaptive associations between irrelevant information in the environment and important events, such as reward or punishment. Pre-exposure effects are believed to prevent the expression of irrelevant associations. For example, learned irrelevance delays the expression of associations between conditioned (CS) and unconditioned (US) stimuli following their uncorrelated presentation. The neuronal substrates of pre-exposure effects in humans are largely unknown because these effects rapidly attenuate when using traditional pre-exposure paradigms. The latter are therefore incompatible with neuroimaging approaches that require many trial repetitions. Moreover, large methodological differences between animal and human research on pre-exposure effects challenge the presumption of shared neurocognitive substrates, and question the prevalent use of pre-exposure effects in animals to model symptoms of human mental disorders. To overcome these limitations, we combined a novel learned irrelevance task with model-based fMRI. We report the results of a model that describes learned irrelevance as a dynamic process, which evolves across trials and integrates the weighting between two state-action values pertaining to 'CS-no US' associations (acquired during pre-exposure) and 'CS-US' associations (acquired during subsequent conditioning). This relative weighting correlated i) positively with the learned irrelevance effect observed in the behavioral task, ii) positively with activity in the entorhinal cortex, and iii) negatively with activity in the nucleus accumbens (NAcc). Furthermore, the model updates the relative weighting of the two state-action values via two separate prediction error (PE) signals that allow the dynamic accumulation of evidence for the CS to predict the 'US' or a 'no US' outcome. One PE signal, designed to increase the relative weight of 'CS-US' associations following 'US' outcomes, correlated with activity in the NAcc, while another PE signal, designed to increase the relative weight of 'CS-no US' associations following 'no US' outcomes, correlated with activity in the basolateral amygdala. By extending previous animal observations to humans, the present study provides a novel approach to foster translational research on pre-exposure effects.

Re-exploring Mechanisms of Exploration.

Deciding when to exploit what is already known and when to explore new possibilities is crucial for adapting to novel and dynamic environments. Using reinforcement-based decision making, Costa et al. (2019) in this issue of Neuron find that neurons in the amygdala and ventral-striatum differentially signal the benefit from exploring new options and exploiting familiar ones.

Trial-by-Trial Modulation of Associative Memory Formation by Reward Prediction Error and Reward Anticipation as Revealed by a Biologically Plausible Computational Model.

Anticipation and delivery of rewards improves memory formation, but little effort has been made to disentangle their respective contributions to memory enhancement. Moreover, it has been suggested that the effects of reward on memory are mediated by dopaminergic influences on hippocampal plasticity. Yet, evidence linking memory improvements to actual reward computations reflected in the activity of the dopaminergic system, i.e., prediction errors and expected values, is scarce and inconclusive. For example, different previous studies reported that the magnitude of prediction errors during a reinforcement learning task was a positive, negative, or non-significant predictor of successfully encoding simultaneously presented images. Individual sensitivities to reward and punishment have been found to influence the activation of the dopaminergic reward system and could therefore help explain these seemingly discrepant results. Here, we used a novel associative memory task combined with computational modeling and showed independent effects of reward-delivery and reward-anticipation on memory. Strikingly, the computational approach revealed positive influences from both reward delivery, as mediated by prediction error magnitude, and reward anticipation, as mediated by magnitude of expected value, even in the absence of behavioral effects when analyzed using standard methods, i.e., by collapsing memory performance across trials within conditions. We additionally measured trait estimates of reward and punishment sensitivity and found that individuals with increased reward (vs. punishment) sensitivity had better memory for associations encoded during positive (vs. negative) prediction errors when tested after 20 min, but a negative trend when tested after 24 h. In conclusion, modeling trial-by-trial fluctuations in the magnitude of reward, as we did here for prediction errors and expected value computations, provides a comprehensive and biologically plausible description of the dynamic interplay between reward, dopamine, and associative memory formation. Our results also underline the importance of considering individual traits when assessing reward-related influences on memory.

Correction: Linking Individual Learning Styles to Approach-Avoidance Motivational Traits and Computational Aspects of Reinforcement Learning.

[This corrects the article DOI: 10.1371/journal.pone.0166675.].

The "Creative Right Brain" Revisited: Individual Creativity and Associative Priming in the Right Hemisphere Relate to Hemispheric Asymmetries in Reward Brain Function.

The idea that creativity resides in the right cerebral hemisphere is persistent in popular science, but has been widely frowned upon by the scientific community due to little empirical support. Yet, creativity is believed to rely on the ability to combine remote concepts into novel and useful ideas, an ability which would depend on associative processing in the right hemisphere. Moreover, associative processing is modulated by dopamine, and asymmetries in dopamine functionality between hemispheres may imbalance the expression of their implemented cognitive functions. Here, by uniting these largely disconnected concepts, we hypothesize that relatively less dopamine function in the right hemisphere boosts creativity by releasing constraining effects of dopamine on remote associations. Indeed, participants with reduced neural responses in the dopaminergic system of the right hemisphere (estimated by functional MRI in a reward task with positive and negative feedback), displayed higher creativity (estimated by convergent and divergent tasks), and increased associative processing in the right hemisphere (estimated by a lateralized lexical decision task). Our findings offer unprecedented empirical support for a crucial and specific contribution of the right hemisphere to creativity. More importantly our study provides a comprehensive view on potential determinants of human creativity, namely dopamine-related activity and associative processing.

Linking Individual Learning Styles to Approach-Avoidance Motivational Traits and Computational Aspects of Reinforcement Learning.

Learning how to gain rewards (approach learning) and avoid punishments (avoidance learning) is fundamental for everyday life. While individual differences in approach and avoidance learning styles have been related to genetics and aging, the contribution of personality factors, such as traits, remains undetermined. Moreover, little is known about the computational mechanisms mediating differences in learning styles. Here, we used a probabilistic selection task with positive and negative feedbacks, in combination with computational modelling, to show that individuals displaying better approach (vs. avoidance) learning scored higher on measures of approach (vs. avoidance) trait motivation, but, paradoxically, also displayed reduced learning speed following positive (vs. negative) outcomes. These data suggest that learning different types of information depend on associated reward values and internal motivational drives, possibly determined by personality traits.

The left hemisphere learns what is right: Hemispatial reward learning depends on reinforcement learning processes in the contralateral hemisphere.

Orienting biases refer to consistent, trait-like direction of attention or locomotion toward one side of space. Recent studies suggest that such hemispatial biases may determine how well people memorize information presented in the left or right hemifield. Moreover, lesion studies indicate that learning rewarded stimuli in one hemispace depends on the integrity of the contralateral striatum. However, the exact neural and computational mechanisms underlying the influence of individual orienting biases on reward learning remain unclear. Because reward-based behavioural adaptation depends on the dopaminergic system and prediction error (PE) encoding in the ventral striatum, we hypothesized that hemispheric asymmetries in dopamine (DA) function may determine individual spatial biases in reward learning. To test this prediction, we acquired fMRI in 33 healthy human participants while they performed a lateralized reward task. Learning differences between hemispaces were assessed by presenting stimuli, assigned to different reward probabilities, to the left or right of central fixation, i.e. presented in the left or right visual hemifield. Hemispheric differences in DA function were estimated through differential fMRI responses to positive vs. negative feedback in the left vs. right ventral striatum, and a computational approach was used to identify the neural correlates of PEs. Our results show that spatial biases favoring reward learning in the right (vs. left) hemifield were associated with increased reward responses in the left hemisphere and relatively better neural encoding of PEs for stimuli presented in the right (vs. left) hemifield. These findings demonstrate that trait-like spatial biases implicate hemisphere-specific learning mechanisms, with individual differences between hemispheres contributing to reinforcing spatial biases.

Hemispheric Asymmetries in Striatal Reward Responses Relate to Approach-Avoidance Learning and Encoding of Positive-Negative Prediction Errors in Dopaminergic Midbrain Regions.

Some individuals are better at learning about rewarding situations, whereas others are inclined to avoid punishments (i.e., enhanced approach or avoidance learning, respectively). In reinforcement learning, action values are increased when outcomes are better than predicted (positive prediction errors [PEs]) and decreased for worse than predicted outcomes (negative PEs). Because actions with high and low values are approached and avoided, respectively, individual differences in the neural encoding of PEs may influence the balance between approach-avoidance learning. Recent correlational approaches also indicate that biases in approach-avoidance learning involve hemispheric asymmetries in dopamine function. However, the computational and neural mechanisms underpinning such learning biases remain unknown. Here we assessed hemispheric reward asymmetry in striatal activity in 34 human participants who performed a task involving rewards and punishments. We show that the relative difference in reward response between hemispheres relates to individual biases in approach-avoidance learning. Moreover, using a computational modeling approach, we demonstrate that better encoding of positive (vs negative) PEs in dopaminergic midbrain regions is associated with better approach (vs avoidance) learning, specifically in participants with larger reward responses in the left (vs right) ventral striatum. Thus, individual dispositions or traits may be determined by neural processes acting to constrain learning about specific aspects of the world.

Increased Reward-Related Behaviors during Sleep and Wakefulness in Sleepwalking and Idiopathic Nightmares.

BACKGROUND: We previously suggested that abnormal sleep behaviors, i.e., as found in parasomnias, may often be the expression of increased activity of the reward system during sleep. Because nightmares and sleepwalking predominate during REM and NREM sleep respectively, we tested here whether exploratory excitability, a waking personality trait reflecting high activity within the mesolimbic dopaminergic (ML-DA) system, may be associated with specific changes in REM and NREM sleep patterns in these two sleep disorders. METHODS: Twenty-four unmedicated patients with parasomnia (12 with chronic sleepwalking and 12 with idiopathic nightmares) and no psychiatric comorbidities were studied. Each patient spent one night of sleep monitored by polysomnography. The Temperament and Character Inventory (TCI) was administered to all patients and healthy controls from the Geneva population (n = 293). RESULTS: Sleepwalkers were more anxious than patients with idiopathic nightmares (Spielberger Trait anxiety/STAI-T), but the patient groups did not differ on any personality dimension as estimated by the TCI. Compared to controls, parasomnia patients (sleepwalkers together with patients with idiopathic nightmares) scored higher on the Novelty Seeking (NS) TCI scale and in particular on the exploratory excitability/curiosity (NS1) subscale, and lower on the Self-directedness (SD) TCI scale, suggesting a general increase in reward sensitivity and impulsivity. Furthermore, parasomnia patients tended to worry about social separation persistently, as indicated by greater anticipatory worry (HA1) and dependence on social attachment (RD3). Moreover, exploratory excitability (NS1) correlated positively with the severity of parasomnia (i.e., the frequency of self-reported occurrences of nightmares and sleepwalking), and with time spent in REM sleep in patients with nightmares. CONCLUSIONS: These results suggest that patients with parasomnia might share common waking personality traits associated to reward-related brain functions. They also provide further support to the notion that reward-seeking networks are active during human sleep.

Trait anxiety and post-learning stress do not affect perceptual learning.

While it is well established that stress can modulate declarative learning, very few studies have investigated the influence of stress on non-declarative learning. Here, we studied the influence of post-learning stress, which effectively modulates declarative learning, on perceptual learning of a visual texture discrimination task (TDT). On day one, participants trained for one session with TDT and were instructed that they, at any time, could be exposed to either a high stressor (ice-water; Cold Pressor Test; CPT) or a low stressor (warm water). Participants did not know when or which stressor they would be exposed to. To determine the impact of the stressor on TDT learning, all participants returned the following day to perform another TDT session. Only participants exposed to the high stressor had significantly elevated cortisol levels. However, there was no difference in TDT improvements from day one to day two between the groups. Recent studies suggested that trait anxiety modulates visual perception under anticipation of stressful events. Here, trait anxiety did neither modulate performance nor influence responsiveness to stress. These results do not support a modulatory role for stress on non-declarative perceptual learning.

Different types of feedback change decision criterion and sensitivity differently in perceptual learning.

In (perceptual) learning, performance improves with practice either by changes in sensitivity or decision criterion. Often, changes in sensitivity are regarded as the appropriate measure of learning while changes in criterion are considered unavoidable nuisances. Very little is known about the distinguishing characteristics of both learning types. Here, we show first that block feedback, which affects sensitivity, does not affect criterion. Second, contrary to changes in sensitivity, changes in decision criterion are limited to the training session and do not transfer overnight. Finally, training with biased trial-wise feedback induces a sensitivity change such that a left offset Vernier may be perceived as a right offset Vernier.