Dopamine regulates decision thresholds in human reinforcement learning in males.

Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.

Erotic cue exposure increases physiological arousal, biases choices toward immediate rewards, and attenuates model-based reinforcement learning.

Computational psychiatry focuses on identifying core cognitive processes that appear altered across distinct psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting may be at least partly under contextual control. Highly arousing cues were shown to increase discounting, although evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues remains unclear. Here, we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n = 39 healthy heterosexual male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic versus neutral cues both on the subjective and autonomic level. Erotic cue exposure increased discounting as reflected by more impatient choices. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation toward immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning in a heterosexual male sample. This highlights how environmental cues can impact core human decision processes and reveal that comprehensive modeling approaches can yield novel insights in reward-based decision processes.

The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting.

Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.

Gambling Environment Exposure Increases Temporal Discounting but Improves Model-Based Control in Regular Slot-Machine Gamblers.

Gambling disorder is a behavioral addiction that negatively impacts personal finances, work, relationships and mental health. In this pre-registered study (https://osf.io/5ptz9/) we investigated the impact of real-life gambling environments on two computational markers of addiction, temporal discounting and model-based reinforcement learning. Gambling disorder is associated with increased temporal discounting and reduced model-based learning. Regular gamblers (n = 30, DSM-5 score range 3-9) performed both tasks in a neutral (café) and a gambling-related environment (slot-machine venue) in counterbalanced order. Data were modeled using drift diffusion models for temporal discounting and reinforcement learning via hierarchical Bayesian estimation. Replicating previous findings, gamblers discounted rewards more steeply in the gambling-related context. This effect was positively correlated with gambling related cognitive distortions (pre-registered analysis). In contrast to our pre-registered hypothesis, model-based reinforcement learning was improved in the gambling context. Here we show that temporal discounting and model-based reinforcement learning are modulated in opposite ways by real-life gambling cue exposure. Results challenge aspects of habit theories of addiction, and reveal that laboratory-based computational markers of psychopathology are under substantial contextual control.

Dopamine release, diffusion and uptake: A computational model for synaptic and volume transmission.

Computational modeling of dopamine transmission is challenged by complex underlying mechanisms. Here we present a new computational model that (I) simultaneously regards release, diffusion and uptake of dopamine, (II) considers multiple terminal release events and (III) comprises both synaptic and volume transmission by incorporating the geometry of the synaptic cleft. We were able to validate our model in that it simulates concentration values comparable to physiological values observed in empirical studies. Further, although synaptic dopamine diffuses into extra-synaptic space, our model reflects a very localized signal occurring on the synaptic level, i.e. synaptic dopamine release is negligibly recognized by neighboring synapses. Moreover, increasing evidence suggests that cognitive performance can be predicted by signal variability of neuroimaging data (e.g. BOLD). Signal variability in target areas of dopaminergic neurons (striatum, cortex) may arise from dopamine concentration variability. On that account we compared spatio-temporal variability in a simulation mimicking normal dopamine transmission in striatum to scenarios of enhanced dopamine release and dopamine uptake inhibition. We found different variability characteristics between the three settings, which may in part account for differences in empirical observations. From a clinical perspective, differences in striatal dopaminergic signaling contribute to differential learning and reward processing, with relevant implications for addictive- and compulsive-like behavior. Specifically, dopaminergic tone is assumed to impact on phasic dopamine and hence on the integration of reward-related signals. However, in humans DA tone is classically assessed using PET, which is an indirect measure of endogenous DA availability and suffers from temporal and spatial resolution issues. We discuss how this can lead to discrepancies with observations from other methods such as microdialysis and show how computational modeling can help to refine our understanding of DA transmission.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making.

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

A potential link between gambling addiction severity and central dopamine levels: Evidence from spontaneous eye blink rates.

Accumulating evidence points at similarities between substance use disorders (SUD) and gambling disorder on the behavioral and neural level. In SUD, attenuation of striatal D2/3-receptor availability is a consistent finding, at least for stimulating substances. For gambling disorder, no clear association with striatal D2/3-receptor availability has been unveiled so far. With its presumably negligible dopaminergic toxicity, possible differences in receptor availability in gambling disorder might constitute a vulnerability marker. Spontaneous eye blink rate (sEBR) is discussed as a potential proxy measure for striatal dopamine D2/3-receptor availability. Here we examined sEBR in 21 male problem gamblers and 20 healthy control participants. In addition, participants completed a screening questionnaire for overall psychopathology and self-reported measures of alcohol and nicotine consumption. We found no significant difference in sEBR between gamblers and controls. However, in gamblers, sEBR was negatively associated with gambling severity and positively associated with psychopathology. A final exploratory analysis revealed that healthy controls with low sEBR displayed higher alcohol and nicotine consumption than healthy participants with high sEBR. Although the exact association between dopamine transmission and sEBR is still debated, our findings reveal that sEBR is sensitive to inter-individual differences in gambling disorder severity in problem gamblers.

Retraining automatic action tendencies in obesity.

Eating behavior in obesity resembles addictive disorders in that individuals have difficulties inhibiting problematic eating behavior. They show an approach bias - a tendency to approach rather than avoid problematic stimuli. Here, we investigate the existence of such a bias towards healthy and unhealthy food in individuals with normal-weight and obesity. We further aimed to assess whether it is possible to retrain a bias, and whether training would differentially affect our two weight groups. 60 participants completed a training form of the Approach-Avoidance Task (AAT) on three consecutive days. Using a joystick, they were implicitly trained to approach healthy and to avoid unhealthy food pictures. Prior to training, individuals with obesity showed stronger approach tendencies towards food pictures than normal-weight individuals. In individuals with obesity, approach tendencies could be diminished for unhealthy food through one training session and stayed weakened for the following days. In normal-weight participants, approach tendencies towards healthy food could be enhanced over the days of training. Findings indicate that automatic approach tendencies can be changed through training, thus offering possibilities for obesity treatment. Future studies should expand on these findings, for example by including pictures of neutral objects or a no-training control condition.

Altered monetary loss processing and reinforcement-based learning in individuals with obesity.

Individuals with obesity are often characterized by alterations in reward processing. This may affect how new information is used to update stimulus values during reinforcement-based learning. Here, we investigated obesity-related changes in non-food reinforcement processing, their impact on learning performance as well as the neural underpinnings of reinforcement-based learning in obesity. Nineteen individuals with obesity (BMI > = 30 kg/m2, 10 female) and 23 lean control participants (BMI 18.5-24.9 kg/m2, 11 female) performed a probabilistic learning task during functional magnetic resonance imaging (fMRI), in which they learned to choose between advantageous and disadvantageous choice options in separate monetary gain, loss, and neutral conditions. During learning individuals with obesity made a significantly lower number of correct choices and accumulated a significantly lower overall monetary outcome than lean control participants. FMRI analyses revealed aberrant medial prefrontal cortex responses to monetary losses in individuals with obesity. There were no significant group differences in the regional representation of prediction errors. However, we found evidence for increased functional connectivity between the ventral striatum and insula in individuals with obesity. The present results suggest that obesity is associated with aberrant value representations for monetary losses, alterations in functional connectivity during the processing of learning outcomes, as well as a decresased reinforcement-based learning performance. This may affect how new information is incorporated to adjust dysfunctional behavior and could be a factor contributing to the maintenance of dysfunctional eating behavior in obesity.

Failing to learn from negative prediction errors: Obesity is associated with alterations in a fundamental neural learning mechanism.

Prediction errors (PEs) encode the difference between expected and actual action outcomes in the brain via dopaminergic modulation. Integration of these learning signals ensures efficient behavioral adaptation. Obesity has recently been linked to altered dopaminergic fronto-striatal circuits, thus implying impairments in cognitive domains that rely on its integrity. 28 obese and 30 lean human participants performed an implicit stimulus-response learning paradigm inside an fMRI scanner. Computational modeling and psycho-physiological interaction (PPI) analysis was utilized for assessing PE-related learning and associated functional connectivity. We show that human obesity is associated with insufficient incorporation of negative PEs into behavioral adaptation even in a non-food context, suggesting differences in a fundamental neural learning mechanism. Obese subjects were less efficient in using negative PEs to improve implicit learning performance, despite proper coding of PEs in striatum. We further observed lower functional coupling between ventral striatum and supplementary motor area in obese subjects subsequent to negative PEs. Importantly, strength of functional coupling predicted task performance and negative PE utilization. These findings show that obesity is linked to insufficient behavioral adaptation specifically in response to negative PEs, and to associated alterations in function and connectivity within the fronto-striatal system. Recognition of neural differences as a central characteristic of obesity hopefully paves the way to rethink established intervention strategies: Differential behavioral sensitivity to negative and positive PEs should be considered when designing intervention programs. Measures relying on penalization of unwanted behavior may prove less effective in obese subjects than alternative approaches.

The role of dopamine in positive and negative prediction error utilization during incidental learning - Insights from Positron Emission Tomography, Parkinson's disease and Huntington's disease.

Incidental learning of appropriate stimulus-response associations is crucial for optimal functioning within our complex environment. Positive and negative prediction errors (PEs) serve as neural teaching signals within distinct ('direct'/'indirect') dopaminergic pathways to update associations and optimize subsequent behavior. Using a computational reinforcement learning model, we assessed learning from positive and negative PEs on a probabilistic task (Weather Prediction Task - WPT) in three populations that allow different inferences on the role of dopamine (DA) signals: (1) Healthy volunteers that repeatedly underwent [11C]raclopride Positron Emission Tomography (PET), allowing for assessment of striatal DA release during learning, (2) Parkinson's disease (PD) patients tested both on and off L-DOPA medication, (3) early Huntington's disease (HD) patients, a disease that is associated with hyper-activation of the 'direct' pathway. Our results show that learning from positive and negative feedback on the WPT is intimately linked to different aspects of dopaminergic transmission. In healthy individuals, the difference in [11C]raclopride binding potential (BP) as a measure for striatal DA release was linearly associated with the positive learning rate. Further, asymmetry between baseline DA tone in the left and right ventral striatum was negatively associated with learning from positive PEs. Female patients with early HD exhibited exaggerated learning rates from positive feedback. In contrast, dopaminergic tone predicted learning from negative feedback, as indicated by an inverted u-shaped association observed with baseline [11C]raclopride BP in healthy controls and the difference between PD patients' learning rate on and off dopaminergic medication. Thus, the ability to learn from positive and negative feedback is a sensitive marker for the integrity of dopaminergic signal transmission in the 'direct' and 'indirect' dopaminergic pathways. The present data are interesting beyond clinical context in that imbalances of dopaminergic signaling have not only been observed for neurological and psychiatric conditions but also been proposed for obesity and adolescence.

Stopping at the sight of food - How gender and obesity impact on response inhibition.

Recent research indicates that reduced inhibitory control is associated with higher body mass index (BMI), higher food craving and increased food intake. However, experimental evidence for the relationship between response inhibition and weight status is inconsistent and to date has been investigated predominantly in women. In the current study, 56 participants (26 obese, 30 lean; 27 female, 29 male) performed a Food Picture Rating Task followed by a Stop Signal Task where pictures of palatable high or low caloric food or non-food items were presented prior to the Go signal. We further assessed participants' self-reported eating behavior and trait impulsivity as potential factors influencing response inhibition, in particular within the food context. Independent of BMI, women showed significantly higher liking for low caloric food items than men. This was accompanied by shorter Stop Signal Reaction Times (SSRT) after high compared to low caloric food pictures for women, and shorter SSRT in women compared to men for high caloric food. No influence of gender on SSRT was observable outside of the food context. While SSRTs did not differ between obese and lean participants across the three picture categories, we found a moderating effect of trait impulsivity on the relationship between BMI and SSRT, specifically in the high caloric food context. Higher BMI was predictive of longer SSRT only for participants with low to normal trait impulsivity, pointing at a complex interplay between response inhibition, general impulsivity and weight status. Our results support the notion that individuals with obesity do not suffer from diminished response inhibition capacity per se. Rather, the ability to withhold a response depends on context and social norms, and strongly interacts with factors like gender and trait impulsivity.

Is it Worth the Effort? Novel Insights into Obesity-Associated Alterations in Cost-Benefit Decision-Making.

Cost-benefit decision-making entails the process of evaluating potential actions according to the trade-off between the expected reward (benefit) and the anticipated effort (costs). Recent research revealed that dopaminergic transmission within the fronto-striatal circuitry strongly modulates cost-benefit decision-making. Alterations within the dopaminergic fronto-striatal system have been associated with obesity, but little is known about cost-benefit decision-making differences in obese compared with lean individuals. With a newly developed experimental task we investigate obesity-associated alterations in cost-benefit decision-making, utilizing physical effort by handgrip-force exertion and both food and non-food rewards. We relate our behavioral findings to alterations in local gray matter volume assessed by structural MRI. Obese compared with lean subjects were less willing to engage in physical effort in particular for high-caloric sweet snack food. Further, self-reported body dissatisfaction negatively correlated with the willingness to invest effort for sweet snacks in obese men. On a structural level, obesity was associated with reductions in gray matter volume in bilateral prefrontal cortex. Nucleus accumbens volume positively correlated with task induced implicit food craving. Our results challenge the common notion that obese individuals are willing to work harder to obtain high-caloric food and emphasize the need for further exploration of the underlying neural mechanisms regarding cost-benefit decision-making differences in obesity.

Slave to habit? Obesity is associated with decreased behavioural sensitivity to reward devaluation.

The motivational value of food is lower during satiety compared to fasting. Dynamic changes in motivational value promote food seeking or meal cessation. In obesity this mechanism might be compromised since obese subjects ingest energy beyond homeostatic needs. Thus, lower adaptation of eating behaviour with respect to changes in motivational value might cause food overconsumption in obesity. To test this hypothesis, we implemented a selective satiation procedure to investigate the relationship between obesity and the size of the behavioural devaluation effect in humans. Lean to obese men (mean age 25.9, range 19-30 years; mean BMI 29.1, range 19.2-45.1 kg/m(2)) were trained on a free operant paradigm and learned to associate cues with the possibility to win different food rewards by pressing a button. After the initial training phase, one of the rewards was devalued by consumption. Response rates for and wanting of the different rewards were measured pre and post devaluation. Behavioural sensitivity to reward devaluation, measured as the magnitude of difference between pre and post responses, was regressed against BMI. Results indicate that (1) higher BMI compared to lower BMI in men led to an attenuated behavioural adjustment to reward devaluation, and (2) the decrease in motivational value was associated with the decrease in response rate between pre and post. Change in explicitly reported motivational value, however, was not affected by BMI. Thus, we conclude that high BMI in men is associated with lower behavioural adaptation with respect to changes in motivational value of food, possibly resulting in automatic overeating patterns that are hard to control in daily life.

Obesity-Related Differences between Women and Men in Brain Structure and Goal-Directed Behavior.

Gender differences in the regulation of body-weight are well documented. Here, we assessed obesity-related influences of gender on brain structure as well as performance in the Iowa Gambling Task. This task requires evaluation of both immediate rewards and long-term outcomes and thus mirrors the trade-off between immediate reward from eating and the long-term effect of overeating on body-weight. In women, but not in men, we show that the preference for salient immediate rewards in the face of negative long-term consequences is higher in obese than in lean subjects. In addition, we report structural differences in the left dorsal striatum (i.e., putamen) and right dorsolateral prefrontal cortex for women only. Functionally, both regions are known to play complimentary roles in habitual and goal-directed control of behavior in motivational contexts. For women as well as men, gray matter volume correlates positively with measures of obesity in regions coding the value and saliency of food (i.e., nucleus accumbens, orbitofrontal cortex) as well as in the hypothalamus (i.e., the brain's central homeostatic center). These differences between lean and obese subjects in hedonic and homeostatic control systems may reflect a bias in eating behavior toward energy-intake exceeding the actual homeostatic demand. Although we cannot infer from our results the etiology of the observed structural differences, our results resemble neural and behavioral differences well known from other forms of addiction, however, with marked differences between women and men. These findings are important for designing gender-appropriate treatments of obesity and possibly its recognition as a form of addiction.