Dopamine and acetylcholine have distinct roles in delay- and effort-based decision-making in humans.

In everyday life, we encounter situations that require tradeoffs between potential rewards and associated costs, such as time and (physical) effort. The literature indicates a prominent role for dopamine in discounting of both delay and effort, with mixed findings for delay discounting in humans. Moreover, the reciprocal antagonistic interaction between dopaminergic and cholinergic transmission in the striatum suggests a potential opponent role of acetylcholine in these processes. We found opposing effects of dopamine D2 (haloperidol) and acetylcholine M1 receptor (biperiden) antagonism on specific components of effort-based decision-making in healthy humans: haloperidol decreased, whereas biperiden increased the willingness to exert physical effort. In contrast, delay discounting was reduced under haloperidol, but not affected by biperiden. Together, our data suggest that dopamine, acting at D2 receptors, modulates both effort and delay discounting, while acetylcholine, acting at M1 receptors, appears to exert a more specific influence on effort discounting only.

Effort-based decision making and motivational deficits in stroke patients.

Motivational deficits in patients recovering from stroke are common and can reduce active participation in rehabilitation and thereby impede functional recovery. We investigated whether stroke patients with clinically reduced drive, initiation, and endurance during functional rehabilitative training (n = 30) display systematic alterations in effort-based decision making compared to age, sex, and severity-matched stroke patients (n = 30) whose drive appeared unaffected. Notably, the two groups did not differ in self-reported ratings of apathy and depression. However, on an effort-based decision-making task, stroke patients with clinically apparent drive impairment showed intact willingness to accept effort for reward, but were more likely to fail to execute the required effort compared to patients without apparent drive impairments. In other words, the decision behavioural assessment revealed that stroke patients that displayed reduced drive, initiation, and endurance during inpatient neurorehabilitation failed to persist in goal-directed effort production, even over very short periods. These findings indicate that reduced drive during rehabilitative therapy in post-stroke patients is not due to a diminished motivation to invest physical effort, but instead is related to a reduced persistence with effortful behaviour.

Removal of reinforcement improves instrumental performance in humans by decreasing a general action bias rather than unmasking learnt associations.

Performance during instrumental learning is commonly believed to reflect the knowledge that has been acquired up to that point. However, recent work in rodents found that instrumental performance was enhanced during periods when reinforcement was withheld, relative to periods when reinforcement was provided. This suggests that reinforcement may mask acquired knowledge and lead to impaired performance. In the present study, we investigated whether such a beneficial effect of removing reinforcement translates to humans. Specifically, we tested whether performance during learning was improved during non-reinforced relative to reinforced task periods using signal detection theory and a computational modelling approach. To this end, 60 healthy volunteers performed a novel visual go/no-go learning task with deterministic reinforcement. To probe acquired knowledge in the absence of reinforcement, we interspersed blocks without feedback. In these non-reinforced task blocks, we found an increased d', indicative of enhanced instrumental performance. However, computational modelling showed that this improvement in performance was not due to an increased sensitivity of decision making to learnt values, but to a more cautious mode of responding, as evidenced by a reduction of a general response bias. Together with an initial tendency to act, this is sufficient to drive differential changes in hit and false alarm rates that jointly lead to an increased d'. To conclude, the improved instrumental performance in the absence of reinforcement observed in studies using asymmetrically reinforced go/no-go tasks may reflect a change in response bias rather than unmasking latent knowledge.

Motor cortical signals reflecting decision making and action preparation.

Decision making often requires accumulating evidence in favour of a particular option. When choices are expressed with a motor response, these actions are preceded by reductions in the power of oscillations in the alpha and beta range in motor cortices. For unimanual movements, these reductions are greater over the hemisphere contralateral to the response side. Such lateralizations are hypothesized to be an online index of the neural state of decisions as they develop over time of processing. In contrast, the lateralized readiness potential (LRP) is considered to selectively activate a response and appears shortly before the motor output. We investigated to what extent these neural signals reflect integration of decision evidence or more motor-related action preparation. Using two different experiments, we found that lateralization of alpha and beta power (APL and BPL, respectively) rapidly emerged after stimulus presentation, even when making an overt response was not yet possible. In contrast, we show that even after prolonged stimulus presentation, no LRP was present. Instead, the LRP emerged only after an imperative cue, prompting participants to indicate their choice. Furthermore, we could show that variations in sensory evidence strength modulate APL and BPL onset times, suggesting that integration of evidence is represented in these motor cortical signals. We conclude that APL and BPL reflect higher cognitive processes rather than pure action preparation, whereas LRP is more closely tied to motor performance. APL and BPL potentially encode decision information in motor areas serving the later preparation of overt decision output.

Reinstatement of Cortical Outcome Representations during Higher-Order Learning.

Naturalistic learning scenarios are characterized by infrequent experience of external feedback to guide behavior. Higher-order learning mechanisms like second-order conditioning (SOC) may allow stimuli that were never experienced together with reinforcement to acquire motivational value. Despite its explanatory potential for real-world learning, surprisingly little is known about the neural mechanism underlying such associative transfer of value in SOC. Here, we used multivariate cross-session, cross-modality searchlight classification on functional magnetic resonance imaging data obtained from humans during SOC. We show that visual first-order conditioned stimuli (CS) reinstate cortical patterns representing previously paired gustatory outcomes in the lateral orbitofrontal cortex (OFC). During SOC, this OFC region showed increased functional covariation with amygdala, where neural pattern similarity between second-order CS and outcomes increased from early to late stages of SOC. Our data suggest a mechanism by which motivational value is conferred to stimuli that were never paired with reinforcement.

Neurophysiology of performance monitoring and adaptive behavior.

Successful goal-directed behavior requires not only correct action selection, planning, and execution but also the ability to flexibly adapt behavior when performance problems occur or the environment changes. A prerequisite for determining the necessity, type, and magnitude of adjustments is to continuously monitor the course and outcome of one's actions. Feedback-control loops correcting deviations from intended states constitute a basic functional principle of adaptation at all levels of the nervous system. Here, we review the neurophysiology of evaluating action course and outcome with respect to their valence, i.e., reward and punishment, and initiating short- and long-term adaptations, learning, and decisions. Based on studies in humans and other mammals, we outline the physiological principles of performance monitoring and subsequent cognitive, motivational, autonomic, and behavioral adaptation and link them to the underlying neuroanatomy, neurochemistry, psychological theories, and computational models. We provide an overview of invasive and noninvasive systemic measures, such as electrophysiological, neuroimaging, and lesion data. We describe how a wide network of brain areas encompassing frontal cortices, basal ganglia, thalamus, and monoaminergic brain stem nuclei detects and evaluates deviations of actual from predicted states indicating changed action costs or outcomes. This information is used to learn and update stimulus and action values, guide action selection, and recruit adaptive mechanisms that compensate errors and optimize goal achievement.

A causal role of estradiol in human reinforcement learning.

The sex hormone estradiol is hypothesized to play a key role in human cognition, and reward processing specifically, via increased dopamine D1-receptor signalling. However, the effect of estradiol on reward processing in men has never been established. To fill this gap, we performed a double-blind placebo-controlled study in which men (N = 100) received either a single dose of estradiol (2 mg) or a placebo. Subjects performed a probabilistic reinforcement learning task where they had to choose between two options with varying reward probabilities to maximize monetary reward. Results showed that estradiol administration increased reward sensitivity compared to placebo. This effect was observed in subjects' choices, how much weight they assigned to their previous choices, and subjective reports about the reward probabilities. Furthermore, effects of estradiol were moderated by reward sensitivity, as measured through the BIS/BAS questionnaire. Using reinforcement learning models, we found that behavioral effects of estradiol were reflected in increased learning rates. These results demonstrate a causal role of estradiol within the framework of reinforcement learning, by enhancing reward sensitivity and learning. Furthermore, they provide preliminary evidence for dopamine-related genetic variants moderating the effect of estradiol on reward processing.

Dorsolateral Prefrontal Cortex Enables Updating of Established Memories.

Updating established memories in light of new information is fundamental for memory to guide future behavior. However, little is known about the brain mechanisms by which existing memories can be updated. Here, we combined functional magnetic resonance imaging and multivariate representational similarity analysis to elucidate the neural mechanisms underlying the updating of consolidated memories. To this end, participants first learned face-city name pairs. Twenty-four hours later, while lying in the MRI scanner, participants were required to update some of these associations, but not others, and to encode entirely new pairs. Updating success was tested again 24 h later. Our results showed increased activity of the dorsolateral prefrontal cortex (dlPFC) specifically during the updating of existing associations that was significantly stronger than when simple retrieval or new encoding was required. The updating-related activity of the dlPFC and its functional connectivity with the hippocampus were directly linked to updating success. Furthermore, neural similarity for updated items was markedly higher in the dlPFC and this increase in dlPFC neural similarity distinguished individuals with high updating performance from those with low updating performance. Together, these findings suggest a key role of the dlPFC, presumably in interaction with the hippocampus, in the updating of established memories.

Differential modulation of reinforcement learning by D2 dopamine and NMDA glutamate receptor antagonism.

The firing pattern of midbrain dopamine (DA) neurons is well known to reflect reward prediction errors (PEs), the difference between obtained and expected rewards. The PE is thought to be a crucial signal for instrumental learning, and interference with DA transmission impairs learning. Phasic increases of DA neuron firing during positive PEs are driven by activation of NMDA receptors, whereas phasic suppression of firing during negative PEs is likely mediated by inputs from the lateral habenula. We aimed to determine the contribution of DA D2-class and NMDA receptors to appetitively and aversively motivated reinforcement learning. Healthy human volunteers were scanned with functional magnetic resonance imaging while they performed an instrumental learning task under the influence of either the DA D2 receptor antagonist amisulpride (400 mg), the NMDA receptor antagonist memantine (20 mg), or placebo. Participants quickly learned to select ("approach") rewarding and to reject ("avoid") punishing options. Amisulpride impaired both approach and avoidance learning, while memantine mildly attenuated approach learning but had no effect on avoidance learning. These behavioral effects of the antagonists were paralleled by their modulation of striatal PEs. Amisulpride reduced both appetitive and aversive PEs, while memantine diminished appetitive, but not aversive PEs. These data suggest that striatal D2-class receptors contribute to both approach and avoidance learning by detecting both the phasic DA increases and decreases during appetitive and aversive PEs. NMDA receptors on the contrary appear to be required only for approach learning because phasic DA increases during positive PEs are NMDA dependent, whereas phasic decreases during negative PEs are not.

Dissociable contributions of ventromedial prefrontal and posterior parietal cortex to value-guided choice.

Two long-standing traditions have highlighted cortical decision mechanisms in the parietal and prefrontal cortices of primates, but it has not been clear how these processes differ, or when each cortical region may influence behaviour. Recent data from ventromedial prefrontal cortex (vmPFC) and posterior parietal cortex (PPC) have suggested one possible axis on which the two decision processes might be delineated. Fast decisions may be resolved primarily by parietal mechanisms, whereas decisions made without time pressure may rely on prefrontal mechanisms. Here, we report direct evidence for such dissociation. During decisions under time pressure, a value comparison process was evident in PPC, but not in vmPFC. Value-related activity was still found in vmPFC under time pressure. However, vmPFC represented overall input value rather than compared output value. In contrast, when decisions were made without time pressure, vmPFC transitioned to encode a value comparison while value-related parameters were entirely absent from PPC. Furthermore, under time pressure, decision performance was primarily governed by PPC, while it was dominated by vmPFC at longer decision times. These data demonstrate that parallel cortical mechanisms may resolve the same choices in differing circumstances, and offer an explanation of the diverse neural signals reported in vmPFC and PPC during value-guided choice.

Tonic and phasic transcutaneous auricular vagus nerve stimulation (taVNS) both evoke rapid and transient pupil dilation.

BACKGROUND: Transcutaneous auricular vagus nerve stimulation (tVNS or taVNS) is a non-invasive method of electrical stimulation of the afferent pathway of the vagus nerve, suggested to drive changes in putative physiological markers of noradrenergic activity, including pupil dilation. OBJECTIVE: However, it is unknown whether different taVNS modes can map onto the phasic and tonic modes of noradrenergic activity. The effects of taVNS on pupil dilation in humans are inconsistent, largely due to differences in stimulation protocols. Here, we attempted to address these issues. METHODS: We investigated pupil dilation under phasic (1 s) and tonic (30 s) taVNS, in a pre-registered, single-blind, sham-controlled, within-subject cross-over design, in the absence of a behavioural task. RESULTS: Phasic taVNS induced a rapid increase in pupil size over baseline, significantly greater than under sham stimulation, which rapidly declined after stimulation offset. Tonic taVNS induced a similarly rapid (and larger than sham) increase in pupil size over baseline, returning to baseline within 5 s, despite the ongoing stimulation. Thus, both active and sham tonic modes closely resembled the phasic effect. There were no differences in tonic baseline pupil size, and no sustained effects of stimulation on tonic baseline pupil size. CONCLUSIONS: These results suggest that both phasic- and tonic-like taVNS under the standard stimulation parameters may modulate primarily the phasic mode of noradrenergic activity, as indexed by evoked pupil dilation, over and above somatosensory effects. This result sheds light on the temporal profile of phasic and tonic stimulation, with implications for their applicability in further research.

Stimulus awareness is necessary for both instrumental learning and instrumental responding to previously learned stimuli.

Instrumental conditioning is a crucial part of adaptive behaviour, allowing agents to selectively interact with stimuli in their environment. Recent evidence suggests that instrumental conditioning cannot proceed without stimulus awareness. However, whether accurate unconscious instrumental responding can emerge from consciously acquired knowledge of the stimulus-action-outcome contingencies is unknown. We studied this question using instrumental trace conditioning, where participants learned to make approach/avoid decisions in two within-subject modes: conscious (stimuli in plain view) and unconscious (visually masked). Both tasks were followed by an unconscious-only instrumental performance task. We show that even when the contingencies are reliably learned in the conscious mode, participants fail to act upon them in the unconscious responding task. We also replicate the previous results that no instrumental learning occurs in the unconscious mode. Consequently, the absence of stimulus awareness not only precludes instrumental conditioning, but also precludes any kind of instrumental responding to already known stimuli. This suggests that instrumental behaviour is entirely supported by conscious awareness of the world, and corroborates the proposals that consciousness may be necessary for adaptive behaviours requiring selective action.

Auditory inputs modulate intrinsic neuronal timescales during sleep.

Functional magnetic resonance imaging (fMRI) studies have demonstrated that intrinsic neuronal timescales (INT) undergo modulation by external stimulation during consciousness. It remains unclear if INT keep the ability for significant stimulus-induced modulation during primary unconscious states, such as sleep. This fMRI analysis addresses this question via a dataset that comprises an awake resting-state plus rest and stimulus states during sleep. We analyzed INT measured via temporal autocorrelation supported by median frequency (MF) in the frequency-domain. Our results were replicated using a biophysical model. There were two main findings: (1) INT prolonged while MF decreased from the awake resting-state to the N2 resting-state, and (2) INT shortened while MF increased during the auditory stimulus in sleep. The biophysical model supported these results by demonstrating prolonged INT in slowed neuronal populations that simulate the sleep resting-state compared to an awake state. Conversely, under sine wave input simulating the stimulus state during sleep, the model's regions yielded shortened INT that returned to the awake resting-state level. Our results highlight that INT preserve reactivity to stimuli in states of unconsciousness like sleep, enhancing our understanding of unconscious brain dynamics and their reactivity to stimuli.

Intermediately synchronised brain states optimise trade-off between subject specificity and predictive capacity.

Functional connectivity (FC) refers to the statistical dependencies between activity of distinct brain areas. To study temporal fluctuations in FC within the duration of a functional magnetic resonance imaging (fMRI) scanning session, researchers have proposed the computation of an edge time series (ETS) and their derivatives. Evidence suggests that FC is driven by a few time points of high-amplitude co-fluctuation (HACF) in the ETS, which may also contribute disproportionately to interindividual differences. However, it remains unclear to what degree different time points actually contribute to brain-behaviour associations. Here, we systematically evaluate this question by assessing the predictive utility of FC estimates at different levels of co-fluctuation using machine learning (ML) approaches. We demonstrate that time points of lower and intermediate co-fluctuation levels provide overall highest subject specificity as well as highest predictive capacity of individual-level phenotypes.

Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices.

A large body of evidence exists on the role of dopamine in reinforcement learning. Less is known about how dopamine shapes the relative impact of positive and negative outcomes to guide value-based choices. We combined administration of the dopamine D(2) receptor antagonist amisulpride with functional magnetic resonance imaging in healthy human volunteers. Amisulpride did not affect initial reinforcement learning. However, in a later transfer phase that involved novel choice situations requiring decisions between two symbols based on their previously learned values, amisulpride improved participants' ability to select the better of two highly rewarding options, while it had no effect on choices between two very poor options. During the learning phase, activity in the striatum encoded a reward prediction error. In the transfer phase, in the absence of any outcome, ventromedial prefrontal cortex (vmPFC) continually tracked the learned value of the available options on each trial. Both striatal prediction error coding and tracking of learned value in the vmPFC were predictive of subjects' choice performance in the transfer phase, and both were enhanced under amisulpride. These findings show that dopamine-dependent mechanisms enhance reinforcement learning signals in the striatum and sharpen representations of associative values in prefrontal cortex that are used to guide reinforcement-based decisions.

Continuous theta-burst stimulation (cTBS) over the lateral prefrontal cortex alters reinforcement learning bias.

The prefrontal cortex is known to play a key role in higher-order cognitive functions. Recently, we showed that this brain region is active in reinforcement learning, during which subjects constantly have to integrate trial outcomes in order to optimize performance. To further elucidate the role of the dorsolateral prefrontal cortex (DLPFC) in reinforcement learning, we applied continuous theta-burst stimulation (cTBS) either to the left or right DLPFC, or to the vertex as a control region, respectively, prior to the performance of a probabilistic learning task in an fMRI environment. While there was no influence of cTBS on learning performance per se, we observed a stimulation-dependent modulation of reward vs. punishment sensitivity: Left-hemispherical DLPFC stimulation led to a more reward-guided performance, while right-hemispherical cTBS induced a more avoidance-guided behavior. FMRI results showed enhanced prediction error coding in the ventral striatum in subjects stimulated over the left as compared to the right DLPFC. Both behavioral and imaging results are in line with recent findings that left, but not right-hemispherical stimulation can trigger a release of dopamine in the ventral striatum, which has been suggested to increase the relative impact of rewards rather than punishment on behavior.

Not giving up: Testosterone promotes persistence against a stronger opponent.

Recent research suggests that when we lack a sense of control, we are prone to motivational failures and early quitting in competitions. Testosterone, on the other hand, is thought to boost competitiveness. Here we investigate the interaction between these factors, testing the testosterone's potential to enhance persistence in a competition against a stronger opponent, depending on experimentally manipulated perceived control. Healthy participants were administered a single dose of testosterone or placebo. They first underwent a task designed to either induce low or high perceived control and then entered a costly competition against a progressively stronger opponent that they could quit at any time. In the placebo group, men with low perceived control quitted twice as early as those with high perceived control. Testosterone countered this effect, making individuals with low control persist in the competition for as long as those with high perceived control, and did so also despite raising participants' explicit awareness of the opponents' advantage. This psychoendocrinological effect was not modulated by basal cortisol levels, CAG repeat polymorphism of the androgen receptor gene, or trait dominance. Our results provide the first causal evidence that testosterone promotes competitive persistence in humans and demonstrate that this effect depends on the psychological state elicited prior to the competition, broadening our understanding of the complex relationships between testosterone and social behaviors.

Dissociable roles of cortical excitation-inhibition balance during patch-leaving versus value-guided decisions.

In a dynamic world, it is essential to decide when to leave an exploited resource. Such patch-leaving decisions involve balancing the cost of moving against the gain expected from the alternative patch. This contrasts with value-guided decisions that typically involve maximizing reward by selecting the current best option. Patterns of neuronal activity pertaining to patch-leaving decisions have been reported in dorsal anterior cingulate cortex (dACC), whereas competition via mutual inhibition in ventromedial prefrontal cortex (vmPFC) is thought to underlie value-guided choice. Here, we show that the balance between cortical excitation and inhibition (E/I balance), measured by the ratio of GABA and glutamate concentrations, plays a dissociable role for the two kinds of decisions. Patch-leaving decision behaviour relates to E/I balance in dACC. In contrast, value-guided decision-making relates to E/I balance in vmPFC. These results support mechanistic accounts of value-guided choice and provide evidence for a role of dACC E/I balance in patch-leaving decisions.

Adaptive coding of action values in the human rostral cingulate zone.

Correctly selecting appropriate actions in an uncertain environment requires gathering experience about the available actions by sampling them over several trials. Recent findings suggest that the human rostral cingulate zone (RCZ) is important for the integration of extended action-outcome associations across multiple trials and in coding the subjective value of each action. During functional magnetic resonance imaging, healthy volunteers performed two versions of a probabilistic reversal learning task with high (HP) or low (LP) reward probabilities that required them to integrate action-outcome relations over lower or higher numbers of trials, respectively. In the HP session, subjects needed fewer trials to adjust their behavior in response to a reversal of response-reward contingencies. Similarly, the learning rate derived from a reinforcement learning model was higher in the HP condition. This was accompanied by a stronger response of the RCZ to negative feedback upon reversals in the HP condition. Furthermore, RCZ activity related to negative reward prediction errors varied as a function of the learning rate, which determines to what extent the prediction error is used to update action values. These data show that RCZ responses vary as a function of the information content provided by the environment. The more likely a negative event indicates the need for behavioral adaptations, the more prominent is the response of the RCZ. Thus, both the window of trials over which reinforcement information is integrated and adjustment of action values in the RCZ covary with the stochastics of the environment.

Dopamine DRD2 polymorphism alters reversal learning and associated neural activity.

In humans, presence of an A1 allele of the DRD2/ANKK1-TaqIa polymorphism is associated with reduced expression of dopamine (DA) D(2) receptors in the striatum. Recently, it was observed that carriers of the A1 allele (A1+ subjects) showed impaired learning from negative feedback in a reinforcement learning task. Here, using functional MRI (fMRI), we investigated carriers and noncarriers of the A1 allele while they performed a probabilistic reversal learning task. A1+ subjects showed subtle deficits in reversal learning. In particular, these deficits consisted of an impairment in sustaining the newly rewarded response after a reversal and in a generally decreased tendency to stick with a rewarded response. Both genetic groups showed increased fMRI signal in response to negative feedback in the rostral cingulate zone (RCZ) and anterior insula. Negative feedback that incurred a change in behavior additionally engaged the ventral striatum and a region of the midbrain consistent with the location of dopaminergic cell groups. The response of the RCZ to negative feedback increased as a function of preceding negative feedback. However, this graded response was not observed in the A1+ group. Furthermore, the A1+ group also showed diminished recruitment of the right ventral striatum and the right lateral orbitofrontal cortex (lOFC) during reversals. Together, these results suggest that a genetically driven reduction in DA D(2) receptors leads to deficient feedback integration in RCZ. This, in turn, was accompanied by impaired recruitment of the ventral striatum and the right lOFC during reversals, which might explain the behavioral differences between the genetic groups.