Investigating associations of delay discounting with brain structure, working memory, and episodic memory.

INTRODUCTION: Delay discounting (DD), the preference for smaller and sooner rewards over larger and later ones, is an important behavioural phenomenon for daily functioning of increasing interest within psychopathology. The neurobiological mechanisms behind DD are not well understood and the literature on structural correlates of DD shows inconsistencies. METHODS: Here we leveraged a large openly available dataset (n = 1196) to investigate associations with memory performance and gray and white matter correlates of DD using linked independent component analysis. RESULTS: Greater DD was related to smaller anterior temporal gray matter volume. Associations of DD with total cortical volume, subcortical volumes, markers of white matter microscopic organization, working memory, and episodic memory scores were not significant after controlling for education and income. CONCLUSION: Effects of size comparable to the one we identified would be unlikely to be replicated with sample sizes common in many previous studies in this domain, which may explain the incongruities in the literature. The paucity and small size of the effects detected in our data underscore the importance of using large samples together with methods that accommodate their statistical structure and appropriate control for confounders, as well as the need to devise paradigms with improved task parameter reliability in studies relating brain structure and cognitive abilities with DD.

Dynamic diversity is the answer to proxy failure.

We argue that a diverse and dynamic pool of agents mitigates proxy failure. Proxy modularity plays a key role in the ongoing production of diversity. We review examples from a range of scales.

Neural signatures of arbitration between Pavlovian and instrumental action selection.

Pavlovian associations drive approach towards reward-predictive cues, and avoidance of punishment-predictive cues. These associations "misbehave" when they conflict with correct instrumental behavior. This raises the question of how Pavlovian and instrumental influences on behavior are arbitrated. We test a computational theory according to which Pavlovian influence will be stronger when inferred controllability of outcomes is low. Using a model-based analysis of a Go/NoGo task with human subjects, we show that theta-band oscillatory power in frontal cortex tracks inferred controllability, and that these inferences predict Pavlovian action biases. Functional MRI data revealed an inferior frontal gyrus correlate of action probability and a ventromedial prefrontal correlate of outcome valence, both of which were modulated by inferred controllability.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [11C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.

Motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function.

Dopaminergic neurotransmission plays a pivotal role in appetitively motivated behavior in mammals, including humans. Notably, action and valence are not independent in motivated tasks, and it is particularly difficult for humans to learn the inhibition of an action to obtain a reward. We have previously observed that the carriers of the DRD2/ANKK1 TaqIA A1 allele, that has been associated with reduced striatal dopamine D2 receptor expression, showed a diminished learning performance when required to learn response inhibition to obtain rewards, a finding that was replicated in two independent cohorts. With our present study, we followed two aims: first, we aimed to replicate our finding on the DRD2/ANKK1 TaqIA polymorphism in a third independent cohort (N = 99) and to investigate the nature of the genetic effects more closely using trial-by-trial behavioral analysis and computational modeling in the combined dataset (N = 281). Second, we aimed to assess a potentially modulatory role of prefrontal dopamine availability, using the widely studied COMT Val108/158Met polymorphism as a proxy. We first report a replication of the above mentioned finding. Interestingly, after combining all three cohorts, exploratory analyses regarding the COMT Val108/158Met polymorphism suggest that homozygotes for the Met allele, which has been linked to higher prefrontal dopaminergic tone, show a lower learning bias. Our results corroborate the importance of genetic variability of the dopaminergic system in individual learning differences of action-valence interaction and, furthermore, suggest that motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning.

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study.

Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models.

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

Latent-Profile Analysis Reveals Behavioral and Brain Correlates of Dopamine-Cognition Associations.

Evidence suggests that associations between the neurotransmitter dopamine and cognition are nonmonotonic and open to modulation by various other factors. The functional implications of a given level of dopamine may therefore differ from person to person. By applying latent-profile analysis to a large (n = 181) sample of adults aged 64-68 years, we probabilistically identified 3 subgroups that explain the multivariate associations between dopamine D2/3R availability (probed with 11C-raclopride-PET, in cortical, striatal, and hippocampal regions) and cognitive performance (episodic memory, working memory, and perceptual speed). Generally, greater receptor availability was associated with better cognitive performance. However, we discovered a subgroup of individuals for which high availability, particularly in striatum, was associated with poor performance, especially for working memory. Relative to the rest of the sample, this subgroup also had lower education, higher body-mass index, and lower resting-state connectivity between caudate nucleus and dorsolateral prefrontal cortex. We conclude that a smaller subset of individuals induces a multivariate non-linear association between dopamine D2/3R availability and cognitive performance in this group of older adults, and discuss potential reasons for these differences that await further empirical scrutiny.

Parsing the Role of the Hippocampus in Approach-Avoidance Conflict.

The hippocampus plays a central role in the approach-avoidance conflict that is central to the genesis of anxiety. However, its exact functional contribution has yet to be identified. We designed a novel gambling task that generated approach-avoidance conflict while controlling for spatial processing. We fit subjects' behavior using a model that quantified the subjective values of choice options, and recorded neural signals using functional magnetic resonance imaging (fMRI). Distinct functional signals were observed in anterior hippocampus, with inferior hippocampus selectively recruited when subjects rejected a gamble, to a degree that covaried with individual differences in anxiety. The superior anterior hippocampus, in contrast, uniquely demonstrated value signals that were potentiated in the context of approach-avoidance conflict. These results implicate the anterior hippocampus in behavioral avoidance and choice monitoring, in a manner relevant to understanding its role in anxiety. Our findings highlight interactions between subregions of the hippocampus as an important focus for future study.

BOLD Variability is Related to Dopaminergic Neurotransmission and Cognitive Aging.

Dopamine (DA) losses are associated with various aging-related cognitive deficits. Typically, higher moment-to-moment brain signal variability in large-scale patterns of voxels in neocortical regions is linked to better cognitive performance and younger adult age, yet the physiological mechanisms regulating brain signal variability are unknown. We explored the relationship among adult age, DA availability, and blood oxygen level-dependent (BOLD) signal variability, while younger and older participants performed a spatial working memory (SWM) task. We quantified striatal and extrastriatal DA D1 receptor density with [(11)C]SCH23390 and positron emission tomography in all participants. We found that BOLD variability in a neocortical region was negatively related to age and positively related to SWM performance. In contrast, BOLD variability in subcortical regions and bilateral hippocampus was positively related to age and slower responses, and negatively related to D1 density in caudate and dorsolateral prefrontal cortex. Furthermore, BOLD variability in neocortical regions was positively associated with task-related disengagement of the default-mode network, a network whose activation needs to be suppressed for efficient SWM processing. Our results show that age-related DA losses contribute to changes in brain signal variability in subcortical regions and suggest a potential mechanism, by which neocortical BOLD variability supports cognitive performance.

Model-Based Reasoning in Humans Becomes Automatic with Training.

Model-based and model-free reinforcement learning (RL) have been suggested as algorithmic realizations of goal-directed and habitual action strategies. Model-based RL is more flexible than model-free but requires sophisticated calculations using a learnt model of the world. This has led model-based RL to be identified with slow, deliberative processing, and model-free RL with fast, automatic processing. In support of this distinction, it has recently been shown that model-based reasoning is impaired by placing subjects under cognitive load--a hallmark of non-automaticity. Here, using the same task, we show that cognitive load does not impair model-based reasoning if subjects receive prior training on the task. This finding is replicated across two studies and a variety of analysis methods. Thus, task familiarity permits use of model-based reasoning in parallel with other cognitive demands. The ability to deploy model-based reasoning in an automatic, parallelizable fashion has widespread theoretical implications, particularly for the learning and execution of complex behaviors. It also suggests a range of important failure modes in psychiatric disorders.

Basal Ganglia Activity Mirrors a Benefit of Action and Reward on Long-Lasting Event Memory.

The expectation of reward is known to enhance a consolidation of long-term memory for events. We tested whether this effect is driven by positive valence or action requirements tied to expected reward. Using a functional magnetic resonance imaging (fMRI) paradigm in young adults, novel images predicted gain or loss outcomes, which in turn were either obtained or avoided by action or inaction. After 24 h, memory for these images reflected a benefit of action as well as a congruence of action requirements and valence, namely, action for reward and inaction for avoidance. fMRI responses in the hippocampus, a region known to be critical for long-term memory function, reflected the anticipation of inaction. In contrast, activity in the putamen mirrored the congruence of action requirement and valence, whereas other basal ganglia regions mirrored overall action benefits on long-lasting memory. The findings indicate a novel type of functional division between the hippocampus and the basal ganglia in the motivational regulation of long-term memory consolidation, which favors remembering events that are worth acting for.

Anterior cingulate cortex instigates adaptive switches in choice by integrating immediate and delayed components of value in ventromedial prefrontal cortex.

Actions can lead to an immediate reward or punishment and a complex set of delayed outcomes. Adaptive choice necessitates the brain track and integrate both of these potential consequences. Here, we designed a sequential task whereby the decision to exploit or forego an available offer was contingent on comparing immediate value and a state-dependent future cost of expending a limited resource. Crucially, the dynamics of the task demanded frequent switches in policy based on an online computation of changing delayed consequences. We found that human subjects choose on the basis of a near-optimal integration of immediate reward and delayed consequences, with the latter computed in a prefrontal network. Within this network, anterior cingulate cortex (ACC) was dynamically coupled to ventromedial prefrontal cortex (vmPFC) when adaptive switches in choice were required. Our results suggest a choice architecture whereby interactions between ACC and vmPFC underpin an integration of immediate and delayed components of value to support flexible policy switching that accommodates the potential delayed consequences of an action.

Pharmacological dissociation of novelty responses in the human brain.

Repeated processing of the same information is associated with decreased neuronal responses, termed repetition suppression (RS). Although RS effects (i.e., the difference in activity between novel and repeated stimuli) have been demonstrated within several brain regions, such as the medial temporal lobe, their precise neural mechanisms still remain unclear. Here, we used functional magnetic resonance imaging together with psychopharmacology in 48 healthy human subjects, demonstrating that RS effects within the mesolimbic system are differentially modulated by cholinergic and dopaminergic stimulation. The dopamine precursor levodopa (100 mg) attenuated RS within the hippocampus, parahippocampal cortex, and substantia nigra/ventral tegmental area, and the degree of this reduction correlated with recognition memory performance 24 h later. The acetylcholinesterase inhibitor galantamine (8 mg), in contrast, reversed RS into repetition enhancement, showing no relationship to subsequent recognition memory. This suggests that novelty sensitive neural populations of the mesolimbic system can dynamically shift their responses depending on the balance of cholinergic and dopaminergic neurotransmission, and these shifts can influence memory retention.

Action controls dopaminergic enhancement of reward representations.

Dopamine is widely observed to signal anticipation of future rewards and thus thought to be a key contributor to affectively charged decision making. However, the experiments supporting this view have not dissociated rewards from the actions that lead to, or are occasioned by, them. Here, we manipulated dopamine pharmacologically and examined the effect on a task that explicitly dissociates action and reward value. We show that dopamine enhanced the neural representation of rewarding actions, without significantly affecting the representation of reward value as such. Thus, increasing dopamine levels with levodopa selectively boosted striatal and substantia nigra/ventral tegmental representations associated with actions leading to reward, but not with actions leading to the avoidance of punishment. These findings highlight a key role for dopamine in the generation of appetitively motivated actions.

Preparing for selective inhibition within frontostriatal loops.

Action inhibition can globally prevent all motor output or selectively cancel specific actions during concurrent motor output. Here we examine the behavioral and neural basis of selective inhibition focusing on the role of preparation. In 18 healthy human participants we manipulated the extent to which they could prepare for selective inhibition by providing or withholding information on what actions might need to be stopped. We show that, on average, information improves both speed and selectivity of inhibition. Functional magnetic resonance imaging data show that preparation for selective inhibition engages the inferior frontal gyrus, supplementary motor area, and striatum. Examining interindividual differences, we find the benefit of proactive control to speed and selectivity of inhibition trade off against each other, such that an improvement in stopping speed leads to a deterioration of selectivity of inhibition, and vice versa. This trade-off is implemented through engagement of the dorsolateral prefrontal cortex and putamen. Our results suggest proactive selective inhibition is implemented within frontostriatal structures, and we provide evidence that a speed-selectivity trade-off might underlie a range of findings reported previously.

Effort and valuation in the brain: the effects of anticipation and execution.

Neural representations of the effort deployed in performing actions, and the valence of the outcomes they yield, form the foundation of action choice. To discover whether brain areas represent effort and outcome valence together or if they represent one but not the other, we examined these variables in an explicitly orthogonal way. We did this by asking human subjects to exert one of two levels of effort to improve their chances of either winning or avoiding the loss of money. Subjects responded faster both when exerting greater effort and when exerting effort in anticipation of winning money. Using fMRI, we inspected BOLD responses during anticipation (before any action was executed) and when the outcome was delivered. In this way, we indexed BOLD signals associated with an anticipated need to exert effort and its affective consequences, as well as the effect of executed effort on the representation of outcomes. Anterior cingulate cortex and dorsal striatum (dorsal putamen) signaled the anticipation of effort independently of the prospect of winning or losing. Activity in ventral striatum (ventral putamen) was greater for better-than-expected outcomes compared with worse-than-expected outcomes, an effect attenuated in the context of having exerted greater effort. Our findings provide evidence that neural representations of anticipated actions are sensitive to the expected demands, but not to the expected value of their consequence, whereas representations of outcome value are discounted by exertion, commensurate with an integration of cost and benefit so as to approximate net value.