Dynamic diversity is the answer to proxy failure.

We argue that a diverse and dynamic pool of agents mitigates proxy failure. Proxy modularity plays a key role in the ongoing production of diversity. We review examples from a range of scales.

Dopamine release in human associative striatum during reversal learning.

The dopaminergic system is firmly implicated in reversal learning but human measurements of dopamine release as a correlate of reversal learning success are lacking. Dopamine release and hemodynamic brain activity in response to unexpected changes in action-outcome probabilities are here explored using simultaneous dynamic [11C]Raclopride PET-fMRI and computational modelling of behavior. When participants encounter reversed reward probabilities during a card guessing game, dopamine release is observed in associative striatum. Individual differences in absolute reward prediction error and sensitivity to errors are associated with peak dopamine receptor occupancy. The fMRI response to perseverance errors at the onset of a reversal spatially overlap with the site of dopamine release. Trial-by-trial fMRI correlates of absolute prediction errors show a response in striatum and association cortices, closely overlapping with the location of dopamine release, and separable from a valence signal in ventral striatum. The results converge to implicate striatal dopamine release in associative striatum as a central component of reversal learning, possibly signifying the need for increased cognitive control when new stimuli-responses should be learned.

Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning.

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

Biphasic patterns of age-related differences in dopamine D1 receptors across the adult lifespan.

Age-related alterations in D1-like dopamine receptor (D1DR) have distinct implications for human cognition and behavior during development and aging, but the timing of these periods remains undefined. Enabled by a large sample of in vivo assessments (n = 180, age 20 to 80 years of age, 50% female), we discover that age-related D1DR differences pivot at approximately 40 years of age in several brain regions. Focusing on the most age-sensitive dopamine-rich region, we observe opposing pre- and post-forties interrelations among caudate D1DR, cortico-striatal functional connectivity, and memory. Finally, particularly caudate D1DR differences in midlife and beyond, but not in early adulthood, associate with manifestation of white matter lesions. The present results support a model by which excessive dopamine modulation in early adulthood and insufficient modulation in aging are deleterious to brain function and cognition, thus challenging a prevailing view of monotonic D1DR function across the adult lifespan.

Investigating associations of delay discounting with brain structure, working memory, and episodic memory.

INTRODUCTION: Delay discounting (DD), the preference for smaller and sooner rewards over larger and later ones, is an important behavioural phenomenon for daily functioning of increasing interest within psychopathology. The neurobiological mechanisms behind DD are not well understood and the literature on structural correlates of DD shows inconsistencies. METHODS: Here we leveraged a large openly available dataset (n = 1196) to investigate associations with memory performance and gray and white matter correlates of DD using linked independent component analysis. RESULTS: Greater DD was related to smaller anterior temporal gray matter volume. Associations of DD with total cortical volume, subcortical volumes, markers of white matter microscopic organization, working memory, and episodic memory scores were not significant after controlling for education and income. CONCLUSION: Effects of size comparable to the one we identified would be unlikely to be replicated with sample sizes common in many previous studies in this domain, which may explain the incongruities in the literature. The paucity and small size of the effects detected in our data underscore the importance of using large samples together with methods that accommodate their statistical structure and appropriate control for confounders, as well as the need to devise paradigms with improved task parameter reliability in studies relating brain structure and cognitive abilities with DD.

A link between age, affect, and predictions?

The prevalence of depressive symptoms decreases from late adolescence to middle age adulthood. Furthermore, despite significant losses in motor and cognitive functioning, overall emotional well-being tends to increase with age, and a bias to positive information has been observed multiple times. Several causes have been discussed for this age-related development, such as improvement in emotion regulation, less regret, and higher socioeconomic status. Here, we explore a further explanation. Our minds host mental models that generate predictions about forthcoming events to successfully interact with our physical and social environment. To keep these models faithful, the difference between the predicted and the actual event, that is, the prediction error, is computed. We argue that prediction errors are attenuated in the middle age and older mind, which, in turn, may translate to less negative affect, lower susceptibility to affective disorders, and possibly, to a bias to positive information. Our proposal is primarily linked to perceptual inferences, but may hold as well for higher-level, cognitive, and emotional forms of error processing.

Dopamine and reward-related vigor in younger and older adults.

Vigor reflects how motivated people are to respond to stimuli. We previously showed that, on average, humans are more vigorous when a higher rate of reward is available, and that this relationship is modulated by the dopamine precursor levodopa. Dopamine signaling and probabilistic reward learning deteriorate across the adult life span, and thus, the relationship between vigor and reward may also change in aging. We tested this assertion and assessed whether it correlates with D1 dopamine receptor availability, measured using Positron Emission Tomography. We registered response times of 30 older and 30 younger participants during an oddball discrimination task where rewards varied systematically between trials. The average reward rate had a similar impact on vigor in both age groups. There was a weak positive association between ventral striatal dopamine receptor availability and the effect of average reward rate on response time. Overall, the effect of reward on response vigor was similar in younger and older adults, and weakly correlated with dopamine D1 receptor availability.

Enhanced Instructed Fear Learning in Delusion-Proneness.

Psychosis is associated with distorted perceptions and deficient bottom-up learning such as classical fear conditioning. This has been interpreted as reflecting imprecise priors in low-level predictive coding systems. Paradoxically, overly strong beliefs, such as overvalued beliefs and delusions, are also present in psychosis-associated states. In line with this, research has suggested that patients with psychosis and associated phenotypes rely more on high-order priors to interpret perceptual input. In this behavioural and fMRI study we studied two types of fear learning, i.e., instructed fear learning mediated by verbal suggestions about fear contingencies and classical fear conditioning mediated by low level associative learning, in delusion proneness-a trait in healthy individuals linked to psychotic disorders. Subjects were shown four faces out of which two were coupled with an aversive stimulation (CS+) while two were not (CS-) in a fear conditioning procedure. Before the conditioning, subjects were informed about the contingencies for two of the faces of each type, while no information was given for the two other faces. We could thereby study the effect of both classical fear conditioning and instructed fear learning. Our main outcome variable was evaluative rating of the faces. Simultaneously, fMRI-measurements were performed to study underlying mechanisms. We postulated that instructed fear learning, measured with evaluative ratings, is stronger in psychosis-related phenotypes, in contrast to classical fear conditioning that has repeatedly been shown to be weaker in these groups. In line with our hypothesis, we observed significantly larger instructed fear learning on a behavioural level in delusion-prone individuals (n = 20) compared to non-delusion-prone subjects (n = 23; n = 20 in fMRI study). Instructed fear learning was associated with a bilateral activation of lateral orbitofrontal cortex that did not differ significantly between groups. However, delusion-prone subjects showed a stronger functional connectivity between right lateral orbitofrontal cortex and regions processing fear and pain. Our results suggest that psychosis-related states are associated with a strong instructed fear learning in addition to previously reported weak classical fear conditioning. Given the similarity between nocebo paradigms and instructed fear learning, our results also have an impact on understanding why nocebo effects differ between individuals.

Motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function.

Dopaminergic neurotransmission plays a pivotal role in appetitively motivated behavior in mammals, including humans. Notably, action and valence are not independent in motivated tasks, and it is particularly difficult for humans to learn the inhibition of an action to obtain a reward. We have previously observed that the carriers of the DRD2/ANKK1 TaqIA A1 allele, that has been associated with reduced striatal dopamine D2 receptor expression, showed a diminished learning performance when required to learn response inhibition to obtain rewards, a finding that was replicated in two independent cohorts. With our present study, we followed two aims: first, we aimed to replicate our finding on the DRD2/ANKK1 TaqIA polymorphism in a third independent cohort (N = 99) and to investigate the nature of the genetic effects more closely using trial-by-trial behavioral analysis and computational modeling in the combined dataset (N = 281). Second, we aimed to assess a potentially modulatory role of prefrontal dopamine availability, using the widely studied COMT Val108/158Met polymorphism as a proxy. We first report a replication of the above mentioned finding. Interestingly, after combining all three cohorts, exploratory analyses regarding the COMT Val108/158Met polymorphism suggest that homozygotes for the Met allele, which has been linked to higher prefrontal dopaminergic tone, show a lower learning bias. Our results corroborate the importance of genetic variability of the dopaminergic system in individual learning differences of action-valence interaction and, furthermore, suggest that motivational learning biases are differentially modulated by genetic determinants of striatal and prefrontal dopamine function.

Decision-making ability, psychopathology, and brain connectivity.

Decision-making is a cognitive process of central importance for the quality of our lives. Here, we ask whether a common factor underpins our diverse decision-making abilities. We obtained 32 decision-making measures from 830 young people and identified a common factor that we call "decision acuity," which was distinct from IQ and reflected a generic decision-making ability. Decision acuity was decreased in those with aberrant thinking and low general social functioning. Crucially, decision acuity and IQ had dissociable brain signatures, in terms of their associated neural networks of resting-state functional connectivity. Decision acuity was reliably measured, and its relationship with functional connectivity was also stable when measured in the same individuals 18 months later. Thus, our behavioral and brain data identify a new cognitive construct that underpins decision-making ability across multiple domains. This construct may be important for understanding mental health, particularly regarding poor social function and aberrant thought patterns.

Overcoming Pavlovian bias in semantic space.

Action is invigorated in the presence of reward-predicting stimuli and inhibited in the presence of punishment-predicting stimuli. Although valuable as a heuristic, this Pavlovian bias can also lead to maladaptive behaviour and is implicated in addiction. Here we explore whether Pavlovian bias can be overcome through training. Across five experiments, we find that Pavlovian bias is resistant to unlearning under most task configurations. However, we demonstrate that when subjects engage in instrumental learning in a verbal semantic space, as opposed to a motoric space, not only do they exhibit the typical Pavlovian bias, but this Pavlovian bias diminishes with training. Our results suggest that learning within the semantic space is necessary, but not sufficient, for subjects to unlearn their Pavlovian bias, and that other task features, such as gamification and spaced stimulus presentation may also be necessary. In summary, we show that Pavlovian bias, whilst robust, is susceptible to change with experience, but only under specific environmental conditions.

Neural signatures of arbitration between Pavlovian and instrumental action selection.

Pavlovian associations drive approach towards reward-predictive cues, and avoidance of punishment-predictive cues. These associations "misbehave" when they conflict with correct instrumental behavior. This raises the question of how Pavlovian and instrumental influences on behavior are arbitrated. We test a computational theory according to which Pavlovian influence will be stronger when inferred controllability of outcomes is low. Using a model-based analysis of a Go/NoGo task with human subjects, we show that theta-band oscillatory power in frontal cortex tracks inferred controllability, and that these inferences predict Pavlovian action biases. Functional MRI data revealed an inferior frontal gyrus correlate of action probability and a ventromedial prefrontal correlate of outcome valence, both of which were modulated by inferred controllability.

Role of dopamine and gray matter density in aging effects and individual differences of functional connectomes.

With increasing age, functional connectomes become dissimilar across normal individuals, reflecting heterogenous aging effects on functional connectivity (FC). We investigated the distribution of these effects across the connectome and their relationship with age-related differences in dopamine (DA) D1 receptor availability and gray matter density (GMD). With this aim, we determined aging effects on mean and interindividual variance of FC using fMRI in 30 younger and 30 older healthy subjects and mapped the contribution of each connection to the patterns of age-related similarity loss. Aging effects on mean FC accounted mainly for the dissimilarity between connectomes of younger and older adults, and were related, across brain regions, to aging effects on DA D1 receptor availability. Aging effects on the variance of FC indicated a global increase in variance with advancing age, explained connectome dissimilarity among older subjects and were related to aging effects on variance of GMD. The relationship between aging and the similarity of connectomes can thus be partly explained by age differences in DA modulation and gray matter structure.

Learning in anticipation of reward and punishment: perspectives across the human lifespan.

Learning to act to receive reward and to withhold to avoid punishment has been found to be easier than learning the opposite contingencies in young adults. To what extent this type of behavioral adaptation might develop during childhood and adolescence and differ during aging remains unclear. We therefore tested 247 healthy individuals across the human life span (7-80 years) with an orthogonalized valenced go/no-go learning task. Computational modeling revealed that peak performance in young adults was attributable to greater sensitivity to both reward and punishment. However, in children and adolescents, we observed an increased bias toward action but not reward sensitivity. By contrast, reduced learning in midlife and older adults was accompanied by decreased reward sensitivity and especially punishment sensitivity along with an age-related increase in the Pavlovian bias. These findings reveal distinct motivation-dependent learning capabilities across the human life span, which cannot be probed using conventional go/reward no-go/punishment style paradigms that have important implications in lifelong education.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning.

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models.

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study.

Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [11C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.