Striatal dopamine integrates cost, benefit, and motivation.

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Multiplexed action-outcome representation by striatal striosome-matrix compartments detected with a mouse cost-benefit foraging task.

Learning about positive and negative outcomes of actions is crucial for survival and underpinned by conserved circuits including the striatum. How associations between actions and outcomes are formed is not fully understood, particularly when the outcomes have mixed positive and negative features. We developed a novel foraging ('bandit') task requiring mice to maximize rewards while minimizing punishments. By 2-photon Ca++ imaging, we monitored activity of visually identified anterodorsal striatal striosomal and matrix neurons. We found that action-outcome associations for reward and punishment were encoded in parallel in partially overlapping populations. Single neurons could, for one action, encode outcomes of opposing valence. Striosome compartments consistently exhibited stronger representations of reinforcement outcomes than matrix, especially for high reward or punishment prediction errors. These findings demonstrate multiplexing of action-outcome contingencies by single identified striatal neurons and suggest that striosomal neurons are particularly important in action-outcome learning.

An association between prediction errors and risk-seeking: Theory and behavioral evidence.

Reward prediction errors (RPEs) and risk preferences have two things in common: both can shape decision making behavior, and both are commonly associated with dopamine. RPEs drive value learning and are thought to be represented in the phasic release of striatal dopamine. Risk preferences bias choices towards or away from uncertainty; they can be manipulated with drugs that target the dopaminergic system. Based on the common neural substrate, we hypothesize that RPEs and risk preferences are linked on the level of behavior as well. Here, we develop this hypothesis theoretically and test it empirically. First, we apply a recent theory of learning in the basal ganglia to predict how RPEs influence risk preferences. We find that positive RPEs should cause increased risk-seeking, while negative RPEs should cause risk-aversion. We then test our behavioral predictions using a novel bandit task in which value and risk vary independently across options. Critically, conditions are included where options vary in risk but are matched for value. We find that our prediction was correct: participants become more risk-seeking if choices are preceded by positive RPEs, and more risk-averse if choices are preceded by negative RPEs. These findings cannot be explained by other known effects, such as nonlinear utility curves or dynamic learning rates.

Peer influence, Frontostriatal connectivity, and delay discounting in African American emerging adults.

Prior research has demonstrated the importance of delay discounting in adverse health behaviors, such as addiction, attention deficit hyperactivity disorder, risk taking, and obesity. Nevertheless, the functional connectivity of neural circuitry associated with delay discounting and the ways in which the social environment may influence frontostriatal connectivity remain largely unknown, particularly in African Americans. Building on recent literature implicating frontostriatal connectivity during active delay discounting decision making and at rest, we used functional magnetic resonance imaging to assess the association between delay discounting and frontostriatal resting state connectivity (rsFC). We also examined the capacity of social relationships with parents and peers to longitudinally predict frontostriatal rsFC. The study cohort was composed of 91 rural African American emerging adults followed over a 6-year period. Greater (i.e., more positive) frontostriatal rsFC was associated with decreased delay discounting (i.e., less impulsive decision making). In addition, peer relationships at ages 20 and 21 significantly predicted frontostriatal rsFC at age 25 above and beyond parental influence. A significant indirect effect of peer affiliation on delay discounting through frontostriatal rsFC also emerged. These results indicate a role of frontostriatal connectivity in delay discounting decision making and highlight peers' unique influence on decision making behaviors through frontostriatal rsFC during emerging adulthood.

Assessment of fatigue-related biochemical alterations in a rat swimming model under hypoxia.

It is well known that exercise-induced fatigue is exacerbated following hypoxia exposure and may arise from central and/or peripheral mechanisms. To assess the relative contribution of peripheral and central factors to exercise-induced fatigue under hypoxia, a rat model of fatigue by a bout of exhaustive swimming was established and fatigue-related biochemical changes in normoxic and severe hypoxic conditions were compared. Rats were randomly divided into four groups: normoxia resting (NR), exhaustive swimming (NE), hypoxia resting (HR) and exhaustive swimming (HE). The swimming time to exhaustion with a weight equal to 2.5% of their body weight reduced under hypoxia. There were lower blood lactate levels, lower gastrocnemius pAMPK/AMPK ratios and higher gastrocnemius glycogen contents in the HE than in the NE groups, which all suggested a lower degree of peripheral fatigue in the HE group than in the NE group. Meanwhile, there was a significant increase in striatal 3,4-dihydroxyphenylacetic acid (DOPAC) caused by exhaustive swimming under normoxia, whereas this increase was almost blunted under severe hypoxia, indicating that hypoxia might exacerbate exercise-induced central fatigue. These biochemical changes suggest that from normoxia to severe hypoxia, the relative contribution of peripheral and central factors to exercise-induced fatigue alters, and central fatigue may play a predominant role in the decline in exercise performance under hypoxia.

The Striatum's Role in Executing Rational and Irrational Economic Behaviors.

The striatum is a critical component of the brain that controls motor, reward, and executive function. This ancient and phylogenetically conserved structure forms a central hub where rapid instinctive, reflexive movements and behaviors in response to sensory stimulation or the retrieval of emotional memory intersect with slower planned motor movements and rational behaviors. This review emphasizes two distinct pathways that begin in the thalamus and converge in the striatum to differentially affect movements, behaviors, and decision making. The convergence of excitatory glutamatergic activity from the thalamus and cortex, along with dopamine release in response to novel stimulation, provide the basis for motor learning, reward seeking, and habit formation. We outline how the rules derived through research on neural pathways may enhance the predictability of reflexive actions and rational responses studied in behavioral economics.

Dissociable contributions of dorsal and ventral striatal regions on a rodent cost/benefit decision-making task requiring cognitive effort.

Cognitive effort is a ubiquitous process, yet surprisingly little is known about the brain mechanisms responsible for evaluating it. Here, we utilize the rat Cognitive Effort Task (rCET) to probe the striatum's role in deciding between options that vary in the amount of cognitive effort required for success. In the rCET, animals choose to perform either an easy trial, in which the attentional demand is low but the potential reward is small, or a difficult trial which is more attentionally demanding but can yield twice the sugar pellets. Twenty-six male Long Evans rats were trained on the rCET and the effects of pharmacologically inactivating the dorsomedial striatum (DMS) and core region of the nucleus accumbens were determined. Temporary inactivation of the DMS decreased all animals' choice of the high-effort, high-reward option, impaired attentional accuracy, and robustly increased premature responding without impairing general indices of motor ability. The DMS therefore appears necessary for the integration of cognitive signals required for optimal performance. In stark contrast, following temporary inactivation of the ventral striatum, subjects were fundamentally unable to perform the task, as reflected by a drastic decrease in the number of trials initiated and an increase in omitted responses. Together, these data suggest the striatum is likely part of a larger cortico-limbic-striatal network whose function is to optimize decisions requiring cognitive effort costs, at least in the attentional domain, and that striatal subregions have dissociable roles in the adjudication and application of this form of cognitive effort.

The costs and benefits of brain dopamine for cognitive control.

Cognitive control helps us attain our goals by resisting distraction and temptations. Dopaminergic drugs are well known to enhance cognitive control. However, there is great variability in the effects of dopaminergic drugs across different contexts, with beneficial effects on some tasks but detrimental effects on other tasks. The mechanisms underlying this variability across cognitive task demands remain unclear. I aim to elucidate this across-task variability in dopaminergic drug efficacy by going beyond classic models that emphasize the importance of dopamine in the prefrontal cortex for cognitive control and working memory. To this end, I build on recent advances in cognitive neuroscience that highlight a role for dopamine in cost-benefit decision making. Specifically, I reconceptualize cognitive control as involving not just prefrontal dopamine but also modulation of cost-benefit decision making by striatal dopamine. This approach will help us understand why we sometimes fail to (choose to) exert cognitive control while also identifying mechanistic factors that predict dopaminergic drug effects on cognitive control. WIREs Cogn Sci 2016, 7:317-329. doi: 10.1002/wcs.1401 For further resources related to this article, please visit the WIREs website.

Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum.

Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the "win-stay, lose-switch" strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences.

Over a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naive, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate an integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates.

Psychosocial versus physiological stress - Meta-analyses on deactivations and activations of the neural correlates of stress reactions.

Stress is present in everyday life in various forms and situations. Two stressors frequently investigated are physiological and psychosocial stress. Besides similar subjective and hormonal responses, it has been suggested that they also share common neural substrates. The current study used activation-likelihood-estimation meta-analysis to test this assumption by integrating results of previous neuroimaging studies on stress processing. Reported results are cluster-level FWE corrected. The inferior frontal gyrus (IFG) and the anterior insula (AI) were the only regions that demonstrated overlapping activation for both stressors. Analysis of physiological stress showed consistent activation of cognitive and affective components of pain processing such as the insula, striatum, or the middle cingulate cortex. Contrarily, analysis across psychosocial stress revealed consistent activation of the right superior temporal gyrus and deactivation of the striatum. Notably, parts of the striatum appeared to be functionally specified: the dorsal striatum was activated in physiological stress, whereas the ventral striatum was deactivated in psychosocial stress. Additional functional connectivity and decoding analyses further characterized this functional heterogeneity and revealed higher associations of the dorsal striatum with motor regions and of the ventral striatum with reward processing. Based on our meta-analytic approach, activation of the IFG and the AI seems to indicate a global neural stress reaction. While physiological stress activates a motoric fight-or-flight reaction, during psychosocial stress attention is shifted towards emotion regulation and goal-directed behavior, and reward processing is reduced. Our results show the significance of differentiating physiological and psychosocial stress in neural engagement. Furthermore, the assessment of deactivations in addition to activations in stress research is highly recommended.

Age-related prefrontal impairments implicate deficient prediction of future reward in older adults.

Foresighted decision-making depends on the ability to learn the value of future outcomes and the sequential choices necessary to achieve them. Using a 3-stage Markov decision task and functional magnetic resonance imaging, we investigated age differences in the ability to extract state transition structures while learning to predict future reward. In younger adults learning was associated with enhanced activity in the prefrontal cortex (PFC). In older adults (OA) we found no evidence for PFC recruitment. However, high-performing OA showed enhanced striatal activity, suggesting that they may engage in a model-free (experience-based) learning strategy. Change point analyses revealed that in younger adults learning was characterized by distinct and abrupt shifts in PFC activity, which were predictive of behavioral change points. In OA PFC activity was less pronounced and not predictive of behavior. Our findings suggest that age-related impairments in learning future reward value can be attributed to a deficit in extracting sequential state transition structures. This deficit may lead to myopic decisions in OA if contextual information has to be temporally integrated.

Direct in vivo assessment of human stem cell graft-host neural circuits.

Despite the potential of stem cell-derived neural transplants for treating intractable neurological diseases, the global effects of a transplant's electrical activity on host circuitry have never been measured directly, preventing the systematic optimization of such therapies. Here, we overcome this problem by combining optogenetics, stem cell biology, and neuroimaging to directly map stem cell-driven neural circuit formation in vivo. We engineered human induced pluripotent stem cells (iPSCs) to express channelrhodopsin-2 and transplanted resulting neurons to striatum of rats. To non-invasively visualize the function of newly formed circuits, we performed high-field functional magnetic resonance imaging (fMRI) during selective stimulation of transplanted cells. fMRI successfully detected local and remote neural activity, enabling the global graft-host neural circuit function to be assessed. These results demonstrate the potential of a novel neuroimaging-based platform that can be used to identify how a graft's electrical activity influences the brain network in vivo.

Selective increase of intention-based economic decisions by noninvasive brain stimulation to the dorsolateral prefrontal cortex.

The intention behind another's action and the impact of the outcome are major determinants of human economic behavior. It is poorly understood, however, whether the two systems share a core neural computation. Here, we investigated whether the two systems are causally dissociable in the brain by integrating computational modeling, functional magnetic resonance imaging, and transcranial direct current stimulation experiments in a newly developed trust game task. We show not only that right dorsolateral prefrontal cortex (DLPFC) activity is correlated with intention-based economic decisions and that ventral striatum and amygdala activity are correlated with outcome-based decisions, but also that stimulation to the DLPFC selectively enhances intention-based decisions. These findings suggest that the right DLPFC is involved in the implementation of intention-based decisions in the processing of cooperative decisions. This causal dissociation of cortical and subcortical backgrounds may indicate evolutionary and developmental differences in the two decision systems.

Frontal-parietal and limbic-striatal activity underlies information sampling in the best choice problem.

Best choice problems have a long mathematical history, but their neural underpinnings remain unknown. Best choice tasks are optimal stopping problem that require subjects to view a list of options one at a time and decide whether to take or decline each option. The goal is to find a high ranking option in the list, under the restriction that declined options cannot be chosen in the future. Conceptually, the decision to take or decline an option is related to threshold crossing in drift diffusion models, when this process is thought of as a value comparison. We studied this task in healthy volunteers using fMRI, and used a Markov decision process to quantify the value of continuing to search versus committing to the current option. Decisions to take versus decline an option engaged parietal and dorsolateral prefrontal cortices, as well ventral striatum, anterior insula, and anterior cingulate. Therefore, brain regions previously implicated in evidence integration and reward representation encode threshold crossings that trigger decisions to commit to a choice.

Multiple memory systems as substrates for multiple decision systems.

It has recently become widely appreciated that value-based decision making is supported by multiple computational strategies. In particular, animal and human behavior in learning tasks appears to include habitual responses described by prominent model-free reinforcement learning (RL) theories, but also more deliberative or goal-directed actions that can be characterized by a different class of theories, model-based RL. The latter theories evaluate actions by using a representation of the contingencies of the task (as with a learned map of a spatial maze), called an "internal model." Given the evidence of behavioral and neural dissociations between these approaches, they are often characterized as dissociable learning systems, though they likely interact and share common mechanisms. In many respects, this division parallels a longstanding dissociation in cognitive neuroscience between multiple memory systems, describing, at the broadest level, separate systems for declarative and procedural learning. Procedural learning has notable parallels with model-free RL: both involve learning of habits and both are known to depend on parts of the striatum. Declarative memory, by contrast, supports memory for single events or episodes and depends on the hippocampus. The hippocampus is thought to support declarative memory by encoding temporal and spatial relations among stimuli and thus is often referred to as a relational memory system. Such relational encoding is likely to play an important role in learning an internal model, the representation that is central to model-based RL. Thus, insofar as the memory systems represent more general-purpose cognitive mechanisms that might subserve performance on many sorts of tasks including decision making, these parallels raise the question whether the multiple decision systems are served by multiple memory systems, such that one dissociation is grounded in the other. Here we investigated the relationship between model-based RL and relational memory by comparing individual differences across behavioral tasks designed to measure either capacity. Human subjects performed two tasks, a learning and generalization task (acquired equivalence) which involves relational encoding and depends on the hippocampus; and a sequential RL task that could be solved by either a model-based or model-free strategy. We assessed the correlation between subjects' use of flexible, relational memory, as measured by generalization in the acquired equivalence task, and their differential reliance on either RL strategy in the decision task. We observed a significant positive relationship between generalization and model-based, but not model-free, choice strategies. These results are consistent with the hypothesis that model-based RL, like acquired equivalence, relies on a more general-purpose relational memory system.

Reward prediction error signal enhanced by striatum-amygdala interaction explains the acceleration of probabilistic reward learning by emotion.

Learning does not only depend on rationality, because real-life learning cannot be isolated from emotion or social factors. Therefore, it is intriguing to determine how emotion changes learning, and to identify which neural substrates underlie this interaction. Here, we show that the task-independent presentation of an emotional face before a reward-predicting cue increases the speed of cue-reward association learning in human subjects compared with trials in which a neutral face is presented. This phenomenon was attributable to an increase in the learning rate, which regulates reward prediction errors. Parallel to these behavioral findings, functional magnetic resonance imaging demonstrated that presentation of an emotional face enhanced reward prediction error (RPE) signal in the ventral striatum. In addition, we also found a functional link between this enhanced RPE signal and increased activity in the amygdala following presentation of an emotional face. Thus, this study revealed an acceleration of cue-reward association learning by emotion, and underscored a role of striatum-amygdala interactions in the modulation of the reward prediction errors by emotion.

The neural mechanisms of affect infusion in social economic decision-making: a mediating role of the anterior insula.

Though emotions have been shown to have sometimes dramatic effects on decision-making, the neural mechanisms mediating these biases are relatively unexplored. Here, we investigated how incidental affect (i.e. emotional states unrelated to the decision at hand) may influence decisions, and how these biases are implemented in the brain. Nineteen adult participants made decisions which involved accepting or rejecting monetary offers from others in an Ultimatum Game while undergoing functional magnetic resonance imaging (fMRI). Prior to each set of decisions, participants watched a short video clip aimed at inducing either a sad or neutral emotional state. Results demonstrated that, as expected, sad participants rejected more unfair offers than those in the neutral condition. Neuroimaging analyses revealed that receiving unfair offers while in a sad mood elicited activity in brain areas related to aversive emotional states and somatosensory integration (anterior insula) and to cognitive conflict (anterior cingulate cortex). Sad participants also showed a diminished sensitivity in neural regions associated with reward processing (ventral striatum). Importantly, insular activation uniquely mediated the relationship between sadness and decision bias. This study is the first to reveal how subtle mood states can be integrated at the neural level to influence decision-making.

Reward-related learning via multiple memory systems.

The application of a neuroeconomic approach to the study of reward-related processes has provided significant insights in our understanding of human learning and decision making. Much of this research has focused primarily on the contributions of the corticostriatal circuitry, involved in trial-and-error reward learning. As a result, less consideration has been allotted to the potential influence of different neural mechanisms such as the hippocampus or to more common ways in human society in which information is acquired and utilized to reach a decision, such as through explicit instruction rather than trial-and-error learning. This review examines the individual contributions of multiple learning and memory neural systems and their interactions during human decision making in both normal and neuropsychiatric populations. Specifically, the anatomical and functional connectivity across multiple memory systems are highlighted to suggest that probing the role of the hippocampus and its interactions with the corticostriatal circuitry via the application of model-based neuroeconomic approaches may provide novel insights into neuropsychiatric populations that suffer from damage to one of these structures and as a consequence have deficits in learning, memory, or decision making.

Reward skewness coding in the insula independent of probability and loss.

Rewards in the natural environment are rarely predicted with complete certainty. Uncertainty relating to future rewards has typically been defined as the variance of the potential outcomes. However, the asymmetry of predicted reward distributions, known as skewness, constitutes a distinct but neuroscientifically underexplored risk term that may also have an impact on preference. By changing only reward magnitudes, we study skewness processing in equiprobable ternary lotteries involving only gains and constant probabilities, thus excluding probability distortion or loss aversion as mechanisms for skewness preference formation. We show that individual preferences are sensitive to not only the mean and variance but also to the skewness of predicted reward distributions. Using neuroimaging, we show that the insula, a structure previously implicated in the processing of reward-related uncertainty, responds to the skewness of predicted reward distributions. Some insula responses increased in a monotonic fashion with skewness (irrespective of individual skewness preferences), whereas others were similarly elevated to both negative and positive as opposed to no reward skew. These data support the notion that the asymmetry of reward distributions is processed in the brain and, taken together with replicated findings of mean coding in the striatum and variance coding in the cingulate, suggest that the brain codes distinct aspects of reward distributions in a distributed fashion.