Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration.

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

Neuroscience: Therapy modulates decision-making in Parkinson's disease.

There is mounting evidence that decision-making can be affected by treatment in Parkinson's disease. A new study shows that dopamine and deep brain stimulation, two mainstay treatments of Parkinson's, differently affect how patients make decisions weighing rewards against effort costs.

Local regulation of striatal dopamine: A diversity of circuit mechanisms for a diversity of behavioral functions?

Striatal dopamine governs a wide range of behavioral functions, yet local dopamine concentrations can be dissociated from somatic activity. Here, we discuss how dopamine's diverse roles in behavior may be driven by local circuit mechanisms shaping dopamine release. We first look at historical and recent work demonstrating that striatal circuits interact with dopaminergic terminals to either initiate the release of dopamine or modulate the release of dopamine initiated by spiking in midbrain dopamine neurons, with particular attention to GABAergic and cholinergic local circuit mechanisms. Then we discuss some of the first in vivo studies of acetylcholine-dopamine interactions in striatum and broadly discuss necessary future work in understanding the roles of midbrain versus striatal dopamine regulation.

Dopamine-independent effect of rewards on choices through hidden-state inference.

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.

Striatal and Behavioral Responses to Reward Vary by Socioeconomic Status in Adolescents.

Disparities in socioeconomic status (SES) lead to unequal access to financial and social support. These disparities are believed to influence reward sensitivity, which in turn are hypothesized to shape how individuals respond to and pursue rewarding experiences. However, surprisingly little is known about how SES shapes reward sensitivity in adolescence. Here, we investigated how SES influenced adolescent responses to reward, both in behavior and the striatum-a brain region that is highly sensitive to reward. We examined responses to both immediate reward (tracked by phasic dopamine) and average reward rate fluctuations (tracked by tonic dopamine) as these distinct signals independently shape learning and motivation. Adolescents (n = 114; 12-14 years; 58 female) performed a gambling task during functional magnetic resonance imaging. We manipulated trial-by-trial reward and loss outcomes, leading to fluctuations between periods of reward scarcity and abundance. We found that a higher reward rate hastened behavioral responses, and increased guess switching, consistent with the idea that reward abundance increases response vigor and exploration. Moreover, immediate reward reinforced previously rewarding decisions (win-stay, lose-switch) and slowed responses (postreward pausing), particularly when rewards were scarce. Notably, lower-SES adolescents slowed down less after rare rewards than higher-SES adolescents. In the brain, striatal activations covaried with the average reward rate across time and showed greater activations during rewarding blocks. However, these striatal effects were diminished in lower-SES adolescents. These findings show that the striatum tracks reward rate fluctuations, which shape decisions and motivation. Moreover, lower SES appears to attenuate reward-driven behavioral and brain responses.

Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons.

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Dopamine's reach: Unlocked by sleep loss.

In this issue of Neuron, Wu et al.1 employ cutting-edge techniques to provide a mechanistic understanding of how sleep deprivation induces an altered affective state. They reveal a key function for dopaminergic signaling, and the formation of cortical spines, in this process.

NMDA- and 6-OHDA-induced Lesions in the Nucleus Accumbens Differently Affect Maternal and Infanticidal Behavior in Pup-naïve Female and Male Mice.

Virgin and pups-naïve female and male adult mice display two opposite responses when they are exposed to pups for the first time. While females generally take care of the pups, males attack them. Since the nucleus accumbens (NA), and its dopaminergic modulation, is critical in integrating information and processing reward and aversion, we investigated if NMDA- and 6-OHDA-induced lesions, damaging mostly NA output and dopaminergic inputs respectively, affected female maternal behavior (MB) or male infanticidal behavior (IB) in mice. Our results revealed minor or no effects of both smaller and larger NMDA-induced lesions in MB and IB. On the other hand, while 6-OHDA-induced lesions in females reduced the incidence of full MB (12.5% 6-OHDA vs. 85.7% SHAM) increasing the latency to retrieve the pups, those lesions did not affect IB in males. There were no differences in locomotor and exploratory activity between the lesioned- and SHAM- females. Despite those lesions did not induce any major effect on IB, NMDA-lesioned males spent less time in the central area of an open field, while dopaminergic-lesioned males showed reduced number of rearing and peripheral crosses. The current study shows that an intact NA is not necessary for the expression of MB and IB. However, dopaminergic inputs to NA play different role in MB and IB. While damaging dopaminergic terminals into the NA did not affect IB, it clearly delayed the more flexible and rewarding expression of parental behavior.

Dopamine regulates attitude toward risk.

Specific brain pathways can lower or raise the willingness of monkeys to take risks.

65 years of research on dopamine's role in classical fear conditioning and extinction: A systematic review.

Dopamine, a catecholamine neurotransmitter, has historically been associated with the encoding of reward, whereas its role in aversion has received less attention. Here, we systematically gathered the vast evidence of the role of dopamine in the simplest forms of aversive learning: classical fear conditioning and extinction. In the past, crude methods were used to augment or inhibit dopamine to study its relationship with fear conditioning and extinction. More advanced techniques such as conditional genetic, chemogenic and optogenetic approaches now provide causal evidence for dopamine's role in these learning processes. Dopamine neurons encode conditioned stimuli during fear conditioning and extinction and convey the signal via activation of D1-4 receptor sites particularly in the amygdala, prefrontal cortex and striatum. The coordinated activation of dopamine receptors allows for the continuous formation, consolidation, retrieval and updating of fear and extinction memory in a dynamic and reciprocal manner. Based on the reviewed literature, we conclude that dopamine is crucial for the encoding of classical fear conditioning and extinction and contributes in a way that is comparable to its role in encoding reward.

Prediction error in dopamine neurons during associative learning.

Dopamine neurons have long been thought to facilitate learning by broadcasting reward prediction error (RPE), a teaching signal used in machine learning, but more recent work has advanced alternative models of dopamine's computational role. Here, I revisit this critical issue and review new experimental evidences that tighten the link between dopamine activity and RPE. First, I introduce the recent observation of a gradual backward shift of dopamine activity that had eluded researchers for over a decade. I also discuss several other findings, such as dopamine ramping, that were initially interpreted to conflict but later found to be consistent with RPE. These findings improve our understanding of neural computation in dopamine neurons.

Does phasic dopamine release cause policy updates?

Phasic dopamine activity is believed to both encode reward-prediction errors (RPEs) and to cause the adaptations that these errors engender. If so, a rat working for optogenetic stimulation of dopamine neurons will repeatedly update its policy and/or action values, thus iteratively increasing its work rate. Here, we challenge this view by demonstrating stable, non-maximal work rates in the face of repeated optogenetic stimulation of midbrain dopamine neurons. Furthermore, we show that rats learn to discriminate between world states distinguished only by their history of dopamine activation. Comparison of these results to reinforcement learning simulations suggests that the induced dopamine transients acted more as rewards than RPEs. However, pursuit of dopaminergic stimulation drifted upwards over a time scale of days and weeks, despite its stability within trials. To reconcile the results with prior findings, we consider multiple roles for dopamine signalling.

An excitatory projection from the basal forebrain to the ventral tegmental area that underlies anorexia-like phenotypes.

Maladaptation in balancing internal energy needs and external threat cues may result in eating disorders. However, brain mechanisms underlying such maladaptations remain elusive. Here, we identified that the basal forebrain (BF) sends glutamatergic projections to glutamatergic neurons in the ventral tegmental area (VTA) in mice. Glutamatergic neurons in both regions displayed correlated responses to various stressors. Notably, in vivo manipulation of BF terminals in the VTA revealed that the glutamatergic BF → VTA circuit reduces appetite, increases locomotion, and elicits avoidance. Consistently, activation of VTA glutamatergic neurons reduced body weight, blunted food motivation, and caused hyperactivity with behavioral signs of anxiety, all hallmarks of typical anorexia symptoms. Importantly, activation of BF glutamatergic terminals in the VTA reduced dopamine release in the nucleus accumbens. Collectively, our results point to overactivation of the glutamatergic BF → VTA circuit as a potential cause of anorexia-like phenotypes involving reduced dopamine release.

Striatal dopamine integrates cost, benefit, and motivation.

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Unraveling the role of Slc10a4 in auditory processing and sensory motor gating: Implications for neuropsychiatric disorders?

BACKGROUND: Psychiatric disorders, such as schizophrenia, are complex and challenging to study, partly due to the lack of suitable animal models. However, the absence of the Slc10a4 gene, which codes for a monoaminergic and cholinergic associated vesicular transporter protein, in knockout mice (Slc10a4-/-), leads to the accumulation of extracellular dopamine. A major challenge for studying schizophrenia is the lack of suitable animal models that accurately represent the disorder. We sought to overcome this challenge by using Slc10a4-/- mice as a potential model, considering their altered dopamine levels. This makes them a potential animal model for schizophrenia, a disorder known to be associated with altered dopamine signaling in the brain. METHODS: The locomotion, auditory sensory filtering and prepulse inhibition (PPI) of Slc10a4-/- mice were quantified and compared to wildtype (WT) littermates. Intrahippocampal electrodes were used to record auditory event-related potentials (aERPs) for quantifying sensory filtering in response to paired-clicks. The channel above aERPs phase reversal was chosen for reliably comparing results between animals, and aERPs amplitude and latency of click responses were quantified. WT and Slc10a4-/- mice were also administered subanesthetic doses of ketamine to provoke psychomimetic behavior. RESULTS: Baseline locomotion during auditory stimulation was similar between Slc10a4-/- mice and WT littermates. In WT animals, normal auditory processing was observed after i.p saline injections, and it was maintained under the influence of 5 mg/kg ketamine, but disrupted by 20 mg/kg ketamine. On the other hand, Slc10a4-/- mice did not show significant differences between N40 S1 and S2 amplitude responses in saline or low dose ketamine treatment. Auditory gating was considered preserved since the second N40 peak was consistently suppressed, but with increased latency. The P80 component showed higher amplitude, with shorter S2 latency under saline and 5 mg/kg ketamine treatment in Slc10a4-/- mice, which was not observed in WT littermates. Prepulse inhibition was also decreased in Slc10a4-/- mice when the longer interstimulus interval of 100 ms was applied, compared to WT littermates. CONCLUSION: The Slc10a4-/- mice responses indicate that cholinergic and monoaminergic systems participate in the PPI magnitude, in the temporal coding (response latency) of the auditory sensory gating component N40, and in the amplitude of aERPs P80 component. These results suggest that Slc10a4-/- mice can be considered as potential models for neuropsychiatric conditions.

Sex-dependent emergence of prepubertal social dysfunction and augmented dopamine activity in a neurodevelopmental rodent model relevant for schizophrenia.

Schizophrenia is a neurodevelopmental psychiatric disorder that often emerges in adolescence, is characterized by social dysfunction, and has an earlier onset in men. These features have been replicated in rats exposed to the mitotoxin methylazoxymethanol acetate (MAM) on gestational day (GD) 17, which as adults exhibit behavioral impairments and dopamine (DA) system changes consistent with a schizophrenia-relevant rodent model. In humans, social withdrawal is a negative symptom that often precedes disease onset and DA system dysfunction and is more pronounced in men. Children and adolescents at high-risk for schizophrenia exhibit social deficits prior to psychotic symptoms (i.e., prodromal phase), which can be used as a predictive marker for future psychopathology. Adult MAM rats also exhibit deficient social interaction, but less is known regarding the emergence of social dysfunction in this model, whether it varies by sex, and whether it is linked to disrupted DA function. To this end, we characterized the ontogeny of social and DA dysfunction in male and female MAM rats during the prepubertal period (postnatal days 33-43) and found sex-specific changes in motivated social behaviors (play, approach) and DA function. Male MAM rats exhibited reduced social approach and increased VTA DA neuron activity compared to saline-treated (SAL) males, whereas female MAM rats exhibited enhanced play behaviors compared to SAL females but no changes in social approach or VTA population activity during this period. These findings demonstrate sex differences in the emergence of social and DA deficits in the MAM model, in which females exhibit delayed emergence.

Fractionation of neural reward processing into independent components by novel decoding principle.

How to retrieve latent neurobehavioural processes from complex neurobiological signals is an important yet unresolved challenge. Here, we develop a novel approach, orthogonal-Decoding multi-Cognitive Processes (DeCoP), to reveal underlying latent neurobehavioural processing and show that its performance is superior to traditional non-orthogonal decoding in terms of both false inference and robustness. Processing value and salience information are two fundamental but mutually confounded pathways of reward reinforcement essential for decision making. During reward/punishment anticipation, we applied DeCoP to decode brain-wide responses into spatially overlapping, yet functionally independent, evaluation and readiness processes, which are modulated differentially by meso­limbic vs nigro-striatal dopamine systems. Using DeCoP, we further demonstrated that most brain regions only encoded abstract information but not the exact input, except for dorsal anterior cingulate cortex and insula. Furthermore, we anticipate our novel analytical principle to be applied generally in decoding multiple latent neurobehavioral processes and thus advance both the design and hypothesis testing for cognitive tasks.

Adolescent neurostimulation of dopamine circuit reverses genetic deficits in frontal cortex function.

Dopamine system dysfunction is implicated in adolescent-onset neuropsychiatric disorders. Although psychosis symptoms can be alleviated by antipsychotics, cognitive symptoms remain unresponsive and novel paradigms investigating the circuit substrates underlying cognitive deficits are critically needed. The frontal cortex and its dopaminergic input from the midbrain are implicated in cognitive functions and undergo maturational changes during adolescence. Here, we used mice carrying mutations in Arc or Disc1 to model mesofrontal dopamine circuit deficiencies and test circuit-based neurostimulation strategies to restore cognitive functions. We found that in a memory-guided spatial navigation task, frontal cortical neurons were activated coordinately at the decision-making point in wild-type but not Arc-/- mice. Chemogenetic stimulation of midbrain dopamine neurons or optogenetic stimulation of frontal cortical dopamine axons in a limited adolescent period consistently reversed genetic defects in mesofrontal innervation, task-coordinated neuronal activity, and memory-guided decision-making at adulthood. Furthermore, adolescent stimulation of dopamine neurons also reversed the same cognitive deficits in Disc1+/- mice. Our findings reveal common mesofrontal circuit alterations underlying the cognitive deficits caused by two different genes and demonstrate the feasibility of adolescent neurostimulation to reverse these circuit and behavioral deficits. These results may suggest developmental windows and circuit targets for treating cognitive deficits in neurodevelopmental disorders.

Linkage and association of variants in the dopamine receptor 2 gene (DRD2) with polycystic ovary syndrome.

Polycystic ovarian syndrome (PCOS) is a disorder with a foundation of neuroendocrine dysfunction, characterized by increased gonadotropin-releasing hormone (GnRH) pulsatility, which is antagonized by dopamine. The dopamine receptor 2 (DRD2), encoded by the DRD2 gene, has been shown to mediate dopamine's inhibition of GnRH neuron excitability through pre- and post-synaptic interactions in murine models. Further, DRD2 is known to mediate prolactin (PRL) inhibition by dopamine, and high blood level of PRL have been found in more than one third of women with PCOS. We recently identified PRL as a gene contributing to PCOS risk and reported DRD2 conferring risk for type 2 diabetes and depression, which can both coexist with PCOS. Given DRD2 mediating dopamine's action on neuroendocrine profiles and association with metabolic-mental states related to PCOS, polymorphisms in DRD2 may predispose to development of PCOS. Therefore, we aimed to investigate whether DRD2 variants are in linkage to and/or linkage disequilibrium (i.e., linkage and association) with PCOS in Italian families. In 212 Italian families, we tested 22 variants within the DRD2 gene for linkage and linkage disequilibrium with PCOS. We identified five novel variants significantly linked to the risk of PCOS. This is the first study to identify DRD2 as a risk gene in PCOS, however, functional studies are needed to confirm these results.

Interaction of gonadal hormones, dopaminergic system, and epigenetic regulation in the generation of sex differences in substance use disorders: A systematic review.

Substance use disorder (SUD) is a chronic condition characterized by pathological drug-taking and seeking behaviors. Remarkably different between males and females, suggesting that drug addiction is a sexually differentiated disorder. The neurobiological bases of sex differences in SUD include sex-specific reward system activation, influenced by interactions between gonadal hormone level changes, dopaminergic reward circuits, and epigenetic modifications of key reward system genes. This systematic review, adhering to PICOS and PRISMA-P 2015 guidelines, highlights the sex-dependent roles of estrogens, progesterone, and testosterone in SUD. In particular, estradiol elevates and progesterone reduces dopaminergic activity in SUD females, whilst testosterone and progesterone augment SUD behavior in males. Finally, SUD is associated with a sex-specific increase in the rate of opioid and monoaminergic gene methylation. The study reveals the need for detailed research on gonadal hormone levels, dopaminergic or reward system activity, and epigenetic landscapes in both sexes for efficient SUD therapy development.