Neural Circuitry of Reward Prediction Error.

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

A comparison of 'pruning' during multi-step planning in depressed and healthy individuals.

BACKGROUND: Real-life decisions are often complex because they involve making sequential choices that constrain future options. We have previously shown that to render such multi-step decisions manageable, people 'prune' (i.e. selectively disregard) branches of decision trees that contain negative outcomes. We have theorized that sub-optimal pruning contributes to depression by promoting an oversampling of branches that result in unsavoury outcomes, which results in a negatively-biased valuation of the world. However, no study has tested this theory in depressed individuals. METHODS: Thirty unmedicated depressed and 31 healthy participants were administered a sequential reinforcement-based decision-making task to determine pruning behaviours, and completed measures of depression and anxiety. Computational, Bayesian and frequentist analyses examined group differences in task performance and relationships between pruning and depressive symptoms. RESULTS: Consistent with prior findings, participants robustly pruned branches of decision trees that began with large losses, regardless of the potential utility of those branches. However, there was no group difference in pruning behaviours. Further, there was no relationship between pruning and levels of depression/anxiety. CONCLUSIONS: We found no evidence that sub-optimal pruning is evident in depression. Future research could determine whether maladaptive pruning behaviours are observable in specific sub-groups of depressed patients (e.g. in treatment-resistant individuals), or whether misuse of other heuristics may contribute to depression.

Learning what to approach.

Most decisions share a common goal: maximize reward and minimize punishment. Achieving this goal requires learning which choices are likely to lead to favorable outcomes. Dopamine is essential for this process, enabling learning by signaling the difference between what we expect to get and what we actually get. Although all animals appear to use this dopamine prediction error circuit, some do so more than others, and this neural heterogeneity correlates with individual variability in behavior. In this issue of PLOS Biology, Lee and colleagues show that manipulating a simple task parameter can bias the animals' behavioral strategy and modulate dopamine release, implying that how we learn is just as flexible as what we learn.

Arithmetic and local circuitry underlying dopamine prediction errors.

Dopamine neurons are thought to facilitate learning by comparing actual and expected reward. Despite two decades of investigation, little is known about how this comparison is made. To determine how dopamine neurons calculate prediction error, we combined optogenetic manipulations with extracellular recordings in the ventral tegmental area while mice engaged in classical conditioning. Here we demonstrate, by manipulating the temporal expectation of reward, that dopamine neurons perform subtraction, a computation that is ideal for reinforcement learning but rarely observed in the brain. Furthermore, selectively exciting and inhibiting neighbouring GABA (γ-aminobutyric acid) neurons in the ventral tegmental area reveals that these neurons are a source of subtraction: they inhibit dopamine neurons when reward is expected, causally contributing to prediction-error calculations. Finally, bilaterally stimulating ventral tegmental area GABA neurons dramatically reduces anticipatory licking to conditioned odours, consistent with an important role for these neurons in reinforcement learning. Together, our results uncover the arithmetic and local circuitry underlying dopamine prediction errors.

The role of dopamine in reward-related behavior: shining new light on an old debate.

The role dopamine plays in reward-related behaviors has been debated for decades. Heymann et al. (Heymann G, Jo YS, Reichard KL, McFarland N, Chavkin C, Palmiter RD, Soden ME, Zweifel LS. Neuron 105: 909-920, 2020) identify subpopulations of dopamine-producing neurons that separately mediate reward association and motivation. Their results help demonstrate that dopamine signaling may partake in both reinforcement learning and incentive salience functions, instantiated by neuropeptide-defined subpopulations of the ventral tegmental area with different projection targets.

The Neural Basis of Aversive Pavlovian Guidance during Planning.

Important real-world decisions are often arduous as they frequently involve sequences of choices, with initial selections affecting future options. Evaluating every possible combination of choices is computationally intractable, particularly for longer multistep decisions. Therefore, humans frequently use heuristics to reduce the complexity of decisions. We recently used a goal-directed planning task to demonstrate the profound behavioral influence and ubiquity of one such shortcut, namely aversive pruning, a reflexive Pavlovian process that involves neglecting parts of the decision space residing beyond salient negative outcomes. However, how the brain implements this important decision heuristic and what underlies individual differences have hitherto remained unanswered. Therefore, we administered an adapted version of the same planning task to healthy male and female volunteers undergoing functional magnetic resonance imaging (fMRI) to determine the neural basis of aversive pruning. Through both computational and standard categorical fMRI analyses, we show that when planning was influenced by aversive pruning, the subgenual cingulate cortex was robustly recruited. This neural signature was distinct from those associated with general planning and valuation, two fundamental cognitive components elicited by our task but which are complementary to aversive pruning. Furthermore, we found that individual variation in levels of aversive pruning was associated with the responses of insula and dorsolateral prefrontal cortices to the receipt of large monetary losses, and also with subclinical levels of anxiety. In summary, our data reveal the neural signatures of an important reflexive Pavlovian process that shapes goal-directed evaluations and thereby determines the outcome of high-level sequential cognitive processes.SIGNIFICANCE STATEMENT Multistep decisions are complex because initial choices constrain future options. Evaluating every path for long decision sequences is often impractical; thus, cognitive shortcuts are often essential. One pervasive and powerful heuristic is aversive pruning, in which potential decision-making avenues are curtailed at immediate negative outcomes. We used neuroimaging to examine how humans implement such pruning. We found it to be associated with activity in the subgenual cingulate cortex, with neural signatures that were distinguishable from those covarying with planning and valuation. Individual variations in aversive pruning levels related to subclinical anxiety levels and insular cortex activation. These findings reveal the neural mechanisms by which basic negative Pavlovian influences guide decision-making during planning, with implications for disrupted decision-making in psychiatric disorders.

Interplay of approximate planning strategies.

Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."

Role of prefrontal cortex and the midbrain dopamine system in working memory updating.

Humans are adept at switching between goal-directed behaviors quickly and effectively. The prefrontal cortex (PFC) is thought to play a critical role by encoding, updating, and maintaining internal representations of task context in working memory. It has also been hypothesized that the encoding of context representations in PFC is regulated by phasic dopamine gating signals. Here we use multimodal methods to test these hypotheses. First we used functional MRI (fMRI) to identify regions of PFC associated with the representation of context in a working memory task. Next we used single-pulse transcranial magnetic stimulation (TMS), guided spatially by our fMRI findings and temporally by previous event-related EEG recordings, to disrupt context encoding while participants performed the same working memory task. We found that TMS pulses to the right dorsolateral PFC (DLPFC) immediately after context presentation, and well in advance of the response, adversely impacted context-dependent relative to context-independent responses. This finding causally implicates right DLPFC function in context encoding. Finally, using the same paradigm, we conducted high-resolution fMRI measurements in brainstem dopaminergic nuclei (ventral tegmental area and substantia nigra) and found phasic responses after presentation of context stimuli relative to other stimuli, consistent with the timing of a gating signal that regulates the encoding of representations in PFC. Furthermore, these responses were positively correlated with behavior, as well as with responses in the same region of right DLPFC targeted in the TMS experiment, lending support to the hypothesis that dopamine phasic signals regulate encoding, and thereby the updating, of context representations in PFC.

Dopamine gates sensory representations in cortex.

The prefrontal cortex (PFC) maintains information about relevant sensory stimuli, in a process thought to rely on dopamine release. In a recent paper, Jacob et al. (J Neurosci 33: 13724-13734, 2013) demonstrated one way in which dopamine might facilitate this process. The authors recorded from PFC neurons in monkeys during local application of dopamine. They found that dopamine increases the gain of sensory-evoked responses in putative pyramidal neurons in PFC, potentially by inhibiting local interneurons.

Striatal dopamine integrates cost, benefit, and motivation.

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience.

The study of complex behaviors is often challenging when using manual annotation due to the absence of quantifiable behavioral definitions and the subjective nature of behavioral annotation. Integration of supervised machine learning approaches mitigates some of these issues through the inclusion of accessible and explainable model interpretation. To decrease barriers to access, and with an emphasis on accessible model explainability, we developed the open-source Simple Behavioral Analysis (SimBA) platform for behavioral neuroscientists. SimBA introduces several machine learning interpretability tools, including SHapley Additive exPlanation (SHAP) scores, that aid in creating explainable and transparent behavioral classifiers. Here we show how the addition of explainability metrics allows for quantifiable comparisons of aggressive social behavior across research groups and species, reconceptualizing behavior as a sharable reagent and providing an open-source framework. We provide an open-source, graphical user interface (GUI)-driven, well-documented package to facilitate the movement toward improved automation and sharing of behavioral classification tools across laboratories.

Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees.

When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal-directed choices in a manner that may be important for theories of decision-making in mood disorders.

Perspective on equitable translational studies and clinical support for an unbiased inclusion of the LGBTQIA2S+community.

Research regarding the mental health of the Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, Asexual, 2 Spirit (LGBTQIA2S+) community has been historically biased by individual and structural homophobia, biphobia, and transphobia, resulting in research that does not represent the best quality science. Furthermore, much of this research does not serve the best interests or priorities of LGBTQIA2S + communities, despite significant mental health disparities and great need for quality mental health research and treatments in these populations. Here, we will highlight how bias has resulted in missed opportunities for advancing understanding of mental health within LGBTQIA2S + communities. We cite up-to-date research on mental health disparities facing the LGBTQIA2S + community and targeted treatment strategies, as well as guidance from health care professionals. Importantly, research is discussed from both preclinical and clinical perspectives, providing common language and research priorities from a translational perspective. Given the rising tide of anti-transgender sentiment among certain political factions, we further emphasize and discuss the impact of historical and present day ciscentrism and structural transphobia in transgender mental health research, from both clinical and translational perspectives, with suggestions for future directions to improve the quality of this field. Finally, we address current best practices for treatment of mental health issues in this community. This approach provides an opportunity to dispel myths regarding the LGBTQIA2S + community as well as inform the scientific community of best practices to work with this community in an equitable manner. Thus, our approach ties preclinical and clinical research within the LGBTQIA2S + community.

Rethinking the Physician-Scientist Pathway.

Physician-scientists have the potential to generate fundamental as well as translational breakthroughs. But many trainees who intend to pursue a hybrid career in research and patient care ultimately leave one or the other behind. In this Invited Commentary, the authors draw from their experience as early-career physician-scientists to frame physician-scientist training as having 2 phases: first, learning to think like a physician-scientist; second, learning to act like a physician-scientist. These phases roughly correspond to (1) clinical training (from medical school through residency or fellowship) that incorporates research exposure, and (2) a structured period of graduated research independence once the physician-scientist has become clinically autonomous. There are many effective ways to pursue each phase; what matters most is flexibility in the first phase and sustained support in the second. Accordingly, the authors suggest many potential reforms, including at the levels of the National Institutes of Health, private funders, as well as universities and research hospitals. The authors argue that rethinking physician-scientist training to support individualized paths to an independent hybrid career can help recruit and retain physician-scientists for years to come.

Modulation of 5-HT release by dynorphin mediates social deficits during opioid withdrawal.

Social isolation during opioid withdrawal is a major contributor to the current opioid addiction crisis. We find that sociability deficits during protracted opioid withdrawal in mice require activation of kappa opioid receptors (KORs) in the nucleus accumbens (NAc) medial shell. Blockade of release from dynorphin (Pdyn)-expressing dorsal raphe neurons (DRPdyn), but not from NAcPdyn neurons, prevents these deficits in prosocial behaviors. Conversely, optogenetic activation of DRPdyn neurons reproduced NAc KOR-dependent decreases in sociability. Deletion of KORs from serotonin (5-HT) neurons, but not from NAc neurons or dopamine (DA) neurons, prevented sociability deficits during withdrawal. Finally, measurements with the genetically encoded GRAB5-HT sensor revealed that during withdrawal KORs block the NAc 5-HT release that normally occurs during social interactions. These results define a neuromodulatory mechanism that is engaged during protracted opioid withdrawal to induce maladaptive deficits in prosocial behaviors, which in humans contribute to relapse.

Neural correlates of anger expression in patients with PTSD.

Anger is a common and debilitating symptom of post-traumatic stress disorder (PTSD). Although studies have identified brain circuits underlying anger experience and expression in healthy individuals, how these circuits interact with trauma remains unclear. Here, we performed the first study examining the neural correlates of anger in patients with PTSD. Using a data-driven approach with resting-state fMRI, we identified two prefrontal regions whose overall functional connectivity was inversely associated with anger: the left anterior middle frontal gyrus (aMFG) and the right orbitofrontal cortex (OFC). We then used concurrent TMS-EEG to target the left aMFG parcel previously identified through fMRI, measuring its cortical excitability and causal connectivity to downstream areas. We found that low-anger PTSD patients exhibited enhanced excitability in the left aMFG and enhanced causal connectivity between this region and visual areas. Together, our results suggest that left aMFG activity may confer protection against the development of anger, and therefore may be an intriguing target for circuit-based interventions for anger in PTSD.