Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors.

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

What Does the Frontopolar Cortex Contribute to Goal-Directed Cognition and Action?

Understanding the unique functions of different subregions of primate prefrontal cortex has been a longstanding goal in cognitive neuroscience. Yet, the anatomy and function of one of its largest subregions (the frontopolar cortex) remain enigmatic and underspecified. Our Society for Neuroscience minisymposium Primate Frontopolar Cortex: From Circuits to Complex Behaviors will comprise a range of new anatomic and functional approaches that have helped to clarify the basic circuit anatomy of the frontal pole, its functional involvement during performance of cognitively demanding behavioral paradigms in monkeys and humans, and its clinical potential as a target for noninvasive brain stimulation in patients with brain disorders. This review consolidates knowledge about the anatomy and connectivity of frontopolar cortex and provides an integrative summary of its function in primates. We aim to answer the question: what, if anything, does frontopolar cortex contribute to goal-directed cognition and action?

Adolescent Dopamine Neurons Represent Reward Differently during Action and State Guided Learning.

Neuronal underpinning of learning cause-and-effect associations in the adolescent brain remains poorly understood. Two fundamental forms of associative learning are Pavlovian (classical) conditioning, where a stimulus is followed by an outcome, and operant (instrumental) conditioning, where outcome is contingent on action execution. Both forms of learning, when associated with a rewarding outcome, rely on midbrain dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SN). We find that, in adolescent male rats, reward-guided associative learning is encoded differently by midbrain dopamine neurons in each conditioning paradigm. Whereas simultaneously recorded VTA and SN adult neurons have a similar phasic response to reward delivery during both forms of conditioning, adolescent neurons display a muted reward response during operant but a profoundly larger reward response during Pavlovian conditioning. These results suggest that adolescent neurons assign a different value to reward when it is not gated by action. The learning rate of adolescents and adults during both forms of conditioning was similar, supporting the notion that differences in reward response in each paradigm may be because of differences in motivation and independent of state versus action value learning. Static characteristics of dopamine neurons, such as dopamine cell number and size, were similar in the VTA and SN of both ages, but there were age-related differences in stimulated dopamine release and correlated spike activity, suggesting that differences in reward responsiveness by adolescent dopamine neurons are not because of differences in intrinsic properties of these neurons but engagement of different dopaminergic networks.SIGNIFICANCE STATEMENT Reckless behavior and impulsive decision-making by adolescents suggest that motivated behavioral states are encoded differently by the adolescent brain. Motivated behavior, which is dependent on the function of the dopamine system, follows learning of cause-and-effect associations in the environment. We find that dopamine neurons in adolescents encode reward differently depending on the cause-and-effect relationship of the means to receive that reward. Compared with adults, reward contingent on action led to a muted response, whereas reward that followed a cue but was not gated by action produced an augmented phasic response. These data demonstrate an age-related difference in dopamine neuron response to reward that is not uniform and is guided by processes that differentiate between state and action values.

Effects of Amygdala Lesions on Object-Based Versus Action-Based Learning in Macaques.

The neural systems that underlie reinforcement learning (RL) allow animals to adapt to changes in their environment. In the present study, we examined the hypothesis that the amygdala would have a preferential role in learning the values of visual objects. We compared a group of monkeys (Macaca mulatta) with amygdala lesions to a group of unoperated controls on a two-armed bandit reversal learning task. The task had two conditions. In the What condition, the animals had to learn to select a visual object, independent of its location. And in the Where condition, the animals had to learn to saccade to a location, independent of the object at the location. In both conditions choice-outcome mappings reversed in the middle of the block. We found that monkeys with amygdala lesions had learning deficits in both conditions. Monkeys with amygdala lesions did not have deficits in learning to reverse choice-outcome mappings. Rather, amygdala lesions caused the monkeys to become overly sensitive to negative feedback which impaired their ability to consistently select the more highly valued action or object. These results imply that the amygdala is generally necessary for RL.

Primate Orbitofrontal Cortex Codes Information Relevant for Managing Explore-Exploit Tradeoffs.

Reinforcement learning (RL) refers to the behavioral process of learning to obtain reward and avoid punishment. An important component of RL is managing explore-exploit tradeoffs, which refers to the problem of choosing between exploiting options with known values and exploring unfamiliar options. We examined correlates of this tradeoff, as well as other RL related variables, in orbitofrontal cortex (OFC) while three male monkeys performed a three-armed bandit learning task. During the task, novel choice options periodically replaced familiar options. The values of the novel options were unknown, and the monkeys had to explore them to see if they were better than other currently available options. The identity of the chosen stimulus and the reward outcome were strongly encoded in the responses of single OFC neurons. These two variables define the states and state transitions in our model that are relevant to decision-making. The chosen value of the option and the relative value of exploring that option were encoded at intermediate levels. We also found that OFC value coding was stimulus specific, as opposed to coding value independent of the identity of the option. The location of the option and the value of the current environment were encoded at low levels. Therefore, we found encoding of the variables relevant to learning and managing explore-exploit tradeoffs in OFC. These results are consistent with findings in the ventral striatum and amygdala and show that this monosynaptically connected network plays an important role in learning based on the immediate and future consequences of choices.SIGNIFICANCE STATEMENT Orbitofrontal cortex (OFC) has been implicated in representing the expected values of choices. Here we extend these results and show that OFC also encodes information relevant to managing explore-exploit tradeoffs. Specifically, OFC encodes an exploration bonus, which characterizes the relative value of exploring novel choice options. OFC also strongly encodes the identity of the chosen stimulus, and reward outcomes, which are necessary for computing the value of novel and familiar options.

Ventral striatum's role in learning from gains and losses.

Adaptive behavior requires animals to learn from experience. Ideally, learning should both promote choices that lead to rewards and reduce choices that lead to losses. Because the ventral striatum (VS) contains neurons that respond to aversive stimuli and aversive stimuli can drive dopamine release in the VS, it is possible that the VS contributes to learning about aversive outcomes, including losses. However, other work suggests that the VS may play a specific role in learning to choose among rewards, with other systems mediating learning from aversive outcomes. To examine the role of the VS in learning from gains and losses, we compared the performance of macaque monkeys with VS lesions and unoperated controls on a reinforcement learning task. In the task, the monkeys gained or lost tokens, which were periodically cashed out for juice, as outcomes for choices. They learned over trials to choose cues associated with gains, and not choose cues associated with losses. We found that monkeys with VS lesions had a deficit in learning to choose between cues that differed in reward magnitude. By contrast, monkeys with VS lesions performed as well as controls when choices involved a potential loss. We also fit reinforcement learning models to the behavior and compared learning rates between groups. Relative to controls, the monkeys with VS lesions had reduced learning rates for gain cues. Therefore, in this task, the VS plays a specific role in learning to choose between rewarding options.

Directional interconnectivity of the human amygdala, fusiform gyrus, and orbitofrontal cortex in emotional scene perception.

The perception of emotionally arousing scenes modulates neural activity in ventral visual areas via reentrant signals from the amygdala. The orbitofrontal cortex (OFC) shares dense interconnections with amygdala and has been strongly implicated in emotional stimulus processing in primates, but our understanding of the functional contribution of this region to emotional perception in humans is poorly defined. In this study we acquired targeted rapid functional imaging from lateral OFC, amygdala, and fusiform gyrus (FG) over multiple scanning sessions (resulting in over 1,000 trials per participant) in an effort to define the activation amplitude and directional connectivity among these regions during naturalistic scene perception. All regions of interest showed enhanced activation during emotionally arousing, compared with neutral scenes. In addition, we identified bidirectional connectivity between amygdala, FG, and OFC in the great majority of individual subjects, suggesting that human emotional perception is implemented in part via nonhierarchical causal interactions across these three regions.NEW & NOTEWORTHY Due to the practical limitations of noninvasive recording methodologies, there is a scarcity of data regarding the interactions of human amygdala and orbitofrontal cortex (OFC). Using rapid functional MRI sampling and directional connectivity, we found that the human amygdala influences emotional perception via distinct interactions with late-stage ventral visual cortex and OFC, in addition to distinct interactions between OFC and fusiform gyrus. Future efforts may leverage these patterns of directional connectivity to noninvasively distinguish clinical groups from controls with respect to network causal hierarchy.

Effects of Ventral Striatum Lesions on Stimulus-Based versus Action-Based Reinforcement Learning.

Learning the values of actions versus stimuli may depend on separable neural circuits. In the current study, we evaluated the performance of rhesus macaques with ventral striatum (VS) lesions on a two-arm bandit task that had randomly interleaved blocks of stimulus-based and action-based reinforcement learning (RL). Compared with controls, monkeys with VS lesions had deficits in learning to select rewarding images but not rewarding actions. We used a RL model to quantify learning and choice consistency and found that, in stimulus-based RL, the VS lesion monkeys were more influenced by negative feedback and had lower choice consistency than controls. Using a Bayesian model to parse the groups' learning strategies, we also found that VS lesion monkeys defaulted to an action-based choice strategy. Therefore, the VS is involved specifically in learning the value of stimuli, not actions.SIGNIFICANCE STATEMENT Reinforcement learning models of the ventral striatum (VS) often assume that it maintains an estimate of state value. This suggests that it plays a general role in learning whether rewards are assigned based on a chosen action or stimulus. In the present experiment, we examined the effects of VS lesions on monkeys' ability to learn that choosing a particular action or stimulus was more likely to lead to reward. We found that VS lesions caused a specific deficit in the monkeys' ability to discriminate between images with different values, whereas their ability to discriminate between actions with different values remained intact. Our results therefore suggest that the VS plays a specific role in learning to select rewarded stimuli.

Learned Value Shapes Responses to Objects in Frontal and Ventral Stream Networks in Macaque Monkeys.

We have an incomplete picture of how the brain links object representations to reward value, and how this information is stored and later retrieved. The orbitofrontal cortex (OFC), medial frontal cortex (MFC), and ventrolateral prefrontal cortex (VLPFC), together with the amygdala, are thought to play key roles in these processes. There is an apparent discrepancy, however, regarding frontal areas thought to encode value in macaque monkeys versus humans. To address this issue, we used fMRI in macaque monkeys to localize brain areas encoding recently learned image values. Each week, monkeys learned to associate images of novel objects with a high or low probability of water reward. Areas responding to the value of recently learned reward-predictive images included MFC area 10 m/32, VLPFC area 12, and inferior temporal visual cortex (IT). The amygdala and OFC, each thought to be involved in value encoding, showed little such effect. Instead, these 2 areas primarily responded to visual stimulation and reward receipt, respectively. Strong image value encoding in monkey MFC compared with OFC is surprising, but agrees with results from human imaging studies. Our findings demonstrate the importance of VLPFC, MFC, and IT in representing the values of recently learned visual images.

Reversal learning and dopamine: a bayesian perspective.

Reversal learning has been studied as the process of learning to inhibit previously rewarded actions. Deficits in reversal learning have been seen after manipulations of dopamine and lesions of the orbitofrontal cortex. However, reversal learning is often studied in animals that have limited experience with reversals. As such, the animals are learning that reversals occur during data collection. We have examined a task regime in which monkeys have extensive experience with reversals and stable behavioral performance on a probabilistic two-arm bandit reversal learning task. We developed a Bayesian analysis approach to examine the effects of manipulations of dopamine on reversal performance in this regime. We find that the analysis can clarify the strategy of the animal. Specifically, at reversal, the monkeys switch quickly from choosing one stimulus to choosing the other, as opposed to gradually transitioning, which might be expected if they were using a naive reinforcement learning (RL) update of value. Furthermore, we found that administration of haloperidol affects the way the animals integrate prior knowledge into their choice behavior. Animals had a stronger prior on where reversals would occur on haloperidol than on levodopa (l-DOPA) or placebo. This strong prior was appropriate, because the animals had extensive experience with reversals occurring in the middle of the block. Overall, we find that Bayesian dissection of the behavior clarifies the strategy of the animals and reveals an effect of haloperidol on integration of prior information with evidence in favor of a choice reversal.

The Role of Frontal Cortical and Medial-Temporal Lobe Brain Areas in Learning a Bayesian Prior Belief on Reversals.

Reversal learning has been extensively studied across species as a task that indexes the ability to flexibly make and reverse deterministic stimulus-reward associations. Although various brain lesions have been found to affect performance on this task, the behavioral processes affected by these lesions have not yet been determined. This task includes at least two kinds of learning. First, subjects have to learn and reverse stimulus-reward associations in each block of trials. Second, subjects become more proficient at reversing choice preferences as they experience more reversals. We have developed a Bayesian approach to separately characterize these two learning processes. Reversal of choice behavior within each block is driven by a combination of evidence that a reversal has occurred, and a prior belief in reversals that evolves with experience across blocks. We applied the approach to behavior obtained from 89 macaques, comprising 12 lesion groups and a control group. We found that animals from all of the groups reversed more quickly as they experienced more reversals, and correspondingly they updated their prior beliefs about reversals at the same rate. However, the initial values of the priors that the various groups of animals brought to the task differed significantly, and it was these initial priors that led to the differences in behavior. Thus, by taking a Bayesian approach we find that variability in reversal-learning performance attributable to different neural systems is primarily driven by different prior beliefs about reversals that each group brings to the task. SIGNIFICANCE STATEMENT: The ability to use prior knowledge to adapt choice behavior is critical for flexible decision making. Reversal learning is often studied as a form of flexible decision making. However, prior studies have not identified which brain regions are important for the formation and use of prior beliefs to guide choice behavior. Here we develop a Bayesian approach that formally characterizes learning set as a concept, and we show that, in macaque monkeys, the amygdala and medial prefrontal cortex have a role in establishing an initial belief about the stability of the reward environment.

Frontal-parietal and limbic-striatal activity underlies information sampling in the best choice problem.

Best choice problems have a long mathematical history, but their neural underpinnings remain unknown. Best choice tasks are optimal stopping problem that require subjects to view a list of options one at a time and decide whether to take or decline each option. The goal is to find a high ranking option in the list, under the restriction that declined options cannot be chosen in the future. Conceptually, the decision to take or decline an option is related to threshold crossing in drift diffusion models, when this process is thought of as a value comparison. We studied this task in healthy volunteers using fMRI, and used a Markov decision process to quantify the value of continuing to search versus committing to the current option. Decisions to take versus decline an option engaged parietal and dorsolateral prefrontal cortices, as well ventral striatum, anterior insula, and anterior cingulate. Therefore, brain regions previously implicated in evidence integration and reward representation encode threshold crossings that trigger decisions to commit to a choice.

Imaging distributed and massed repetitions of natural scenes: spontaneous retrieval and maintenance.

Repetitions that are distributed (spaced) across time prompt enhancement of a memory-related event-related potential, compared to when repetitions are massed (contiguous). Here, we used fMRI to investigate neural enhancement and suppression effects during free viewing of natural scenes that were either novel or repeated four times with massed or distributed repetitions. Distributed repetition was uniquely associated with a repetition enhancement effect in a bilateral posterior parietal cluster that included the precuneus and posterior cingulate and which has previously been implicated in episodic memory retrieval. Unique to massed repetition, conversely, was enhancement in a right dorsolateral prefrontal cluster that has been implicated in short-term maintenance. Repetition suppression effects for both types of spacing were widespread in regions activated during novel picture processing. Taken together, the data are consistent with a hypothesis that distributed repetition prompts spontaneous retrieval of prior occurrences, whereas massed repetition prompts short-term maintenance of the episodic representation, due to contiguous presentation. These processing differences may mediate the classic spacing effect in learning and memory.

Contribution of amygdala to dynamic model arbitration under uncertainty.

Intrinsic uncertainty in the reward environment requires the brain to run multiple models simultaneously to predict outcomes based on preceding cues or actions, commonly referred to as stimulus- and action-based learning. Ultimately, the brain also must adopt appropriate choice behavior using reliability of these models. Here, we combined multiple experimental and computational approaches to quantify concurrent learning in monkeys performing tasks with different levels of uncertainty about the model of the environment. By comparing behavior in control monkeys and monkeys with bilateral lesions to the amygdala or ventral striatum, we found evidence for dynamic, competitive interaction between stimulus-based and action-based learning, and for a distinct role of the amygdala. Specifically, we demonstrate that the amygdala adjusts the initial balance between the two learning systems, thereby altering the interaction between arbitration and learning that shapes the time course of both learning and choice behaviors. This novel role of the amygdala can account for existing contradictory observations and provides testable predictions for future studies into circuit-level mechanisms of flexible learning and choice under uncertainty.

Transcriptomic diversity of amygdalar subdivisions across humans and nonhuman primates.

The amygdaloid complex mediates learning, memory, and emotions. Understanding the cellular and anatomical features that are specialized in the amygdala of primates versus other vertebrates requires a systematic, anatomically-resolved molecular analysis of constituent cell populations. We analyzed five nuclear subdivisions of the primate amygdala with single-nucleus RNA sequencing in macaques, baboons, and humans to examine gene expression profiles for excitatory and inhibitory neurons and confirmed our results with single-molecule FISH analysis. We identified distinct subtypes of FOXP2 + interneurons in the intercalated cell masses and protein-kinase C-δ interneurons in the central nucleus. We also establish that glutamatergic, pyramidal-like neurons are transcriptionally specialized within the basal, lateral, or accessory basal nuclei. Understanding the molecular heterogeneity of anatomically-resolved amygdalar neuron types provides a cellular framework for improving existing models of how amygdalar neural circuits contribute to cognition and mental health in humans by using nonhuman primates as a translational bridge.

Lesions to the mediodorsal thalamus, but not orbitofrontal cortex, enhance volatility beliefs linked to paranoia.

Beliefs-attitudes toward some state of the environment-guide action selection and should be robust to variability but sensitive to meaningful change. Beliefs about volatility (expectation of change) are associated with paranoia in humans, but the brain regions responsible for volatility beliefs remain unknown. The orbitofrontal cortex (OFC) is central to adaptive behavior, whereas the magnocellular mediodorsal thalamus (MDmc) is essential for arbitrating between perceptions and action policies. We assessed belief updating in a three-choice probabilistic reversal learning task following excitotoxic lesions of the MDmc (n = 3) or OFC (n = 3) and compared performance with that of unoperated monkeys (n = 14). Computational analyses indicated a double dissociation: MDmc, but not OFC, lesions were associated with erratic switching behavior and heightened volatility belief (as in paranoia in humans), whereas OFC, but not MDmc, lesions were associated with increased lose-stay behavior and reward learning rates. Given the consilience across species and models, these results have implications for understanding paranoia.

Adverse childhood experiences and hormonal contraception: Interactive impact on sexual reward function.

The current literature suggests that some women are uniquely vulnerable to negative effects of hormonal contraception (HC) on affective processes. However, little data exists as to which factors contribute to such vulnerability. The present study evaluated the impact of prepubertal adverse childhood experiences (ACEs) on reward processing in women taking HC (N = 541) compared to naturally cycling women (N = 488). Participants completed an online survey assessing current and past HC use and exposure to 10 different adverse childhood experiences (ACEs) before puberty (ACE Questionnaire), with participants categorized into groups of low (0-1) versus high (≥2) prepubertal ACE exposure. Participants then completed a reward task rating their expected and experienced valence for images that were either erotic, pleasant (non-erotic), or neutral. Significant interactions emerged between prepubertal ACE exposure and HC use on expected (p = 0.028) and experienced (p = 0.025) valence ratings of erotic images but not pleasant or neutral images. Importantly, follow-up analyses considering whether women experienced HC-induced decreases in sexual desire informed the significant interaction for expected valence ratings of erotic images. For current HC users, prepubertal ACEs interacted with HC-induced decreased sexual desire (p = 0.008), such that high ACE women reporting decreased sexual desire on HC showed substantially decreased ratings for anticipated erotic images compared to both high prepubertal ACE women without decreased sexual desire (p < 0.001) and low prepubertal ACE women also reporting decreased sexual desire (p = 0.010). The interaction was not significant in naturally cycling women reporting previous HC use, suggesting that current HC use could be impacting anticipatory reward processing of sexual stimuli among certain women (e.g., high prepubertal ACE women reporting HC-induced decreases in sexual desire). The study provides rationale for future randomized, controlled trials to account for prepubertal ACE exposure to promote contraceptive selection informed by behavioral evidence.

Efficient viral expression of a chemogenetic receptor in the old-world monkey amygdala.

Genetically encoded synthetic receptors, such as the chemogenetic and optogenetic proteins, are powerful tools for functional brain studies in animals. In the primate brain, with its comparatively large, intricate anatomical structures, it can be challenging to express transgenes, such as the hM4Di chemogenetic receptor, in a defined anatomical structure with high penetrance. Here, we compare parameters for lentivirus vector injections in the rhesus monkey amygdala. We find that four injections of 20 µl, infused at 0.5 µl/min, can achieve neuronal hM4Di expression in 50-100% of neurons within a 60 mm3 volume, without observable damage from overexpression. Increasing the number of hM4Di_CFP lentivirus injections to up to 12 sites per hemisphere, resulted in 30%-40% neuronal coverage of the overall amygdala volume, with coverage reaching 60% in some subnuclei. Manganese Chloride was mixed with lentivirus and used as an MRI marker to verify targeting accuracy and correct unsuccessful injections in these experiments. In a separate monkey we visualized, in vivo, viral expression of the hM4Di receptor protein in the amygdala, using Positron Emission Tomography. Together, these data show efficient and verifiable expression of a chemogenetic receptor in old-world monkey amygdala.

The amygdala is not necessary for the familiarity aspect of recognition memory.

Dual-process accounts of item recognition posit two memory processes: slow but detailed recollection, and quick but vague familiarity. It has been proposed, based on prior rodent work, that the amygdala is critical for the familiarity aspect of item recognition. Here, we evaluated this proposal in male rhesus monkeys (Macaca mulatta) with selective bilateral excitotoxic amygdala damage. We used four established visual memory tests designed to assess different aspects of familiarity, all administered on touchscreen computers. Specifically, we assessed monkeys' tendencies to make low-latency false alarms, to make false alarms to recently seen lures, to produce curvilinear ROC curves, and to discriminate stimuli based on repetition across days. Three of the four tests showed no familiarity impairment and the fourth was explained by a deficit in reward processing. Consistent with this, amygdala damage did produce an anticipated deficit in reward processing in a three-arm-bandit gambling task, verifying the effectiveness of the lesions. Together, these results contradict prior rodent work and suggest that the amygdala is not critical for the familiarity aspect of item recognition.