Dopamine Neurons Encoding Long-Term Memory of Object Value for Habitual Behavior.

Dopamine neurons promote learning by processing recent changes in reward values, such that reward may be maximized. However, such a flexible signal is not suitable for habitual behaviors that are sustained regardless of recent changes in reward outcome. We discovered a type of dopamine neuron in the monkey substantia nigra pars compacta (SNc) that retains past learned reward values stably. After reward values of visual objects are learned, these neurons continue to respond differentially to the objects, even when reward is not expected. Responses are strengthened by repeated learning and are evoked upon presentation of the objects long after learning is completed. These "sustain-type" dopamine neurons are confined to the caudal-lateral SNc and project to the caudate tail, which encodes long-term value memories of visual objects and guides gaze automatically to stably valued objects. This population of dopamine neurons thus selectively promotes learning and retention of habitual behavior.

Environment-based object values learned by local network in the striatum tail.

Basal ganglia contribute to object-value learning, which is critical for survival. The underlying neuronal mechanism is the association of each object with its rewarding outcome. However, object values may change in different environments and we then need to choose different objects accordingly. The mechanism of this environment-based value learning is unknown. To address this question, we created an environment-based value task in which the value of each object was reversed depending on the two scene-environments (X and Y). After experiencing this task repeatedly, the monkeys became able to switch the choice of object when the scene-environment changed unexpectedly. When we blocked the inhibitory input from fast-spiking interneurons (FSIs) to medium spiny projection neurons (MSNs) in the striatum tail by locally injecting IEM-1460, the monkeys became unable to learn scene-selective object values. We then studied the mechanism of the FSI-MSN connection. Before and during this learning, FSIs responded to the scenes selectively, but were insensitive to object values. In contrast, MSNs became able to discriminate the objects (i.e., stronger response to good objects), but this occurred clearly in one of the two scenes (X or Y). This was caused by the scene-selective inhibition by FSI. As a whole, MSNs were divided into two groups that were sensitive to object values in scene X or in scene Y. These data indicate that the local network of striatum tail controls the learning of object values that are selective to the scene-environment. This mechanism may support our flexible switching behavior in various environments.

Basal ganglia circuits for reward value-guided behavior.

The basal ganglia are equipped with inhibitory and disinhibitory mechanisms that enable a subject to choose valuable objects and actions. Notably, a value can be determined flexibly by recent experience or stably by prolonged experience. Recent studies have revealed that the head and tail of the caudate nucleus selectively and differentially process flexible and stable values of visual objects. These signals are sent to the superior colliculus through different parts of the substantia nigra so that the animal looks preferentially at high-valued objects, but in different manners. Thus, relying on short-term value memories, the caudate head circuit allows the subject's gaze to move expectantly to recently valued objects. Relying on long-term value memories, the caudate tail circuit allows the subject's gaze to move automatically to previously valued objects. The basal ganglia also contain an equivalent parallel mechanism for action values. Such flexible-stable parallel mechanisms for object and action values create a highly adaptable system for decision making.

Multiple neuronal circuits for variable object-action choices based on short- and long-term memories.

At each time in our life, we choose one or few behaviors, while suppressing many other behaviors. This is the basic mechanism in the basal ganglia, which is done by tonic inhibition and selective disinhibition. Dysfunctions of the basal ganglia then cause 2 types of disorders (difficulty in initiating necessary actions and difficulty in suppressing unnecessary actions) that occur in Parkinson's disease. The basal ganglia generate such opposite outcomes through parallel circuits: The direct pathway for initiation and indirect pathway for suppression. Importantly, the direct pathway processes good information and the indirect pathway processes bad information, which enables the choice of good behavior and the rejection of bad behavior. This is mainly enabled by dopaminergic inputs to these circuits. However, the value judgment is complex because the world is complex. Sometimes, the value must be based on recent events, thus is based on short-term memories. Or, the value must be based on historical events, thus is based on long-term memories. Such memory-based value judgment is generated by another parallel circuit originating from the caudate head and caudate tail. These circuit-information mechanisms allow other brain areas (e.g., prefrontal cortex) to contribute to decisions by sending information to these basal ganglia circuits. Moreover, the basal ganglia mechanisms (i.e., what to choose) are associated with cerebellum mechanisms (i.e., when to choose). Overall, multiple levels of parallel circuits in and around the basal ganglia are essential for coordinated behaviors. Understanding these circuits is useful for creating clinical treatments of disorders resulting from the failure of these circuits.

Visualization of iron-rich subcortical structures in non-human primates in vivo by quantitative susceptibility mapping at 3T MRI.

Magnetic resonance imaging (MRI) is now an essential tool in the field of neuroscience involving non-human primates (NHP). Structural MRI scanning using T1-weighted (T1w) or T2-weighted (T2w) images provides anatomical information, particularly for experiments involving deep structures such as the basal ganglia and cerebellum. However, for certain subcortical structures, T1w and T2w image contrasts are insufficient for their detection of important anatomical details. To better visualize such structures in the macaque brain, we applied a relatively new method called quantitative susceptibility mapping (QSM), which enhances tissue contrast based on the local tissue magnetic susceptibility. The QSM significantly improved the visualization of important structures, including the ventral pallidum (VP), globus pallidus external and internal segments (GPe and GPi), substantia nigra (SN), subthalamic nucleus (STN) in the basal ganglia and the dentate nucleus (DN) in the cerebellum. We quantified this the contrast enhancement by systematically comparing of contrast-to-noise ratios (CNRs) of QSM images relative to the corresponding T1w and T2w images. In addition, QSM values of some structures were correlated to the age of the macaque subjects. These results identify the QSM method as a straightforward and useful tool for clearly visualizing details of subcortical structures that are invisible with more traditional scanning sequences.

Amygdala activity for the modulation of goal-directed behavior in emotional contexts.

Choosing valuable objects and rewarding actions is critical for survival. While such choices must be made in a way that suits the animal's circumstances, the neural mechanisms underlying such context-appropriate behavior are unclear. To address this question, we devised a context-dependent reward-seeking task for macaque monkeys. Each trial started with the appearance of one of many visual scenes containing two or more objects, and the monkey had to choose the good object by saccade to get a reward. These scenes were categorized into two dimensions of emotional context: dangerous versus safe and rich versus poor. We found that many amygdala neurons were more strongly activated by dangerous scenes, by rich scenes, or by both. Furthermore, saccades to target objects occurred more quickly in dangerous than in safe scenes and were also quicker in rich than in poor scenes. Thus, amygdala neuronal activity and saccadic reaction times were negatively correlated in each monkey. These results suggest that amygdala neurons facilitate targeting saccades predictably based on aspects of emotional context, as is necessary for goal-directed and social behavior.

Temporal-prefrontal cortical network for discrimination of valuable objects in long-term memory.

Remembering and discriminating objects based on their previously learned values are essential for goal-directed behaviors. While the cerebral cortex is known to contribute to object recognition, surprisingly little is known about its role in retaining long-term object-value associations. To address this question, we trained macaques to arbitrarily associate small or large rewards with many random fractal objects (>100) and then used fMRI to study the long-term retention of value-based response selectivity across the brain. We found a pronounced long-term value memory in core subregions of temporal and prefrontal cortex where, several months after training, fractals previously associated with high reward ("good" stimuli) elicited elevated fMRI responses compared with those associated with low reward ("bad" stimuli). Similar long-term value-based modulation was also observed in subregions of the striatum, amygdala, and claustrum, but not in the hippocampus. The value-modulated temporal-prefrontal subregions showed strong resting-state functional connectivity to each other. Moreover, for areas outside this core, the magnitude of long-term value responses was predicted by the strength of resting-state functional connectivity to the core subregions. In separate testing, free-viewing gaze behavior indicated that the monkeys retained stable long-term memory of object value. These results suggest an implicit and high-capacity memory mechanism in the temporal-prefrontal circuitry and its associated subcortical regions for long-term retention of object-value memories that can guide value-oriented behavior.

The Caudal Part of Putamen Represents the Historical Object Value Information.

The basal ganglia, especially the circuits originating from the putamen, are essential for controlling normal body movements. Notably, the putamen receives inputs not only from motor cortical areas but also from multiple sensory cortices. However, how these sensory signals are processed in the putamen remains unclear. We recorded the activity of tentative medium spiny neurons in the caudal part of the putamen when the monkey viewed many fractal objects. We found many neurons that responded to these objects, mostly in the ventral region. We called this region "putamen tail" (PUTt), as it is dorsally adjacent to "caudate tail" (CDt). Although PUTt and CDt are mostly separated by a thin layer of white matter, their neurons shared several features. Almost all of them had receptive fields in the contralateral hemifield. Moreover, their responses were object selective (i.e., variable across objects). The object selectivity was higher in the ventral region (i.e., CDt > PUTt). Some neurons above PUTt, which we called the caudal-dorsal putamen (cdPUT), also responded to objects, but less selectively than PUTt. Next, we examined whether these visual neurons changed their responses based on the reward outcome. We found that many neurons encoded the values of many objects based on long-term memory, but not based on short-term memory. Such stable value responses were stronger in PUTt and CDt than in cdPUT. These results suggest that PUTt, together with CDt, controls saccade/attention among objects with different historical values, and may control other motor actions as well.SIGNIFICANCE STATEMENT Although the putamen receives inputs not only from motor cortical areas but also from sensory cortical areas, how these sensory signals are processed remains unclear. Here we found that neurons in the caudal-ventral part of the putamen (putamen tail) process visual information including spatial and object features. These neurons discriminate many objects, first by their visual features and later by their reward values as well. Importantly, the value discrimination was based on long-term memory, but not on short-term memory. These results suggest that the putamen tail controls saccade/attention among objects with different historical values and might control other motor actions as well.

Medial thalamus in the territory of oculomotor basal ganglia represents stable object value.

Many visual objects are attached with values which were created by our long rewarding history. Such stable object values attract gaze. We previously found that the output pathway of basal ganglia from caudal-dorsal-lateral portion of substantia nigra pars reticulata (cdlSNr) to superior colliculus (SC) carries robust stable value signal to execute the automatic choice of valuable objects. An important question here is whether stable value signal in basal ganglia can influence on other inner processing such as perception, attention, emotion, or arousal than motor execution. The key brain circuit is another output path of basal ganglia: the pathway from SNr to temporal and frontal lobes through thalamus. To examine the existence of stable value signal in this pathway, we explored thalamus in a wide range. We found that many neurons in the medial thalamus represented stable value. Histological examination showed that the recorded sites of those neurons included ventral anterior nucleus, pars magnocellularis (VAmc) which is the main target of nigrothalamic projection. Consistent with the SNr GABArgic projection, the latency of value signal in the medial thalamus was later than cdlSNr, and the sign of value coding in the medial thalamus was opposite to cdlSNr. As is the case with cdlSNr neurons, the medial thalamus neurons showed no sensitivity to frequently updated value (flexible value). These results suggest that the pathway from cdlSNr to the medial thalamus influences on various aspects of cognitive processing by propagating stable value signal to the wide cortical area.

Neuronal connections of direct and indirect pathways for stable value memory in caudal basal ganglia.

Direct and indirect pathways in the basal ganglia work together for controlling behavior. However, it is still a controversial topic whether these pathways are segregated or merged with each other. To address this issue, we studied the connections of these two pathways in the caudal parts of the basal ganglia of rhesus monkeys using anatomical tracers. Our previous studies showed that the caudal basal ganglia control saccades by conveying long-term values (stable values) of many visual objects toward the superior colliculus. In experiment 1, we injected a tracer in the caudate tail (CDt), and found local dense plexuses of axon terminals in the caudal-dorsal-lateral part of substantia nigra pars reticulata (cdlSNr) and the caudal-ventral part of globus pallidus externus (cvGPe). These anterograde projections may correspond to the direct and indirect pathways, respectively. To verify this in experiment 2, we injected different tracers into cdlSNr and cvGPe, and found many retrogradely labeled neurons in CDt and, in addition, the caudal-ventral part of the putamen (cvPut). These cdlSNr-projecting and cvGPe-projecting neurons were found intermingled in both CDt and cvPut (which we call "striatum tail"). A small but significant proportion of neurons (<15%) were double-labeled, indicating that they projected to both cdlSNr and cvGPe. These anatomical results suggest that stable value signals (good vs. bad) are sent from the striatum tail to cdlSNr and cvGPe in a biased (but not exclusive) manner. These connections may play an important role in biasing saccades toward higher valued objects and away from lower valued objects.

Direct and indirect pathways for choosing objects and actions.

A prominent target of the basal ganglia is the superior colliculus (SC) which controls gaze orientation (saccadic eye movement in primates) to an important object. This 'object choice' is crucial for choosing an action on the object. SC is innervated by the substantia nigra pars reticulata (SNr) which is controlled mainly by the caudate nucleus (CD). This CD-SNr-SC circuit is sensitive to the values of individual objects and facilitates saccades to good objects. The object values are processed differently in two parallel circuits: flexibly by the caudate head (CDh) and stably by the caudate tail (CDt). To choose good objects, we need to reject bad objects. In fact, these contrasting functions are accomplished by the circuit originating from CDt: The direct pathway focuses on good objects and facilitates saccades to them; the indirect pathway focuses on bad objects and suppresses saccades to them. Inactivation of CDt deteriorated the object choice, because saccades to bad objects were no longer suppressed. This suggests that the indirect pathway is important for object choice. However, the direct and indirect pathways for 'object choice', which aim at the same action (i.e., saccade), may not work for 'action choice'. One possibility is that circuits controlling different actions are connected through the indirect pathway. Additional connections of the indirect pathway with brain areas outside the basal ganglia may also provide a wider range of behavioral choice. In conclusion, basal ganglia circuits are composed of the basic direct/indirect pathways and additional connections and thus have acquired multiple functions.

Parallel basal ganglia circuits for decision making.

The basal ganglia control body movements, mainly, based on their values. Critical for this mechanism is dopamine neurons, which sends unpredicted value signals, mainly, to the striatum. This mechanism enables animals to change their behaviors flexibly, eventually choosing a valuable behavior. However, this may not be the best behavior, because the flexible choice is focused on recent, and, therefore, limited, experiences (i.e., short-term memories). Our old and recent studies suggest that the basal ganglia contain separate circuits that process value signals in a completely different manner. They are insensitive to recent changes in value, yet gradually accumulate the value of each behavior (i.e., movement or object choice). These stable circuits eventually encode values of many behaviors and then retain the value signals for a long time (i.e., long-term memories). They are innervated by a separate group of dopamine neurons that retain value signals, even when no reward is predicted. Importantly, the stable circuits can control motor behaviors (e.g., hand or eye) quickly and precisely, which allows animals to automatically acquire valuable outcomes based on historical life experiences. These behaviors would be called 'skills', which are crucial for survival. The stable circuits are localized in the posterior part of the basal ganglia, separately from the flexible circuits located in the anterior part. To summarize, the flexible and stable circuits in the basal ganglia, working together but independently, enable animals (and humans) to reach valuable goals in various contexts.

What do eye movements tell us about patients with neurological disorders? - An introduction to saccade recording in the clinical setting.

Non-invasive and readily implemented in the clinical setting, eye movement studies have been conducted extensively not only in healthy human subjects but also in patients with neurological disorders. The purpose of saccade studies is to "read out" the pathophysiology underlying neurological disorders from the saccade records, referring to known primate physiology. In the current review, we provide an overview of studies in which we attempted to elucidate the patterns of saccade abnormalities in over 250 patients with neurological disorders, including cerebellar ataxia and brainstem pathology due to neurodegenerative disorders, and what they tell about the pathophysiology of patients with neurological disorders. We also discuss how interventions, such as deep brain stimulation, affect saccade performance and provide further insights into the workings of the oculomotor system in humans. Finally, we argue that it is important to understand the functional significance and behavioral correlate of saccade abnormalities in daily life, which could require eye tracking methodologies to be performed in settings similar to daily life.

Neurons in the Primate Medial Basal Forebrain Signal Combined Information about Reward Uncertainty, Value, and Punishment Anticipation.

It has been suggested that the basal forebrain (BF) exerts strong influences on the formation of memory and behavior. However, what information is used for the memory-behavior formation is unclear. We found that a population of neurons in the medial BF (medial septum and diagonal band of Broca) of macaque monkeys encodes a unique combination of information: reward uncertainty, expected reward value, anticipation of punishment, and unexpected reward and punishment. The results were obtained while the monkeys were expecting (often with uncertainty) a rewarding or punishing outcome during a Pavlovian procedure, or unexpectedly received an outcome outside the procedure. In vivo anterograde tracing using manganese-enhanced MRI suggested that the major recipient of these signals is the intermediate hippocampal formation. Based on these findings, we hypothesize that the medial BF identifies various contexts and outcomes that are critical for memory processing in the hippocampal formation.

Parallel basal ganglia circuits for voluntary and automatic behaviour to reach rewards.

The basal ganglia control body movements, value processing and decision-making. Many studies have shown that the inputs and outputs of each basal ganglia structure are topographically organized, which suggests that the basal ganglia consist of separate circuits that serve distinct functions. A notable example is the circuits that originate from the rostral (head) and caudal (tail) regions of the caudate nucleus, both of which target the superior colliculus. These two caudate regions encode the reward values of visual objects differently: flexible (short-term) values by the caudate head and stable (long-term) values by the caudate tail. These value signals in the caudate guide the orienting of gaze differently: voluntary saccades by the caudate head circuit and automatic saccades by the caudate tail circuit. Moreover, separate groups of dopamine neurons innervate the caudate head and tail and may selectively guide the flexible and stable learning/memory in the caudate regions. Studies focusing on manual handling of objects also suggest that rostrocaudally separated circuits in the basal ganglia control the action differently. These results suggest that the basal ganglia contain parallel circuits for two steps of goal-directed behaviour: finding valuable objects and manipulating the valuable objects. These parallel circuits may underlie voluntary behaviour and automatic skills, enabling animals (including humans) to adapt to both volatile and stable environments. This understanding of the functions and mechanisms of the basal ganglia parallel circuits may inform the differential diagnosis and treatment of basal ganglia disorders.

Object-finding skill created by repeated reward experience.

For most animals, survival depends on rapid detection of rewarding objects, but search for an object surrounded by many others is known to be difficult and time consuming. However, there is neuronal evidence for robust and rapid differentiation of objects based on their reward history in primates (Hikosaka, Kim, Yasuda, & Yamamoto, 2014). We hypothesized that such robust coding should support efficient search for high-value objects, similar to a pop-out mechanism. To test this hypothesis, we let subjects (n = 4, macaque monkeys) view a large number of complex objects with consistently biased rewards with variable training durations (1, 5, or 30 + days). Following training, subjects searched for a high-value object (Good) among a variable number of low-value objects (Bad). Consistent with our hypothesis, we found that Good objects were accurately and quickly targeted, often by a single and direct saccade with a very short latency (<200 ms). The dependence of search times on display size reduced significantly with longer reward training, giving rise to a more efficient search (40 ms/item to 16 ms/item). This object-finding skill showed a large capacity for value-biased objects and was maintained in the long-term memory with no interference from reward learning with other objects. Such object-finding skill, and in particular its large capacity and long term retention, would be crucial for maximizing rewards and biological fitness throughout life where many objects are experienced continuously and/or intermittently.

Attention, reward, and information seeking.

Decision making is thought to be guided by the values of alternative options and involve the accumulation of evidence to an internal bound. However, in natural behavior, evidence accumulation is an active process whereby subjects decide when and which sensory stimulus to sample. These sampling decisions are naturally served by attention and rapid eye movements (saccades), but little is known about how saccades are controlled to guide future actions. Here we review evidence that was discussed at a recent symposium, which suggests that information selection involves basal ganglia and cortical mechanisms and that, across different contexts, it is guided by two central factors: the gains in reward and gains in information (uncertainty reduction) associated with sensory cues.

Functional territories in primate substantia nigra pars reticulata separately signaling stable and flexible values.

Gaze is strongly attracted to visual objects that have been associated with rewards. Key to this function is a basal ganglia circuit originating from the caudate nucleus (CD), mediated by the substantia nigra pars reticulata (SNr), and aiming at the superior colliculus (SC). Notably, subregions of CD encode values of visual objects differently: stably by CD tail [CD(T)] vs. flexibly by CD head [CD(H)]. Are the stable and flexible value signals processed separately throughout the CD-SNr-SC circuit? To answer this question, we identified SNr neurons by their inputs from CD and outputs to SC and examined their sensitivity to object values. The direct input from CD was identified by SNr neuron's inhibitory response to electrical stimulation of CD. We found that SNr neurons were separated into two groups: 1) neurons inhibited by CD(T) stimulation, located in the caudal-dorsal-lateral SNr (cdlSNr), and 2) neurons inhibited by CD(H) stimulation, located in the rostral-ventral-medial SNr (rvmSNr). Most of CD(T)-recipient SNr neurons encoded stable values, whereas CD(H)-recipient SNr neurons tended to encode flexible values. The output to SC was identified by SNr neuron's antidromic response to SC stimulation. Among the antidromically activated neurons, many encoded only stable values, while some encoded only flexible values. These results suggest that CD(T)-cdlSNr-SC circuit and CD(H)-rvmSNr-SC circuit transmit stable and flexible value signals, largely separately, to SC. The speed of signal transmission was faster through CD(T)-cdlSNr-SC circuit than through CD(H)-rvmSNr-SC circuit, which may reflect automatic and controlled gaze orienting guided by these circuits.

The habenula: from stress evasion to value-based decision-making.

Surviving in a world with hidden rewards and dangers requires choosing the appropriate behaviours. Recent discoveries indicate that the habenula plays a prominent part in such behavioural choice through its effects on neuromodulator systems, in particular the dopamine and serotonin systems. By inhibiting dopamine-releasing neurons, habenula activation leads to the suppression of motor behaviour when an animal fails to obtain a reward or anticipates an aversive outcome. Moreover, the habenula is involved in behavioural responses to pain, stress, anxiety, sleep and reward, and its dysfunction is associated with depression, schizophrenia and drug-induced psychosis. As a highly conserved structure in the brain, the habenula provides a fundamental mechanism for both survival and decision-making.

Two types of dopamine neuron distinctly convey positive and negative motivational signals.

Midbrain dopamine neurons are activated by reward or sensory stimuli predicting reward. These excitatory responses increase as the reward value increases. This response property has led to a hypothesis that dopamine neurons encode value-related signals and are inhibited by aversive events. Here we show that this is true only for a subset of dopamine neurons. We recorded the activity of dopamine neurons in monkeys (Macaca mulatta) during a Pavlovian procedure with appetitive and aversive outcomes (liquid rewards and airpuffs directed at the face, respectively). We found that some dopamine neurons were excited by reward-predicting stimuli and inhibited by airpuff-predicting stimuli, as the value hypothesis predicts. However, a greater number of dopamine neurons were excited by both of these stimuli, inconsistent with the hypothesis. Some dopamine neurons were also excited by both rewards and airpuffs themselves, especially when they were unpredictable. Neurons excited by the airpuff-predicting stimuli were located more dorsolaterally in the substantia nigra pars compacta, whereas neurons inhibited by the stimuli were located more ventromedially, some in the ventral tegmental area. A similar anatomical difference was observed for their responses to actual airpuffs. These findings suggest that different groups of dopamine neurons convey motivational signals in distinct manners.