Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys.

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Monitoring and Updating of Action Selection for Goal-Directed Behavior through the Striatal Direct and Indirect Pathways.

The basal ganglia play key roles in adaptive behaviors guided by reward and punishment. However, despite accumulating knowledge, few studies have tested how heterogeneous signals in the basal ganglia are organized and coordinated for goal-directed behavior. In this study, we investigated neuronal signals of the direct and indirect pathways of the basal ganglia as rats performed a lever push/pull task for a probabilistic reward. In the dorsomedial striatum, we found that optogenetically and electrophysiologically identified direct pathway neurons encoded reward outcomes, whereas indirect pathway neurons encoded no-reward outcome and next-action selection. Outcome coding occurred in association with the chosen action. In support of pathway-specific neuronal coding, light activation induced a bias on repeat selection of the same action in the direct pathway, but on switch selection in the indirect pathway. Our data reveal the mechanisms underlying monitoring and updating of action selection for goal-directed behavior through basal ganglia circuits.

Roles of centromedian parafascicular nuclei of thalamus and cholinergic interneurons in the dorsal striatum in associative learning of environmental events.

The thalamus provides a massive input to the striatum, but despite accumulating evidence, the functions of this system remain unclear. It is known, however, that the centromedian (CM) and parafascicular (Pf) nuclei of the thalamus can strongly influence particular striatal neuron subtypes, notably including the cholinergic interneurons of the striatum (CINs), key regulators of striatal function. Here, we highlight the thalamostriatal system through the CM-Pf to striatal CINs. We consider how, by virtue of the direct synaptic connections of the CM and PF, their neural activity contributes to the activity of CINs and striatal projection neurons (SPNs). CM-Pf neurons are strongly activated at sudden changes in behavioral context, such as switches in action-outcome contingency or sequence of behavioral requirements, suggesting that their activity may represent change of context operationalized as associability. Striatal CINs, on the other hand, acquire and loose responses to external events associated with particular contexts. In light of this physiological evidence, we propose a hypothesis of the CM-Pf-CINs system, suggesting that it augments associative learning by generating an associability signal and promotes reinforcement learning guided by reward prediction error signals from dopamine-containing neurons. We discuss neuronal circuit and synaptic organizations based on in vivo/in vitro studies that we suppose to underlie our hypothesis. Possible implications of CM-Pf-CINs dysfunction (or degeneration) in brain diseases are also discussed by focusing on Parkinson's disease.

Distinct Functions of the Primate Putamen Direct and Indirect Pathways in Adaptive Outcome-Based Action Selection.

Cortico-basal ganglia circuits are critical regulators of reward-based decision making. Reinforcement learning models posit that action reward value is encoded by the firing activity of striatal medium spiny neurons (MSNs) and updated upon changing reinforcement contingencies by dopamine (DA) signaling to these neurons. However, it remains unclear how the anatomically distinct direct and indirect pathways through the basal ganglia are involved in updating action reward value under changing contingencies. MSNs of the direct pathway predominantly express DA D1 receptors and those of the indirect pathway predominantly D2 receptors, so we tested for distinct functions in behavioral adaptation by injecting D1 and D2 receptor antagonists into the putamen of two macaque monkeys performing a free choice task for probabilistic reward. In this task, monkeys turned a handle toward either a left or right target depending on an asymmetrically assigned probability of large reward. Reward probabilities of left and right targets changed after 30-150 trials, so the monkeys were required to learn the higher-value target choice based on action-outcome history. In the control condition, the monkeys showed stable selection of the higher-value target (that more likely to yield large reward) and kept choosing the higher-value target regardless of less frequent small reward outcomes. The monkeys also made flexible changes of selection away from the high-value target when two or three small reward outcomes occurred randomly in succession. DA D1 antagonist injection significantly increased the probability of the monkey switching to the alternate target in response to successive small reward outcomes. Conversely, D2 antagonist injection significantly decreased the switching probability. These results suggest distinct functions of D1 and D2 receptor-mediated signaling processes in action selection based on action-outcome history, with D1 receptor-mediated signaling promoting the stable choice of higher-value targets and D2 receptor-mediated signaling promoting a switch in action away from small reward outcomes. Therefore, direct and indirect pathways appear to have complementary functions in maintaining optimal goal-directed action selection and updating action value, which are dependent on D1 and D2 DA receptor signaling.

Separation of metalloproteins using a novel metal ion contaminant sweeping technique and detection of protein-bound copper by a metal ion probe in polyacrylamide gel electrophoresis: distribution of copper in human serum.

A polyacrylamide gel electrophoresis (PAGE)-based method has been developed, consisting of two types of gel electrophoresis, to obtain an accurate distribution of protein-bound metal ions in biological samples. First, proteins are separated by PAGE without the uptake of contaminant metal ions in the separation field and dissociation of metal ions from the proteins. This is followed by another PAGE for the separation and detection of protein-bound metal ions in small volume samples with high sensitivity in the ppt range using a fluorescent metal probe. The former is a new technique using blue-native (BN) PAGE to electrophoretically sweep all metal contaminants by employing two kinds of chelating agents. These agents form complexes with contaminants in the gel and the separation buffer solution, which migrate towards opposite pole directions, thus lowering the contaminants to below the ppt level during separation. This is termed "Metal Ion Contaminant Sweeping BN-PAGE (MICS-BN-PAGE)". After the separation of proteins under these first metal-free conditions, the metal ions in the gel fractions are eluted, followed by derivatization of copper ions into the metal probe complexes to be separated and determined by fluorescence detection in the second PAGE. In this PAGE-based method, the copper ions bound to ceruloplasmin and superoxide dismutase were quantitatively determined, in addition to the exchangeable albumin-bound copper ions. This system successfully provided distribution maps of protein-copper in human serum. The precise distribution of copper in human serum was investigated, and found to be different from that which is widely accepted.

Coding of the long-term value of multiple future rewards in the primate striatum.

Decisions maximizing benefits involve a tradeoff between the quantity of a reward and the cost of elapsed time until an animal receives it. The estimation of long-term reward values is critical to attain the most desirable outcomes over a certain period of time. Reinforcement learning theories have established algorithms to estimate the long-term reward values of multiple future rewards in which the values of future rewards are discounted as a function of how many steps of choices are necessary to achieve them. Here, we report that presumed striatal projection neurons represent the long-term values of multiple future rewards estimated by a standard reinforcement learning model while monkeys are engaged in a series of trial-and-error choices and adaptive decisions for multiple rewards. We found that the magnitude of activity of a subset of neurons was positively correlated with the long-term reward values, and that of another subset of neurons was negatively correlated throughout the entire decision-making process in individual trials: from the start of the task trial, estimation of the values and their comparison among alternatives, choice execution, and evaluation of the received rewards. An idiosyncratic finding was that neurons showing negative correlations represented reward values in the near future (high discounting), while neurons showing positive correlations represented reward values not only in the near future, but also in the far future (low discounting). These findings provide a new insight that long-term value signals are embedded in two subsets of striatal neurons as high and low discounting of multiple future rewards.

Dopamine neurons learn to encode the long-term value of multiple future rewards.

Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multistep choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.