Dopamine neurons learn to encode the long-term value of multiple future rewards.

Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multistep choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.

Age-dependent interactive changes in serotonergic and noradrenergic cortical axon terminals in F344 rats.

In the frontal cortex of aging rats, we found an increase in sprouting of the noradrenergic (NA) axons originated from the locus coeruleus (LC). The serotonergic (5-HT) axons originating from the dorsal raphe (DR) share the same cortical area and their age-dependent changes and interactions with NA axons were still unclear. To compare quantitatively the extent of axonal sprouting of DR and LC neurons in the frontal cortex, we extracellularly recorded from both DR and LC neurons in the same animals and antidromically stimulated 32 cortical sites (a pair of stimulating electrodes was moved at 100-mum intervals from 500 to 2000 microm in depth). In addition, to examine the effects of degeneration of 5-HT axons on NA axons, and vice versa, we used specific neurotoxins for 5-HT (PCA) or NA (DSP-4) axons. We also used noradrenaline uptake inhibitor (maprotiline) to verify the effects of NA on degeneration of 5-HT axons. Results suggested that 5-HT axons sprouted between 15 and 17 months of age and noradrenaline accelerated the age-dependent change of 5-HT axons.

Effects of BDNF infusion on the axon terminals of locus coeruleus neurons of aging rats.

Using in vivo electrophysiological techniques and continuous local infusion methods, we examined the effects of brain-derived neurotrophic factor (BDNF) and its specific antibody (anti-BDNF) on the noradrenergic axon terminals of the locus coeruleus (LC) neurons in the frontal cortex of aging rats. Recently, we observed that LC neurons with multiple-threshold antidromic responses (multi-threshold LC neurons) increased critically between 15 and 17 months of age. To examine whether the BDNF is involved in this change occurred in the aging brain, we continuously infused BDNF into the frontal cortex for 14 days. Exogenous BDNF produced a marked increase in the multi-threshold LC neurons in the 13-month-old brain, accompanied with a decrease in threshold current. However, no morphological change in the noradrenergic axons was observed in the BDNF-infused cortex. In contrast, infusion of anti-BDNF led to a dose-dependent reduction of the multi-threshold LC neurons in the 19-month-old brain, accompanied with an increase in threshold current. These findings suggest that BDNF may contribute to functional changes in the presynaptic axon terminals of LC neurons in the aging brain.

Correlated coding of motivation and outcome of decision by dopamine neurons.

We recorded the activity of midbrain dopamine neurons in an instrumental conditioning task in which monkeys made a series of behavioral decisions on the basis of distinct reward expectations. Dopamine neurons responded to the first visual cue that appeared in each trial [conditioned stimulus (CS)] through which monkeys initiated trial for decision while expecting trial-specific reward probability and volume. The magnitude of neuronal responses to the CS was approximately proportional to reward expectations but with considerable discrepancy. In contrast, CS responses appear to represent motivational properties, because their magnitude at trials with identical reward expectation had significant negative correlation with reaction times of the animal after the CS. Dopamine neurons also responded to reinforcers that occurred after behavioral decisions, and the responses precisely encoded positive and negative reward expectation errors (REEs). The gain of coding REEs by spike frequency increased during learning act-outcome contingencies through a few months of task training, whereas coding of motivational properties remained consistent during the learning. We found that the magnitude of CS responses was positively correlated with that of reinforcers, suggesting a modulation of the effectiveness of REEs as a teaching signal by motivation. For instance, rate of learning could be faster when animals are motivated, whereas it could be slower when less motivated, even at identical REEs. Therefore, the dual correlated coding of motivation and REEs suggested the involvement of the dopamine system, both in reinforcement in more elaborate ways than currently proposed and in motivational function in reward-based decision-making and learning.

[Plasticity of central monoaminergic systems during aging].

The locus coeruleus (LC), located within the caudal pontine central gray, is composed of noradrenaline-containing neurons. The axons of these neurons form extensive collateral branches that project widely to many brain sites. The function of the LC is still unclear at present, however, LC neurons are known to exhibit marked axonal regeneration and sprouting in response to brain damage. We investigated the age-related changes in noradrenergic innervations of the frontal cortex, using in vivo electrophysiological techniques and immunohistochemistry. While noradrenergic innervations gradually decreased with age in the frontal cortex, a high degree of sprouting occurred in the LC axon terminals in middle age. Neither the electrophysiological properties of LC neurons nor NA levels in the frontal cortex changed with age. These findings suggested that the LC neurons preserve a strong capacity to remodel their axon terminals even in the aging brain. Exogenous brain-derived neurotrophic factor (BDNF) infusion caused a marked increase in the density of noradrenergic axon in the aged brain, but no trophic action of BDNF was observed in the young or middle-aged brain. The result suggests that BDNF is necessary for the maintenance of noradrenergic innervations in the aged brain.

Brain activation during whole body cooling in humans studied with functional magnetic resonance imaging.

Regional activation of the brain was studied in humans using functional magnetic resonance imaging during whole body cooling that produced thermal comfort/discomfort. Eight normal male subjects lay in a sleeping bag through which air was blown, exposing subjects to cold air (8 degrees C) for 22 min. Each subject scored their degree of thermal comfort and discomfort every min. As the subjects reported more discomfort the blood oxygen level dependent response in the bilateral amygdala increased. There was no activation in the thalamus, somatosensory, cingulate, or insula cortices. This result suggests that the amygdala plays a role in the genesis of thermal discomfort due to cold.

Effects of fasting on thermoregulatory processes and the daily oscillations in rats.

To investigate the mechanism involved in the reduction of body core temperature (T(core)) during fasting in rats, which is selective in the light phase, we measured T(core), surface temperature, and oxygen consumption rate in fed control animals and in fasted animals on day 3 of fasting and day 4 of recovery at an ambient temperature (T(a)) of 23 degrees C by biotelemetry, infrared thermography, and indirect calorimetry, respectively. On the fasting day, 1) T(core) in the light phase decreased (P < 0.05) from the control; however, T(core) in the dark phase was unchanged, 2) tail temperature fell from the control (P < 0.05, from 30.7 +/- 0.1 to 23.9 +/- 0.1 degrees C in the dark phase and from 29.4 +/- 0.1 to 25.2 +/- 0.2 degrees C in the light phase), 3) oxygen consumption rate decreased from the control (P < 0.05, from 24.37 +/- 1.06 to 16.24 +/- 0.69 ml. min(-1). kg body wt(-0.75) in the dark phase and from 18.91 +/- 0.64 to 14.00 +/- 0.41 ml. min(-1). kg body wt(-0.75) in the light phase). All these values returned to the control levels on the recovery day. The results suggest that, in the fasting condition, T(core) in the dark phase was maintained by suppression of the heat loss mechanism, despite the reduction of metabolic heat production. In contrast, the response was weakened in the light phase, decreasing T(core) greatly. Moreover, the change in the regulation of tail blood flow was a likely mechanism to suppress heat loss.