Behavioral reactions reflecting differential reward expectations in monkeys.

Learning theory emphasizes the importance of expectations in the control of instrumental action. This study investigated the variation of behavioral reactions toward different rewards as an expression of differential expectations of outcomes in primates. We employed several versions of two basic behavioral paradigms, the spatial delayed response task and the delayed reaction task. These tasks are commonly used in neurobiological studies of working memory, movement preparation, and event expectation involving the frontal cortex and basal ganglia. An initial visual instruction stimulus indicated to the animal which one of several food or liquid rewards would be delivered after each correct behavioral response, or whether or not a reward could be obtained. We measured the reaction times of the operantly conditioned arm movement necessary for obtaining the reward, and the durations of anticipatory licking prior to liquid reward delivery as a Pavlovian conditioned response. The results showed that both measures varied depending on the reward predicted by the initial instruction. Arm movements were performed with significantly shorter reaction times for foods or liquids that were more preferred by the animal than for less preferred ones. Still larger differences were observed between rewarded and unrewarded trials. An interesting effect was found in unrewarded trials, in which reaction times were significantly shorter when a highly preferred reward was delivered in the alternative rewarded trials of the same trial block as compared to a less preferred reward. Anticipatory licks preceding the reward were significantly longer when highly preferred rather than less preferred rewards, or no rewards, were predicted. These results demonstrate that behavioral reactions preceding rewards may vary depending on the predicted future reward and suggest that monkeys differentially expect particular outcomes in the presently investigated tasks.

Involvement of basal ganglia and orbitofrontal cortex in goal-directed behavior.

An impressive array of neural processing appears to be dedicated to the extraction of reward-related information from environmental stimuli and use of this information in the generation of goal-directed behaviors. While other structures are certainly involved in these processes, the characteristics of activations seen in mesencephalic dopamine neurons, striatal neurons and neurons of the orbitofrontal cortex provide distinct examples of the different ways in which reward-related information is processed. In addition, the differences in activations seen in these three regions demonstrate the different roles they may play in goal-directed behavior. A principal role played by dopamine neurons is that of a detector of an error in reward prediction. The homogeneity of responsiveness across the population of dopamine neurons indicates that this error signal is widely broadcast to dopamine terminal regions where it could provide a teaching signal for synaptic modifications underlying the learning of goal-directed appetitive behaviors. The responses of these same neurons to conditioned stimuli associated with reward could also serve as a signal of prediction error useful for the learning of sequences of environmental stimuli leading to reward. Dopamine neuron responses to both rewards and conditioned stimuli are not contingent on the behavior executed to obtain the reward and thus appear to reflect a relatively pure signal of a reward prediction error. It is not yet clear whether these activations, and responses to novel stimuli, have an additional function in engaging neural systems involved in the representation and execution of goal-directed behaviors. This representation of goal-directed behaviors may involve the striatal regions studied, where processing of reward-related information appears to be much more heterogeneous. Different subpopulations of striatal neurons are activated at different stages in the course of goal-directed behaviors, with largely separate populations activated following presentation of conditioned stimuli, preceding reinforcers, and following reinforcers. Neurons exhibiting each of these types of activation appear to differentiate between rewarding and non-rewarding outcomes of behavioral acts and, as a population, appear to be biased towards processing reward vs. non-reward. These activations observed in the striatum were often contingent on the behavioral act associated with obtaining reward, reflecting an integration of information not observed in dopamine neurons. Another difference between reward processing in striatal neurons and dopamine neurons is the influence of predictability on neuronal responsiveness. Unlike dopamine neurons, many striatal neurons respond to predicted rewards, although at least some may reflect the relative degree of predictability in the magnitude of the responses to reward. Thus, striatal processing of reward-related information is in some ways more complex than that observed in dopamine neurons, incorporating information on behavior and potentially providing more detailed information regarding predictability. These activations could serve as a component of the neural representation of the goal, and/or the behavioral aspects of goal-directed behaviors. As such they would be of use for the execution of appropriate goal-directed behaviors in response to known environmental stimuli, as well as for generating behaviors in response to novel stimuli that may be associated with desirable goals. Neuronal activations in the orbitofrontal cortex appear to involve less integration of behavioral and reward-related information, but rather incorporate another aspect of reward, the relative motivational significance of different rewards. These activations would serve a function similar to those striatal neurons that encode exclusively reward-related information in situations in which only a single outcome is obtainable. (ABSTRACT TRUNCATED)

Reward processing in primate orbitofrontal cortex and basal ganglia.

This article reviews and interprets neuronal activities related to the expectation and delivery of reward in the primate orbitofrontal cortex, in comparison with slowly discharging neurons in the striatum (caudate, putamen and ventral striatum, including nucleus accumbens) and midbrain dopamine neurons. Orbitofrontal neurons showed three principal forms of reward-related activity during the performance of delayed response tasks, namely responses to reward-predicting instructions, activations during the expectation period immediately preceding reward and responses following reward. These activations discriminated between different rewards, often on the basis of the animals' preferences. Neurons in the striatum were also activated in relation to the expectation and detection of reward but in addition showed activities related to the preparation, initiation and execution of movements which reflected the expected reward. Dopamine neurons responded to rewards and reward-predicting stimuli, and coded an error in the prediction of reward. Thus, the investigated cortical and basal ganglia structures showed multiple, heterogeneous, partly simultaneous activations which were related to specific aspects of rewards. These activations may represent the neuronal substrates of rewards during learning and established behavioral performance. The processing of reward expectations suggests an access to central representations of rewards which may be used for the neuronal control of goaldirected behavior.

Influence of reward expectation on behavior-related neuronal activity in primate striatum.

Rewards constitute important goals for voluntary behavior. This study aimed to investigate how expected rewards influence behavior-related neuronal activity in the anterior striatum. In a delayed go-nogo task, monkeys executed or withheld a reaching movement and obtained liquid or sound as reinforcement. An initial instruction picture indicated the behavioral reaction to be performed and the reinforcer to be obtained after a subsequent trigger stimulus. Movements varied according to the reinforcers predicted by the instructions, suggesting that animals differentially expected the two outcomes. About 250 of nearly 1,500 neurons in anterior parts of caudate nucleus, putamen, and ventral striatum showed typical task-related activations that reflected the expectation of instructions and trigger, and the preparation, initiation, and execution of behavioral reactions. Strikingly, most task-related activations occurred only when liquid reward was delivered at trial end, rather than the reinforcing sound. Activations close to the time of reward showed similar preferences for liquid reward over the reinforcing sound, suggesting a relationship to the expectation or detection of the motivational outcome of the trial rather than to a "correct" or "end-of-trial" signal. By contrast, relatively few activations in the present task occurred irrespective of the type of reinforcement. In conclusion, many of the behavior-related neurons investigated in the anterior striatum were influenced by an upcoming primary liquid reward and did not appear to code behavioral acts in a motivationally neutral manner. Rather, these neurons incorporated information about the expected outcome into their behavior-related activity. The activations influenced by reward several seconds before its occurrence may constitute a neuronal basis for the retrograde effects of rewards on behavioral reactions.

Modifications of reward expectation-related neuronal activity during learning in primate striatum.

This study investigated neuronal activity in the anterior striatum while monkeys repeatedly learned to associate new instruction stimuli with known behavioral reactions and reinforcers. In a delayed go-nogo task with several trial types, an initial picture instructed the animal to execute or withhold a reaching movement and to expect a liquid reward or not. During learning, new instruction pictures were presented, and animals guessed and performed one of the trial types according to a trial-and-error strategy. Learning of a large number of pictures resulted in a learning set in which learning took place in a few trials and correct performance exceeded 80% in the first 60-90 trials. About 200 task-related striatal neurons studied in both familiar and learning conditions showed three forms of changes during learning. Activations related to the preparation and execution of behavioral reactions and the expectation of reward were maintained in many neurons but occurred in inappropriate trial types when behavioral errors were made. The activations became appropriate for individual trial types when the animals' behavior adapted to the new task contingencies. In particular, reward expectation-related activations occurred initially in both rewarded and unrewarded movement trials and became subsequently restricted to rewarded trials. These changes occurred in parallel with the visible adaptation of reward expectations by the animals. The second learning change consisted in decreases of task-related activations that were either restricted to the initial trials of new learning problems or persisted during the subsequent consolidation phase. They probably reflected reductions in the expectation and preparation of upcoming task events, including reward. The third learning change consisted in transient or sustained increases of activations. These might reflect the increased attention accompanying learning and serve to induce synaptic changes underlying the behavioral adaptations. Both decreases and increases often induced changes in the trial selective occurrence of activations. In conclusion, neurons in anterior striatum showed changes related to adaptations or reductions of expectations in new task situations and displayed activations that might serve to induce structural changes during learning.

Reward prediction in primate basal ganglia and frontal cortex.

Reward information is processed in a limited number of brain structures, including fronto-basal ganglia systems. Dopamine neurons respond phasically to primary rewards and reward-predicting stimuli depending on reward unpredictability but without discriminating between rewards. These responses reflect 'errors' in the prediction of rewards in correspondence to learning theories and thus may constitute teaching signals for appetitive learning. Neurons in the striatum (caudate, putamen, ventral striatum) code reward predictions in a different manner. They are activated during several seconds when animals expect predicted rewards. During learning, these activations occur initially in rewarded and unrewarded trials and become subsequently restricted to rewarded trials. This occurs in parallel with the adaptation of reward expectations by the animals, as inferred from their behavioral reactions. Neurons in orbitofrontal cortex respond differentially to stimuli predicting different liquid rewards, without coding spatial or visual features. Thus, different structures process reward information processed in different ways. Whereas dopamine neurons emit a reward teaching signal without indicating the specific reward, striatal neurons adapt expectation activity to new reward situations, and orbitofrontal neurons process the specific nature of rewards. These reward signals need to cooperate in order for reward information to be used for learning and maintaining approach behavior.

Dopamine neurons report an error in the temporal prediction of reward during learning.

Many behaviors are affected by rewards, undergoing long-term changes when rewards are different than predicted but remaining unchanged when rewards occur exactly as predicted. The discrepancy between reward occurrence and reward prediction is termed an 'error in reward prediction'. Dopamine neurons in the substantia nigra and the ventral tegmental area are believed to be involved in reward-dependent behaviors. Consistent with this role, they are activated by rewards, and because they are activated more strongly by unpredicted than by predicted rewards they may play a role in learning. The present study investigated whether monkey dopamine neurons code an error in reward prediction during the course of learning. Dopamine neuron responses reflected the changes in reward prediction during individual learning episodes; dopamine neurons were activated by rewards during early trials, when errors were frequent and rewards unpredictable, but activation was progressively reduced as performance was consolidated and rewards became more predictable. These neurons were also activated when rewards occurred at unpredicted times and were depressed when rewards were omitted at the predicted times. Thus, dopamine neurons code errors in the prediction of both the occurrence and the time of rewards. In this respect, their responses resemble the teaching signals that have been employed in particularly efficient computational learning models.

Subthalamic nucleus cell firing in the 6-OHDA-treated rat: basal activity and response to haloperidol.

Single unit recordings from neurons of the subthalamic nucleus were made in control and 6-hydroxydopamine (6-OHDA)-treated rats anesthetized with chloral hydrate. Subthalamic nucleus cells in this preparation exhibited a wide range of firing rates and three different firing patterns. These patterns were defined as 'burst', 'normal', and 'mixed' based on comparisons of their interspike interval histograms. Four to 6 weeks after 6-OHDA treatment there was no change in the basal firing rates of subthalamic nucleus cells, but there was a significant shift in firing pattern, with a smaller proportion of cells exhibiting the 'normal' firing pattern. The response of subthalamic nucleus neurons to acute administration of haloperidol was also altered in 6-OHDA-treated rats tested 4-6 weeks post-lesion, with a significantly greater proportion of cells responding to doses of haloperidol as low as 0.2 mg/kg (i.v.) with increases in firing rate of 20% or more. These results suggest that the subthalamic nucleus is probably not involved in the increases in basal levels of dopamine cell activity observed previously in the 6-OHDA-treated rat, but may play a role in the acute induction of depolarization block of dopamine cell firing in response to haloperidol administration in this model.

Electrophysiological, biochemical, and behavioral studies of acute haloperidol-induced depolarization block of nigral dopamine neurons.

The electrophysiological, biochemical and behavioral responses produced by administration of haloperidol were studied in intact rats and in rats with 6-hydroxydopamine-induced partial lesions of the nigrostriatal dopamine pathway. In both control rats and rates tested four to 10 days postlesion, the electrophysiological response of nigral dopamine neurons to increasing doses of haloperidol consisted of either: (1) an increase in firing rate which reached a plateau at six to 10 spikes per second, or (2) no response (i.e., less than 20% change in firing rate). Administration of additional doses of haloperidol up to lethal levels did not elicit further changes in dopamine cell firing in these rats. In contrast, in 6-hydroxydopamine-treated rats tested four to six weeks postlesion, acute administration of haloperidol was not only more consistent in producing increases in dopamine cell firing rate, but also caused six out of seven dopamine neurons tested to cease firing upon entering a state of depolarization block. In all cases in which depolarization block was observed, dopamine cell firing was reinstated by either iontophoretic application of gamma-aminobutyric acid or intravenous administration of apomorphine. In parallel studies, haloperidol caused an increase in the extracellular dopamine levels measured by microdialysis in the striatum of control rats, whereas administration of the same dose of haloperidol to 6-hydroxydopamine-treated rats four to six weeks postlesion did not elicit any change in extracellular dopamine levels. In addition, administration of haloperidol at a dose which was ineffective in control rats produced gross motor deficits in the 6-hydroxydopamine-treated rats when tested four to six weeks postlesion. These results show that 6-hydroxydopamine-induced dopamine depletions produce a time-dependent change in the responsivity of the nigrostriatal dopamine system to acute haloperidol administration. In this altered system, the induction of depolarization block of spike activity in nigral dopamine neurons by haloperidol was not associated with a corresponding decrease in extracellular dopamine levels measured in the striatum. However, it appeared that depolarization block did prevent haloperidol-induced increases in extracellular dopamine levels. The occurrence of depolarization block in the dopamine-depleted animal may limit the capacity of this system to respond to additional compromise, in spite of the compensatory processes that contribute to maintaining motor function.

The effects of dopamine-depleting brain lesions on the electrophysiological activity of rat substantia nigra dopamine neurons.

Extensive damage to the nigrostriatal dopamine (DA) system initiates a number of compensatory changes that may be involved in counterbalancing the effects of the lesion. In this study, we examined whether changes in the electrophysiological activity of the remaining DA cells play a role in compensating for 6-hydroxydopamine (6-OHDA)-induced depletions of striatal DA. DA cell activity in lesioned rats was assessed along three dimensions: (1) the relative proportion of the remaining DA neurons firing spontaneously, (2) their firing rate, and (3) their firing pattern. Histofluorescence studies revealed a sparing of DA neurons in the midbrain of 6-OHDA-lesioned rats relative to the levels of DA remaining in the striatum. With respect to DA cell activity, depletions of up to 96% of striatal DA did not result in substantial alterations in the proportion of DA neurons active, their mean firing rate, or their firing pattern. Increases in these parameters only occurred when striatal DA depletions exceeded 96%. These results suggest that the biochemical and receptor compensations produced in the DA system in response to injury are of sufficient magnitude to allow the DA cells to maintain baseline levels of activity. In this way, the remaining DA neurons would maintain the wide dynamic range of electrophysiological responsivity that may be necessary for the normal function of the extrapyramidal motor system.

Acute haloperidol administration induces depolarization block of nigral dopamine neurons in rats after partial dopamine lesions.

The effects of the antipsychotic drug haloperidol (HAL) on the electrophysiological activity of dopamine (DA)-containing cells in the substantia nigra was assessed in rats 6 weeks after partial 6-hydroxydopamine (6-OHDA)-induced lesions of the nigrostriatal DA pathway. Depleting 75% or more of striatal DA altered the response of DA neurons to acute HAL administration. Whereas acute HAL administration generally accelerates DA neuron firing in control rats, similar HAL doses given to lesioned rats not only increased firing rate but induced depolarization block of DA neuron spike generation similar to that resulting from chronic neuroleptic administration. In contrast, acute administration of doses of HAL up to lethal levels typically could not induce depolarization block of DA neurons in non-lesioned rats. This preparation thus could be an effective model for investigating the exacerbation of behavioral deficits produced by an increased demand placed upon a compromised DA system, as may occur in Parkinson's disease or with antipsychotic drug treatment.

Yaw direction neurons in the cat inferior olive.

1. Single units that responded to yaw rotation were recorded extracellularly in the caudal inferior olive (IO) of barbiturate-anesthetized cats. Of 276 neurons, 55 responded reliably to yaw, and extensive quantitative data were recorded from 25. 2. No yaw-sensitive IO neuron responded to somatosensory or auditory stimuli but two responded, though unreliably, to flash. 3. Yaw-sensitive IO cells fired at low (1-4 spikes/s), irregular rates during one direction of rotation. Though cells responded reliably during yaw, firing rates varied considerably from cycle to cycle. Rotation speed and acceleration were not represented in any cell's firing rate. 4. Eighty five percent (47/55) of yaw-sensitive cells fired during contralateral rotation, 9% (5/55) during ipsilateral rotation, and 6% (3/55) fired from late in the ipsilateral phase of a sinusoidal oscillation to the middle of the contralateral phase. 5. Responses were tested to 0.1-Hz sinusoidal yaw oscillations with a range of peak angular velocities (1-200 degrees/s). Thresholds were not sharp because of the cycle to cycle variability in response rates but were estimated using averaged responses. The peak rate of the most sensitive cell was driven to criterion (2 SD above spontaneous rate) by an oscillation with a peak velocity of 1 degrees/s. Other cells reached criterion between 5 and 50 degrees/s. 6. Sinusoidal oscillation at all frequencies tested (0.01-0.5 Hz) elicited approximately the same firing rates. Even at 0.01 Hz cells responded well. Responses lagged acceleration by approximately 25 degrees at 0.01 Hz and shifted to later parts of the cycle as frequency increased so that firing lagged acceleration by approximately 200 degrees at 0.5 Hz. 7. Histological reconstruction showed that yaw-sensitive neurons were recorded in olivary subnucleus beta (N beta), the dorsal cap of Kooy (DC), the posterior medial region of the medial accessory division of the inferior olive (MAO), and in the medial-lateral center of the caudal MAO. 8. Yaw-sensitive neurons in the inferior olive provide a signal to the cerebellum that indicates the direction of passive rotation over a wide range of velocity and acceleration. The signal from individual neurons does not reliably encode either rotation velocity or acceleration. Yaw-sensitive IO neurons are therefore unlike other central vestibular neurons but are similar to somatosensory IO cells which signal the presence, but not the intensity of a stimulus.