Midbrain dopamine neurons encode a quantitative reward prediction error signal.

The midbrain dopamine neurons are hypothesized to provide a physiological correlate of the reward prediction error signal required by current models of reinforcement learning. We examined the activity of single dopamine neurons during a task in which subjects learned by trial and error when to make an eye movement for a juice reward. We found that these neurons encoded the difference between the current reward and a weighted average of previous rewards, a reward prediction error, but only for outcomes that were better than expected. Thus, the firing rate of midbrain dopamine neurons is quantitatively predicted by theoretical descriptions of the reward prediction error signal used in reinforcement learning models for circumstances in which this signal has a positive value. We also found that the dopamine system continued to compute the reward prediction error even when the behavioral policy of the animal was only weakly influenced by this computation.

Statistics of midbrain dopamine neuron spike trains in the awake primate.

Work in behaving primates indicates that midbrain dopamine neurons encode a prediction error, the difference between an obtained reward and the reward expected. Studies of dopamine action potential timing in the alert and anesthetized rat indicate that dopamine neurons respond in tonic and phasic modes, a distinction that has been less well characterized in the primates. We used spike train models to examine the relationship between the tonic and burst modes of activity in dopamine neurons while monkeys were performing a reinforced visuo-saccadic movement task. We studied spiking activity during four task-related intervals; two of these were intervals during which no task-related events occurred, whereas two were periods marked by task-related phasic activity. We found that dopamine neuron spike trains during the intervals when no events occurred were well described as tonic. Action potentials appeared to be independent, to occur at low frequency, and to be almost equally well described by Gaussian and Poisson-like (gamma) processes. Unlike in the rat, interspike intervals as low as 20 ms were often observed during these presumptively tonic epochs. Having identified these periods of presumptively tonic activity, we were able to quantitatively define phasic modulations (both increases and decreases in activity) during the intervals in which task-related events occurred. This analysis revealed that the phasic modulations of these neurons include both bursting, as has been described previously, and pausing. Together bursts and pauses seemed to provide a continuous, although nonlinear, representation of the theoretically defined reward prediction error of reinforcement learning.

Physiological utility theory and the neuroeconomics of choice.

Over the past half century economists have responded to the challenges of Allais [Econometrica (1953) 53], Ellsberg [Quart. J. Econ. (1961) 643] and others raised to neoclassicism either by bounding the reach of economic theory or by turning to descriptive approaches. While both of these strategies have been enormously fruitful, neither has provided a clear programmatic approach that aspires to a complete understanding of human decision making as did neoclassicism. There is, however, growing evidence that economists and neurobiologists are now beginning to reveal the physical mechanisms by which the human neuroarchitecture accomplishes decision making. Although in their infancy, these studies suggest both a single unified framework for understanding human decision making and a methodology for constraining the scope and structure of economic theory. Indeed, there is already evidence that these studies place mathematical constraints on existing economic models. This article reviews some of those constraints and suggests the outline of a neuroeconomic theory of decision.

Eye position and memory saccade related responses in substantia nigra pars reticulata.

The substantia nigra pars reticulata (SNr), a major output nucleus of the basal ganglia, has been implicated anatomically, pharmacologically and physiologically in the generation of saccadic eye movements. However, the unique contribution of the SNr to saccade generation remains elusive. We studied the activity of SNr neurons while rhesus monkeys made saccades from different initial orbital positions, to determine what effects, if any, eye position had on SNr neuronal activity. We found that there was no effect of eye position on SNr neuronal responses. We also examined the responses of SNr neurons during memory-guided saccades to determine whether SNr discharges were affected by whether the target of the upcoming saccade was visible. We found that there was no change in response properties during memory saccade trials as compared to otherwise identical visually guided trials. SNr neurons appear to carry no information about either eye position or whether a movement is guided by a visible or remembered target. These results suggest that nigral signals are encoded in the same coordinate frame as those in the SC and FEF, but that unlike neuronal responses in these areas, SNr activity is not influenced by whether the saccade target remains visible until the movement is executed.