Does phasic dopamine release cause policy updates?

Phasic dopamine activity is believed to both encode reward-prediction errors (RPEs) and to cause the adaptations that these errors engender. If so, a rat working for optogenetic stimulation of dopamine neurons will repeatedly update its policy and/or action values, thus iteratively increasing its work rate. Here, we challenge this view by demonstrating stable, non-maximal work rates in the face of repeated optogenetic stimulation of midbrain dopamine neurons. Furthermore, we show that rats learn to discriminate between world states distinguished only by their history of dopamine activation. Comparison of these results to reinforcement learning simulations suggests that the induced dopamine transients acted more as rewards than RPEs. However, pursuit of dopaminergic stimulation drifted upwards over a time scale of days and weeks, despite its stability within trials. To reconcile the results with prior findings, we consider multiple roles for dopamine signalling.

The priming effect of rewarding brain stimulation in rats depends on both the cost and strength of reward but survives blockade of D2-like dopamine receptors.

Receipt of an intense reward boosts motivation to work for more of that reward. This phenomenon is called the priming effect of rewards. Using a novel measurement method, we show that the priming effect of rewarding electrical brain stimulation depends on the cost, as well as on the strength, of the anticipated reward. Previous research on the priming effect of electrical brain stimulation utilized a runway paradigm in which running speed serves as the measure of motivation. In the present study, the measure of motivation was the vigour with which rats executed a two-lever response chain, in a standard operant-conditioning chamber, to earn rewarding electrical stimulation of the lateral hypothalamus. In a second experiment, we introduced a modification that entails self-administered priming stimulation and alternating blocks of primed and unprimed trials. Reliable, consistent priming effects of substantial magnitude were obtained in the modified paradigm, which is closely analogous to the runway paradigm. In a third experiment, the modified paradigm served to assess the dependence of the priming effect on dopamine D2-like receptors. The priming effect proved resilient to the effect of eticlopride, a selective D2-like receptor antagonist. These results are discussed within the framework of a new model of brain reward circuitry in which non-dopaminergic medial forebrain bundle fibers and dopamine axons provide parallel inputs to the final common paths for reward and incentive motivation.

Learning to use past evidence in a sophisticated world model.

Humans and other animals are able to discover underlying statistical structure in their environments and exploit it to achieve efficient and effective performance. However, such structure is often difficult to learn and use because it is obscure, involving long-range temporal dependencies. Here, we analysed behavioural data from an extended experiment with rats, showing that the subjects learned the underlying statistical structure, albeit suffering at times from immediate inferential imperfections as to their current state within it. We accounted for their behaviour using a Hidden Markov Model, in which recent observations are integrated with evidence from the past. We found that over the course of training, subjects came to track their progress through the task more accurately, a change that our model largely attributed to improved integration of past evidence. This learning reflected the structure of the task, decreasing reliance on recent observations, which were potentially misleading.

The priming effect of food persists following blockade of dopamine receptors.

The priming effect of rewards is a boost in the vigor of reward seeking resulting from the previous receipt of a reward. Extensive work has been carried out on the priming effect of electrical brain stimulation, but much less research exists on the priming effect of natural rewards, such as food. While both reinforcement and motivation are linked with dopamine transmission in the brain, the priming effect of rewards does not appear to be dopamine-dependent. In the present study, an operant method was developed to measure the priming effect of food and then applied to investigate whether it is affected by dopamine receptor antagonism. Long-Evans rats were administered saline or one of the three doses (0.01, 0.05, 0.075 mg/kg) of the dopamine D1 receptor family antagonist, SCH23390, or the dopamine D2 receptor family antagonist, eticlopride. Although dopamine receptor antagonism affected pursuit of food, it did not eliminate the priming effect. These data suggest that despite the involvement of dopamine transmission in reinforcement and motivation, the priming effect of food does not depend on dopamine transmission.

Some work and some play: microscopic and macroscopic approaches to labor and leisure.

Given the option, humans and other animals elect to distribute their time between work and leisure, rather than choosing all of one and none of the other. Traditional accounts of partial allocation have characterised behavior on a macroscopic timescale, reporting and studying the mean times spent in work or leisure. However, averaging over the more microscopic processes that govern choices is known to pose tricky theoretical problems, and also eschews any possibility of direct contact with the neural computations involved. We develop a microscopic framework, formalized as a semi-Markov decision process with possibly stochastic choices, in which subjects approximately maximise their expected returns by making momentary commitments to one or other activity. We show macroscopic utilities that arise from microscopic ones, and demonstrate how facets such as imperfect substitutability can arise in a more straightforward microscopic manner.

Role of dopamine tone in the pursuit of brain stimulation reward.

Dopaminergic neurons contribute to intracranial self-stimulation (ICSS) and other reward-seeking behaviors, but it is not yet known where dopaminergic neurons intervene in the neural circuitry underlying reward pursuit or which psychological processes are involved. In rats working for electrical stimulation of the medial forebrain bundle, we assessed the effect of GBR-12909 (1-[2-[bis(4-fluorophenyl)-methoxy]ethyl]-4-[3- phenylpropyl]piperazine), a specific blocker of the dopamine transporter. Operant performance was measured as a function of the strength and cost of electrical stimulation. GBR-12909 increased the opportunity cost most subjects were willing to pay for a reward of a given intensity. However, this effect was smaller than that produced by a regimen of cocaine administration that drove similar increases in nucleus accumbens (NAc) dopamine levels in unstimulated rats. Delivery of rewarding stimulation to drug-treated rats caused an additional increase in dopamine concentration in the NAc shell in cocaine-treated, but not GBR-12909-treated, rats. These behavioral and neurochemical differences may reflect blockade of the norepinephrine transporter by cocaine but not by GBR-12909. Whereas the effect of psychomotor stimulants on ICSS has long been attributed to dopaminergic action at early stages of the reward pathway, the results reported here imply that increased dopamine tone boosts reward pursuit by acting at or beyond the output of the circuitry that temporally and spatially summates the output of the directly stimulated neurons underlying ICSS. The observed enhancement of reward seeking could be attributable to a decrease in the value of competing behaviors, a decrease in subjective effort costs, or an increase in reward-system gain.

Cannabinoid receptor blockade reduces the opportunity cost at which rats maintain operant performance for rewarding brain stimulation.

There is ample evidence that blockade of CB(1) receptors reduces reward seeking. However, the reported effects of CB(1) blockade on performance for rewarding electrical brain stimulation stand out as an exception. By applying a novel method for conceptualizing and measuring reward seeking, we show that AM-251, a CB(1) receptor antagonist, does indeed decrease performance for rewarding electrical stimulation of the medial forebrain bundle in rats. Reward seeking depends on multiple sets of variables, including the intensity of the reward, its cost, and the value of competing rewards. In turn, reward intensity depends both on the sensitivity and gain of brain reward circuitry. We show that drug-induced changes in sensitivity cannot account for the suppressive effect of AM-251 on reward seeking. Therefore, the role of CB(1) receptors must be sought among the remaining determinants of performance. Our analysis provides an explanation of the inconsistencies between prior reports, which likely arose from the following: (1) the averaging of data across subjects showing heterogeneous effects and (2) the use of methods that cannot distinguish between the different determinants of reward pursuit. By means of microdialysis, we demonstrate that blockade of CB(1) receptors attenuates nucleus accumbens dopamine release in response to rewarding medial forebrain bundle stimulation, and we propose that this action is responsible for the ability of the drug to decrease performance for the electrical reward.

The Convergence Model of Brain Reward Circuitry: Implications for Relief of Treatment-Resistant Depression by Deep-Brain Stimulation of the Medial Forebrain Bundle.

Deep-brain stimulation of the medial forebrain bundle (MFB) can provide effective, enduring relief of treatment-resistant depression. Panksepp provided an explanatory framework: the MFB constitutes the core of the neural circuitry subserving the anticipation and pursuit of rewards: the "SEEKING" system. On that view, the SEEKING system is hypoactive in depressed individuals; background electrical stimulation of the MFB alleviates symptoms by normalizing activity. Panksepp attributed intracranial self-stimulation to excitation of the SEEKING system in which the ascending projections of midbrain dopamine neurons are an essential component. In parallel with Panksepp's qualitative work, intracranial self-stimulation has long been studied quantitatively by psychophysical means. That work argues that the predominant directly stimulated substrate for MFB self-stimulation are myelinated, non-dopaminergic fibers, more readily excited by brief electrical current pulses than the thin, unmyelinated axons of the midbrain dopamine neurons. The series-circuit hypothesis reconciles this view with the evidence implicating dopamine in MFB self-stimulation as follows: direct activation of myelinated MFB fibers is rewarding due to their trans-synaptic activation of midbrain dopamine neurons. A recent study in which rats worked for optogenetic stimulation of midbrain dopamine neurons challenges the series-circuit hypothesis and provides a new model of intracranial self-stimulation in which the myelinated non-dopaminergic neurons and the midbrain dopamine projections access the behavioral final common path for reward seeking via separate, converging routes. We explore the potential implications of this convergence model for the interpretation of the antidepressant effect of MFB stimulation. We also discuss the consistent finding that psychomotor stimulants, which boost dopaminergic neurotransmission, fail to provide a monotherapy for depression. We propose that non-dopaminergic MFB components may contribute to the therapeutic effect in parallel to, in synergy with, or even instead of, a dopaminergic component.

Dopamine and Beyond: Implications of Psychophysical Studies of Intracranial Self-Stimulation for the Treatment of Depression.

Major depressive disorder is a leading cause of disability and suicide worldwide. Consecutive rounds of conventional interventions are ineffective in a significant sub-group of patients whose disorder is classified as treatment-resistant depression. Significant progress in managing this severe form of depression has been achieved through the use of deep brain stimulation of the medial forebrain bundle (MFB). The beneficial effect of such stimulation appears strong, safe, and enduring. The proposed neural substrate for this promising clinical finding includes midbrain dopamine neurons and a subset of their cortical afferents. Here, we aim to broaden the discussion of the candidate circuitry by exploring potential implications of a new "convergence" model of brain reward circuitry in rodents. We chart the evolution of the new model from its predecessors, which held that midbrain dopamine neurons constituted an obligatory stage of the final common path for reward seeking. In contrast, the new model includes a directly activated, non-dopaminergic pathway whose output ultimately converges with that of the dopaminergic neurons. On the basis of the new model and the relative ineffectiveness of dopamine agonists in the treatment of depression, we ask whether non-dopaminergic circuitry may contribute to the clinical efficacy of deep brain stimulation of the MFB.

The trade-off between pulse duration and power in optical excitation of midbrain dopamine neurons approximates Bloch's law.

Optogenetic experiments reveal functional roles of specific neurons. However, functional inferences have been limited by widespread adoption of a restricted set of stimulation parameters. Broader exploration of the parameter space can deepen insight into the mapping between selective neural activity and behavior. In this way, characteristics of the activated neural circuit, such as temporal integration, can be inferred. Our objective was to determine whether an equal-energy principle accounts for the interaction of pulse duration and optical power in optogenetic excitation. Six male TH::Cre rats worked for optogenetic (ChannelRhodopsin-2) stimulation of VTA dopamine neurons. We used a within-subject design to describe the trade-off between pulse duration and optical power in determining reward seeking. Parameters were customized for each subject based on behavioral effectiveness. Within a useful range of powers (~12.6-31.6 mW) the product of optical power and pulse duration required to produce a given level of reward seeking was roughly constant. Such reciprocity is consistent with Bloch's law, which posits an equal-energy principle of temporal summation over short durations in human vision. The trade-off between pulse duration and power broke down at higher powers. Thus, optical power and duration can be adjusted reciprocally for brief durations and lower powers, and power can be substituted for pulse duration to scale the region of excitation in behavioral optogenetic experiments. The findings demonstrate the utility of within-subject and trade-off designs in optogenetics and of parameter adjustment based on functional endpoints instead of physical properties of the stimulation.

Dopamine neurons do not constitute an obligatory stage in the final common path for the evaluation and pursuit of brain stimulation reward.

The neurobiological study of reward was launched by the discovery of intracranial self-stimulation (ICSS). Subsequent investigation of this phenomenon provided the initial link between reward-seeking behavior and dopaminergic neurotransmission. We re-evaluated this relationship by psychophysical, pharmacological, optogenetic, and computational means. In rats working for direct, optical activation of midbrain dopamine neurons, we varied the strength and opportunity cost of the stimulation and measured time allocation, the proportion of trial time devoted to reward pursuit. We found that the dependence of time allocation on the strength and cost of stimulation was similar formally to that observed when electrical stimulation of the medial forebrain bundle served as the reward. When the stimulation is strong and cheap, the rats devote almost all their time to reward pursuit; time allocation falls off as stimulation strength is decreased and/or its opportunity cost is increased. A 3D plot of time allocation versus stimulation strength and cost produces a surface resembling the corner of a plateau (the "reward mountain"). We show that dopamine-transporter blockade shifts the mountain along both the strength and cost axes in rats working for optical activation of midbrain dopamine neurons. In contrast, the same drug shifted the mountain uniquely along the opportunity-cost axis when rats worked for electrical MFB stimulation in a prior study. Dopamine neurons are an obligatory stage in the dominant model of ICSS, which positions them at a key nexus in the final common path for reward seeking. This model fails to provide a cogent account for the differential effect of dopamine transporter blockade on the reward mountain. Instead, we propose that midbrain dopamine neurons and neurons with non-dopaminergic, MFB axons constitute parallel limbs of brain-reward circuitry that ultimately converge on the final-common path for the evaluation and pursuit of rewards.

The reinforcement mountain: allocation of behavior as a function of the rate and intensity of rewarding brain stimulation.

The single-operant matching law has been used to describe the relationship between time allocated to pursuit of brain stimulation reward (BSR) and the obtained rate of reinforcement. We generalize this relationship to a third dimension by including the strength of the stimulation (the number of pulses per train) as an independent dimension, and we dub the resulting 3-dimensional structure "the reinforcement mountain." The validity of generalizing the single-operant matching law in this way was assessed by determining the changes in the position of the mountain produced by increasing the stimulation current or the train duration. Most of the predictions were supported, and the mountain model fitted the data closely. It is argued that application of this model can remove ambiguity inherent in 2-dimensional descriptions of operant performance and can reveal whether lesions, drugs, or physiological manipulations that alter performance for BSR act before or after the output of the ("reward-growth") function that translates the electrically induced impulse flow into the intensity of the BSR.

Potentiation of intracranial self-stimulation during prolonged subcutaneous infusion of cocaine.

Subcutaneous administration of cocaine yields a longer duration of action than administration via the intraperitoneal or intravenous routes. However, cocaine is a powerful vasoconstrictor, and thus injection of this drug at a single subcutaneous locus entails significant risk of necrotic skin lesions. This paper introduces a new method for subcutaneous administration of cocaine that reduces the probability of dermonecrosis by dispersing the drug under a large area of skin. Two experiments were conducted to evaluate the new method. In the first, changes in dopamine tone in the nucleus accumbens were measured by means of microdialysis during prolonged subcutaneous infusions of cocaine. The dopamine concentration attained a fairly stable, elevated level, suggesting that absorption, distribution, and excretion of the drug approached steady state. In a second experiment, performance for rewarding electrical stimulation was measured during repeated prolonged infusions of cocaine. The pulse frequency required to sustain responding was decreased by the drug, in a manner that was stable both within and across test sessions. Thus, the new method is appropriate for studies requiring stable neurochemical and behavioral conditions during repeated long test sessions, high rates of drug delivery and alternation between administration of the drug and the vehicle.

Dopamine tone increases similarly during predictable and unpredictable administration of rewarding brain stimulation at short inter-train intervals.

Unpredicted rewards, but not predicted ones, trigger strong phasic changes in the firing rates of midbrain dopamine (DA). In contrast, neurochemical measurements of DA tone have failed to reveal an influence of reward predictability. However, the subjects of the neurochemical experiments were asked to predict reward onset over longer intervals (12s, on average) than the subjects of the electrophysiological studies (typically, 2s). Thus, the contrasting effects of reward predictability could reflect the difference in the duration of the interval separating the predictor from the reward rather than a difference in the influence of reward predictability on phasic and tonic DA signaling. This hypothesis was tested in rats receiving trains of rewarding electrical brain stimulation with either a predictable or unpredictable onset. The mean inter-train interval was 1.5s, a value close to the 2-s CS-US interval that has been used in electrophysiological studies demonstrating the dependence of phasic DA responses on reward predictability. Despite the shortened inter-train interval, the time courses of the observed stimulation-induced elevations in DA levels were very similar, regardless of whether train onset was predictable. This finding is consistent with the idea that tonic DA signaling is insensitive to the predictability of rewards.

17ß-estradiol locally increases phasic dopamine release in the dorsal striatum.

Studies using in vivo microdialysis have shown that 17ß-estradiol (E2) increases dopamine (DA) transmission in the dorsal striatum. Both systemic administration of E2 and local infusion into the dorsal striatum rapidly enhance amphetamine-induced DA release. However, it is not known to what degree these effects reflect tonic and/or phasic DA release. It was hypothesized that E2 acts directly within the DS to rapidly increase phasic DA transmission. In urethane-anesthetized (1.5mL/kg) female rats, we used fast-scan cyclic voltammetry to study the effects of E2 on phasic, electrically-evoked release of DA in the dorsal striatum. Rats were ovariectomized and implanted with a silastic tube containing 5% E2 in cholesterol, previously shown to mimic low physiological serum concentrations of∼20-25pg/mL. DA release was evoked every 1min by delivering biphasic electrical stimulation in the substantia nigra. Local infusions of E2 (244.8pg/µl) into the dorsal striatum increased the amplitude of the electrically evoked DA transients. Behaviorally significant stimuli and events trigger phasic release of DA. The present findings predict that E2 would boost such signaling in behaving subjects.

Predictable and unpredictable rewards produce similar changes in dopamine tone.

Unpredicted rewards trigger more vigorous phasic responses in midbrain dopamine (DA) neurons than predicted rewards. However, recent evidence suggests that reward predictability may fail to influence DA signaling over longer scales: In rats passively receiving rewarding electrical brain stimulation, the concentration of DA in dialysate obtained from nucleus accumbens probes was similar regardless of whether reward onset was predictable (G. Hernandez et al., 2006). The present experiment followed up on these findings by requiring the rats to work for the rewarding stimulation, thus confirming whether they indeed learned the timing and predictability of reward delivery. Performance under fixed-interval and variable-interval schedules was compared, and DA levels in the nucleus accumbens were measured by means of in vivo microdialysis. The observed patterns of operant responding indicate that the rats working under the fixed-interval schedule learned to predict the time of reward availability, whereas the rats working under the variable-interval schedule did not. Nonetheless, indistinguishable changes in DA concentration were observed in the 2 groups. Thus, reward predictability had no discernable effect on a measure believed to track the slower components of DA signaling.

The effects of reinforcer magnitude on timing in rats.

The relation between reinforcer magnitude and timing behavior was studied using a peak procedure. Four rats received multiple consecutive sessions with both low and high levels of brain stimulation reward (BSR). Rats paused longer and had later start times during sessions when their responses were reinforced with low-magnitude BSR. When estimated by a symmetric Gaussian function, peak times also were earlier; when estimated by a better-fitting asymmetric Gaussian function or by analyzing individual trials, however, these peak-time changes were determined to reflect a mixture of large effects of BSR on start times and no effect on stop times. These results pose a significant dilemma for three major theories of timing (SET, MTS, and BeT), which all predict no effects for chronic manipulations of reinforcer magnitude. We conclude that increased reinforcer magnitude influences timing in two ways: through larger immediate after-effects that delay responding and through anticipatory effects that elicit earlier responding.

Valuation of opportunity costs by rats working for rewarding electrical brain stimulation.

Pursuit of one goal typically precludes simultaneous pursuit of another. Thus, each exclusive activity entails an "opportunity cost:" the forgone benefits from the next-best activity eschewed. The present experiment estimates, in laboratory rats, the function that maps objective opportunity costs into subjective ones. In an operant chamber, rewarding electrical brain stimulation was delivered when the cumulative time a lever had been depressed reached a criterion duration. The value of the activities forgone during this duration is the opportunity cost of the electrical reward. We determined which of four functions best describes how objective opportunity costs, expressed as the required duration of lever depression, are translated into their subjective equivalents. The simplest account is the identity function, which equates subjective and objective opportunity costs. A variant of this function called the "sigmoidal-slope function," converges on the identity function at longer durations but deviates from it at shorter durations. The sigmoidal-slope function has the form of a hockey stick. The flat "blade" denotes a range over which opportunity costs are subjectively equivalent; these durations are too short to allow substitution of more beneficial activities. The blade extends into an upward-curving portion over which costs become discriminable and finally into the straight "handle," over which objective and subjective costs match. The two remaining functions are based on hyperbolic and exponential temporal discounting, respectively. The results are best described by the sigmoidal-slope function. That this is so suggests that different principles of intertemporal choice are involved in the evaluation of time spent working for a reward or waiting for its delivery. The subjective opportunity-cost function plays a key role in the evaluation and selection of goals. An accurate description of its form and parameters is essential to successful modeling and prediction of instrumental performance and reward-related decision making.

Prolonged rewarding stimulation of the rat medial forebrain bundle: neurochemical and behavioral consequences.

Extracellular dopamine levels were measured in the rat nucleus accumbens by means of in vivo microdialysis. Delivery of rewarding medial forebrain bundle stimulation at a low rate (5 trains/min) produced a sustained elevation of dopamine levels, regardless of whether train onset was predictable. When the rate of train delivery was increased to 40 trains/min, dopamine levels rose rapidly during the first 40 min but then declined toward the baseline range. The rewarding impact of the stimulation was reduced following prior delivery of stimulation at the high, but not the low, rate. These results support the idea that dopamine tone plays an enabling role in brain stimulation reward and is elevated similarly by predictable and unpredictable stimulation.

Potentiation of brain stimulation reward by weight loss: evidence for functional heterogeneity in brain reward circuitry.

Physiological need states can influence goal-directed behavior by modulating the neural circuitry underlying the rewarding effects of stimuli and behaviors. Direct electrical stimulation of this circuitry produces a powerful rewarding effect, which is called brain stimulation reward. Chronic food restriction resulting in body weight loss potentiates brain stimulation reward, but in only in a subset of cases. This could reflect individual differences between rats or subtle difference in electrode location that lead to differential excitation of functionally different components of the neural circuitry underlying brain stimulation reward. To distinguish between these views, the influence of chronic food restriction on brain stimulation reward was assessed in rats bearing multiple stimulation electrodes. In 5 of 13 cases, the rewarding effects produced by stimulating different sites in the same rat were altered differentially by weight loss. This finding is interpreted in terms of the subdivision of brain reward circuitry along functional and anatomical lines.