From Prediction to Action: Dissociable Roles of Ventral Tegmental Area and Substantia Nigra Dopamine Neurons in Instrumental Reinforcement.

Reward seeking requires the coordination of motor programs to achieve goals. Midbrain dopamine neurons are critical for reinforcement, and their activation is sufficient for learning about cues, actions, and outcomes. Here we examine in detail the mechanisms underlying the ability of ventral tegmental area (VTA) and substantia nigra (SNc) dopamine neurons to support instrumental learning. By exploiting numerous behavioral tasks in combination with time-limited optogenetic manipulations in male and female rats, we reveal that VTA and SNc dopamine neurons generate reinforcement through separable psychological processes. VTA dopamine neurons imbue actions and their associated cues with motivational value that allows flexible and persistent pursuit, whereas SNc dopamine neurons support time-limited, precise, action-specific learning that is nonscalable and inflexible. This architecture is reminiscent of actor-critic reinforcement learning models with VTA and SNc instructing the critic and actor, respectively. Our findings indicate that heterogeneous dopamine systems support unique forms of instrumental learning that ultimately result in disparate reward-seeking strategies.SIGNIFICANCE STATEMENT Dopamine neurons in the midbrain are essential for learning, motivation, and movement. Here we describe in detail the ability of VTA and SNc dopamine neurons to generate instrumental reinforcement, a process where an agent learns about actions they can emit to earn reward. While rats will avidly work and learn to respond for activation of VTA and SNc dopamine neurons, we find that only VTA dopamine neurons imbue actions and their associated cues with motivational value that spur continued pursuit of reward. Our data support a hypothesis that VTA and SNc dopamine neurons engage distinct psychological processes that have consequences for our understanding of these neurons in health and disease.

Prior Cocaine Exposure Increases Firing to Immediate Reward While Attenuating Cue and Context Signals Related to Reward Value in the Insula.

The insula contributes to behavioral control and is disrupted by substance abuse, yet we know little about the neural signals underlying these functions or how they are disrupted after chronic drug self-administration. Here, male and female rats self-administered either cocaine (experimental group) or sucrose (control) for 12 consecutive days. After a 1 month withdrawal period, we recorded from insula while rats performed a previously learned reward-guided decision-making task. Cocaine-exposed rats were more sensitive to value manipulations and were faster to respond. These behavioral changes were accompanied by elevated counts of neurons in the insula that increased firing to reward. These neurons also fired more strongly at the start of long-delay trials, when a more immediate reward would be expected, and fired less strongly in anticipation of the actual delivery of delayed rewards. Although reward-related firing to immediate reward was enhanced after cocaine self-administration, reward-predicting cue and context signals were attenuated. In addition to revealing novel firing patterns unique to insula, our data suggest changes in such neural activity likely contribute to impaired decision making observed after drug use.SIGNIFICANCE STATEMENT The insula plays a clear role in drug addiction and drug-induced impairments of decision making, yet there is little understanding of its underlying neural signals. We found that chronic cocaine self-administration reduces cue and context encoding in insula while enhancing signals related to immediate reward. These changes in neural activity likely contribute to impaired decision making and impulsivity observed after drug use.

Overexpressing Histone Deacetylase 5 in Rat Dorsal Striatum Alters Reward-Guided Decision-Making and Associated Neural Encoding.

Accumulating evidence in the past decade implicates histone-modifying enzymes, such as class I histone deacetylases (HDACs), in learning and memory and, recently, habit formation. However, it is unclear whether HDACs play roles in complex cognitive function. To address this issue, we examined the role of dorsal striatal HDAC5, a class II HDAC, in reward-guided decision-making and associated neural encoding in rats. We first injected adeno-associated virus to overexpress a nuclear-localized HDAC5 in dorsal striatum (DS). We then recorded neural correlates from dorsolateral striatum (DLS) as rats performed two reward-guided choice tasks, in which we manipulated either the size of or delay to reward. During these tasks, rats first learned which of two options led to the better reward and then reversed those contingencies in a second block of trials. We found that rats with HDAC5 overexpression in DS responded faster and chose higher value reward more often during the first block of trials but were less able to reverse those contingencies in the second block of trials. At the neural level, HDAC5 overexpression in DS elevated and reduced the number of cells in DLS that increased firing to stimuli and reward, respectively, and shifted encoding toward cues that predicted more immediate reward. These results suggest that the HDAC5 overexpression in DS contributes to inflexible decision-making, demonstrating a role of histone-modifying enzymes in complex cognitive function.SIGNIFICANCE STATEMENT HDACs are important for learning and habit formation. Here, we expanded on these functions and found that overexpression of HDAC5 produced faster and more automatic behavior, and related changes in dorsolateral striatal neural firing in rats performing a value-based decision-making task. These results implicate HDAC5 as a potential therapeutic target for psychiatric conditions that impair decision-making and executive function.

NMDAR inhibition-independent antidepressant actions of ketamine metabolites.

Major depressive disorder affects around 16 per cent of the world population at some point in their lives. Despite the availability of numerous monoaminergic-based antidepressants, most patients require several weeks, if not months, to respond to these treatments, and many patients never attain sustained remission of their symptoms. The non-competitive, glutamatergic NMDAR (N-methyl-d-aspartate receptor) antagonist (R,S)-ketamine exerts rapid and sustained antidepressant effects after a single dose in patients with depression, but its use is associated with undesirable side effects. Here we show that the metabolism of (R,S)-ketamine to (2S,6S;2R,6R)-hydroxynorketamine (HNK) is essential for its antidepressant effects, and that the (2R,6R)-HNK enantiomer exerts behavioural, electroencephalographic, electrophysiological and cellular antidepressant-related actions in mice. These antidepressant actions are independent of NMDAR inhibition but involve early and sustained activation of AMPARs (α-amino-3-hydroxy-5-methyl-4-isoxazole propionic acid receptors). We also establish that (2R,6R)-HNK lacks ketamine-related side effects. Our data implicate a novel mechanism underlying the antidepressant properties of (R,S)-ketamine and have relevance for the development of next-generation, rapid-acting antidepressants.

The impact of drugs of abuse on executive function: characterizing long-term changes in neural correlates following chronic drug exposure and withdrawal in rats.

Addiction has long been characterized by diminished executive function, control, and impulsivity management. In particular, these deficits often manifest themselves as impairments in reversal learning, delay discounting, and response inhibition. Understanding the neurobiological substrates of these behavioral deficits is of paramount importance to our understanding of addiction. Within the cycle of addiction, periods during and after withdrawal represent a particularly difficult point of intervention in that the negative physical symptoms associated with drug removal and drug craving increase the likelihood that the patient will relapse and return to drug use in order to abate these symptoms. Moreover, it is often during this time that drug induced deficits in executive function hinder the ability of the patient to refrain from drug use. Thus, it is necessary to understand the physiological and behavioral changes associated with withdrawal and drug craving-largely manifesting as deficits in executive control-to develop more effective treatment strategies. In this review, we address the long-term impact that drugs of abuse have on the behavioral and neural correlates that give rise to executive control as measured by reversal learning, delay discounting, and stop-signal tasks, focusing particularly on our work using rats as a model system.

The Prodrug 4-Chlorokynurenine Causes Ketamine-Like Antidepressant Effects, but Not Side Effects, by NMDA/GlycineB-Site Inhibition.

Currently approved antidepressant drug treatment typically takes several weeks to be effective. The noncompetitive N-methyl-d-aspartate (NMDA) receptor antagonist ketamine has shown efficacy as a rapid-acting treatment of depression, but its use is associated with significant side effects. We assessed effects following blockade of the glycineB co-agonist site of the NMDA receptor, located on the GluN1 subunit, by the selective full antagonist 7-chloro-kynurenic acid (7-Cl-KYNA), delivered by systemic administration of its brain-penetrant prodrug 4-chlorokynurenine (4-Cl-KYN) in mice. Following administration of 4-Cl-KYN, 7-Cl-KYNA was promptly recovered extracellularly in hippocampal microdialysate of freely moving animals. The behavioral responses of the animals were assessed using measures of ketamine-sensitive antidepressant efficacy (including the 24-hour forced swim test, learned helplessness test, and novelty-suppressed feeding test). In these tests, distinct from fluoxetine, and similar to ketamine, 4-Cl-KYN administration resulted in rapid, dose-dependent and persistent antidepressant-like effects following a single treatment. The antidepressant effects of 4-Cl-KYN were prevented by pretreatment with glycine or the α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) receptor antagonist 2,3-dihydroxy-6-nitro-7-sulfamoyl-benzo[f]quinoxaline-2,3-dione (NBQX). 4-Cl-KYN administration was not associated with the rewarding and psychotomimetic effects of ketamine, and did not induce locomotor sensitization or stereotypic behaviors. Our results provide further support for antagonism of the glycineB site for the rapid treatment of treatment-resistant depression without the negative side effects seen with ketamine or other channel-blocking NMDA receptor antagonists.

Insula lesions reduce stimulus-driven control of behavior during odor-guided decision-making and autoshaping.

The insula has become a significant brain region in the study of both normal and impaired behavior and decision-making and has emerged as an important contributor to drug addiction. Consistent with this literature, in a previous study, we found that neural signals in rat insula encode anticipation and contextual global reward value during performance of an odor-guided delay/size choice task, and that these signals are disrupted by prior cocaine self-administration. Still, it is unknown if insula is critical for performance of this task under normal circumstances. Here, we sought to elucidate the functional role of these signals by lesioning the same region of anterior insula we previously recorded from. In addition to examining behavior during decision-making, we characterized behavior during autoshaping to further assess insula's role in behavior. We found insula damage resulted in reduced accuracy and faster reaction times, without affecting rats' choice of high-value reward, and that insula lesions reduced sign-tracking behavior. These results suggest that insula contributes to our odor-guided delay/size choice task via mechanisms that impact the control that environmental stimuli have on behavior.

Rats delay gratification during a time-based diminishing returns task.

The rat is a common animal model used to uncover the neural underpinnings of decision making and their disruption in psychiatric illness. Here, we ask if rats can perform a decision-making task that assesses self-control by delayed gratification in the context of diminishing returns. In this task, rats could choose to press one of two levers. One lever was associated with a fixed delay (FD) schedule that delivered reward after a fixed time delay (10 s). The other lever was associated with a progressive delay (PD) schedule; the delay increased by a fixed amount of time (1 s) after each PD lever press. Rats were tested under two conditions: a reset condition where rats could reset the PD schedule back to its initial 0-s delay by pressing the FD lever and a no-reset condition in which resetting the PD schedule was unavailable. We found that rats adapted behavior within reset sessions by delaying gratification to obtain more reward in the long run. That is, they selected the FD lever with the longer delay to reset the PD delay back to zero prior to the equality point, thus achieving more reward over the course of the session. These results are consistent with other species, demonstrating that rats can also maximize the net rate of reward by selecting an option that is not immediately beneficial. Moreover, use of this task in rodents might provide insights into how the brain governs normal and abnormal behavior, as well as treatments that can improve self-control. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Prior cocaine self-administration impairs attention signals in anterior cingulate cortex.

Although maladaptive decision-making is a defining feature of drug abuse and addiction, we have yet to ascertain how cocaine self-administration disrupts neural signals in anterior cingulate cortex (ACC), a brain region thought to contribute to attentional control. To address this issue, rats were trained on a reward-guided decision-making task; reward value was manipulated by independently varying the size of or the delay to reward over several trial blocks. Subsequently, rats self-administered either a cocaine (experimental group) or sucrose (control) during 12 consecutive days, after which they underwent a 1-month withdrawal period. Upon completion of this period, rats performed the previously learned reward-guided decision-making task while we recorded from single neurons in ACC. We demonstrate that prior cocaine self-administration attenuates attention and attention-related ACC signals in an intake-dependent manner, and that changes in attention are decoupled from ACC firing. These effects likely contribute to the impaired decision-making-typified by chronic substance abuse and relapse-observed after drug use.

Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions.

Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) encode reward prediction errors (RPEs) and are proposed to mediate error-driven learning. However, the learning strategy engaged by DA-RPEs remains controversial. RPEs might imbue predictive cues with pure value, independently of representations of their associated outcome. Alternatively, RPEs might promote learning about the sensory features (the identity) of the rewarding outcome. Here, we show that, although both VTA and SNc DA neuron activation reinforces instrumental responding, only VTA DA neuron activation during consumption of expected sucrose reward restores error-driven learning and promotes formation of a new cue→sucrose association. Critically, expression of VTA DA-dependent Pavlovian associations is abolished following sucrose devaluation, a signature of identity-based learning. These findings reveal that activation of VTA- or SNc-DA neurons engages largely dissociable learning processes with VTA-DA neurons capable of participating in outcome-specific predictive learning, and the role of SNc-DA neurons appears limited to reinforcement of instrumental responses.

Lever Insertion as a Salient Stimulus Promoting Insensitivity to Outcome Devaluation.

Flexible and efficient decision-making in complex environments can be achieved through constant interactions between the goal-directed and habitual systems. While goal-directed behavior is considered dependent upon Response-Outcome (R-O) associations, habits instead rely on Stimulus-Response (S-R) associations. However, the stimuli that support the S-R association underlying habitual responding in typical instrumental procedures are poorly defined. To resolve this issue, we designed a discrete-trials procedure, in which rats must wait for lever insertion and complete a sequence of five lever presses to obtain a reward (20% sucrose or grain-based pellets). Lever insertion thus constituted an audio-visual stimulus signaling the opportunity for reward. Using sensory-specific satiety-induced devaluation, we found that rats trained with grain-based pellets remained sensitive to outcome devaluation over the course of training with this procedure whereas rats trained with a solution of 20% sucrose rapidly developed habit, and that insensitivity to outcome devaluation in rats trained with sucrose did not result from a bias in general satiety. Importantly, although rats trained with pellets were sensitive to satiety-induced devaluation, their performance was not affected by degradation of instrumental contingency and devaluation by conditioned taste aversion (CTA), suggesting that these rats may also have developed habitual responding. To test whether the discrete-trials procedure biases subjects towards habitual responding, we compared discrete-trials to free-running instrumental responding, and found that rats trained with sucrose in a fixed-ratio 5 (FR5) procedure with continuous presentation of the lever were goal-directed. Together, these results demonstrate that discrete presentations of a stimulus predictive of reward availability promoted the formation of S-R habit in rats trained with liquid sucrose. Further research is necessary to explain inconsistencies in sensitivity to outcome devaluation when rats are trained with grain-based pellets.