What Role Does Striatal Dopamine Play in Goal-directed Action?

Evidence suggests that dopamine activity provides a US-related prediction error for Pavlovian conditioning and the reinforcement signal supporting the acquisition of habits. However, its role in goal-directed action is less clear. There are currently few studies that have assessed dopamine release as animals acquire and perform self-paced instrumental actions. Here we briefly review the literature documenting the psychological, behavioral and neural bases of goal-directed actions in rats and mice, before turning to describe recent studies investigating the role of dopamine in instrumental learning and performance. Plasticity in dorsomedial striatum, a central node in the network supporting goal-directed action, clearly requires dopamine release, the timing of which, relative to cortical and thalamic inputs, determines the degree and form of that plasticity. Beyond this, bilateral release appears to reflect reward prediction errors as animals experience the consequences of an action. Such signals feedforward to update the value of the specific action associated with that outcome during subsequent performance, with dopamine release at the time of action reflecting the updated predicted action value. More recently, evidence has also emerged for a hemispherically lateralised signal associated with the action; dopamine release is greater in the hemisphere contralateral to the spatial target of the action. This effect emerges over the course of acquisition and appears to reflect the strength of the action-outcome association. Thus, during goal-directed action, dopamine release signals the action, the outcome and their association to shape the learning and performance processes necessary to support this form of behavioral control.

What does dopamine release reveal about latent inhibition?

A recent paper by Kutlu et al. (2022) argues that changes in dopamine release during stimulus pre-exposure reflect non-associative changes in attention to the conditioned stimulus that are causally related to latent inhibition effects. Associative accounts of pre-exposure-induced changes in associability suggest, however, that such conclusions may be premature.

Response-independent outcome presentations weaken the instrumental response-outcome association.

The present article explored the fate of previously formed response-outcome associations when the relation between R and O was disrupted by arranging for O to occur independently of R. In each of three experiments response independent outcome delivery selectively reduced the R earning that O. Nevertheless, in Experiments 1 and 2, the R continued to show sensitivity to outcome devaluation, suggesting that the strength of the R-O association was undiminished by this treatment. These experiments used a two-lever, two-outcome design introducing the possibility that devaluation reflected the influence of specific Pavlovian lever-outcome associations. In an attempt to nullify the influence of these incidental Pavlovian cues Experiment 3 used a single bidirectional vertical lever that rats could press left or right for different outcomes. Again, response-independent outcome presentations selectively depressed the performance of the R that delivered the response-independent O. However, in this situation, the response independent O also reduced the sensitivity of R to outcome devaluation; whereas the nondegraded R was sensitive to devaluation, the degraded R was not. We conclude that selective degradation of the instrumental contingency can weaken a specific R-O association while leaving other R-O associations intact. Furthermore, the use of a bidirectional vertical lever in Experiment 3 revealed that unidirectional and spatially separated instrumental manipulanda, such as levers or chains, may produce Pavlovian cues capable of forming incidental associations with the instrumental outcome that can obscure the relative influence of R-O associations after various manipulations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Striatal direct and indirect pathway neurons differentially control the encoding and updating of goal-directed learning.

The posterior dorsomedial striatum (pDMS) is necessary for goal-directed action; however, the role of the direct (dSPN) and indirect (iSPN) spiny projection neurons in the pDMS in such actions remains unclear. In this series of experiments, we examined the role of pDMS SPNs in goal-directed action in rats and found that whereas dSPNs were critical for goal-directed learning and for energizing the learned response, iSPNs were involved in updating that learning to support response flexibility. Instrumental training elevated expression of the plasticity marker Zif268 in dSPNs only, and chemogenetic suppression of dSPN activity during training prevented goal-directed learning. Unilateral optogenetic inhibition of dSPNs induced an ipsilateral response bias in goal-directed action performance. In contrast, although initial goal-directed learning was unaffected by iSPN manipulations, optogenetic inhibition of iSPNs, but not dSPNs, impaired the updating of this learning and attenuated response flexibility after changes in the action-outcome contingency.

Rodent medial and lateral orbitofrontal cortices represent unique components of cognitive maps of task space.

The orbitofrontal cortex (OFC) has been proposed to function as a cognitive map of task space: a mental model of the steps involved in a task. This idea has proven popular because it provides a cohesive explanation for a number of disparate findings regarding the OFC's role in a broad array of tasks. Concurrently, evidence has begun to reveal the functional heterogeneity of OFC subregions, particularly the medial and lateral OFC. How these subregions uniquely contribute to the OFC's role as a cognitive map of task space, however, has not been explored. Here we propose that, in rodents, the lateral OFC represents the agent's initial position within that task map (i.e. initial state), determining which actions are available as a consequence of that position, whereas the medial OFC represents the agent's future position within the task map (i.e. terminal state), influencing which actions are selected to achieve that position. We argue that these processes are achieved somewhat independently and somewhat interdependently, and are achieved through similar but non-identical circuitry.

From learning to action: the integration of dorsal striatal input and output pathways in instrumental conditioning.

Considerable evidence suggests that the learning and performance of instrumental actions depend on activity in basal ganglia circuitry; however, these two functions have generally been considered independently. Whereas research investigating the associative mechanisms underlying instrumental conditioning has identified critical cortical and limbic input pathways to the dorsal striatum, the performance of instrumental actions has largely been attributed to activity in the dorsal striatal output pathways, with direct and indirect pathway projection neurons mediating action initiation, perseveration and cessation. Here, we discuss evidence that the dorsal striatal input and basal ganglia output pathways mediate the learning and performance of instrumental actions, respectively, with the dorsal striatum functioning as a transition point. From this perspective, the issue of how multiple striatal inputs are integrated at the level of the dorsal striatum and converted into relatively restricted outputs becomes one of critical significance for understanding how learning is translated into action. So too does the question of how learning signals are modulated by recent experience. We propose that this occurs through recurrent corticostriatothalamic feedback circuits that serve to integrate performance signals by updating ongoing action-related learning.

Open-field PET: Simultaneous brain functional imaging and behavioural response measurements in freely moving small animals.

A comprehensive understanding of how the brain responds to a changing environment requires techniques capable of recording functional outputs at the whole-brain level in response to external stimuli. Positron emission tomography (PET) is an exquisitely sensitive technique for imaging brain function but the need for anaesthesia to avoid motion artefacts precludes concurrent behavioural response studies. Here, we report a technique that combines motion-compensated PET with a robotically-controlled animal enclosure to enable simultaneous brain imaging and behavioural recordings in unrestrained small animals. The technique was used to measure in vivo displacement of [11C]raclopride from dopamine D2 receptors (D2R) concurrently with changes in the behaviour of awake, freely moving rats following administration of unlabelled raclopride or amphetamine. The timing and magnitude of [11C]raclopride displacement from D2R were reliably estimated and, in the case of amphetamine, these changes coincided with a marked increase in stereotyped behaviours and hyper-locomotion. The technique, therefore, allows simultaneous measurement of changes in brain function and behavioural responses to external stimuli in conscious unrestrained animals, giving rise to important applications in behavioural neuroscience.

Inferring action-dependent outcome representations depends on anterior but not posterior medial orbitofrontal cortex.

Although studies examining orbitofrontal cortex (OFC) often treat it as though it were functionally homogeneous, recent evidence has questioned this assumption. Not only are the various subregions of OFC (lateral, ventral, and medial) hetereogeneous, but there is further evidence of heterogeneity within those subregions. For example, several studies in both humans and monkeys have revealed a functional subdivision along the anterior-posterior gradient of the medial OFC (mOFC). Given our previous findings suggesting that, in rats, the mOFC is responsible for inferring the likelihood of unobservable action outcomes (Bradfield, Dezfouli, van Holstein, Chieng, & Balleine, 2015), and given the anterior nature of the placements of our prior manipulations, we decided to assess whether the rat mOFC also differs in connection and function along its anteroposterior axis. We first used retrograde tracing to compare the density of efferents from mOFC to several structures known to contribute to goal-directed action: the mediodorsal thalamus, basolateral amygdala, posterior dorsomedial striatum, nucleus accumbens core and ventral tegmental area. We then compared the functional effects of anterior versus posterior mOFC excitotoxic lesions on tests of Pavlovian-instrumental transfer, instrumental outcome devaluation and outcome-specific reinstatement. We found evidence that the anterior mOFC had greater connectivity with the accumbens core and greater functional involvement in goal-directed action than the posterior mOFC. Consistent with previous findings across species, therefore, these results suggest that the anterior and posterior mOFC of the rat are indeed functionally distinct, and that it is the anterior mOFC that is particularly critical for inferring unobservable action outcomes.

The Bilateral Prefronto-striatal Pathway Is Necessary for Learning New Goal-Directed Actions.

The acquisition of new goal-directed actions requires the encoding of action-outcome associations. At a neural level, this encoding has been hypothesized to involve a prefronto-striatal circuit extending between the prelimbic cortex (PL) and the posterior dorsomedial striatum (pDMS); however, no research identifying this pathway with any precision has been reported. We started by mapping the prelimbic input to the dorsal and ventral striatum using a combination of retrograde and anterograde tracing with CLARITY and established that PL-pDMS projections share some overlap with projections to the nucleus accumbens core (NAc) in rats. We then tested whether each of these pathways were functionally required for goal-directed learning; we used a pathway-specific dual-virus chemogenetic approach to selectively silence pDMS-projecting or NAc-projecting PL neurons during instrumental training and tested rats for goal-directed action. We found that silencing PL-pDMS projections abolished goal-directed learning, whereas silencing PL-NAc projections left goal-directed learning intact. Finally, we used a three-virus approach to silence bilateral and contralateral pDMS-projecting PL neurons and again blocked goal-directed learning. These results establish that the acquisition of new goal-directed actions depends on the bilateral PL-pDMS pathway driven by intratelencephalic cortical neurons.

Prefrontal Corticostriatal Disconnection Blocks the Acquisition of Goal-Directed Action.

The acquisition of goal-directed action requires encoding of the association between an action and its specific consequences or outcome. At a neural level, this encoding has been hypothesized to involve a prefrontal corticostriatal circuit involving the projection from the prelimbic cortex (PL) to the posterior dorsomedial striatum (pDMS); however, no direct evidence for this claim has been reported. In a series of experiments, we performed functional disconnection of this pathway using targeted lesions of the anterior corpus callosum to disrupt contralateral corticostriatal projections with asymmetrical lesions of the PL and/or pDMS to block plasticity in this circuit in rats. We first demonstrated that unilaterally blocking the PL input to the pDMS prevented the phosphorylation of extracellular signal-related kinase/mitogen activated protein kinase (pERK/pMAPK) induced by instrumental training. Next, we used a full bilateral disconnection of the PL from the pDMS and assessed goal-directed action using an outcome-devaluation test. Importantly, we found evidence that rats maintaining an ipsilateral and/or contralateral connection between the PL and the pDMS were able to acquire goal-directed actions. In contrast, bilateral PL-pDMS disconnection abolished the acquisition of goal-directed actions. Finally, we used a temporary pharmacological disconnection to disrupt PL inputs to the pDMS by infusing the NMDA antagonist dl-2-amino-5-phosphonopentanoic acid into the pDMS during instrumental training and found that this manipulation also disrupted goal-directed learning. These results establish that, in rats, the acquisition of new goal-directed actions depends on a prefrontal-corticostriatal circuit involving a connection between the PL and the pDMS.SIGNIFICANCE STATEMENT It has been hypothesized that the prelimbic cortex (PL) and posterior dorsomedial striatum (pDMS) in rodents interact in a corticostriatal circuit to mediate goal-directed learning. However, no direct evidence supporting this claim has been reported. Using targeted lesions, we performed functional disconnection of the PL-pDMS pathway to assess its role in goal-directed learning. In the first experiment, we demonstrated that PL input to the pDMS is necessary for instrumental training-induced neuronal activity. Next, we disrupted ipsilateral, contralateral, or bilateral PL-pDMS connections and found that only bilateral PL-pDMS disconnection disrupted the acquisition of goal-directed actions, a finding we replicated in our final study using a pharmacological disconnection procedure.

Consolidation of Goal-Directed Action Depends on MAPK/ERK Signaling in Rodent Prelimbic Cortex.

The prelimbic prefrontal cortex (PL) has consistently been found to be necessary for the acquisition of goal-directed actions in rodents. Nevertheless, the specific cellular processes underlying this learning remain unknown. We assessed changes in learning-related expression of mitogen-activated protein kinase/extracellular signal-related kinase (MAPK/ERK1/2) phosphorylation (pERK) in layers 2-3 and 5-6 of the anterior and posterior PL and in the population of neurons projecting to posterior dorsomedial striatum (pDMS), also implicated in goal-directed learning. Rats were given either a single session of training to press a lever for a pellet reward or yoked reward deliveries without instrumental training and assessed 5 or 60 min after training for pERK expression. Relative to yoked training, instrumental training produced an increase in pERK expression in all regions of the PL both at 5 and 60 min, and this was accompanied by an increase in nuclear pERK expression in the posterior PL in rats given instrumental training. pDMS-projecting neurons showed a transient increase in pERK expression in posterior layer 5-6 projection neurons after 5 min, and a delayed increase in anterior layer 2-3 neurons after 60 min, suggesting that ERK expression in the PL is necessary for the consolidation of goal-directed learning. Consistent with this claim, we found that rats trained on two lever press actions for distinct outcomes and then infused with the MEK inhibitor PD98059 into the PL immediately after training failed to acquire specific action-outcome associations, suggesting that persistent pERK signaling in the PL is necessary for goal-directed learning. SIGNIFICANCE STATEMENT: The prelimbic cortex is implicated in goal-directed learning in rodents; however, it is unclear whether it is involved in the consolidation of this learning, and what cellular processes are involved. We used pERK as a marker of activity-related synaptic plasticity to assess learning-induced changes in distinct layers and neuronal populations of the prelimbic prefrontal cortex (PL). Training produced long-lasting upregulation of pERK throughout the PL and specifically within neurons that project to the pDMS, another region critical for goal-directed learning. Next, we demonstrated that pERK signaling in the PL was necessary for the consolidation of goal-directed learning. Together, these results indicate that instrumental training induces ERK signaling in distinct layers and populations in the PL and this signaling underlies the consolidation of goal-directed learning.

Benzodiazepine administration prevents the use of error-correction mechanisms during fear extinction.

Three experiments examined the effect of systemic administration of the benzodiazepine midazolam on extinction and re-extinction of conditioned fear. Experiment 1 demonstrated that midazolam administration prior to extinction of a conditioned stimulus (CS) impaired that extinction when rats were subsequently tested drug free; however, extinction was spared if rats were extinguished, reconditioned, and re-extinguished under midazolam. Experiment 2 provided a replication of this effect within-subjects; rats were conditioned to two CSs (A and B), extinguished to one (A-), reconditioned to both, and then extinguished/re-extinguished to both stimuli in compound (AB-), under either vehicle or midazolam. On the drug-free test, rats given midazolam froze more to the CS that had been extinguished (B) than the one that been re-extinguished (A). The final experiment examined whether extinction under midazolam was regulated by prediction error. Rats were trained with three CSs (A, B, C) and extinguished to two (A-, C-). These stimuli then underwent additional extinction under midazolam or vehicle, with one CS now presented in compound with the non-extinguished CS (AB-, C-). Rats were then tested for fear of A relative to C. Rats given vehicle showed a deepening of extinction to A relative to C, as is predicted from error-correction models; however, rats given midazolam failed to show any such discrepancy in responding. The results are interpreted to indicate that the drug reduced prediction error during extinction by reducing fear, and rats were able to re-extinguish fear via a retrieval mechanism that is independent of prediction error.

Benzodiazepine treatment can impair or spare extinction, depending on when it is given.

BACKGROUND: Benzodiazepines reduce the effectiveness of fear extinction in rodents and of exposure therapy in people suffering from anxiety disorders if given concomitantly with the behavioral treatment from its onset. The present experiments used rats to examine whether benzodiazepines had the same detrimental effect when given after some initial extinction had been conducted drug-free. METHODS: Rats were trained to fear a context (Experiments 1 and 2) or discrete cue (Experiment 3) and were extinguished to the context or cue under a benzodiazepine (midazolam) or vehicle. Extinction occurred either continuously in one session, with the drug or vehicle administered prior to the onset, or divided into two sessions, with the drug or vehicle administered prior to the second session. Rats were then tested, drug-free, for fear of the context or CS. RESULTS: Midazolam disrupted context and cue extinction when administered prior to the initial session but failed to disrupt extinction when given prior to the second session. CONCLUSIONS: The results in an animal model confirm that the effectiveness of extinction can be reduced when combined with benzodiazepines. They also suggest that the effectiveness of extinction will not be reduced when combined with a benzodiazepine if the patient has undergone some initial extinction drug-free.

Dorsal and ventral streams: the distinct role of striatal subregions in the acquisition and performance of goal-directed actions.

Considerable evidence suggests that distinct neural processes mediate the acquisition and performance of goal-directed instrumental actions. Whereas a cortical-dorsomedial striatal circuit appears critical for the acquisition of goal-directed actions, a cortical-ventral striatal circuit appears to mediate instrumental performance, particularly the motivational control of performance. Here we review evidence that these distinct mechanisms of learning and performance constitute two distinct 'streams' controlling instrumental conditioning. From this perspective, the regulation of the interaction between these 'streams' becomes a matter of considerable importance. We describe evidence that the basolateral amygdala, which is heavily interconnected with both the dorsal and ventral subregions of the striatum, coordinates this interaction providing input to the final common path to action as a critical component of the limbic-motor interface.

The role of the anterior, mediodorsal, and parafascicular thalamus in instrumental conditioning.

The traditional animal model of instrumental behavior has focused almost exclusively on structures within the cortico-striatal network and ignored the contributions of various thalamic nuclei despite large and specific connections with each of these structures. One possible reason for this is that the thalamus has been conventionally viewed as a mediator of general processes, such as attention, arousal and movement, that are not easily separated from more cognitive aspects of instrumental behavior. Recent research has, however, begun to separate these roles. Here we review the role of three thalamic nuclei in instrumental conditioning: the anterior thalamic nuclei (ANT), the mediodorsal (MD), and parafascicular thalamic nuclei (PF). Early research suggested that ANT might regulate aspects of instrumental behavior but, on review, we suggest that the types of tasks used in these studies were more likely to recruit Pavlovian processes. Indeed lesions of ANT have been found to have no effect on performance in instrumental free-operant tasks. By contrast the mediodorsal thalamus (MD) has been found to play a specific and important role in the acquisition of goal-directed action. We propose this role is related to its connections with prelimbic cortex (PL) and present new data that directly implicates this circuit in the acquisition of goal-directed actions. Finally we review evidence suggesting the PF, although not critical for the acquisition or performance of instrumental actions, plays a specific role in regulating action flexibility.

Systemic or intra-amygdala infusion of the benzodiazepine, midazolam, impairs learning, but facilitates re-learning to inhibit fear responses in extinction.

A series of experiments used rats to study the effect of a systemic or intra-amygdala infusion of the benzodiazepine, midazolam, on learning and re-learning to inhibit context conditioned fear (freezing) responses. Rats were subjected to two context-conditioning episodes followed by extinction under drug or vehicle, or to two cycles of context conditioning and extinction with the second extinction under drug or vehicle. A 20-min extinction under vehicle resulted in better long-term inhibition on a subsequent drug-free retention test than a 4-min extinction under vehicle, or a 20-min, as well as a 4-min, extinction under drug. However, a 20-min, as well as a 4-min, second extinction under drug was just as effective in producing long-term inhibition as a 20-min second extinction under vehicle and this inhibition was greater than that produced by a 4-min second extinction under vehicle. Initial extinction of 5, 10, or 20 min were equally effective in producing long-term inhibition when the second extinction under drug was 20 min; and 5-, 10-, or 20-min second extinction under drug were equally effective in producing long-term inhibition when the initial extinction was 5 min. A 4- or 20-min second extinction under an infusion of drug into the basolateral amygdala (BLA) was as effective in producing long-term inhibition as a 20-min second extinction under vehicle and was more effective than a 4-min second extinction under vehicle. The results show that midazolam impairs learning to inhibit fear responses but spares and even facilitates re-learning this inhibition.

Systemic or intra-amygdala injection of a benzodiazepine (midazolam) impairs extinction but spares re-extinction of conditioned fear responses.

Rats were subjected to one or two cycles of fear conditioning and extinction, injected with a benzodiazepine, midazolam, before the first or second extinction, and tested for long-term inhibition of fear responses (freezing). In Experiment 1, inhibition of context-conditioned fear was spared when midazolam was injected before the second extinction, but impaired when injected before the first. In Experiment 2, it was spared when midazolam was injected before the second extinction, but only if vehicle had been injected before the first: Inhibition was impaired when the drug was injected before both. In Experiment 3, inhibition of a discrete conditioned stimulus (CS A) was spared when midazolam was injected before its second extinction, but impaired when injected before extinction of CS A in rats that had undergone extinction of CS B. In Experiment 4, inhibition was spared when midazolam was injected into the basolateral amygdala before the second extinction, but impaired when injected before the first extinction of context-conditioned fear. The results show that midazolam impairs learning, but not relearning to inhibit fear responses, and are discussed in terms of state dependency, error correction, and memory retrieval, whereby the drug's anxiolytic effects on the second extinction reactivate and strengthen the original inhibitory memory.