Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release.

Recent computational models of sign tracking (ST) and goal tracking (GT) have accounted for observations that dopamine (DA) is not necessary for all forms of learning and have provided a set of predictions to further their validity. Among these, a central prediction is that manipulating the intertrial interval (ITI) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses. Here, we tested these predictions and found that lengthening the ITI increased ST, i.e., behavioral engagement with conditioned stimuli (CS) and cue-induced phasic DA release. Importantly, DA release was also present at the time of reward delivery, even after learning, and DA release was correlated with time spent in the food cup during the ITI. During conditioning with shorter ITIs, GT was prominent (i.e., engagement with food cup), and DA release responded to the CS while being absent at the time of reward delivery after learning. Hence, shorter ITIs restored the classical DA reward prediction error (RPE) pattern. These results validate the computational hypotheses, opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling.

The effect of high-fat diet consumption on appetitive instrumental behavior in rats.

Evidence now indicates that the chronic consumption of high-calorie foods, such as a high-fat diet (HFD), is associated with impaired control over food-seeking, yet the extent of this alteration is not fully understood. Using different reinforcement schedules, we evaluated whether HFD intake from weaning to adulthood modifies instrumental responding and induces a shift from goal-directed actions to habitual responding. We first observed reduced instrumental performance and motivation for a food reward in HFD-fed rats trained under schedules of reinforcement that facilitate habitual responding [Random Interval (RI)]. However, this deficit was alleviated if rats trained under RI were subsequently trained with reinforcement schedules that promote goal-directed strategies [Random Ratio (RR)]. Using an outcome devaluation procedure, we then demonstrated that consumption of a HFD promoted habitual behavior in rats trained under RI but not RR schedules. Finally, extended HFD exposure did not interfere with the ability of RR training to overcome impaired RI instrumental performance and to favor goal-directed behavior. These results indicate that chronic consumption of a HFD changes the co-ordination of goal-directed actions and habits and that alteration of food-seeking may be reversed under particular behavioral conditions.

Flexible Use of Predictive Cues beyond the Orbitofrontal Cortex: Role of the Submedius Thalamic Nucleus.

The orbitofrontal cortex (OFC) is known to play a crucial role in learning the consequences of specific events. However, the contribution of OFC thalamic inputs to these processes is largely unknown. Using a tract-tracing approach, we first demonstrated that the submedius nucleus (Sub) shares extensive reciprocal connections with the OFC. We then compared the effects of excitotoxic lesions of the Sub or the OFC on the ability of rats to use outcome identity to direct responding. We found that neither OFC nor Sub lesions interfered with the basic differential outcomes effect. However, more specific tests revealed that OFC rats, but not Sub rats, were disproportionally relying on the outcome, rather than on the discriminative stimulus, to guide behavior, which is consistent with the view that the OFC integrates information about predictive cues. In subsequent experiments using a Pavlovian contingency degradation procedure, we found that both OFC and Sub lesions produced a severe deficit in the ability to update Pavlovian associations. Altogether, the submedius therefore appears as a functionally relevant thalamic component in a circuit dedicated to the integration of predictive cues to guide behavior, previously conceived as essentially dependent on orbitofrontal functions. Significance statement: In the present study, we identify a largely unknown thalamic region, the submedius nucleus, as a new functionally relevant component in a circuit supporting the flexible use of predictive cues. Such abilities were previously conceived as largely dependent on the orbitofrontal cortex. Interestingly, this echoes recent findings in the field showing, in research involving an instrumental setup, an additional involvement of another thalamic nuclei, the parafascicular nucleus, when correct responding requires an element of flexibility (Bradfield et al., 2013a). Therefore, the present contribution supports the emerging view that limbic thalamic nuclei may contribute critically to adaptive responding when an element of flexibility is required after the establishment of initial learning.

Parallel inputs from the mediodorsal thalamus to the prefrontal cortex in the rat.

There is a growing interest in determining the functional contribution of thalamic inputs to cortical functions. In the context of adaptive behaviours, identifying the precise role of the mediodorsal thalamus (MD) in particular remains difficult despite the large amount of experimental data available. A better understanding of the thalamocortical connectivity of this region may help to capture its functional role. To address this issue, this study focused exclusively on the specific connections from the MD to the prefrontal cortex (PFC) by means of direct comparisons of labelling produced by single and dual injections of retrograde tracers in the different subdivisions of the PFC in the rat. We show that at least three parallel and essentially separate thalamocortical pathways originate from the MD, as follows: projections to the dorsal (1) and the ventral (2) subdivisions of the mPFC follow a mediolateral topography at the thalamic level (i.e. medial thalamic neurons target the mPFC ventrally whereas lateral thalamic neurons project dorsally), whereas a considerable innervation to the OFC (3) includes thalamic cells projecting to both the lateral and the ventral OFC subdivisions. These observations provide new insight on the functions of the MD and suggest a specific focus on each of these pathways for future functional studies.

A time course analysis of satiety-induced instrumental outcome devaluation.

Sensory-specific satiety is commonly used in studies of decision making to selectively devalue a food reward. Devaluation is reflected in an immediate reduction in the subsequent intake of the food and in the performance of actions that gain access to that food. Despite its frequent use, the lasting effects of satiety-induced devaluation on instrumental actions are unknown. Here, we examined the time course and contextual dependency of sensory-specific satiety-induced devaluation on instrumental responding and consumption. Rats were trained to perform two instrumental actions for two distinct food rewards. Then, one of the instrumental outcomes was provided ad libitum for 1 hour in separate feeding cages and the effect of this devaluation was assessed 0, 2, or 5 hours after satiation. At a delay of 0 or 2 hours, both intake and instrumental responding were sensitive to the satiety treatment. That is, rats consumed less of the devalued outcome and responded less for the devalued outcome than for the valued outcome. By contrast, after 5 hours, rats showed sensitivity to devaluation in consumption but not in instrumental responding. Strikingly, sensitivity to devaluation was restored for the instrumental response after a 5 hour delay when devaluation was performed in the instrumental context. These results indicate that, in rats, specific satiety-induced devaluation endures and is context-independent for up to 2 hours post-satiation. At longer delays, the impact of sensory-specific satiety on instrumental responding is context-dependent, suggesting that contextual cues may be required for the value of specific outcomes to control instrumental responding.

Mediodorsal but not anterior thalamic nuclei lesions impair acquisition of a conditional discrimination task.

The limbic thalamus is a heterogeneous structure with distinctive cortical connectivity. A recent review suggests that the mediodorsal thalamic nucleus (MD), unlike the anterior thalamic nuclei (ATN), may be involved in selecting relevant information in tasks relying on executive functions. We compared the effects of excitotoxic lesions of the MD or the ATN on the acquisition of a simple conditional discrimination in rats. When required to choose from two levers according to auditory or visual cues, ATN rats and sham-lesioned rats performed to the same levels and displayed similar acquisition curves. Under the same conditions, MD rats' acquisition of the task was markedly delayed. This group nevertheless attained nearly normal performances after more extensive training. Furthermore, all rats learned reversal of the original discrimination at the same rate. These results highlight functional specialization within the limbic thalamus and support the notion that MD contributes to the identification of relevant dimensions in conditional tasks during the initial stages of acquisition.

Parallel maturation of goal-directed behavior and dopaminergic systems during adolescence.

Adolescence is a crucial developmental period characterized by specific behaviors reflecting the immaturity of decision-making abilities. However, the maturation of precise cognitive processes and their neurobiological correlates at this period remain poorly understood. Here, we investigate whether a differential developmental time course of dopamine (DA) pathways during late adolescence could explain the emergence of particular executive and motivational components of goal-directed behavior. First, using a contingency degradation protocol, we demonstrate that adolescent rats display a specific deficit when the causal relationship between their actions and their consequences is changed. When the rats become adults, this deficit disappears. In contrast, actions of adolescents remain sensitive to outcome devaluation or to the influence of a pavlovian-conditioned stimulus. This aspect of cognitive maturation parallels a delayed development of the DA system, especially the mesocortical pathway involved in action adaptation to rule changes. Unlike in striatal and nucleus accumbens regions, DA fibers and DA tissue content continue to increase in the medial prefrontal cortex from juvenile to adult age. Moreover, a sustained overexpression of DA receptors is observed in the prefrontal region until the end of adolescence. These findings highlight the relationship between the emergence of specific cognitive processes, in particular the adaptation to changes in action consequences, and the delayed maturation of the mesocortical DA pathway. Similar developmental processes in humans could contribute to the adolescent vulnerability to the emergence of several psychiatric disorders characterized by decision-making deficits.

A role for anterior thalamic nuclei in affective cognition: interaction with environmental conditions.

Damage to anterior thalamic nuclei (ATN) is a well-known cause of diencephalic pathology that produces a range of cognitive deficits reminiscent of a hippocampal syndrome. Anatomical connections of the ATN also extend to cerebral areas that support affective cognition. Enriched environments promote recovery of declarative/relational memory after ATN lesions and are known to downregulate emotional behaviors. Hence, the performance of standard-housed and enriched ATN rats in a range of behavioral tasks engaging affective cognition was compared. ATN rats exhibited reduced anxiety responses in the elevated plus maze, increased activity and reduced corticosterone responses when exploring an open field, and delayed acquisition of a conditioned contextual fear response. ATN rats also exhibited reduced c-Fos and phosphorylated cAMP response element-binding protein (pCREB) immunoreactivity in the hippocampal formation and the amygdala after completion of the contextual fear test. Marked c-Fos hypoactivity and reduced pCREB levels were also evident in the granular retrosplenial cortex and, to a lesser extent, in the anterior cingulate cortex. Unlike standard-housed ATN rats, enriched ATN rats expressed virtually no fear of the conditioned context. These results show that the ATN regulate affective cognition and that damage to this region may produce markedly different behavioral effects as a function of environmental housing conditions.

A cholinergic-dependent role for the entorhinal cortex in trace fear conditioning.

Trace conditioning is considered a model of higher cognitive involvement in simple associative tasks. Studies of trace conditioning have shown that cortical areas and the hippocampal formation are required to associate events that occur at different times. However, the mechanisms that bridge the trace interval during the acquisition of trace conditioning remain unknown. In four experiments with fear conditioning in rats, we explored the involvement of the entorhinal cortex (EC) in the acquisition of fear under a trace-30 s protocol. We first determined that pretraining neurotoxic lesions of the EC selectively impaired trace-, but not delay-conditioned fear as evaluated by freezing behavior. A local cholinergic deafferentation of the EC using 192-IgG-saporin did not replicate this deficit, presumably because cholinergic interneurons were spared by the toxin. However, pretraining local blockade of EC muscarinic receptors with the M1 antagonist pirenzepine yielded a specific and dose-dependent deficit in trace-conditioned responses. The same microinjections performed after conditioning were without effect on trace fear responses. These effects of blocking M1 receptors are consistent with the notion that conditioned stimulus (CS)-elicited, acetylcholine-dependent persistent activities in the EC are needed to maintain a representation of a tone CS across the trace interval during the acquisition of trace conditioning. This function of the EC is consistent with recent views of this region as a short-term stimulus buffer.

A role for medial prefrontal dopaminergic innervation in instrumental conditioning.

To investigate the involvement of dopaminergic projections to the prelimbic and infralimbic cortex in the control of goal-directed responses, a first experiment examined the effect of pretraining 6-OHDA lesions of these cortices. We used outcome devaluation and contingency degradation procedures to separately assess the representation of the outcome as a goal or the encoding of the contingency between the action and its outcome. All groups acquired the instrumental response at a normal rate, indicating that dopaminergic activity in the medial prefrontal cortex is not necessary for the acquisition of instrumental learning. Sham-operated animals showed sensitivity to both outcome devaluation and contingency degradation. Animals with dopaminergic lesions of the prelimbic cortex, but not the infralimbic cortex, failed to adapt their instrumental response to changes in contingency, whereas their response remained sensitive to outcome devaluation. In a second experiment, aimed at determining whether dopamine was specifically needed during contingency changes, we performed microinfusions of the dopamine D(1)/D(2) receptor antagonist flupenthixol in the prelimbic cortex only before contingency degradation sessions. Animals with infusions of flupenthixol failed to adapt their response to changes in contingency, thus replicating the deficit of animals with dopaminergic lesions in Experiment 1. These results demonstrate that dissociable neurobiological mechanisms support action-outcome relationships and goal representation, dopamine signaling in the prelimbic cortex being necessary for the former but not the latter.

Targeting Catecholaminergic Systems in Transgenic Rats With a CAV-2 Vector Harboring a Cre-Dependent DREADD Cassette.

Techniques that allow the manipulation of specific neural circuits have greatly increased in the past few years. DREADDs (Designer receptors exclusively activated by designer drugs) provide an elegant way to manipulate individual brain structures and/or neural circuits, including neuromodulatory pathways. Considerable efforts have been made to increase cell-type specificity of DREADD expression while decreasing possible limitations due to multiple viral vectors injections. In line with this, a retrograde canine adenovirus type 2 (CAV-2) vector carrying a Cre-dependent DREADD cassette has been recently developed. In combination with Cre-driver transgenic animals, the vector allows one to target neuromodulatory pathways with cell-type specificity. In the present study, we specifically targeted catecholaminergic pathways by injecting the vector in knock-in rat line containing Cre recombinase cassette under the control of the tyrosine hydroxylase promoter. We assessed the efficacy of infection of the nigrostriatal pathway and the catecholaminergic pathways ascending to the orbitofrontal cortex (OFC) and found cell-type-specific DREADD expression.

Differential contribution of dorsal and ventral hippocampus to trace and delay fear conditioning.

Trace conditioning relies on the maintained representation of a stimulus across a trace interval, and may involve a persistent trace of the conditioned stimulus (CS) and/or a contribution of contextual conditioning. The role of hippocampal structures in these two types of conditioning was studied by means of pretraining lesions and reversible inactivation of the hippocampus in rats. Similar levels of conditioning to a tone CS and to the context were obtained with a trace interval of 30 s. Neurotoxic lesions of the whole hippocampus or reversible muscimol inactivation of the ventral hippocampus impaired both contextual and tone freezing in both trace- and delay-conditioned rats. Dorsal hippocampal injections impaired contextual freezing and trace conditioning, but not delay conditioning. No dissociation between trace and contextual conditioning was observed under any of these conditions. Altogether, these data indicate that the ventral and dorsal parts of the hippocampus compute different aspects of trace conditioning, with the ventral hippocampus being involved in fear and anxiety processes, and the dorsal hippocampus in the temporal and contextual aspects of event representation.

Transient role of the rat prelimbic cortex in goal-directed behaviour.

Lesion studies show that goal-directed actions mediated by action-outcome (A-O) associations and habits mediated by stimulus-response (S-R) associations can be dissociated during instrumental training, with the prelimbic region of the medial prefrontal cortex being involved in the former and the infralimbic region in the latter. The present work further investigates the role of the prelimbic region in acquisition vs. expression of goal-directed instrumental behaviour, using reversible neuronal inactivation and outcome devaluation procedures. In a first experiment, inactivating the prelimbic cortex at the time of testing did not alter the sensitivity to devaluation, indicating that this region was not essential for the expression of A-O associations. In a second experiment, the prelimbic cortex was inactivated throughout the training phase. At the time of testing the performance was insensitive to devaluation, indicating that the acquired response was not goal-directed but mediated by an S-R association. These data challenge the view that the habit system replaces the goal-directed system as training progresses. They show that the prelimbic cortex plays a transient but crucial role in the acquisition of goal-directed responding and that the A-O and S-R systems can operate in a competitive fashion early in training.

Goal-directed responding is sensitive to lesions to the prelimbic cortex or basolateral nucleus of the amygdala but not to their disconnection.

The current view of instrumental conditioning indicates that performance in the early stage of training is maintained by a representation of the outcome, as indexed by its sensitivity to changes in the value of the reward. In the present study, the authors tested the effects of a disconnection of the prelimbic cortex (PL) and the basolateral nucleus of the amygdale (BLA), using an asymmetric lesion procedure, to determine whether these structures interact sequentially as part of a corticolimbic system. In marked contrast to the effects of bilateral lesions of the PL or the BLA, which both altered rats' sensitivity to outcome devaluation, the disconnection of these 2 brain areas was without an effect on outcome devaluation. These results demonstrate that the PL and the BLA mediate different aspects of outcome representation in goal-directed responding.

The basolateral amygdala is necessary for learning but not relearning extinction of context conditioned fear.

Extinction of conditioned fear involves new learning that inhibits but does not eliminate the original fear memory. This inhibitory learning is thought to require activation of NMDA receptors (NMDAr) within the basolateral amygdala (BLA). However, once extinction has been learned, the role played by the BLA during subsequent extinction procedures remains unknown. The present study examined the role of neuronal activity and NMDAr activation in rats receiving their first or second extinction of context fear. We found that BLA infusion of DL-APV, a competitive antagonist of NMDAr, depressed fear responses at both the first and second extinction. It impaired learning extinction but spared and even facilitated relearning extinction. BLA infusion of muscimol, a GABA(A) agonist, produced a similar outcome, suggesting that DL-APV not only blocked NMDAr-dependent plasticity but also disrupted neuronal activity. In contrast, infusion of ifenprodil, a more selective antagonist of NMDAr containing the NR2B subunit, did not depress fear responses but impaired short- and long-term inhibition of fear at both the first and second extinction. Therefore, we suggest that relearning extinction normally requires NMDAr containing the NR2B subunit in the BLA. However, simultaneous blockade of these receptors and neuronal activity in the BLA results in compensatory learning that is able to promote long-term re-extinction. These data are consistent with a current model that attributes fear extinction to interactions between several neural substrates, including the amygdala and the medial prefrontal cortex.

Distinct regional patterns in noradrenergic innervation of the rat prefrontal cortex.

The anatomy and functions of the rodent prefrontal cortex (PFC) have been extensively studied. It is now clear that the PFC is at the core of various executive functions and that these functions depend on monoaminergic neuromodulation. The PFC receives extensive projections from monoaminergic nuclei and, in particular, from the locus cœruleus (LC) which is the major source of noradrenaline (NA) in the cortex. Projections of this nucleus have long been considered to act diffusely and uniformly throughout the entire brain. However, recent studies have revealed a separate innervation of prefrontal sub-regions by non-collateralizing LC neurons, suggesting a specific modulation of their functions. Following this idea, we aimed at describing more precisely the pattern of noradrenergic innervation into different orbital (OFC) and medial (mPFC) sub-regions of the PFC. We focused on the lateral (LO), ventral (VO) and medial (MO) portions of the OFC, and on areas 32d (A32d), 32v (A32v) and 25 (A25) in the mPFC. Using Dopamine-ß-Hydroxylase as a specific noradrenergic marker, we performed an automatic quantification of noradrenergic fibers and varicosities in each of these sub-regions. The results indicate that noradrenergic innervation is heterogeneous in some prefrontal sub-regions along the rostro-caudal axis. Functional dissociations have been recently reported in prefrontal sub-regions along the rostro-caudal direction. Our findings add neuroanatomical support to this emergent idea.

Impacts of inter-trial interval duration on a computational model of sign-tracking vs. goal-tracking behaviour.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53-57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

Dopamine blockade impairs the exploration-exploitation trade-off in rats.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

Thalamocortical and corticothalamic pathways differentially contribute to goal-directed behaviors in the rat.

Highly distributed neural circuits are thought to support adaptive decision-making in volatile and complex environments. Notably, the functional interactions between prefrontal and reciprocally connected thalamic nuclei areas may be important when choices are guided by current goal value or action-outcome contingency. We examined the functional involvement of selected thalamocortical and corticothalamic pathways connecting the dorsomedial prefrontal cortex (dmPFC) and the mediodorsal thalamus (MD) in the behaving rat. Using a chemogenetic approach to inhibit projection-defined dmPFC and MD neurons during an instrumental learning task, we show that thalamocortical and corticothalamic pathways differentially support goal attributes. Both pathways participate in adaptation to the current goal value, but only thalamocortical neurons are required to integrate current causal relationships. These data indicate that antiparallel flow of information within thalamocortical circuits may convey qualitatively distinct aspects of adaptive decision-making and highlight the importance of the direction of information flow within neural circuits.