Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Targeting Catecholaminergic Systems in Transgenic Rats With a CAV-2 Vector Harboring a Cre-Dependent DREADD Cassette.

Techniques that allow the manipulation of specific neural circuits have greatly increased in the past few years. DREADDs (Designer receptors exclusively activated by designer drugs) provide an elegant way to manipulate individual brain structures and/or neural circuits, including neuromodulatory pathways. Considerable efforts have been made to increase cell-type specificity of DREADD expression while decreasing possible limitations due to multiple viral vectors injections. In line with this, a retrograde canine adenovirus type 2 (CAV-2) vector carrying a Cre-dependent DREADD cassette has been recently developed. In combination with Cre-driver transgenic animals, the vector allows one to target neuromodulatory pathways with cell-type specificity. In the present study, we specifically targeted catecholaminergic pathways by injecting the vector in knock-in rat line containing Cre recombinase cassette under the control of the tyrosine hydroxylase promoter. We assessed the efficacy of infection of the nigrostriatal pathway and the catecholaminergic pathways ascending to the orbitofrontal cortex (OFC) and found cell-type-specific DREADD expression.

Impacts of inter-trial interval duration on a computational model of sign-tracking vs. goal-tracking behaviour.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53-57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

Dopamine blockade impairs the exploration-exploitation trade-off in rats.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

Distinct regional patterns in noradrenergic innervation of the rat prefrontal cortex.

The anatomy and functions of the rodent prefrontal cortex (PFC) have been extensively studied. It is now clear that the PFC is at the core of various executive functions and that these functions depend on monoaminergic neuromodulation. The PFC receives extensive projections from monoaminergic nuclei and, in particular, from the locus cœruleus (LC) which is the major source of noradrenaline (NA) in the cortex. Projections of this nucleus have long been considered to act diffusely and uniformly throughout the entire brain. However, recent studies have revealed a separate innervation of prefrontal sub-regions by non-collateralizing LC neurons, suggesting a specific modulation of their functions. Following this idea, we aimed at describing more precisely the pattern of noradrenergic innervation into different orbital (OFC) and medial (mPFC) sub-regions of the PFC. We focused on the lateral (LO), ventral (VO) and medial (MO) portions of the OFC, and on areas 32d (A32d), 32v (A32v) and 25 (A25) in the mPFC. Using Dopamine-ß-Hydroxylase as a specific noradrenergic marker, we performed an automatic quantification of noradrenergic fibers and varicosities in each of these sub-regions. The results indicate that noradrenergic innervation is heterogeneous in some prefrontal sub-regions along the rostro-caudal axis. Functional dissociations have been recently reported in prefrontal sub-regions along the rostro-caudal direction. Our findings add neuroanatomical support to this emergent idea.

Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release.

Recent computational models of sign tracking (ST) and goal tracking (GT) have accounted for observations that dopamine (DA) is not necessary for all forms of learning and have provided a set of predictions to further their validity. Among these, a central prediction is that manipulating the intertrial interval (ITI) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses. Here, we tested these predictions and found that lengthening the ITI increased ST, i.e., behavioral engagement with conditioned stimuli (CS) and cue-induced phasic DA release. Importantly, DA release was also present at the time of reward delivery, even after learning, and DA release was correlated with time spent in the food cup during the ITI. During conditioning with shorter ITIs, GT was prominent (i.e., engagement with food cup), and DA release responded to the CS while being absent at the time of reward delivery after learning. Hence, shorter ITIs restored the classical DA reward prediction error (RPE) pattern. These results validate the computational hypotheses, opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling.

Thalamocortical and corticothalamic pathways differentially contribute to goal-directed behaviors in the rat.

Highly distributed neural circuits are thought to support adaptive decision-making in volatile and complex environments. Notably, the functional interactions between prefrontal and reciprocally connected thalamic nuclei areas may be important when choices are guided by current goal value or action-outcome contingency. We examined the functional involvement of selected thalamocortical and corticothalamic pathways connecting the dorsomedial prefrontal cortex (dmPFC) and the mediodorsal thalamus (MD) in the behaving rat. Using a chemogenetic approach to inhibit projection-defined dmPFC and MD neurons during an instrumental learning task, we show that thalamocortical and corticothalamic pathways differentially support goal attributes. Both pathways participate in adaptation to the current goal value, but only thalamocortical neurons are required to integrate current causal relationships. These data indicate that antiparallel flow of information within thalamocortical circuits may convey qualitatively distinct aspects of adaptive decision-making and highlight the importance of the direction of information flow within neural circuits.

Personality and predisposition to form habit behaviours during instrumental conditioning in horses (Equus caballus).

The relationship between personality and learning abilities has become a growing field of interest. Studies have mainly focused on the relationship with performance, such as the speed of acquisition. In this study, we hypothesised that personality could in part also be related to a certain predisposition of an individual to switch more easily from a goal-directed process to a habit process during learning. To identify these processes, we conducted a contingency degradation protocol. This study investigated 1/ whether in general horses are able to adjust their response according to the contingency between their action and the reward, 2/ whether there are any relationships between certain personality profiles and a predisposition to switch more rapidly to habitual processes, and 3/ whether emotional states experienced during the learning procedure play a role in this switching. Personality tests were conducted on 29 horses, followed by a degradation contingency protocol. Overall, results show that horses were sensitive to contingency degradation between their action and the reward. Nevertheless, there was inter-individual variability: the horses presenting high fearfulness, and to a lesser extent low sensory sensitivity and low gregariousness were less sensitive to the degradation, demonstrating that they were more likely to switch to a habitual process. Contrary to our expectations, the emotional state experienced during the procedure did not seem to explain this switching. We conclude that personality is not only related to learning performance, but also in part to the process involved during learning, independently of the emotion experienced during the process. This study provides new theoretical knowledge on cognitive skills in ungulates.

The effect of high-fat diet consumption on appetitive instrumental behavior in rats.

Evidence now indicates that the chronic consumption of high-calorie foods, such as a high-fat diet (HFD), is associated with impaired control over food-seeking, yet the extent of this alteration is not fully understood. Using different reinforcement schedules, we evaluated whether HFD intake from weaning to adulthood modifies instrumental responding and induces a shift from goal-directed actions to habitual responding. We first observed reduced instrumental performance and motivation for a food reward in HFD-fed rats trained under schedules of reinforcement that facilitate habitual responding [Random Interval (RI)]. However, this deficit was alleviated if rats trained under RI were subsequently trained with reinforcement schedules that promote goal-directed strategies [Random Ratio (RR)]. Using an outcome devaluation procedure, we then demonstrated that consumption of a HFD promoted habitual behavior in rats trained under RI but not RR schedules. Finally, extended HFD exposure did not interfere with the ability of RR training to overcome impaired RI instrumental performance and to favor goal-directed behavior. These results indicate that chronic consumption of a HFD changes the co-ordination of goal-directed actions and habits and that alteration of food-seeking may be reversed under particular behavioral conditions.

Parallel inputs from the mediodorsal thalamus to the prefrontal cortex in the rat.

There is a growing interest in determining the functional contribution of thalamic inputs to cortical functions. In the context of adaptive behaviours, identifying the precise role of the mediodorsal thalamus (MD) in particular remains difficult despite the large amount of experimental data available. A better understanding of the thalamocortical connectivity of this region may help to capture its functional role. To address this issue, this study focused exclusively on the specific connections from the MD to the prefrontal cortex (PFC) by means of direct comparisons of labelling produced by single and dual injections of retrograde tracers in the different subdivisions of the PFC in the rat. We show that at least three parallel and essentially separate thalamocortical pathways originate from the MD, as follows: projections to the dorsal (1) and the ventral (2) subdivisions of the mPFC follow a mediolateral topography at the thalamic level (i.e. medial thalamic neurons target the mPFC ventrally whereas lateral thalamic neurons project dorsally), whereas a considerable innervation to the OFC (3) includes thalamic cells projecting to both the lateral and the ventral OFC subdivisions. These observations provide new insight on the functions of the MD and suggest a specific focus on each of these pathways for future functional studies.

A time course analysis of satiety-induced instrumental outcome devaluation.

Sensory-specific satiety is commonly used in studies of decision making to selectively devalue a food reward. Devaluation is reflected in an immediate reduction in the subsequent intake of the food and in the performance of actions that gain access to that food. Despite its frequent use, the lasting effects of satiety-induced devaluation on instrumental actions are unknown. Here, we examined the time course and contextual dependency of sensory-specific satiety-induced devaluation on instrumental responding and consumption. Rats were trained to perform two instrumental actions for two distinct food rewards. Then, one of the instrumental outcomes was provided ad libitum for 1 hour in separate feeding cages and the effect of this devaluation was assessed 0, 2, or 5 hours after satiation. At a delay of 0 or 2 hours, both intake and instrumental responding were sensitive to the satiety treatment. That is, rats consumed less of the devalued outcome and responded less for the devalued outcome than for the valued outcome. By contrast, after 5 hours, rats showed sensitivity to devaluation in consumption but not in instrumental responding. Strikingly, sensitivity to devaluation was restored for the instrumental response after a 5 hour delay when devaluation was performed in the instrumental context. These results indicate that, in rats, specific satiety-induced devaluation endures and is context-independent for up to 2 hours post-satiation. At longer delays, the impact of sensory-specific satiety on instrumental responding is context-dependent, suggesting that contextual cues may be required for the value of specific outcomes to control instrumental responding.

Dissociable effects of anterior and mediodorsal thalamic lesions on spatial goal-directed behavior.

Goal-directed behaviors are thought to be supported by a neural circuit encompassing the prefrontal cortex, the dorsomedial striatum, the amygdala, and, as more recently suggested, the limbic thalamus. Since evidence indicates that the various thalamic nuclei contribute to dissociable functions, we directly compared the functional contribution of the mediodorsal thalamus (MD) and of the anterior thalamic nuclei (ATN) in a new task assessing spatial goal-directed behavior in a cross-maze. Rats sustaining lesions of the mediodorsal or the anterior thalamus were trained to associate each of the two goal arms with a distinctive food reward. Unlike control rats, both lesioned groups failed to express a bias for the goal arm corresponding to the non-devalued outcome following devaluation by sensory-specific satiety. In addition, MD rats were slower than the other groups to complete the trials. When tested for spatial working memory using a standard non-matching-to-place procedure in the same apparatus, ATN rats were severely impaired but MD rats performed as well as controls, even when spatial or temporal challenges were introduced. Finally, all groups displayed comparable breaking points in a progressive ratio test, indicating that the slower choice performance of MD rats did not result from motivational factors. Thus, a spatial task requiring the integration of instrumental and Pavlovian contingencies reveals a fundamental deficit of MD rats in adapting their choice according to goal value. By contrast, the deficit associated with anterior thalamic lesions appears to simply reflect the inability to process spatial information.

Flexible Use of Predictive Cues beyond the Orbitofrontal Cortex: Role of the Submedius Thalamic Nucleus.

The orbitofrontal cortex (OFC) is known to play a crucial role in learning the consequences of specific events. However, the contribution of OFC thalamic inputs to these processes is largely unknown. Using a tract-tracing approach, we first demonstrated that the submedius nucleus (Sub) shares extensive reciprocal connections with the OFC. We then compared the effects of excitotoxic lesions of the Sub or the OFC on the ability of rats to use outcome identity to direct responding. We found that neither OFC nor Sub lesions interfered with the basic differential outcomes effect. However, more specific tests revealed that OFC rats, but not Sub rats, were disproportionally relying on the outcome, rather than on the discriminative stimulus, to guide behavior, which is consistent with the view that the OFC integrates information about predictive cues. In subsequent experiments using a Pavlovian contingency degradation procedure, we found that both OFC and Sub lesions produced a severe deficit in the ability to update Pavlovian associations. Altogether, the submedius therefore appears as a functionally relevant thalamic component in a circuit dedicated to the integration of predictive cues to guide behavior, previously conceived as essentially dependent on orbitofrontal functions. Significance statement: In the present study, we identify a largely unknown thalamic region, the submedius nucleus, as a new functionally relevant component in a circuit supporting the flexible use of predictive cues. Such abilities were previously conceived as largely dependent on the orbitofrontal cortex. Interestingly, this echoes recent findings in the field showing, in research involving an instrumental setup, an additional involvement of another thalamic nuclei, the parafascicular nucleus, when correct responding requires an element of flexibility (Bradfield et al., 2013a). Therefore, the present contribution supports the emerging view that limbic thalamic nuclei may contribute critically to adaptive responding when an element of flexibility is required after the establishment of initial learning.

Mediodorsal but not anterior thalamic nuclei lesions impair acquisition of a conditional discrimination task.

The limbic thalamus is a heterogeneous structure with distinctive cortical connectivity. A recent review suggests that the mediodorsal thalamic nucleus (MD), unlike the anterior thalamic nuclei (ATN), may be involved in selecting relevant information in tasks relying on executive functions. We compared the effects of excitotoxic lesions of the MD or the ATN on the acquisition of a simple conditional discrimination in rats. When required to choose from two levers according to auditory or visual cues, ATN rats and sham-lesioned rats performed to the same levels and displayed similar acquisition curves. Under the same conditions, MD rats' acquisition of the task was markedly delayed. This group nevertheless attained nearly normal performances after more extensive training. Furthermore, all rats learned reversal of the original discrimination at the same rate. These results highlight functional specialization within the limbic thalamus and support the notion that MD contributes to the identification of relevant dimensions in conditional tasks during the initial stages of acquisition.

Functional heterogeneity of the limbic thalamus: From hippocampal to cortical functions.

Today, the idea that the integrity of the limbic thalamus is necessary for normal memory functions is well established. However, if the study of thalamic patients emphasized the anterior and the mediodorsal thalamus as the critical thalamic loci supporting cognitive functions, clinical studies have so far failed to attribute a specific role to each of these regions. In view of these difficulties, we review here the experimental data conducted in rodents harboring specific lesions of each thalamic region. These data clearly indicate a major functional dissociation within the limbic thalamus. The anterior thalamus provides critical support for hippocampal functions due to its cardinal location in the Papez circuit, while the mediodorsal thalamus may signal relevant information in a circuit encompassing the basolateral amygdala and the prefrontal cortex. Interestingly, while clinical studies have suggested that diencephalic pathologies may disconnect the medial temporal lobe from the cortex, experimental studies conducted in rodent show how this may differently affect distinct temporo-thalamo-cortical circuits, sharing the same general organization but supporting dissociable functions.

Elucidating poor decision-making in a rat gambling task.

Although poor decision-making is a hallmark of psychiatric conditions such as attention deficit/hyperactivity disorder, pathological gambling or substance abuse, a fraction of healthy individuals exhibit similar poor decision-making performances in everyday life and specific laboratory tasks such as the Iowa Gambling Task. These particular individuals may provide information on risk factors or common endophenotypes of these mental disorders. In a rodent version of the Iowa gambling task--the Rat Gambling Task (RGT), we identified a population of poor decision makers, and assessed how these rats scored for several behavioral traits relevant to executive disorders: risk taking, reward seeking, behavioral inflexibility, and several aspects of impulsivity. First, we found that poor decision-making could not be well predicted by single behavioral and cognitive characteristics when considered separately. By contrast, a combination of independent traits in the same individual, namely risk taking, reward seeking, behavioral inflexibility, as well as motor impulsivity, was highly predictive of poor decision-making. Second, using a reinforcement-learning model of the RGT, we confirmed that only the combination of extreme scores on these traits could induce maladaptive decision-making. Third, the model suggested that a combination of these behavioral traits results in an inaccurate representation of rewards and penalties and inefficient learning of the environment. Poor decision-making appears as a consequence of the over-valuation of high-reward-high-risk options in the task. Such a specific psychological profile could greatly impair clinically healthy individuals in decision-making tasks and may predispose to mental disorders with similar symptoms.

Adolescent stimulation of D2 receptors alters the maturation of dopamine-dependent goal-directed behavior.

Adolescence is a period of high sensitivity to drugs and rewards, characterized by the immaturity of decision-making abilities. A chronic stimulation of reward systems during this period might constitute a factor of vulnerability to the development of psychiatric disorders. However, the long-term consequences of such an exposure have seldom been explored. Here, we investigate at the adult age the effects of chronic dopamine (DA) stimulation during adolescence on both the maturation of DA systems and the cognitive processes underlying goal-directed actions. We first demonstrate that chronic stimulation of D2 receptors by quinpirole during adolescence alters the development of DA systems. This treatment has particularly prominent effects on the mesocortical DA pathway where it decreases DA fibers density, DA concentration, and DA receptors expression. Furthermore, we show that quinpirole-treated rats exhibit specific impairments in instrumental goal-directed behavior, as they fail to adapt their action when action-outcome relationships change in a contingency degradation procedure. These results therefore highlight the vulnerability of DA system and prefrontal areas to prolonged stimulation during adolescence, and its potential long-term impact on cognitive functions.

A role for anterior thalamic nuclei in affective cognition: interaction with environmental conditions.

Damage to anterior thalamic nuclei (ATN) is a well-known cause of diencephalic pathology that produces a range of cognitive deficits reminiscent of a hippocampal syndrome. Anatomical connections of the ATN also extend to cerebral areas that support affective cognition. Enriched environments promote recovery of declarative/relational memory after ATN lesions and are known to downregulate emotional behaviors. Hence, the performance of standard-housed and enriched ATN rats in a range of behavioral tasks engaging affective cognition was compared. ATN rats exhibited reduced anxiety responses in the elevated plus maze, increased activity and reduced corticosterone responses when exploring an open field, and delayed acquisition of a conditioned contextual fear response. ATN rats also exhibited reduced c-Fos and phosphorylated cAMP response element-binding protein (pCREB) immunoreactivity in the hippocampal formation and the amygdala after completion of the contextual fear test. Marked c-Fos hypoactivity and reduced pCREB levels were also evident in the granular retrosplenial cortex and, to a lesser extent, in the anterior cingulate cortex. Unlike standard-housed ATN rats, enriched ATN rats expressed virtually no fear of the conditioned context. These results show that the ATN regulate affective cognition and that damage to this region may produce markedly different behavioral effects as a function of environmental housing conditions.

Parallel maturation of goal-directed behavior and dopaminergic systems during adolescence.

Adolescence is a crucial developmental period characterized by specific behaviors reflecting the immaturity of decision-making abilities. However, the maturation of precise cognitive processes and their neurobiological correlates at this period remain poorly understood. Here, we investigate whether a differential developmental time course of dopamine (DA) pathways during late adolescence could explain the emergence of particular executive and motivational components of goal-directed behavior. First, using a contingency degradation protocol, we demonstrate that adolescent rats display a specific deficit when the causal relationship between their actions and their consequences is changed. When the rats become adults, this deficit disappears. In contrast, actions of adolescents remain sensitive to outcome devaluation or to the influence of a pavlovian-conditioned stimulus. This aspect of cognitive maturation parallels a delayed development of the DA system, especially the mesocortical pathway involved in action adaptation to rule changes. Unlike in striatal and nucleus accumbens regions, DA fibers and DA tissue content continue to increase in the medial prefrontal cortex from juvenile to adult age. Moreover, a sustained overexpression of DA receptors is observed in the prefrontal region until the end of adolescence. These findings highlight the relationship between the emergence of specific cognitive processes, in particular the adaptation to changes in action consequences, and the delayed maturation of the mesocortical DA pathway. Similar developmental processes in humans could contribute to the adolescent vulnerability to the emergence of several psychiatric disorders characterized by decision-making deficits.

The role of the rat medial prefrontal cortex in adapting to changes in instrumental contingency.

In order to select actions appropriate to current needs, a subject must identify relationships between actions and events. Control over the environment is determined by the degree to which action consequences can be predicted, as described by action-outcome contingencies--i.e. performing an action should affect the probability of the outcome. We evaluated in a first experiment adaptation to contingency changes in rats with neurotoxic lesions of the medial prefrontal cortex. Results indicate that this brain region is not critical to adjust instrumental responding to a negative contingency where the rats must refrain from pressing a lever, as this action prevents reward delivery. By contrast, this brain region is required to reduce responding in a non-contingent situation where the same number of rewards is freely delivered and actions do not affect the outcome any more. In a second experiment, we determined that this effect does not result from a different perception of temporal relationships between actions and outcomes since lesioned rats adapted normally to gradually increasing delays in reward delivery. These data indicate that the medial prefrontal cortex is not directly involved in evaluating the correlation between action--and reward--rates or in the perception of reward delays. The deficit in lesioned rats appears to consist of an abnormal response to the balance between contingent and non-contingent rewards. By highlighting the role of prefrontal regions in adapting to the causal status of actions, these data contribute to our understanding of the neural basis of choice tasks.