Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Adaptive Responding to Stimulus-Outcome Associations Requires Noradrenergic Transmission in the Medial Prefrontal Cortex.

A dynamic environment, such as the one we inhabit, requires organisms to continuously update their knowledge of the setting. While the prefrontal cortex is recognized for its pivotal role in regulating such adaptive behavior, the specific contribution of each prefrontal area remains elusive. In the current work, we investigated the direct involvement of two major prefrontal subregions, the medial prefrontal cortex (mPFC, A32D + A32V) and the orbitofrontal cortex (OFC, VO + LO), in updating pavlovian stimulus-outcome (S-O) associations following contingency degradation in male rats. Specifically, animals had to learn that a particular cue, previously fully predicting the delivery of a specific reward, was no longer a reliable predictor. First, we found that chemogenetic inhibition of mPFC, but not of OFC, neurons altered the rats' ability to adaptively respond to degraded and non-degraded cues. Next, given the growing evidence pointing at noradrenaline (NA) as a main neuromodulator of adaptive behavior, we decided to investigate the possible involvement of NA projections to the two subregions in this higher-order cognitive process. Employing a pair of novel retrograde vectors, we traced NA projections from the locus ceruleus (LC) to both structures and observed an equivalent yet relatively segregated amount of inputs. Then, we showed that chemogenetic inhibition of NA projections to the mPFC, but not to the OFC, also impaired the rats' ability to adaptively respond to the degradation procedure. Altogether, our findings provide important evidence of functional parcellation within the prefrontal cortex and point at mPFC NA as key for updating pavlovian S-O associations.

Obesogenic diet induces circuit-specific memory deficits in mice.

Obesity is associated with neurocognitive dysfunction, including memory deficits. This is particularly worrisome when obesity occurs during adolescence, a maturational period for brain structures critical for cognition. In rodent models, we recently reported that memory impairments induced by obesogenic high-fat diet (HFD) intake during the periadolescent period can be reversed by chemogenetic manipulation of the ventral hippocampus (vHPC). Here, we used an intersectional viral approach in HFD-fed male mice to chemogenetically inactivate specific vHPC efferent pathways to nucleus accumbens (NAc) or medial prefrontal cortex (mPFC) during memory tasks. We first demonstrated that HFD enhanced activation of both pathways after training and that our chemogenetic approach was effective in normalizing this activation. Inactivation of the vHPC-NAc pathway rescued HFD-induced deficits in recognition but not location memory. Conversely, inactivation of the vHPC-mPFC pathway restored location but not recognition memory impairments produced by HFD. Either pathway manipulation did not affect exploration or anxiety-like behaviour. These findings suggest that HFD intake throughout adolescence impairs different types of memory through overactivation of specific hippocampal efferent pathways and that targeting these overactive pathways has therapeutic potential.

Inhibition of noradrenergic signalling in rodent orbitofrontal cortex impairs the updating of goal-directed actions.

In a constantly changing environment, organisms must track the current relationship between actions and their specific consequences and use this information to guide decision-making. Such goal-directed behaviour relies on circuits involving cortical and subcortical structures. Notably, a functional heterogeneity exists within the medial prefrontal, insular, and orbitofrontal cortices (OFC) in rodents. The role of the latter in goal-directed behaviour has been debated, but recent data indicate that the ventral and lateral subregions of the OFC are needed to integrate changes in the relationships between actions and their outcomes. Neuromodulatory agents are also crucial components of prefrontal functions and behavioural flexibility might depend upon the noradrenergic modulation of the prefrontal cortex. Therefore, we assessed whether noradrenergic innervation of the OFC plays a role in updating action-outcome relationships in male rats. We used an identity-based reversal task and found that depletion or chemogenetic silencing of noradrenergic inputs within the OFC rendered rats unable to associate new outcomes with previously acquired actions. Silencing of noradrenergic inputs in the prelimbic cortex or depletion of dopaminergic inputs in the OFC did not reproduce this deficit. Together, our results suggest that noradrenergic projections to the OFC are required to update goal-directed actions.

The mediodorsal thalamus supports adaptive responding based on stimulus-outcome associations.

The ability to engage into flexible behaviors is crucial in dynamic environments. We recently showed that in addition to the well described role of the orbitofrontal cortex (OFC), its thalamic input from the submedius thalamic nucleus (Sub) also contributes to adaptive responding during Pavlovian degradation. In the present study, we examined the role of the mediodorsal thalamus (MD) which is the other main thalamic input to the OFC. To this end, we assessed the effect of both pre- and post-training MD lesions in rats performing a Pavlovian contingency degradation task. Pre-training lesions mildly impeded the establishment of stimulus-outcome associations during the initial training of Pavlovian conditioning without interfering with Pavlovian degradation training when the sensory feedback provided by the outcome rewards were available to animals. However, we found that both pre- and post-training MD lesions produced a selective impairment during a test conducted under extinction conditions, during which only current mental representation could guide behavior. Altogether, these data suggest a role for the MD in the successful encoding and representation of Pavlovian associations.

Dopamine and noradrenaline modulation of goal-directed behavior in orbital and medial prefrontal cortex: Toward a division of labor?

The prefrontal cortex is considered to be at the core of goal-directed behaviors. Notably, the medial prefrontal cortex (mPFC) is known to play an important role in learning action-outcome (A-O) associations, as well as in detecting changes in this contingency. Previous studies have also highlighted a specific engagement of the dopaminergic pathway innervating the mPFC in adapting to changes in action causality. While previous research on goal-directed actions has primarily focused on the mPFC region, recent findings have revealed a distinct and specific role of the ventral and lateral orbitofrontal cortex (vlOFC). Indeed, vlOFC is not necessary to learn about A-O associations but appears specifically involved when outcome identity is unexpectedly changed. Unlike the mPFC, the vlOFC does not receive a strong dopaminergic innervation. However, it receives a dense noradrenergic innervation which might indicate a crucial role for this neuromodulator. In addition, several lines of evidence highlight a role for noradrenaline in adapting to changes in the environment. We, therefore, propose that the vlOFC's function in action control might be under the strong influence of the noradrenergic system. In the present article, we review anatomical and functional evidence consistent with this proposal and suggest a direction for future studies that aim to shed light on the orbitofrontal mechanisms for flexible action control. Specifically, we suggest that dopaminergic modulation in the mPFC and noradrenergic modulation in the vlOFC may underlie distinct processes related to updating one's actions. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Chemogenetic silencing of hippocampus and amygdala reveals a double dissociation in periadolescent obesogenic diet-induced memory alterations.

In addition to numerous metabolic comorbidities, obesity is associated with several adverse neurobiological outcomes, especially learning and memory alterations. Obesity prevalence is rising dramatically in youth and is persisting in adulthood. This is especially worrying since adolescence is a crucial period for the maturation of certain brain regions playing a central role in memory processes such as the hippocampus and the amygdala. We previously showed that periadolescent, but not adult, exposure to obesogenic high-fat diet (HFD) had opposite effects on hippocampus- and amygdala-dependent memory, impairing the former and enhancing the latter. However, the causal role of these two brain regions in periadolescent HFD-induced memory alterations remains unclear. Here, we first showed that periadolescent HFD induced long-term, but not short-term, object recognition memory deficits, specifically when rats were exposed to a novel context. Using chemogenetic approaches to inhibit targeted brain regions, we then demonstrated that recognition memory deficits are dependent on the activity of the ventral hippocampus, but not the basolateral amygdala. On the contrary, the HFD- induced enhancement of conditioned odor aversion specifically requires amygdala activity. Taken together, these findings suggest that HFD consumption throughout adolescence impairs long-term object recognition memory through alterations of ventral hippocampal activity during memory acquisition. Moreover, these results further highlight the bidirectional effects of adolescent HFD on hippocampal and amygdala functions.

Targeting Catecholaminergic Systems in Transgenic Rats With a CAV-2 Vector Harboring a Cre-Dependent DREADD Cassette.

Techniques that allow the manipulation of specific neural circuits have greatly increased in the past few years. DREADDs (Designer receptors exclusively activated by designer drugs) provide an elegant way to manipulate individual brain structures and/or neural circuits, including neuromodulatory pathways. Considerable efforts have been made to increase cell-type specificity of DREADD expression while decreasing possible limitations due to multiple viral vectors injections. In line with this, a retrograde canine adenovirus type 2 (CAV-2) vector carrying a Cre-dependent DREADD cassette has been recently developed. In combination with Cre-driver transgenic animals, the vector allows one to target neuromodulatory pathways with cell-type specificity. In the present study, we specifically targeted catecholaminergic pathways by injecting the vector in knock-in rat line containing Cre recombinase cassette under the control of the tyrosine hydroxylase promoter. We assessed the efficacy of infection of the nigrostriatal pathway and the catecholaminergic pathways ascending to the orbitofrontal cortex (OFC) and found cell-type-specific DREADD expression.

Dopamine blockade impairs the exploration-exploitation trade-off in rats.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

A thalamocortical circuit for updating action-outcome associations.

The ability to flexibly use knowledge is one cardinal feature of goal-directed behaviors. We recently showed that thalamocortical and corticothalamic pathways connecting the medial prefrontal cortex and the mediodorsal thalamus (MD) contribute to adaptive decision-making (Alcaraz et al., 2018). In this study, we examined the impact of disconnecting the MD from its other main cortical target, the orbitofrontal cortex (OFC) in a task assessing outcome devaluation after initial instrumental training and after reversal of action-outcome contingencies. Crossed MD and OFC lesions did not impair instrumental performance. Using the same approach, we found however that disconnecting the OFC from its other main thalamic afferent, the submedius nucleus, produced a specific impairment in adaptive responding following action-outcome reversal. Altogether, this suggests that multiple thalamocortical circuits may act synergistically to achieve behaviorally relevant functions.

Distinct regional patterns in noradrenergic innervation of the rat prefrontal cortex.

The anatomy and functions of the rodent prefrontal cortex (PFC) have been extensively studied. It is now clear that the PFC is at the core of various executive functions and that these functions depend on monoaminergic neuromodulation. The PFC receives extensive projections from monoaminergic nuclei and, in particular, from the locus cœruleus (LC) which is the major source of noradrenaline (NA) in the cortex. Projections of this nucleus have long been considered to act diffusely and uniformly throughout the entire brain. However, recent studies have revealed a separate innervation of prefrontal sub-regions by non-collateralizing LC neurons, suggesting a specific modulation of their functions. Following this idea, we aimed at describing more precisely the pattern of noradrenergic innervation into different orbital (OFC) and medial (mPFC) sub-regions of the PFC. We focused on the lateral (LO), ventral (VO) and medial (MO) portions of the OFC, and on areas 32d (A32d), 32v (A32v) and 25 (A25) in the mPFC. Using Dopamine-ß-Hydroxylase as a specific noradrenergic marker, we performed an automatic quantification of noradrenergic fibers and varicosities in each of these sub-regions. The results indicate that noradrenergic innervation is heterogeneous in some prefrontal sub-regions along the rostro-caudal axis. Functional dissociations have been recently reported in prefrontal sub-regions along the rostro-caudal direction. Our findings add neuroanatomical support to this emergent idea.

Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release.

Recent computational models of sign tracking (ST) and goal tracking (GT) have accounted for observations that dopamine (DA) is not necessary for all forms of learning and have provided a set of predictions to further their validity. Among these, a central prediction is that manipulating the intertrial interval (ITI) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses. Here, we tested these predictions and found that lengthening the ITI increased ST, i.e., behavioral engagement with conditioned stimuli (CS) and cue-induced phasic DA release. Importantly, DA release was also present at the time of reward delivery, even after learning, and DA release was correlated with time spent in the food cup during the ITI. During conditioning with shorter ITIs, GT was prominent (i.e., engagement with food cup), and DA release responded to the CS while being absent at the time of reward delivery after learning. Hence, shorter ITIs restored the classical DA reward prediction error (RPE) pattern. These results validate the computational hypotheses, opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling.

Thalamocortical and corticothalamic pathways differentially contribute to goal-directed behaviors in the rat.

Highly distributed neural circuits are thought to support adaptive decision-making in volatile and complex environments. Notably, the functional interactions between prefrontal and reciprocally connected thalamic nuclei areas may be important when choices are guided by current goal value or action-outcome contingency. We examined the functional involvement of selected thalamocortical and corticothalamic pathways connecting the dorsomedial prefrontal cortex (dmPFC) and the mediodorsal thalamus (MD) in the behaving rat. Using a chemogenetic approach to inhibit projection-defined dmPFC and MD neurons during an instrumental learning task, we show that thalamocortical and corticothalamic pathways differentially support goal attributes. Both pathways participate in adaptation to the current goal value, but only thalamocortical neurons are required to integrate current causal relationships. These data indicate that antiparallel flow of information within thalamocortical circuits may convey qualitatively distinct aspects of adaptive decision-making and highlight the importance of the direction of information flow within neural circuits.

Protracted motivational dopamine-related deficits following adolescence sugar overconsumption.

Adolescence represents a critical period characterized by major neurobiological changes. Chronic stimulation of the reward system during adolescence might constitute an important factor of vulnerability to pathological development. Increasing evidences suggest that adolescent overconsumption of sweet palatable foods impact reward-based processes. However, the neurobiological bases of these deficits remain poorly understood. Previous studies have demonstrated motivational deficits for palatable foods after sweet diet exposure during adolescence that might involve the dopamine (DA) system, a central actor in incentive processes. In the present study, the impact of adolescent sugar overconsumption on the sensitivity of the DA system was tested using pharmacological (Experiment 1) and receptor expression approaches (Experiment 2). Adolescent rats received free and continuous access to 5% sucrose solution from post-natal day 30-46. At adulthood, the functionality of the DA system in motivational processes was tested using systemic injections of specific DA receptors D1R or D2R agonists and antagonists during a motivation-dependent progressive ratio task (Experiment 1). Sucrose-exposed rats showed a lower motivation for saccharin and a decreased sensitivity to the effects of both D1R and D2R stimulation and blockade. In Experiment 2, Sucrose-exposed animals presented a lower expression of both D1R and D2R in the nucleus accumbens, a central brain region for incentive processes, but not in dorsal striatum or prefrontal cortex. These findings highlight the impact of sucrose overconsumption during adolescence on DA system that may support deficits in reward-related disorders.

Inter-individual differences in decision-making, flexible and goal-directed behaviors: novel insights within the prefronto-striatal networks.

Inflexible behavior is a hallmark of several decision-making-related disorders such as ADHD and addiction. As in humans, a subset of healthy rats makes poor decisions and prefers immediate larger rewards despite suffering large losses in a rat gambling task (RGT). They also display a combination of traits reminiscent of addiction, notably inflexible behavior and perseverative responses. The goal of the present work was twofold: (1) to elucidate if behavioral inflexibility of poor decision-makers could be related to a lower quality of goal-directed behavior (action-outcome associations); (2) to uncover the neural basis of inter-individual differences in goal-directed behavior. We specifically assessed inter-individual differences in decision-making in the RGT, flexibility in the RGT-reversed version and goal-directed behavior in a contingency degradation test, i.e., response adaptation when dissociating reward delivery from the animal's action. The contributions of the medial prefrontal cortex and the dorsal striatum to action-outcome associations were assessed using Zif268 immunodetection. Inflexible behavior was related to a lower sensitivity to contingency degradation in all poor decision-makers and only in a few good decision-makers. This poorer sensitivity was associated with a lower immunoreactivity in prelimbic and infralimbic cortices and a higher one in the dorsomedial and dorsolateral striatum. These findings suggest that an imbalanced prefronto-striatal activity could underlie inaccurate goal representation in changing environments and may promote maladaptive habit formation among poor decision-makers. These data strengthen our previous work identifying biomarkers of vulnerability to develop psychiatric disorders and demonstrate the relevance of inter-individual differences to model maladaptive behaviors.

Insular and Ventrolateral Orbitofrontal Cortices Differentially Contribute to Goal-Directed Behavior in Rodents.

The medial prefrontal cortex (mPFC) has long been considered a critical site in action control. However, recent evidence indicates that the contribution of cortical areas to goal-directed behavior likely extends beyond mPFC. Here, we examine the function of both insular (IC) and ventrolateral orbitofrontal (vlOFC) cortices in action-dependent learning. We used chemogenetics to study the consequences of IC or vlOFC inhibition on acquisition and performance of instrumental actions using the outcome devaluation task. Rats first learned to associate actions with desirable outcomes. Then, one of these outcomes was devalued and we assessed the rats' choice between the 2 actions. Typically, rats will bias their selection towards the action that delivers the still valued outcome. We show that chemogenetic-induced inhibition of IC during choice abolishes goal-directed control whereas inhibition during instrumental acquisition is without effect. IC is therefore necessary for action selection based on current outcome value. By contrast, vlOFC inhibition during acquisition or the choice test impaired goal-directed behavior but only following a shift in the instrumental contingencies. Our results provide clear evidence that vlOFC plays a critical role in action-dependent learning, which challenges the popular idea that this region of OFC is exclusively involved in stimulus-dependent behaviors.

Personality and predisposition to form habit behaviours during instrumental conditioning in horses (Equus caballus).

The relationship between personality and learning abilities has become a growing field of interest. Studies have mainly focused on the relationship with performance, such as the speed of acquisition. In this study, we hypothesised that personality could in part also be related to a certain predisposition of an individual to switch more easily from a goal-directed process to a habit process during learning. To identify these processes, we conducted a contingency degradation protocol. This study investigated 1/ whether in general horses are able to adjust their response according to the contingency between their action and the reward, 2/ whether there are any relationships between certain personality profiles and a predisposition to switch more rapidly to habitual processes, and 3/ whether emotional states experienced during the learning procedure play a role in this switching. Personality tests were conducted on 29 horses, followed by a degradation contingency protocol. Overall, results show that horses were sensitive to contingency degradation between their action and the reward. Nevertheless, there was inter-individual variability: the horses presenting high fearfulness, and to a lesser extent low sensory sensitivity and low gregariousness were less sensitive to the degradation, demonstrating that they were more likely to switch to a habitual process. Contrary to our expectations, the emotional state experienced during the procedure did not seem to explain this switching. We conclude that personality is not only related to learning performance, but also in part to the process involved during learning, independently of the emotion experienced during the process. This study provides new theoretical knowledge on cognitive skills in ungulates.

The effect of high-fat diet consumption on appetitive instrumental behavior in rats.

Evidence now indicates that the chronic consumption of high-calorie foods, such as a high-fat diet (HFD), is associated with impaired control over food-seeking, yet the extent of this alteration is not fully understood. Using different reinforcement schedules, we evaluated whether HFD intake from weaning to adulthood modifies instrumental responding and induces a shift from goal-directed actions to habitual responding. We first observed reduced instrumental performance and motivation for a food reward in HFD-fed rats trained under schedules of reinforcement that facilitate habitual responding [Random Interval (RI)]. However, this deficit was alleviated if rats trained under RI were subsequently trained with reinforcement schedules that promote goal-directed strategies [Random Ratio (RR)]. Using an outcome devaluation procedure, we then demonstrated that consumption of a HFD promoted habitual behavior in rats trained under RI but not RR schedules. Finally, extended HFD exposure did not interfere with the ability of RR training to overcome impaired RI instrumental performance and to favor goal-directed behavior. These results indicate that chronic consumption of a HFD changes the co-ordination of goal-directed actions and habits and that alteration of food-seeking may be reversed under particular behavioral conditions.

Switching Adolescent High-Fat Diet to Adult Control Diet Restores Neurocognitive Alterations.

In addition to metabolic and cardiovascular disorders, obesity is associated with adverse cognitive and emotional outcomes. Its growing prevalence in adolescents is particularly alarming since this is a period of ongoing maturation for brain structures (including the hippocampus and amygdala) and for the hypothalamic-pituitary-adrenal (HPA) stress axis, which is required for cognitive and emotional processing. We recently demonstrated that adolescent, but not adult, high-fat diet (HF) exposure leads to impaired hippocampal function and enhanced amygdala function through HPA axis alteration (Boitard et al., 2012, 2014, 2015). Here, we assessed whether the effects of adolescent HF consumption on brain function are permanent or reversible. After adolescent exposure to HF, switching to a standard control diet restored levels of hippocampal neurogenesis and normalized enhanced HPA axis reactivity, amygdala activity and avoidance memory. Therefore, while the adolescent period is highly vulnerable to the deleterious effects of diet-induced obesity, adult exposure to a standard diet appears sufficient to reverse alterations of brain function.