Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement.

Reinforcement learning inspires much theorizing in neuroscience, cognitive science, machine learning, and AI. A central question concerns the conditions that produce the perception of a contingency between an action and reinforcement-the assignment-of-credit problem. Contemporary models of associative and reinforcement learning do not leverage the temporal metrics (measured intervals). Our information-theoretic approach formalizes contingency by time-scale invariant temporal mutual information. It predicts that learning may proceed rapidly even with extremely long action-reinforcer delays. We show that rats can learn an action after a single reinforcement, even with a 16-min delay between the action and reinforcement (15-fold longer than any delay previously shown to support such learning). By leveraging metric temporal information, our solution obviates the need for windows of associability, exponentially decaying eligibility traces, microstimuli, or distributions over Bayesian belief states. Its three equations have no free parameters; they predict one-shot learning without iterative simulation.

Nucleus accumbens and dorsal medial striatal dopamine and neural activity are essential for action sequence performance.

Separable striatal circuits have unique functions in Pavlovian and instrumental behaviors but how these roles relate to performance of sequences of actions with and without associated cues are less clear. Here, we tested whether dopamine transmission and neural activity more generally in three striatal subdomains are necessary for performance of an action chain leading to reward delivery. Male and female Long-Evans rats were trained to press a series of three spatially distinct levers to receive reward. We assessed the contribution of neural activity or dopamine transmission within each striatal subdomain when progression through the action sequence was explicitly cued and in the absence of cues. Behavior in both task variations was substantially impacted following microinfusion of the dopamine antagonist, flupenthixol, into nucleus accumbens core (NAc) or dorsomedial striatum (DMS), with impairments in sequence timing and numbers of rewards earned after NAc flupenthixol. In contrast, after pharmacological inactivation to suppress overall activity, there was minimal impact on total rewards earned. Instead, inactivation of both NAc and DMS impaired sequence timing and led to sequence errors in the uncued, but not cued task. There was no impact of dopamine antagonism or reversible inactivation of dorsolateral striatum on either cued or uncued action sequence completion. These results highlight an essential contribution of NAc and DMS dopamine systems in motivational and performance aspects of chains of actions, whether cued or internally generated, as well as the impact of intact NAc and DMS function for correct sequence performance.

Adolescent social isolation shifts the balance of decision-making strategy from goal-directed action to habitual response in adulthood via suppressing the excitatory neurotransmission onto the direct pathway of the dorsomedial striatum.

Adverse experience, such as social isolation, during adolescence is one of the major causes of neuropsychiatric disorders that extend from adolescence into adulthood, such as substance addiction, obsessive-compulsive disorder, and eating disorders leading to obesity. A common behavioral feature of these neuropsychiatric disorders is a shift in the balance of decision-making strategy from goal-directed action to habitual response. This study has verified that adolescent social isolation directly shifts the balance of decision-making strategy from goal-directed action to habitual response, and that it cannot be reversed by simple regrouping. This study has further revealed that adolescent social isolation induces a suppression in the excitatory neurotransmission onto the direct-pathway medium spiny neurons of the dorsomedial striatum (DMS), and that chemogenetically compensating this suppression effect shifts the balance of decision-making strategy from habitual response back to goal-directed action. These findings suggest that the plasticity in the DMS causes the shift in the balance of decision-making strategy, which would potentially help to develop a general therapy to treat the various neuropsychiatric disorders caused by adolescent social isolation. Such a study is especially necessary under the circumstances that social distancing and lockdown have caused during times of world-wide, society-wide pandemic.

Unconscious knowledge of rewards guides instrumental behaviors via conscious judgments.

The demonstration that unconscious learning supports instrumental behaviors (i.e., choosing the stimuli that lead to rewards) is central for the tenet that unconscious cognition sustains human adaptation. Recent studies, using reliable subliminal conditioning paradigms and improved awareness measurements have found evidence against unconscious knowledge sustaining accurate instrumental responses. The present preregistered study proposes a paradigm, in which unconscious processing is stimulated not by subliminally exposing the predictive (conditioned) stimuli, but by employing predictive regularities that are complex and difficult to detect consciously. Participants (N = 211) were exposed to letter strings that, unknown to them, were built from two complex artificial grammars: a "rewarded" or a "non-rewarded" grammar. On each trial, participants memorized a string, and subsequently had to discriminate the memorized string from a distractor. Correct discriminations were rewarded only when the identified string followed the rewarded grammar, but not when it followed the non-rewarded grammar. In a subsequent test phase, participants were presented with new strings from the rewarded and from the unrewarded grammar. Their task was now to directly choose the strings from the rewarded grammar, in order to collect more rewards. A trial-by-trial awareness measure revealed that participants accurately choose novel strings from the rewarded grammar when they had no conscious knowledge of the grammar. The awareness measure also showed that participants were accurate only when the unconsciously learned grammar led to conscious judgments. The present study shows that unconscious knowledge can guide instrumental responses, but only to the extent it supports conscious judgments.

Instrumental learning and behavioral persistence in children with attention-deficit/hyperactivity-disorder: does reinforcement frequency matter?

BACKGROUND: Prominent theoretical accounts of attention-deficit/hyperactivity-disorder (ADHD) hypothesize that reinforcement learning deficits underlie symptoms of ADHD. The Dynamic Developmental Theory and the Dopamine Transfer Deficit hypothesis assume impairments in both the acquisition and extinction of behavior, especially when learning occurs under partial (non-continuous) reinforcement, and subsequently the Partial Reinforcement Extinction Effect (PREE). Few studies have evaluated instrumental learning in ADHD and the results are inconsistent. The current study investigates instrumental learning under partial and continuous reinforcement schedules and subsequent behavioral persistence when reinforcement is withheld (extinction) in children with and without ADHD. METHODS: Large well-defined samples of children with ADHD (n = 93) and typically developing (TD) children (n = 73) completed a simple instrumental learning task. The children completed acquisition under continuous (100%) or partial (20%) reinforcement, followed by a 4-min extinction phase. Two-way (diagnosis by condition) ANOVAs evaluated responses needed to reach the learning criterion during acquisition, and target and total responses during extinction. RESULTS: Children with ADHD required more trials to reach criterion compared to TD children under both continuous and partial reinforcement. After partial reinforcement, children with ADHD executed fewer target responses during extinction than TD children. Children with ADHD executed more responses than TD children during extinction, irrespective of learning condition. CONCLUSIONS: The findings demonstrate general difficulties in instrumental learning in ADHD, that is, slower learning irrespective of reinforcement schedule. They also show faster extinction following learning under partial reinforcement in those with ADHD, that is, a diminished PREE. Children with ADHD executed more responses during extinction. Results are theoretically important, with clinical implications for understanding and managing learning difficulties in those with ADHD, as they suggest poorer reinforcement learning and lower behavioral persistence.

Learning about me and you: Only deterministic stimulus associations elicit self-prioritization.

Self-relevant material has been shown to be prioritized over stimuli relating to others (e.g., friend, stranger), generating benefits in attention, memory, and decision-making. What is not yet understood, however, is whether the conditions under which self-related knowledge is acquired impacts the emergence of self-bias. To address this matter, here we used an associative-learning paradigm in combination with a stimulus-classification task to explore the effects of different learning experiences (i.e., deterministic vs. probabilistic) on self-prioritization. The results revealed an effect of prior learning on task performance, with self-prioritization only emerging when participants acquired target-related associations (i.e., self vs. friend) under conditions of certainty (vs. uncertainty). A further computational (i.e., drift diffusion model) analysis indicated that differences in the efficiency of stimulus processing (i.e., rate of information uptake) underpinned this self-prioritization effect. The implications of these findings for accounts of self-function are considered.

Evaluating instrumental learning and striatal-cortical functional connectivity in adolescent alcohol and cannabis use.

Adolescence is a vulnerable time for the acquisition of substance use disorders, potentially relating to ongoing development of neural circuits supporting instrumental learning. Striatal-cortical circuits undergo dynamic changes during instrumental learning and are implicated in contemporary addiction theory. Human studies have not yet investigated these dynamic changes in relation to adolescent substance use. Here, functional magnetic resonance imaging was used while 135 adolescents without (AUD-CUDLow ) and with significant alcohol (AUDHigh ) or cannabis use disorder symptoms (CUDHigh ) performed an instrumental learning task. We assessed how cumulative experience with instrumental cues altered cue selection preferences and functional connectivity strength between reward-sensitive striatal and cortical regions. Adolescents in AUDHigh and CUDHigh groups were slower in learning to select optimal instrumental cues relative to AUD-CUDLow adolescents. The relatively fast learning observed for AUD-CUDLow adolescents coincided with stronger functional connectivity between striatal and frontoparietal regions during early relative to later periods of task experience, whereas the slower learning for the CUDHigh group coincided with the opposite pattern. The AUDHigh group not only exhibited slower learning but also produced more instrumental choice errors relative to AUD-CUDLow adolescents. For the AUDHigh group, Bayesian analyses evidenced moderate support for no experience-related changes in striatal-frontoparietal connectivity strength during the task. Findings suggest that adolescent cannabis use is related to slowed instrumental learning and delays in peak functional connectivity strength between the striatal-frontoparietal regions that support this learning, whereas adolescent alcohol use may be more closely linked to broader impairments in instrumental learning and a general depression of the neural circuits supporting it.

The impact of ADHD symptom severity on reinforcement and punishment learning among adults.

Introduction: Aberrations in feedback learning are hypothesised to contribute to the behavioural disruptions and impairment of attention-deficit/hyperactivity disorder (ADHD). However, few studies have evaluated the relation of reward/punishment feedback and ADHD symptom severity on learning. The current study evaluates the differential effects of reward and punishment feedback on learning among adults with elevated ADHD. Methods: One hundred five participants self-reported their level of current ADHD symptoms and completed an innovative instrumental learning task. Results: Consistent with predictions, participants with low self-reported ADHD symptom severity benefitted equally from reward and punishment feedback during the learning task, whereas participants with high self-reported symptom severity performed better (indexed by accuracy on learning task) from reward than punishment feedback trials. Conclusions: Overall, adults with high self-reported symptom severity of ADHD learned more from reward-based feedback, which provides critical implications for motivational theories about ADHD, as well as for treatment protocols. Future work should examine the translatability of results within a treatment setting.

Depression in the Direct Pathway of the Dorsomedial Striatum Permits the Formation of Habitual Action.

In order to achieve optimal outcomes in an ever-changing environment, humans and animals generally manage their action control via either goal-directed action or habitual action. These two action strategies are thought to be encoded in distinct parallel circuits in the dorsal striatum, specifically, the posterior dorsomedial striatum (DMS) and the dorsolateral striatum (DLS), respectively. The striatum is primarily composed of two subtypes of medium spiny neurons (MSNs): the direct-pathway striatonigral and the indirect-pathway striatopallidal MSNs. MSN-subtype-specific synaptic plasticity in the DMS and the DLS has been revealed to underlie goal-directed action and habitual action, respectively. However, whether any MSN-subtype-specific synaptic plasticity in the DMS is associated with habitual action, and if so, whether the synaptic plasticity affects the formation of habitual action, are not known. This study demonstrates that postsynaptic depression in the excitatory synapses of the direct-pathway striatonigral MSNs in the DMS is formed after habit learning. Moreover, chemogenetically rescuing this depression compromises the acquisition, but not the expression, of habitual action. These findings reveal that an MSN-subtype-specific synaptic plasticity in the DMS affects habitual action and suggest that plasticity in the DMS as well as in the DLS contributes to the formation of habitual action.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [11C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.

The effects of induced positive and negative affect on Pavlovian-instrumental interactions.

Across species, animals have an intrinsic drive to approach appetitive stimuli and to withdraw from aversive stimuli. In affective science, influential theories of emotion link positive affect with strengthened behavioural approach and negative affect with avoidance. Based on these theories, we predicted that individuals' positive and negative affect levels should particularly influence their behaviour when innate Pavlovian approach/avoidance tendencies conflict with learned instrumental behaviours. Here, across two experiments - exploratory Experiment 1 (N = 91) and a preregistered confirmatory Experiment 2 (N = 335) - we assessed how induced positive and negative affect influenced Pavlovian-instrumental interactions in a reward/punishment Go/No-Go task. Contrary to our hypotheses, we found no evidence for a main effect of positive/negative affect on either approach/avoidance behaviour or Pavlovian-instrumental interactions. However, we did find evidence that the effects of induced affect on behaviour were moderated by individual differences in self-reported behavioural inhibition and gender. Exploratory computational modelling analyses explained these demographic moderating effects as arising from positive correlations between demographic factors and individual differences in the strength of Pavlovian-instrumental interactions. These findings serve to sharpen our understanding of the effects of positive and negative affect on instrumental behaviour.

Putamen volume predicts real-time fMRI neurofeedback learning success across paradigms and neurofeedback target regions.

Real-time fMRI guided neurofeedback training has gained increasing interest as a noninvasive brain regulation technique with the potential to modulate functional brain alterations in therapeutic contexts. Individual variations in learning success and treatment response have been observed, yet the neural substrates underlying the learning of self-regulation remain unclear. Against this background, we explored potential brain structural predictors for learning success with pooled data from three real-time fMRI data sets. Our analysis revealed that gray matter volume of the right putamen could predict neurofeedback learning success across the three data sets (n = 66 in total). Importantly, the original studies employed different neurofeedback paradigms during which different brain regions were trained pointing to a general association with learning success independent of specific aspects of the experimental design. Given the role of the putamen in associative learning this finding may reflect an important role of instrumental learning processes and brain structural variations in associated brain regions for successful acquisition of fMRI neurofeedback-guided self-regulation.

The Role of the Striatum in Learning to Orthogonalize Action and Valence: A Combined PET and 7 T MRI Aging Study.

Pavlovian biases influence instrumental learning by coupling reward seeking with action invigoration and punishment avoidance with action suppression. Using a probabilistic go/no-go task designed to orthogonalize action (go/no-go) and valence (reward/punishment), recent studies have shown that the interaction between the two is dependent on the striatum and its key neuromodulator dopamine. Using this task, we sought to identify how structural and neuromodulatory age-related differences in the striatum may influence Pavlovian biases and instrumental learning in 25 young and 31 older adults. Computational modeling revealed a significant age-related reduction in reward and punishment sensitivity and marked (albeit not significant) reduction in learning rate and lapse rate (irreducible noise). Voxel-based morphometry analysis using 7 Tesla MRI images showed that individual differences in learning rate in older adults were related to the volume of the caudate nucleus. In contrast, dopamine synthesis capacity in the dorsal striatum, assessed using [18F]-DOPA positron emission tomography in 22 of these older adults, was not associated with learning performance and did not moderate the relationship between caudate volume and learning rate. This multiparametric approach suggests that age-related differences in striatal volume may influence learning proficiency in old age.

Striatopallidal Pathway Distinctly Modulates Goal-Directed Valuation and Acquisition of Instrumental Behavior via Striatopallidal Output Projections.

The striatopallidal pathway is specialized for control of motor and motivational behaviors, but its causal role in striatal control of instrumental learning remains undefined (partly due to the confounding motor effects). Here, we leveraged the transient and "time-locked" optogenetic manipulations with the reward delivery to minimize motor confounding effect, to better define the striatopallidal control of instrumental behaviors. Optogenetic (Arch) silencing of the striatopallidal pathway in the dorsomedial striatum (DMS) and dorsolateral striatum (DLS) promoted goal-directed and habitual behaviors, respectively, without affecting acquisition of instrumental behaviors, indicating striatopallidal pathway suppression of instrumental behaviors under physiological condition. Conversely, striatopallidal pathway activation mainly affected the acquisition of instrumental behaviors with the acquisition suppression achieved by either optogenetic (ChR2) or chemicogenetic (hM3q) activation, by strong (10 mW, but not weak 1 mW) optogenetic activation, by the time-locked (but not random) optogenetic activation with the reward and by the DMS (but not DLS) striatopallidal pathway. Lastly, striatopallidal pathway modulated instrumental behaviors through striatopallidal output projections into the external globus pallidus (GPe) since optogenetic activation of the striatopallidal pathway in the DMS and of the striatopallidal output projections in the GPe similarly suppressed goal-directed behavior. Thus, the striatopallidal pathway confers distinctive and inhibitory controls of animal's sensitivity to goal-directed valuation and acquisition of instrumental behaviors under normal and over-activation conditions, through the output projections into GPe.

Context, attention, and the switch between habit and goal-direction in behavior.

This article reviews recent findings from the author's laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) "in mind." Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

Characterization of striatum-mediated behavior and neurochemistry in the DJ-1 knock-out rat model of Parkinson's disease.

The recently developed DJ-1 knockout (KO) rat models the DJ-1 (or PARK7) loss-of-function mutation responsible for one form of early-onset familial Parkinson's disease (PD). Prior studies demonstrate that DJ-1 KO rats present progressive dopamine (DA) cell body degeneration in the substantia nigra pars compacta between 4 and 8 months of age. Furthermore, as some motor deficits emerge before the significant loss of DA cells, this mutation may yield a period of DA neuron dysfunction preceding cell death that may also contribute to cognitive impairments in early PD. However, cognitive functions subserved by corticostriatal circuitry, as well as additional alterations to the neurochemistry of monoamine systems, are largely uncharacterized in the DJ-1 KO rat. We therefore assessed a variety of striatally-mediated behavioral tasks, as well as the integrity of dopamine and serotonin systems, in male DJ-1 KO rats and wild-type (WT) controls at 4, 6, and 8 months of age. We demonstrate that DJ-1 KO rats exhibited motor impairments, but have intact goal-directed control over behavior in an appetitive instrumental learning task. Further, preprotachykinin mRNA expression, a post-synaptic indicator of DA signaling, was significantly decreased in 4-month DJ-1 KO rats, while DA transporter binding in the dorsal striatum did not differ between genotypes at any of the ages examined. Striatal tyrosine hydroxylase levels were significantly increased in 8-month DJ-1 KO rats and tended to be higher than WT at 4 and 6 months. Lastly, serotonin transporter binding was increased in the medial and orbitofrontal cortices of 4-month old DJ-1 KO rats. These results suggest that the nigrostriatal dopaminergic and prefrontal serotoninergic systems are altered early in the progression of DJ-1 KO pathology, despite no overt loss of the DA innervation of the striatum, and thus may be associated with early alterations in the functions of corticostriatal systems.

Early-life stress is associated with a preponderance of habitual responding in a novel instrumental avoidance learning paradigm.

There is substantial evidence linking early-life stress (ELS) to negative health outcomes in adulthood, including addiction. However, the neurocognitive and behavioral mechanisms through which ELS increases these risks remain unclear. To address this gap in knowledge, we developed a novel instrumental learning paradigm to explore the effects of ELS on the balance of habitual versus goal-directed learning. Habits efficiently reproduce repetitive behaviors but are inflexible whenreward contingencies related to those behaviors change. Persisting in performing a response after its outcome has been devalued is the hallmark of habitual behavior in instrumental learning. Participants with a history of higher ELS were significantly more likely to make habitual responses in this instrumental avoidance learning paradigm than individuals with a history of lower ELS. Logistic regression analysis showed that ELS is significantly related to habitual responding over and above the effects of retrospective socioeconomic status, trait and state anxiety, depression and recent levels of stress. Analysis of the differential impacts of the type of ELS suggested that these effects are largely driven by experiences of physical neglect.

Chemogenetic inhibition in the dorsal striatum reveals regional specificity of direct and indirect pathway control of action sequencing.

Animals engage in intricate action sequences that are constructed during instrumental learning. There is broad consensus that the basal ganglia play a crucial role in the formation and fluid performance of action sequences. To investigate the role of the basal ganglia direct and indirect pathways in action sequencing, we virally expressed Cre-dependent Gi-DREADDs in either the dorsomedial (DMS) or dorsolateral (DLS) striatum during and/or after action sequence learning in D1 and D2 Cre rats. Action sequence performance in D1 Cre rats was slowed down early in training when DREADDs were activated in the DMS, but sped up when activated in the DLS. Acquisition of the reinforced sequence was hindered when DREADDs were activated in the DLS of D2 Cre rats. Outcome devaluation tests conducted after training revealed that the goal-directed control of action sequence rates was immune to chemogenetic inhibition-rats suppressed the rate of sequence performance when rewards were devalued. Sequence initiation latencies were generally sensitive to outcome devaluation, except in the case where DREADD activation was removed in D2 Cre rats that previously experienced DREADD activation in the DMS during training. Sequence completion latencies were generally not sensitive to outcome devaluation, except in the case where D1 Cre rats experienced DREADD activation in the DMS during training and test. Collectively, these results suggest that the indirect pathway originating from the DLS is part of a circuit involved in the effective reinforcement of action sequences, while the direct and indirect pathways originating from the DMS contribute to the goal-directed control of sequence completion and initiation, respectively.

Reduction of conditioned avoidance via contingency reversal.

There is an increased interest in how excessive avoidance can diminish. Avoidance reduction is typically tested by using Extinction with Response Prevention (ExRP) protocols, where feared stimuli are presented without any aversive outcome while avoidance is prevented. These effects, however, often do not persist. Here, we tested whether pairing an avoidance response with the presence of an aversive event would reduce avoidance more than ExRP. Participants (N = 58) first saw a picture of a square (A) being paired with a shock whereas another picture of a square (B) not being paired with a shock. Then, they learned to press a button during the presentation of A to avoid the shock. Afterwards, the ExRP group saw unreinforced presentations of A and B without being able to press the avoidance button, whereas the Contingency Reversal group (ConR) received a shock whenever they pressed the button in presence of A. In the test phase, participants saw unreinforced presentations of A and B. Results showed that after successful acquisition of fear and avoidance, in the test phase the ConR group avoided A less often than did the ExRP group. Research on contingency reversal could prove helpful for developing avoidance reduction protocols.

Stress-induced impairment in goal-directed instrumental behaviour is moderated by baseline working memory.

Acute stress has been found to impair goal-directed instrumental behaviour, a cognitively flexible behaviour that requires cognitive control. The current study aimed to investigate the role of individual differences in baseline and stress-induced changes in working memory (WM) on the shift to less goal-directed responding under stress. To this end, 112 healthy participants performed an instrumental learning task. In phase 1, participants learned instrumental actions that were associated with two different food rewards. In phase 2, one of these food rewards was devalued by eating until satiety. Before the extinction test in phase 3, participants were subjected to the Maastricht Acute Stress Test or a no-stress control procedure. Results showed that the effect of stress on instrumental behaviour is modulated by baseline, but not stress-induced changes in WM capacity. Specifically, only at low baseline WM capacity did stress induce a shift to less goal-directed behaviour. These findings highlight that our cognitive resources are limited and for those who already have limited resources at baseline taking into account motivational value is impaired under stress.