BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING.

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Infralimbic cortex plays a similar role in the punishment and extinction of instrumental behavior.

Learning to stop responding is a fundamental process in instrumental learning. Animals may learn to stop responding under a variety of conditions that include punishment-where the response earns an aversive stimulus in addition to a reinforcer-and extinction-where a reinforced response now earns nothing at all. Recent research suggests that punishment and extinction may be related manifestations of a common retroactive interference process. In both paradigms, animals learn to stop performing a specific response in a specific context, suggesting direct inhibition of the response by the context. This process may depend on the infralimbic cortex (IL), which has been implicated in a variety of interference-based learning paradigms including extinction and habit learning. Despite the behavioral parallels between extinction and punishment, a corresponding role for IL in punishment has not been identified. Here we report that, in a simple arrangement where either punishment or extinction was conducted in a context that differed from the context in which the behavior was first acquired, IL inactivation reduced response suppression in the inhibitory context, but not responding when it "renewed" in the original context. In a more complex arrangement in which two responses were first trained in different contexts and then extinguished or punished in the opposite one, IL inactivation had no effect. The results advance our understanding of the effects of IL in retroactive interference and the behavioral mechanisms that can produce suppression of a response.

Prelimbic cortex inactivation prevents ABA renewal based on satiety state.

We have previously shown that the rat prelimbic cortex (PL) is necessary for contexts to promote the performance of instrumental behaviors that have been learned in them, whether the context is physical (operant chamber) or behavioral (recent performance of a behavior that has historically preceded the target in a behavior chain). In the present experiment, we investigated the role of the PL in satiety level as an interoceptive acquisition context. Rats were trained to lever-press for sweet/fat pellets while sated (22 hrs continuous food access) followed by the extinction of the response while hungry (22 hrs food deprived). Pharmacological inactivation of the PL (with baclofen/muscimol infusion) attenuated renewal of the response that occurred upon a return to the sated context. In contrast, animals that received a vehicle (saline) infusion showed renewal of the previously extinguished response. These results support the hypothesis that the PL monitors the relevant contextual elements (physical, behavioral, or satiety state) associated with reinforcement of a response and promotes the subsequent performance of that response in their presence.

Early-life seizures alter habit behavior formation and fronto-striatal circuit dynamics.

Obsessive compulsive disorder (OCD) can occur comorbidly with epilepsy; both are complex, disruptive disorders that lower quality of life. Both OCD and epilepsy are disorders of hyperexcitable circuits, but it is unclear whether common circuit pathology may underlie the co-occurrence of these two neuropsychiatric disorders. Here, we induced early-life seizures (ELS) in rats to examine habit formation as a model for compulsive behaviors. Compulsive, repetitive behaviors in OCD utilize the same circuitry as habit formation. We hypothesized that rats with ELS could be more susceptible to habit formation than littermate controls, and that altered behavior would correspond to altered signaling in fronto-striatal circuits that underlie decision-making and action initiation. Here, we show instead that rats with ELS were significantly less likely to form habit behaviors compared with control rats. This behavioral difference corresponded with significant alterations to temporal coordination within and between brain regions that underpin the action to habit transition: 1) phase coherence between the lateral orbitofrontal cortex and dorsomedial striatum (DMS) and 2) theta-gamma coupling within DMS. Finally, we used cortical electrical stimulation as a model of transcranial magnetic stimulation (TMS) to show that temporal coordination of fronto-striatal circuits in control and ELS rats are differentially susceptible to potentiating and suppressive stimulation, suggesting that altered underlying circuit physiology may lead to altered response to therapeutic interventions such as TMS.

A comparison of renewal, spontaneous recovery, and reacquisition after punishment and extinction.

Punishment and extinction are both effective methods of reducing instrumental responding and may involve similar learning mechanisms. To characterize the similarities and differences between them, we examined three well-established recovery or "relapse" effects -renewal, spontaneous recovery, and reacquisition - following either punishment or extinction of an instrumental response. In Experiment 1a, both punished and extinguished responses renewed to similar degrees following a context change at test (ABA renewal). In Experiment 1b, responding spontaneously recovered to similar degrees following punishment or extinction. In Experiment 2, responding was rapidly reacquired when the response was reinforced again following extinction but not following punishment, as predicted by the idea that the reinforcer delivered in reacquisition is part of the context of punishment, but not extinction. The results collectively suggest that both punishment and extinction produce similar context-dependent retroactive interference effects. More broadly, they also suggest that punished and extinguished responses may be equally likely to return following a change of context despite the intuition that punishment might provide a more extreme and effective means of suppressing behavior. To our knowledge, this is the first direct behavioral comparison of response recovery after punishment and extinction within individual experiments.

Partial reinforcement effects on acquisition and extinction of a conditioned taste aversion.

Four experiments with rat subjects asked whether a partial reinforcement extinction effect (PREE) occurs in taste aversion learning. The question has received little attention in the literature, and to our knowledge no taste aversion experiment has previously demonstrated a PREE. In each of the present experiments, experimental groups received a taste mixed in drinking water for 20 min; such taste exposures were sometimes paired with a lithium chloride (LiCl) injection and sometimes not. Control groups received only taste-LiCl pairings. There was evidence that each reinforced and non-reinforced trial produced increments and decrements in aversion strength (respectively), and trials mattered more than accumulated time during the conditioned stimulus and during the background (as emphasized in time-accumulation models like those of Gallistel and Gibbon, Psychological Review, 107, 289-344, 2000, and Gibbon and Balsam, Autoshaping and conditioning theory, Academic Press, New York, pp. 219-235, 1981). In addition, a partial reinforcement extinction effect was observed when there was a relatively large number of conditioning trials. The results extend our understanding of extinction in taste aversion learning and provide more evidence that aversion learning might follow rules that are qualitatively similar to those of other forms of learning.

New functions of the rodent prelimbic and infralimbic cortex in instrumental behavior.

The prelimbic and infralimbic cortices of the rodent medial prefrontal cortex mediate the effects of context and goals on instrumental behavior. Recent work from our laboratory has expanded this understanding. Results have shown that the prelimbic cortex is important for the modulation of instrumental behavior by the context in which the behavior is learned (but not other contexts), with context potentially being broadly defined (to include at least previous behaviors). We have also shown that the infralimbic cortex is important in the expression of extensively-trained instrumental behavior, regardless of whether that behavior is expressed as a stimulus-response habit or a goal-directed action. Some of the most recent data suggest that infralimbic cortex may control the currently active behavioral state (e.g., habit vs. action or acquisition vs. extinction) when two states have been learned. We have also begun to examine prelimbic and infralimbic cortex function as key nodes of discrete circuits and have shown that prelimbic cortex projections to an anterior region of the dorsomedial striatum are important for expression of minimally-trained instrumental behavior. Overall, the use of an associative learning perspective on instrumental learning has allowed the research to provide new perspectives on how these two "cognitive" brain regions contribute to instrumental behavior.

Context, attention, and the switch between habit and goal-direction in behavior.

This article reviews recent findings from the author's laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) "in mind." Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

Some factors that restore goal-direction to a habitual behavior.

Recent findings from our laboratory suggest that an extensively-practiced instrumental behavior can appear to be a goal-directed action (rather than a habit) when a second behavior is added and reinforced during intermixed final sessions (Shipman et al., 2018). The present experiments were designed to explore and understand this finding. All used the taste aversion method of devaluing the reinforcer to distinguish between goal-directed actions and habits. Experiment 1 confirmed that reinforcing a second response in a separate context (but not mere exposure to that context) can return an extensively-trained habit to the status of goal-directed action. Experiment 2 showed that training of the second response needs to be intermixed with training of the first response to produce this effect; training the second response after the first-response training was complete preserved the first response as a habit. Experiment 3 demonstrated that reinforcing the second response with a different reinforcer breaks the habit status of the first response. Experiment 4 found that free reinforcers (that were not response-contingent) were sufficient to restore goal-directed performance. Together, the results suggest that unexpected reinforcer delivery can render a habitual response goal-directed again.

Unexpected food outcomes can return a habit to goal-directed action.

Three experiments examined the return of a habitual instrumental response to the status of goal-directed action. In all experiments, rats received extensive training in which lever pressing was reinforced with food pellets on a random-interval schedule of reinforcement. In Experiment 1, the extensively-trained response was not affected by conditioning a taste aversion to the reinforcer, and was therefore considered a habit. However, if the response had earned a new and unexpected food pellet during the final training session, the response was affected by taste aversion conditioning to the (first) reinforcer, and had thus been converted to a goal-directed action. In Experiment 3, 30 min of prefeeding with an irrelevant food pellet immediately before the test also converted a habit back to action, as judged by the taste-aversion devaluation method. That result was consistent with difficulty in finding evidence of habit with the sensory-specific satiety method after extensive instrumental training (Experiment 2). The results suggest that an instrumental behavior's status as a habit is not permanent, and that a habit can be returned to action status by associating it with a surprising reinforcer (Experiment 1) or by giving the animal an unexpected prefeeding immediately prior to the action/habit test (Experiment 3).

Inactivation of the prelimbic cortex attenuates operant responding in both physical and behavioral contexts.

The present experiments aimed to expand our understanding of the role of the prelimbic cortex (PL) in the contextual control of instrumental behavior. Research has previously shown that the PL is involved when the "physical context," or chamber in which an instrumental behavior is trained, facilitates performance of the instrumental response (Trask, Shipman, Green, & Bouton, 2017). Recently, evidence has suggested that when a sequence of two instrumental behaviors is required to earn a reinforcing outcome, the first response (rather than the physical chamber) can be the "behavioral context" for the second response (Thrailkill, Trott, Zerr, and Bouton, 2016). Could the PL also be involved in this kind of contextual control? Here rats first learned a heterogenous behavior chain in which the first response (i.e., pressing a lever or pulling a chain) was cued by a discriminative stimulus and led to a second stimulus which cued a second response (i.e., pulling a chain or pressing a lever); the second response led to a sucrose reward. When the first and second responses were tested in isolation in the training context, pharmacological inactivation of the PL resulted in a reduction of the first response, but not the second response. When the second response was performed in the "context" of the first response (i.e., as part of the behavior chain) however, PL inactivation reduced the second response. Overall, these results support the idea that the PL is important for mediating the effects of a training context on instrumental responding, whether the context is physical or behavioral.

Stress as a context: Stress causes relapse of inhibited food seeking if it has been associated with prior food seeking.

Three experiments with rats explored the hypothesis that inhibited food-seeking can be reinstated by stress if stress has been part of the context of earlier food-seeking. In all experiments, rats first learned to lever press for sucrose pellets and then had the response inhibited through extinction (where responding no longer yielded sucrose pellets). In a final test, inhibited responding was tested after exposure to a stressor or not. Previous research indicates that stress during testing does not normally reinstate extinguished food-seeking, although it reliably does so when animals are responding for drugs. In Experiment 1, stress caused a reinstatement of food seeking if and only if the rats had been exposed to stressors prior to sessions of lever press training. In Experiment 2, a new stressor that had not been associated with response acquisition also caused reinstatement if other stressors had been associated with response acquisition. Experiment 3 then established that stressors must be associated with the acquisition of lever pressing, rather than extinction, in order to allow a stressor to cause relapse of extinguished food seeking. The results support the view that stress can cause relapse of inhibited food seeking if it has been part of the context of original food seeking. The effect is therefore an example of the ABA renewal effect in which inhibited responding recovers after extinction when the response is returned to its training context. Implications for understanding relapse to overeating and other "addictive" behaviors are discussed.

Inactivation of the Prelimbic Cortex Attenuates Context-Dependent Operant Responding.

Operant responding in rats provides an analog to voluntary behavior in humans and is used to study maladaptive behaviors, such as overeating, drug taking, or relapse. In renewal paradigms, extinguished behavior recovers when tested outside the context where extinction was learned. Inactivation of the prelimbic (PL) region of the medial prefrontal cortex by baclofen/muscimol (B/M) during testing attenuates renewal when tested in the original acquisition context after extinction in another context (ABA renewal). Two experiments tested the hypothesis that the PL is important in context-dependent responding learned during conditioning. In the first, rats learned to lever-press for a sucrose-pellet reward. Following acquisition, animals were infused with either B/M or vehicle in the PL and tested in the acquisition context (A) and in a different context (B). All rats showed a decrement in responding when switched from Context A to Context B, but PL inactivation decreased responding only in Context A. Experiment 2a examined the effects of PL inactivation on ABC renewal in the same rats. Here, following reacquisition of the response, responding was extinguished in a new context (C). Following infusions of B/M or vehicle in the PL, responding was tested in Context C and another new context (D). The rats exhibited ACD renewal regardless of PL inactivation. Experiment 2b demonstrated that PL inactivation attenuated the ABA renewal effect in the same animals, replicating earlier results and demonstrating that cannulae were still functional. The results suggest that, rather than attenuating renewal generally, PL inactivation specifically affects ABA renewal by reducing responding in the conditioning context.SIGNIFICANCE STATEMENT Extinguished operant behavior can recover ("renew") when tested outside the extinction context. This suggests that behaviors, such as overeating or drug taking, might be especially prone to relapse following treatment. In rats, inactivation of the prelimbic cortex (PL) attenuates renewal. However, we report that PL inactivation after training attenuates responding in the context in which responding was acquired, but not in another one. A similar inactivation has no impact on renewal when testing occurs in a new, rather than the original, context following extinction. The PL thus has a more specific role in controlling contextually dependent operant behavior than has been previously reported.

Inactivation of prelimbic and infralimbic cortex respectively affects minimally-trained and extensively-trained goal-directed actions.

Several studies have examined a role for the prelimbic cortex (PL) and infralimbic cortex (IL) in free operant behavior. The general conclusion has been that PL controls goal-directed actions (instrumental behaviors that are sensitive to reinforcer devaluation) whereas IL controls habits (instrumental behaviors that are not sensitive to reinforcer devaluation). To further examine the involvement of these regions in the expression of instrumental behavior, we first implanted male rats with bilateral guide cannulae into their PL, then trained two responses to produce a sucrose pellet reinforcer, R1 and R2, each in a distinct context. R1 received extensive training and R2 received minimal training. Rats then received lithium chloride injections either paired or unpaired with sucrose pellets in both contexts until paired rats rejected all pellets. Following acquisition, in Experiment 1, rats received either an infusion of saline or baclofen/muscimol into the PL and were tested (in extinction) on both R1 and R2. In vehicle controls, both responses were goal-directed actions, as indicated by their sensitivity to reinforcer devaluation. PL inactivation decreased expression of the minimally-trained action without affecting expression of the extensively-trained action. Experiment 2 utilized the same experimental design but with IL inactivation at test. The extensively-trained response was again a goal-directed action. However, now expression of the extensively-trained goal-directed action was suppressed by IL inactivation. The overall pattern of results suggests that the PL is involved in expression of minimally trained goal-directed behavior while the IL is involved in expression of extensively trained goal-directed behavior. This implies that the PL does not control all types of actions and the IL can control some types of actions. These results expand upon the traditional view that the PL controls action while the IL controls habit.

Retrieval practice after multiple context changes, but not long retention intervals, reduces the impact of a final context change on instrumental behavior.

Recent evidence from this laboratory suggests that a context switch after operant learning consistently results in a decrement in responding. One way to reduce this decrement is to train the response in multiple contexts. One interpretation of this result, rooted in stimulus sampling theory, is that conditioning of a greater number of common stimulus elements arising from more contexts causes better generalization to new contexts. An alternative explanation is that each change of context causes more effortful retrieval, and practice involving effortful retrieval results in learning that is better able to transfer to new situations. The current experiments were designed to differentiate between these two explanations for the first time in an animal learning and memory task. Experiment 1 demonstrated that the detrimental impact of a context change on an instrumental nose-poking response can be reduced by training the response in multiple contexts. Experiment 2 then found that a training procedure which inserted extended retention intervals between successive training sessions did not reduce the detrimental impact of a final context change. This occurred even though the inserted retention intervals had a detrimental impact on responding (and, thus, presumably retrieval) similar to the effect that context switches had in Experiment 1. Together, the results suggest that effortful retrieval practice may not be sufficient to reduce the negative impact of a context change on instrumental behavior. A common elements explanation which supposes that physical and temporal contextual cues do not overlap may account for the findings more readily.

Hunger as a Context: Food Seeking That Is Inhibited During Hunger Can Renew in the Context of Satiety.

At the end of a diet, even a successful one, people often return to overeating. One potential reason is that the behavioral inhibition that people learn while dieting might not readily transfer outside the context in which it is learned: Basic research indicates that after a behavior is inhibited, a return to the conditioning context or simple removal from the treatment context can cause the behavior to return (i.e., to renew). Can states of hunger and satiety play the role of context? In two experiments, rats learned a food-seeking response that earned sucrose or sweet, fatty food pellets while they were satiated. Responding was then inhibited (i.e., extinguished) while the rats were hungry. On the rats' return to the satiated state, their food seeking was renewed. Additional results suggest that associations with hunger or satiety stimuli were learned more readily than associations with other potentially useful exteroceptive stimuli. The findings have implications for understanding the role of interoceptive contexts in controlling the inhibition of motivated behavior.

Extinction and the associative structure of heterogeneous instrumental chains.

Drug abuse, overeating, and smoking are all examples of instrumental behaviors that often involve chains or sequences of behavior. A behavior chain is minimally composed of a procurement response that is required in order for a subsequent consumption response to be reinforced. Despite the translational importance of behavior chains, few studies have attempted to understand what binds them together and takes them apart. This article surveys the development of the heterogeneous instrumental chain method and introduces recent findings that have used extinction to analyze the associative content of (what is learned in) the chain. Chained responses that are occasion-set by their own discriminative stimuli may be directly associated; extinction of the procurement response weakens its associated consumption response, and extinction of the consumption response weakens its associated procurement response. Extinction itself involves learning to inhibit the response. Extinguished chained responses are subject to renewal when they are tested either back in the acquisition context or in a new context. In addition, a consumption response that is extinguished outside its chain is renewed when returned to the context of the preceding response in the chain. Research on heterogeneous behavior chains can provide important insights into an important but often overlooked aspect of instrumental learning.

Medial prefrontal cortex involvement in the expression of extinction and ABA renewal of instrumental behavior for a food reinforcer.

Instrumental renewal, the return of extinguished instrumental responding after removal from the extinction context, is an important model of behavioral relapse that is poorly understood at the neural level. In two experiments, we examined the role of the dorsomedial prefrontal cortex (dmPFC) and the ventromedial prefrontal cortex (vmPFC) in extinction and ABA renewal of instrumental responding for a sucrose reinforcer. Previous work, exclusively using drug reinforcers, has suggested that the roles of the dmPFC and vmPFC in expression of extinction and ABA renewal may depend at least in part on the type of drug reinforcer used. The current experiments used a food reinforcer because the behavioral mechanisms underlying the extinction and renewal of instrumental responding are especially well worked out in this paradigm. After instrumental conditioning in context A and extinction in context B, we inactivated dmPFC, vmPFC, or a more ventral medial prefrontal cortex region by infusing baclofen/muscimol (B/M) just prior to testing in both contexts. In rats with inactivated dmPFC, ABA renewal was still present (i.e., responding increased when returned to context A); however responding was lower (less renewal) than controls. Inactivation of vmPFC increased responding in context B (the extinction context) and decreased responding in context A, indicating no renewal in these animals. There was no effect of B/M infusion on rats with cannula placements ventral to the vmPFC. Fluorophore-conjugated muscimol was infused in a subset of rats following test to visualize infusion spread. Imaging suggested that the infusion spread was minimal and mainly constrained to the targeted area. Together, these experiments suggest that there is a region of medial prefrontal cortex encompassing both dmPFC and vmPFC that is important for ABA renewal of extinguished instrumental responding for a food reinforcer. In addition, vmPFC, but not dmPFC, is important for expression of extinction of responding for a food reinforcer. The role of the medial prefrontal cortex in renewal in the original conditioning context may depend in part on control over excitatory context-response or context-(response-outcome) relations that might be learned in acquisition. The role of the vmPFC in expression of extinction may depend on its control over inhibitory context-response or context-(response-outcome) relations that are learned in extinction.

Role of the discriminative properties of the reinforcer in resurgence.

In three experiments with rat subjects, we examined the effects of the discriminative effects of reinforcers that were presented during or after operant extinction. Experiments 1 and 2 examined resurgence, in which an extinguished operant response (R1) recovers when a second behavior (R2) that has been reinforced to replace it is also placed in extinction. The results of Experiment 1 suggest that the amount of R1's resurgence is a decreasing linear function of the interreinforcement interval used during the reinforcement of R2. In Experiment 2, R1 was reinforced with one outcome (O1), and R2 was then reinforced with a second outcome (O2) while R1 was extinguished. In resurgence tests, response-independent (noncontingent) presentations of O2 prevented resurgence of R1, which otherwise occurred when testing was conducted with either no reinforcers or noncontingent presentations of O1. In Experiment 3, we then examined the effects of noncontingent O1 and O2 presentations after simple extinction in either the presence or the absence of noncontingent presentations of O2. Overall, the results are consistent with a role for the discriminative properties of the reinforcer in controlling operant behavior. In resurgence, the reinforcer used during response elimination provides a distinct context that controls the inhibition of R1. The results are less consistent with an alternative view emphasizing the disrupting effects of alternative reinforcement.

Discriminative properties of the reinforcer can be used to attenuate the renewal of extinguished operant behavior.

Previous research on the resurgence effect has suggested that reinforcers that are presented during the extinction of an operant behavior can control inhibition of the response. To further test this hypothesis, in three experiments with rat subjects we examined the effectiveness of using reinforcers that were presented during extinction as a means of attenuating or inhibiting the operant renewal effect. In Experiment 1, lever pressing was reinforced in Context A, extinguished in Context B, and then tested in Context A. Renewal of responding that occurred during the final test was attenuated when a distinct reinforcer that had been presented independent of responding during extinction was also presented during the renewal test. Experiment 2 established that this effect depended on the reinforcer being featured as a part of extinction (and thus associated with response inhibition). Experiment 3 then showed that the reinforcers presented during extinction suppressed performance in both the extinction and renewal contexts; the effects of the physical and reinforcer contexts were additive. Together, the results further suggest that reinforcers associated with response inhibition can serve a discriminative role in suppressing behavior and may be an effective stimulus that can attenuate operant relapse.