Stimulus preexposure speeds or slows subsequent acquisition of associative learning depending on learning test procedures and response measure.

Prior exposure to a conditioned stimulus (CS) typically results in latent inhibition-slower acquisition of associative learning about that stimulus in subsequent training. Here, we found that CS preexposure had different effects on the appetitive conditioning of rats with a sucrose unconditioned stimulus (US) depending on training test procedures, the similarity of preexposure and training procedures, and the choice of response measure. Preexposure to a visual or an auditory stimulus produced facilitation of acquisition of food-cup-directed responding when both of those cues were (separately) paired with sucrose delivery in the training test (Experiments 1 and 3). By contrast, the same preexposure procedure resulted in latent inhibition of food-cup learning if the second stimulus in the test phase was of the same modality as the preexposed stimulus (Experiment 2). In Experiment 3, latent inhibition was enhanced if both phases included a single CS or both phases included both auditory and visual CSs, compared to treatments in which only one CS was presented in one phase but two CSs were presented in the other phase. In Experiment 4, preexposure of an auditory cue slowed subsequent learning about it if the context was salient but enhanced learning if the context was of weaker salience. Finally, a measure of general activity revealed latent inhibition after preexposure in all conditions in all 4 experiments. We discuss the results within several classes of latent inhibition theories, none of which provides a comprehensive account.

Secondary visual cortex is critical to the expression of surprise-induced enhancements in cue associability in rats.

Considerable evidence indicates that reinforcement prediction error, the difference between the obtained and expected reinforcer values, modulates attention to potential cues for reinforcement. The surprising delivery or omission of a reinforcer enhances the associability of the stimuli that were present when the error was induced, so that they more readily enter into new associations in the future. Previous research from our laboratory identified brain circuit elements critical to the enhancement of stimulus associability by omission of an expected event and to the subsequent expression of that altered associability in more rapid learning. A key finding was that the rat posterior parietal cortex was essential during the encoding, consolidation and retrieval of associability memories that were altered by the surprising omission of an expected event in a serial prediction task. Here, we found that the function of adjacent secondary visual cortex was critical only to the expression of altered cue associability in that same task. This specialization of function is discussed in the context of broader cortical and subcortical networks for modulation of attention in associative learning, as well as recent anatomical investigations that suggest that the rodent posterior parietal cortex overlaps with and may subsume secondary visual cortex.

Consolidation of altered associability information by amygdala central nucleus.

The surprising omission of a reinforcer can enhance the associability of the stimuli that were present when the reward prediction error was induced, so that they more readily enter into new associations in the future. Previous research from this laboratory identified brain circuit elements critical to the enhancement of stimulus associability by the omission of an expected event and to the subsequent expression of that altered associability in more rapid learning. These elements include the amygdala, the midbrain substantia nigra, the basal forebrain substantia innominata, the dorsolateral striatum, the secondary visual cortex, and the posterior parietal cortex. Here, we found that consolidation of a surprise-enhanced associability memory in a serial prediction task depends on processing in the amygdala central nucleus (CeA) after completion of sessions that included the surprising omission of an expected event. Post-surprise infusions of anisomycin, lidocaine, or muscimol prevented subsequent display of surprise-enhanced associability. Because previous studies indicated that CeA function is unnecessary for the expression of associability enhancements that were induced previously when CeA function was intact (Holland & Gallagher, 2006), we interpreted these results as indicating that post-surprise activity of CeA ("surprise replay") is necessary for the consolidation of altered associability memories elsewhere in the brain, such as the posterior parietal cortex (Schiffino et al., 2014a).

Mini-review: Prediction errors, attention and associative learning.

Most modern theories of associative learning emphasize a critical role for prediction error (PE, the difference between received and expected events). One class of theories, exemplified by the Rescorla-Wagner (1972) model, asserts that PE determines the effectiveness of the reinforcer or unconditioned stimulus (US): surprising reinforcers are more effective than expected ones. A second class, represented by the Pearce-Hall (1980) model, argues that PE determines the associability of conditioned stimuli (CSs), the rate at which they may enter into new learning: the surprising delivery or omission of a reinforcer enhances subsequent processing of the CSs that were present when PE was induced. In this mini-review we describe evidence, mostly from our laboratory, for PE-induced changes in the associability of both CSs and USs, and the brain systems involved in the coding, storage and retrieval of these altered associability values. This evidence favors a number of modifications to behavioral models of how PE influences event processing, and suggests the involvement of widespread brain systems in animals' responses to PE.

Dorsolateral striatum is critical for the expression of surprise-induced enhancements in cue associability.

The dorsolateral striatum (DLS) is frequently implicated in sensory-motor integration, including the performance of sensory orienting responses (ORs) and learned stimulus-response habits. Our laboratory previously identified a role for the DLS in rats' performance of conditioned ORs to Pavlovian cues for food delivery. Here, we considered whether DLS is also critical to another aspect of attention in associative learning, the surprise-induced enhancement of cue associability. A large behavioral literature shows that a cue present when an expected event is omitted enters into new associations more rapidly when that cue is subsequently paired with food. Research from our laboratory has shown that both cue associability enhancements and conditioned ORs depend on the function of a circuit that includes the amygdala central nucleus and the substantia nigra pars compacta. In three experiments, we explored the involvement of DLS in surprise-induced associability enhancements, using a three-stage serial prediction task that permitted separation of DLS function in registering surprise (prediction error) and enhancing cue associability, and in using that increased associability to learn more rapidly about that cue later. The results showed that DLS is critical to the expression, but not the establishment, of the enhanced cue associability normally produced by surprise in this task. They extend the role of DLS and the amygdalo-nigro-striatal circuit underlying learned orienting to more subtle aspects of attention in associative learning, but are consistent with the general notion that DLS is more important in the expression of previously acquired tendencies than in their acquisition.

Dorsolateral striatum implicated in the acquisition, but not expression, of immediate response learning in rodent submerged T-maze.

Animals can use multiple strategies when learning about, and navigating within, their environment. Typically, in the frequently-studied food-rewarded T-maze, rats initially adopt a flexible, hippocampal-dependent place strategy. However, as learning progresses, rats switch to an automatic, striatal-dependent response strategy (Packard & McGaugh, 1996). Interestingly, in a similar but aversively motivating water-submerged T-maze, rats exhibit the opposite behavioral pattern, initially adopting a response strategy but switching to a place strategy with extended training (Asem & Holland, 2013). Here, we examined the effects of transient lidocaine inactivation of the dorsolateral striatum (DLS) on rats' acquisition and expression of place and response strategies in the submerged T-maze. DLS inactivation prior to probe tests had no effect on rats' initial expression of a response strategy nor on their transition to the use of a place strategy with further training. Nevertheless, in a second experiment using the same rats, identical inactivation parameters significantly affected performance in an appetitively motivating positive control task, which required a response strategy. Furthermore, in a third experiment, DLS inactivation prior to early learning trials interfered with the acquisition of the response strategy in the submerged T-maze. These differences in DLS inactivation effects across appetitive and aversive tasks support the view that task motivation plays crucial roles in guiding learning, memory, and behavior. Additionally, differences in DLS inactivation effects between tests of acquisition and expression suggest that the DLS is required during early acquisition but not expression of the response learning strategy.

The basolateral amygdala is necessary for negative prediction errors to enhance cue salience, but not to produce conditioned inhibition.

Behavioral evidence shows that prediction errors (PEs) not only drive associative learning, but also enhance the salience of predictive cues, making them better able to capture attention when they are next encountered. Research from our laboratory suggests that this latter consequence of PEs depends on a neural circuit that includes the amygdala. Lesions of the basolateral complex of the amygdala (BLA), for instance, selectively disrupt enhancements in cue processing that are normally induced by positive PEs without compromising simple excitatory learning. This result is consistent with electrophysiological evidence showing that BLA neurons track positive PEs. Interestingly, the same neurons also seem to track negative PEs, suggesting the possibility that the BLA might also use these errors to drive enhancements in cue processing. Here, we examined the role of the BLA in the processing (Experiment 1) and utilization (Experiment 2) of negative PEs in increasing cue salience in an unblocking procedure. Using FOS expression as an index of neural activity, Experiment 1 confirmed that BLA neurons track negative PEs with reinforcement downshifts. This tracking was evident both when these errors were generated by decreasing the concentration of a sucrose reinforcer (which encourages the development of conditioned inhibition) and when they were generated by decreasing the number of sucrose reinforcers (which encourages excitatory learning - unblocking - and allows the detection of enhancements in cue processing). Experiment 2 demonstrated that BLA lesions abolished enhancements in cue processing while sparing inhibitory learning. These results suggest a general role of the BLA in utilizing PEs, whatever their sign, for boosting cue processing.

Posterior parietal cortex is critical for the encoding, consolidation, and retrieval of a memory that guides attention for learning.

Within most contemporary learning theories, reinforcement prediction error, the difference between the obtained and expected reinforcer value, critically influences associative learning. In some theories, this prediction error determines the momentary effectiveness of the reinforcer itself, such that the same physical event produces more learning when its presentation is surprising than when it is expected. In other theories, prediction error enhances attention to potential cues for that reinforcer by adjusting cue-specific associability parameters, biasing the processing of those stimuli so that they more readily enter into new associations in the future. A unique feature of these latter theories is that such alterations in stimulus associability must be represented in memory in an enduring fashion. Indeed, considerable data indicate that altered associability may be expressed days after its induction. Previous research from our laboratory identified brain circuit elements critical to the enhancement of stimulus associability by the omission of an expected event, and to the subsequent expression of that altered associability in more rapid learning. Here, for the first time, we identified a brain region, the posterior parietal cortex, as a potential site for a memorial representation of altered stimulus associability. In three experiments using rats and a serial prediction task, we found that intact posterior parietal cortex function was essential during the encoding, consolidation, and retrieval of an associability memory enhanced by surprising omissions. We discuss these new results in the context of our previous findings and additional plausible frontoparietal and subcortical networks.

Role of lateral hypothalamus in two aspects of attention in associative learning.

Orexin (hypocretin) and melanin-concentrating hormone (MCH) neurons are unique to the lateral hypothalamic (LH) region, but project throughout the brain. These cell groups have been implicated in a variety of functions, including reward learning, responses to stimulants, and the modulation of attention, arousal and the sleep/wakefulness cycle. Here, we examined roles for LH in two aspects of attention in associative learning shown previously to depend on intact function in major targets of orexin and MCH neurons. In experiments 1 and 2, unilateral orexin-saporin lesions of LH impaired the acquisition of conditioned orienting responses (ORs) and bilaterally suppressed FOS expression in the amygdala central nucleus (CeA) normally observed in response to food cues that provoke conditioned ORs. Those cues also induced greater FOS expression than control cues in LH orexin neurons, but not in MCH neurons. In experiment 3, unilateral orexin-saporin lesions of LH eliminated the cue associability enhancements normally produced by the surprising omission of an expected event. The magnitude of that impairment was positively correlated with the amount of LH damage and with the loss of orexin neurons in particular, but not with the loss of MCH neurons. We suggest that the effects of the LH orexin-saporin lesions were mediated by their effect on information processing in the CeA, known to be critical to both behavioral phenomena examined here. The results imply close relations between LH motivational amplification functions and attention, and may inform our understanding of disorders in which motivational and attentional impairments co-occur.

Stimuli associated with the cancellation of food and its cues enhance eating but display negative incentive value.

Initially neutral conditioned stimuli paired with food often acquire motivating properties, including serving as secondary reinforcers, enhancing instrumental responding in Pavlovian-instrumental transfer procedures, and potentiating food consumption under conditions of food satiation. Interestingly, cues associated with the cancellation of food and food cues may also potentiate food consumption (e.g., Galarce and Holland, 2009), despite their apparent negative correlations with food delivery. In three experiments with rats, we investigated conditions under which potentiation of feeding by such "interruption stimuIi" (ISs) develops, and some aspects of the content of that learning. Although in all three experiments ISs enhanced food consumption beyond control levels, they were found to act as conditioned inhibitors for anticipatory food cup entry (Experiment 1), to serve as conditioned punishers of instrumental responding (Experiment 2), and to suppress instrumental lever press responding in a Pavlovian instrumental transfer procedure (Experiment 3). Furthermore, when given concurrent choice between different foods, an IS enhanced consumption of the food whose interruption it had previously signaled, but when given a choice between performing two instrumental responses, the IS shifted rats' choice away from the response that had previously yielded the food whose interruption had been signaled by IS (Experiment 3). Thus, the effects of an IS on appetitive responses were opposite to its effects on consummatory responding. Implications for our understanding of learned incentive motivation and the control of overeating are discussed.

Blocking in autoshaped lever-pressing procedures with rats.

Rats will approach and contact a lever whose insertion into the chamber signals response-independent food delivery. This "autoshaping" or "sign-tracking" phenomenon has recently attracted considerable attention as a platform for studying individual differences in impulsivity, drug sensitization, and other traits associated with vulnerability to drug addiction. Here, we examined two basic stimulus selection phenomena-blocking and overshadowing-in the autoshaped lever pressing of rats. Blocking and overshadowing were decidedly asymmetrical. Previously reinforced lever-extension conditioned stimuli (CSs) completely blocked conditioning to auditory cues (Exps. 1 and 2), and previously nonreinforced lever-extension CSs overshadowed conditioning to auditory cues. By contrast, conditioning to lever-extension CSs was not blocked by either auditory (Exp. 3) or lever-insertion (Exp. 4) cues, and was not overshadowed by auditory cues. Conditioning to a lever-insertion cue was somewhat overshadowed by the presence of another lever, especially in terms of food cup behavior displayed after lever withdrawal. We discuss several frameworks in which the apparent immunity of autoshaped lever pressing to blocking might be understood. Given evidence that different brain systems are engaged when different kinds of cues are paired with food delivery, it is worth considering the possibility that interactions among them in learning and performance may follow different rules. In particular, it is intriguing to speculate that the roles of simple cue-reinforcer contiguity, as well as of individual and aggregate reinforcer prediction errors, may differ across stimulus classes.

Role of medial prefrontal cortex Narp in the extinction of morphine conditioned place preference.

Narp knockout (KO) mice demonstrate an impaired extinction of morphine conditioned place preference (CPP). Because the medial prefrontal cortex (mPFC) has been implicated in extinction learning, we tested whether Narp cells in this region play a role in the extinction of morphine CPP. We found that intracranial injections of adenoassociated virus (AAV) expressing wild-type (WT) Narp into the mPFC of Narp KO mice rescued the extinction and the injection of AAV expressing a dominant negative form of Narp (NarpN) into the mPFC of WT mice impaired the extinction of morphine CPP. These findings suggest that Narp in the mPFC mediates the extinction of morphine CPP.

Role of amygdala central nucleus in aversive learning produced by shock or by unexpected omission of food.

Many psychological learning theories have noted commonalities between aversive states produced by presentation of negative reinforcers, such as electric shock, and the omission of expected positive reinforcers, such as food. Here, three groups of rats received training with one auditory cue paired with shock and another with the omission of expected food, a shock-paired cue and a food-omission control cue, or a food-omission cue and a shock control cue. Food-omission cues were established by contrast with food delivery; after extensive light-food pairings, the light was followed by the food-omission cue instead of food. Aversiveness of the food-omission cue was assessed with a conditioned punishment procedure, in which presentation of that cue was made contingent on performance of one previously trained instrumental response, whereas a second response had no consequences. We found that rats with lesions of amygdala central nucleus (CeA) showed impaired acquisition of freezing to the cue paired with shock and no evidence for acquisition of aversive properties by the cue that accompanied the omission of expected food. Furthermore, analyses of Arc and Homer1a mRNAs after rats were exposed to a two-epoch test procedure that allowed assessment of gene expression produced by two different test stimuli showed that both food-omission and shock-paired cues generated more neuronal activity in CeA than appropriate control cues. However, the number of neurons that were activated by both shock and food-omission cues was not significantly greater than expected by chance. Thus, under these test conditions, different subsets of CeA neurons represented these two aversive states.

Effects of ventral striatal lesions on first- and second-order appetitive conditioning.

Rats with bilateral lesions of the ventral striatal nucleus accumbens failed to acquire Pavlovian second-order conditioning to auditory stimuli paired with visual stimuli that had previously received first-order pairings with food. This deficit in second-order conditioning was specific to learning driven by incentive properties of the first-order cues, and was observed whether the first-order training had occurred prior to or after lesion surgery. Lesions also produced deficits in the display of conditioned responses to the first-order conditioned stimulus, but only when they were made after first-order training. These results suggest a specific role for the ventral striatum in acquiring and expressing incentive properties of conditioned stimuli through second-order conditioning, as well as a more general role in expressing previously acquired Pavlovian conditioned responses.

Odor-mediated taste learning requires dorsal hippocampus, but not basolateral amygdala activity.

Mediated learning is a unique cognitive phenomenon in which mental representations of physically absent stimuli enter into associations with directly-activated representations of physically present stimuli. Three experiments investigated the functional physiology of mediated learning involving the use of odor-taste associations. In Experiments 1a and 1b, basolateral amygdala lesions failed to attenuate mediated taste aversion learning. In Experiment 2, dorsal hippocampus inactivation impaired mediated learning, but left direct learning intact. Considered with past studies, the results implicate the dorsal hippocampus in mediated learning generally, and suggest a limit on the importance of the basolateral amygdala.

The role of melanin-concentrating hormone in conditioned reward learning.

The orexigenic neuropeptide melanin-concentrating hormone (MCH) is well positioned to play a key role in connecting brain reward and homeostatic systems due to its synthesis in hypothalamic circuitry and receptor expression throughout the cortico-striatal reward circuit. Here we examined whether targeted-deletion of the MCH receptor (MCH-1R) in gene-targeted heterozygote and knockout mice (KO), or systemic treatment with pharmacological agents designed to antagonise MCH-1R in C57BL/6J mice would disrupt two putative consequences of reward learning that rely on different neural circuitries: conditioned reinforcement (CRf) and Pavlovian-instrumental transfer (PIT). Mice were trained to discriminate between presentations of a reward-paired cue (CS+) and an unpaired CS-. Following normal acquisition of the Pavlovian discrimination in all mice, we assessed the capacity for the CS+ to act as a reinforcer for new nose-poke learning (CRf). Pharmacological disruption in control mice and genetic deletion in KO mice impaired CRf test performance, suggesting MCH-1R is necessary for initiating and maintaining behaviors that are under the control of conditioned reinforcers. To examine a dissociable form of reward learning (PIT), a naïve group of mice were trained in separate Pavlovian and instrumental lever training sessions followed by the PIT test. For all mice the CS+ was capable of augmenting ongoing lever responding relative to CS- periods. These results suggest a role for MCH in guiding behavior based on the conditioned reinforcing value of a cue, but not on its incentive motivational value.

Roles of nucleus accumbens and basolateral amygdala in autoshaped lever pressing.

Initially-neutral cues paired with rewards are thought to acquire motivational significance, as if the incentive motivational value of the reward is transferred to the cue. Such cues may serve as secondary reinforcers to establish new learning, modulate the performance of instrumental action (Pavlovian-instrumental transfer, PIT), and be the targets of approach and other cue-directed behaviors. Here we examined the effects of lesions of the ventral striatal nucleus accumbens (ACb) and the basolateral amygdala (BLA) on the acquisition of discriminative autoshaped lever-pressing in rats. Insertion of one lever into the experimental chamber was reinforced by sucrose delivery, but insertion of another lever was not reinforced. Although sucrose was delivered independently of the rats' behavior, sham-lesioned rats rapidly came to press the reinforced but not the nonreinforced lever. Bilateral ACb lesions impaired the initial acquisition of sign-tracking but not its terminal levels. In contrast, BLA lesions produced substantial deficits in terminal levels of sign-tracking. Furthermore, whereas ACb lesions primarily affected the probability of lever press responses, BLA lesions mostly affected the rate of responding once it occurred. Finally, disconnection lesions that disrupted communication between ACb and BLA produced both sets of deficits. We suggest that ACb is important for initial acquisition of consummatory-like responses that incorporate hedonic aspects of the reward, while BLA serves to enhance such incentive salience once it is acquired.

Interactions between amygdala central nucleus and the ventral tegmental area in the acquisition of conditioned cue-directed behavior in rats.

Rats orient to and approach localizable visual cues paired with food delivery. Previous studies from this laboratory show that the acquisition and expression of these learned cue-directed responses depend on integrity of a system including the central nucleus of the amygdala (CeA), the substantia nigra pars compacta (SNc) and the dorsolateral striatum (DLS). Other investigators have suggested that cue-directed behaviors may also depend on interaction between CeA and the ventral striatum, perhaps via CeA projections to the ventral tegmental area (VTA). In Experiment 1, we examined the effects of unilateral lesions of CeA and/or VTA on rats' acquisition of conditioned responses to visual cues paired with food. Contrary to the results of previous studies that examined interactions of CeA with either SNc or DLS, rats with contralateral disconnection lesions of CeA and VTA were unimpaired in their acquisition of cue-directed responses. By contrast, rats with lesions of both structures in the same hemisphere failed to learn cue-directed responses, but were normal in their acquisition of conditioned responses directed to the food cup. In Experiment 2, we attempted to characterize the influence of VTA on CeA by examining FOS induction in CeA by a visual cue for food in rats with unilateral lesions of VTA. The results suggested an excitatory influence of VTA on CeA in the presence of food cues. Implications of these results for brain circuits involved in learned orienting and incentive motivation are discussed.

The central amygdala projection to the substantia nigra reflects prediction error information in appetitive conditioning.

The central amygdala nucleus (CeA) plays a critical role in cognitive processes beyond fear conditioning. For example, intact CeA function is essential for enhancing attention to conditioned stimuli (CSs). Furthermore, this enhanced attention depends on the CeA's connections to the nigrostriatal system. In the current study, we examined the role of the CeA's connections to two midbrain dopamine regions, the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA), in processing CS information when predictions of reward or nonreward were confirmed or disconfirmed. Initially, two different retrograde tracers were injected into the SNc and the VTA of rats, to label CeA cells. Different groups of rats then received a visual CS either paired or unpaired with food. Finally, Fos induction was assessed after a test session in which rats were exposed to the visual CS alone or paired with food. Colabeling of Fos and the retrograde tracer(s) showed that CeA neurons projecting to the SNc, but not to the VTA, were engaged in processing CS information when the training and testing conditions differed. These results suggest that the CeA-nigral pathway represents prediction error information during appetitive conditioning.

Deficits in sensory-specific devaluation task performance following genetic deletions of cannabinoid (CB1) receptor.

Cannabinoid CB1 receptor is abundantly expressed throughout the CNS and is implicated in numerous physiological and behavioral functions, including appetite and feeding. In the present study, wild-type and CB1 heterozygous and homozygous knockout mice were tested on an instrumental outcome-selective devaluation task to assess changes in acquired instrumental response levels for a distinct food reward following selective satiation. Deletion of CB1 receptor, as well as reduction in CB1 expression (HET), produced deficits in outcome-selective instrumental devaluation. These results identify a critical role for CB1 receptor in the ability of animals to represent, update, and/or use sensory-specific outcome representations to alter appetitive behaviors.