Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related.

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Decreased Ventral Tegmental Area CB1R Signaling Reduces Sign Tracking and Shifts Cue-Outcome Dynamics in Rat Nucleus Accumbens.

Sign-tracking (ST) rats show enhanced cue sensitivity before drug experience that predicts greater discrete cue-induced drug seeking compared with goal-tracking or intermediate rats. Cue-evoked dopamine in the nucleus accumbens (NAc) is a neurobiological signature of sign-tracking behaviors. Here, we examine a critical regulator of the dopamine system, endocannabinoids, which bind the cannabinoid receptor-1 (CB1R) in the ventral tegmental area (VTA) to control cue-evoked striatal dopamine levels. We use cell type-specific optogenetics, intra-VTA pharmacology, and fiber photometry to test the hypothesis that VTA CB1R receptor signaling regulates NAc dopamine levels to control sign tracking. We trained male and female rats in a Pavlovian lever autoshaping (PLA) task to determine their tracking groups before testing the effect of VTA → NAc dopamine inhibition. We found that this circuit is critical for mediating the vigor of the ST response. Upstream of this circuit, intra-VTA infusions of rimonabant, a CB1R inverse agonist, during PLA decrease lever and increase food cup approach in sign-trackers. Using fiber photometry to measure fluorescent signals from a dopamine sensor, GRABDA (AAV9-hSyn-DA2m), we tested the effects of intra-VTA rimonabant on NAc dopamine dynamics during autoshaping in female rats. We found that intra-VTA rimonabant decreased sign-tracking behaviors, which was associated with increases in NAc shell, but not core, dopamine levels during reward delivery [unconditioned stimulus (US)]. Our results suggest that CB1R signaling in the VTA influences the balance between the conditioned stimulus-evoked and US-evoked dopamine responses in the NAc shell and biases behavioral responding to cues in sign-tracking rats.SIGNIFICANCE STATEMENT Substance use disorder (SUD) is a chronically relapsing psychological disorder that affects a subset of individuals who engage in drug use. Recent research suggests that there are individual behavioral and neurobiological differences before drug experience that predict SUD and relapse vulnerabilities. Here, we investigate how midbrain endocannabinoids regulate a brain pathway that is exclusively involved in driving cue-motivated behaviors of sign-tracking rats. This work contributes to our mechanistic understanding of individual vulnerabilities to cue-triggered natural reward seeking that have relevance for drug-motivated behaviors.

Manipulating the revision of reward value during the intertrial interval increases sign tracking and dopamine release.

Recent computational models of sign tracking (ST) and goal tracking (GT) have accounted for observations that dopamine (DA) is not necessary for all forms of learning and have provided a set of predictions to further their validity. Among these, a central prediction is that manipulating the intertrial interval (ITI) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses. Here, we tested these predictions and found that lengthening the ITI increased ST, i.e., behavioral engagement with conditioned stimuli (CS) and cue-induced phasic DA release. Importantly, DA release was also present at the time of reward delivery, even after learning, and DA release was correlated with time spent in the food cup during the ITI. During conditioning with shorter ITIs, GT was prominent (i.e., engagement with food cup), and DA release responded to the CS while being absent at the time of reward delivery after learning. Hence, shorter ITIs restored the classical DA reward prediction error (RPE) pattern. These results validate the computational hypotheses, opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling.

Effects of 5-HT2A receptor stimulation on economic demand for fentanyl after intermittent and continuous access self-administration in male rats.

The relative value of and motivation for abused drugs often increases with drug experience and differs based on drug availability. Here, we determined how different intake patterns of fentanyl, a µ-opioid agonist, alter economic demand for fentanyl and how 5-HT2A receptor stimulation affects economic demand for fentanyl. We used a within-session demand threshold procedure to characterize changes in economic demand for fentanyl before and after intermittent or continuous access schedules. We subsequently tested the acute effects of 5-HT2A receptor stimulation with psychedelic 2,5-dimethoxy-4-iodoamphetamine (DOI) on economic demand for fentanyl. Extended fentanyl experience with both intermittent and continuous schedules increased fentanyl consumption at low cost (Q0 ), increased total fentanyl consumption, and decreased demand elasticity (α), indicating both schedules elevated motivation to self-administer fentanyl. Overall, the two schedules produced similar alterations in economic demand for fentanyl, although low-cost consumption (Q0 ) increased more in the continuous access group. Systemic injections of DOI (0.0-0.4 mg/kg, i.p.) dose-dependently produced economic demand changes in the opposite direction produced by fentanyl experience. DOI decreased motivation (increased "α"), decreased Q0 , and decreased total fentanyl consumption. The selective 5-HT2A antagonist, M100907 (0.3 mg/kg, i.p.), blocked the effects of DOI, indicating that DOI is acting through 5-HT2A receptors to alter economic demand for fentanyl. In an economic food demand experiment, DOI (0.4 mg/kg) also increased demand elasticity and reduced food consumption. These results demonstrate that both intermittent and continuous fentanyl experience raise the economic demand for fentanyl, and acute 5-HT2A receptor activation reduces economic demand for fentanyl and food.

Intrinsic plasticity: an emerging player in addiction.

Exposure to drugs of abuse, such as cocaine, leads to plastic changes in the activity of brain circuits, and a prevailing view is that these changes play a part in drug addiction. Notably, there has been intense focus on drug-induced changes in synaptic excitability and much less attention on intrinsic excitability factors (that is, excitability factors that are remote from the synapse). Accumulating evidence now suggests that intrinsic factors such as K+ channels are not only altered by cocaine but may also contribute to the shaping of the addiction phenotype.

Disconnection of basolateral amygdala and insular cortex disrupts conditioned approach in Pavlovian lever autoshaping.

Previously established individual differences in appetitive approach and devaluation sensitivity observed in goal- and sign-trackers may be attributed to differences in the acquisition, modification, or use of associative information in basolateral amygdala (BLA) pathways. Here, we sought to determine the extent to which communication of associative information between BLA and anterior portions of insular cortex (IC) supports ongoing Pavlovian conditioned approach behaviors in sign- and goal-tracking rats, in the absence of manipulations to outcome value. We hypothesized that the BLA mediates goal-, but not sign- tracking approach through interactions with the IC, a brain region involved in supporting flexible behavior. We first trained rats in Pavlovian lever autoshaping to determine their sign- or goal-tracking tendency. During alternating test sessions, we gave unilateral intracranial injections of vehicle or a cocktail of gamma-aminobutyric acid (GABA) receptor agonists, baclofen and muscimol, unilaterally into the BLA and contralaterally or ipsilaterally into the IC prior to reinforced lever autoshaping sessions. Consistent with our hypothesis we found that contralateral inactivation of BLA and IC increased the latency to approach the food cup and decreased the number of food cup contacts in goal-trackers. While contralateral inactivation of BLA and IC did not affect the total number of lever contacts in sign-trackers, this manipulation increased the latency to approach the lever. Ipsilateral inactivation of BLA and IC did not impact approach behaviors in Pavlovian lever autoshaping. These findings, contrary to our hypothesis, suggest that communication between BLA and IC maintains a representation of initially learned appetitive associations that commonly support the initiation of Pavlovian conditioned approach behavior regardless of whether it is directed at the cue or the location of reward delivery.

Effect of yohimbine on reinstatement of operant responding in rats is dependent on cue contingency but not food reward history.

Yohimbine is an alpha-2 adrenoceptor antagonist that has been used in numerous studies as a pharmacological stressor in rodents, monkeys and humans. Recently, yohimbine has become the most common stress manipulation in studies on reinstatement of drug and food seeking. However, the wide range of conditions under which yohimbine promotes reward seeking is significantly greater than that of stressors like intermittent footshock. Here, we addressed two fundamental questions regarding yohimbine's effect on reinstatement of reward seeking: (1) whether the drug's effect on operant responding is dependent on previous reward history or cue contingency, and (2) whether yohimbine is aversive or rewarding under conditions typically used in reinstatement studies. We also used in vivo microdialysis to determine yohimbine's effect on dopamine levels in nucleus accumbens (NAc) and medial prefrontal cortex (mPFC). We found that the magnitude of yohimbine-induced (0.5, 1.0, 2.0 mg/kg) operant responding during the reinstatement tests was critically dependent on the contingency between lever pressing and discrete tone-light cue delivery but not the previous history with food reward during training. We also found that yohimbine (2 mg/kg) did not cause conditioned place aversion. Finally, we found that yohimbine modestly increased dopamine levels in mPFC but not NAc. Results suggest that yohimbine's effects on operant responding in reinstatement studies are likely independent of the history of contingent self-administration of food or drug rewards and may not be related to the commonly assumed stress-like effects of yohimbine.

Optogenetic inhibition of dorsal medial prefrontal cortex attenuates stress-induced reinstatement of palatable food seeking in female rats.

Relapse to maladaptive eating habits during dieting is often provoked by stress. Recently, we identified a role of dorsal medial prefrontal cortex (mPFC) neurons in stress-induced reinstatement of palatable food seeking in male rats. It is unknown whether endogenous neural activity in dorsal mPFC drives stress-induced reinstatement in female rats. Here, we used an optogenetic approach, in which female rats received bilateral dorsal mPFC microinjections of viral constructs coding light-sensitive eNpHR3.0-eYFP or control eYFP protein and intracranial fiber optic implants. Rats were food restricted and trained to lever press for palatable food pellets. Subsequently, pellets were removed, and lever pressing was extinguished; then the effect of bilateral dorsal mPFC light delivery on reinstatement of food seeking was assessed after injections of the pharmacological stressor yohimbine (an α-2 andrenoceptor antagonist) or pellet priming, a manipulation known to provoke food seeking in hungry rats. Dorsal mPFC light delivery attenuated yohimbine-induced reinstatement of food seeking in eNpHR3.0-injected but not eYFP-injected rats. This optical manipulation had no effect on pellet-priming-induced reinstatement or ongoing food-reinforced responding. Dorsal mPFC light delivery attenuated yohimbine-induced Fos immunoreactivity and disrupted neural activity during in vivo electrophysiological recording in awake rats. Optical stimulation caused significant outward currents and blocked electrically evoked action potentials in eNpHR3.0-injected but not eYFP-injected mPFC hemispheres. Light delivery alone caused no significant inflammatory response in mPFC. These findings indicate that intracranial light delivery in eNpHR3.0 rats disrupts endogenous dorsal mPFC neural activity that plays a role in stress-induced relapse to food seeking in female rats.

Effects of psychedelic, DOI, on nucleus accumbens dopamine signaling to predictable rewards and cues in rats.

Psychedelics produce lasting therapeutic responses in neuropsychiatric diseases suggesting they may disrupt entrenched associations and catalyze learning. Here, we examine psychedelic 5-HT2A/2C agonist, DOI, effects on dopamine signaling in the nucleus accumbens (NAc) core, a region extensively linked to reward learning, motivation, and drug-seeking. We measure phasic dopamine transients following acute DOI administration in rats during well learned Pavlovian tasks in which sequential cues predict rewards. We find that DOI (0.0-1.2 mg/kg, i.p.) increases dopamine signals, photometrically measured using GRABDA optical sensor, to rewards and proximal reward cues, but not to the distal cues that predict these events. We determine that the elevated dopamine produced by DOI to reward cues occurs independently of DOI-induced changes in reward value. The increased dopamine associated with predictable reward cues and rewards supports DOI-induced increases in prediction error signaling. These findings lay a foundation for developing psychedelic strategies aimed at engaging error-driven learning mechanisms to disrupt entrenched associations or produce new associations.

Dorsomedial Striatum CB1R signaling is required for Pavlovian outcome devaluation in male Long Evans rats and reduces inhibitory synaptic transmission in both sexes.

Cannabinoid-1 receptor (CB1R) signaling in the dorsal striatum regulates the shift from flexible to habitual behavior in instrumental outcome devaluation. Based on prior work establishing individual, sex, and experience-dependent differences in Pavlovian behaviors, we predicted a role for dorsomedial striatum CB1R signaling in driving rigid responding in Pavlovian autoshaping and outcome devaluation. We trained male and female Long Evans rats in Pavlovian Lever Autoshaping (PLA). We gave intra-dorsomedial striatum (DMS) infusions of the CB1R inverse agonist, rimonabant, before satiety-induced outcome devaluation test sessions, where we sated rats on training pellets or home cage chow and tested them in brief nonreinforced Pavlovian Lever Autoshaping sessions. Overall, inhibition of DMS CB1R signaling prevented Pavlovian outcome devaluation but did not affect behavior in reinforced PLA sessions. Males were sensitive to devaluation while females were not and DMS CB1R inhibition impaired devaluation sensitivity in males. We then investigated how DMS CB1R signaling impacts local inhibitory synaptic transmission in male and female Long Evans rats. We recorded spontaneous inhibitory postsynaptic currents (sIPSC) from DMS neurons at baseline and before and after application of a CB1R agonist, WIN 55,212-2. We found that male rats showed decreased sIPSC frequency compared to females, and that CB1R activation reduced DMS inhibitory transmission independent of sex. Altogether our results demonstrate that DMS CB1Rs regulate Pavlovian devaluation sensitivity and inhibitory synaptic transmission and suggest that basal sex differences in inhibitory synaptic transmission may underly sex differences in DMS function and behavioral flexibility.

Medial prefrontal cortex neuronal activation and synaptic alterations after stress-induced reinstatement of palatable food seeking: a study using c-fos-GFP transgenic female rats.

Relapse to maladaptive eating habits during dieting is often provoked by stress and there is evidence for a role of ovarian hormones in stress responses and feeding. We studied the role of these hormones in stress-induced reinstatement of food seeking and medial prefrontal cortex (mPFC) neuronal activation in c-fos-GFP transgenic female rats, which express GFP in strongly activated neurons. Food-restricted ovariectomized or sham-operated c-fos-GFP rats were trained to lever-press for palatable food pellets. Subsequently, lever-pressing was extinguished and reinstatement of food seeking and mPFC neuronal activation was assessed after injections of the pharmacological stressor yohimbine (0.5-2 mg/kg) or pellet priming (1-4 noncontingent pellets). Estrous cycle effects on reinstatement were also assessed in wild-type rats. Yohimbine- and pellet-priming-induced reinstatement was associated with Fos and GFP induction in mPFC; both reinstatement and neuronal activation were minimally affected by ovarian hormones in both c-fos-GFP and wild-type rats. c-fos-GFP transgenic rats were then used to assess glutamatergic synaptic alterations within activated GFP-positive and nonactivated GFP-negative mPFC neurons following yohimbine-induced reinstatement of food seeking. This reinstatement was associated with reduced AMPA receptor/NMDA receptor current ratios and increased paired-pulse facilitation in activated GFP-positive but not GFP-negative neurons. While ovarian hormones do not appear to play a role in stress-induced relapse of food seeking in our rat model, this reinstatement was associated with unique synaptic alterations in strongly activated mPFC neurons. Our paper introduces the c-fos-GFP transgenic rat as a new tool to study unique synaptic changes in activated neurons during behavior.

Inactivation of the Basolateral Amygdala to Insular Cortex Pathway Makes Sign-Tracking Sensitive to Outcome Devaluation.

Goal-tracking (GT) rats are sensitive to Pavlovian outcome devaluation while sign-tracking (ST) rats are devaluation insensitive. During outcome devaluation, GT rats flexibly modify responding to cues based on the current value of the associated outcome. However, ST rats rigidly respond to cues regardless of the current outcome value. Prior work demonstrated disconnection of the basolateral amygdala (BLA) and anterior insular cortex (aIC) decreased both GT and ST behaviors. Given the role of these regions in appetitive motivation and behavioral flexibility, we predicted that disrupting BLA to aIC pathway during outcome devaluation would reduce flexibility in GT rats and reduce rigid appetitive motivation in ST rats. We inhibited the BLA to aIC pathway by infusing inhibitory DREADDs (hM4Di-mcherry) or control (mCherry) virus into the BLA and implanted cannulae into the aIC to inhibit BLA terminals using intracranial injections of clozapine N-oxide (CNO). After training, we used a within-subject satiety-induced outcome devaluation procedure in which we sated rats on training pellets (devalued condition) or homecage chow (valued condition). All rats received bilateral CNO infusions into the aIC before brief nonreinforced test sessions. Contrary to our hypothesis, BLA-IC inhibition did not interfere with devaluation sensitivity in GT rats but did make ST behaviors sensitive to devaluation. Intermediate rats showed the opposite effect, showing rigid responding to cues with BLA-aIC pathway inactivation. Together, these results demonstrate BLA-IC projections mediate tracking-specific Pavlovian devaluation sensitivity and highlights the importance of considering individual differences in Pavlovian approach when evaluating circuitry contributions to behavioral flexibility.

Investigating discriminative stimulus modulation of opioid seeking after conflict-induced abstinence in sign- and goal-tracking rats.

RATIONALE: Discriminative stimuli (DS) are cues that predict reward availability. DS are resistant to extinction and motivate drug seeking even after long periods of abstinence. Previous studies have demonstrated that sign-tracking (ST) and goal-tracking (GT) differences in Pavlovian approach predict distinct cue-modulated vulnerabilities to cocaine reinstatement. GT rats show heightened reinstatement to contextual and DS, while ST rats show heightened reinstatement to discrete stimuli. Here we examine whether DS modulate reinstatement after electric barrier-induced abstinence and whether tracking-related relapse vulnerabilities generalize to opioid relapse. OBJECTIVES: We examine whether DS-modulated reinstatement to fentanyl seeking persists in the presence of reduced adverse consequences after electric barrier-induced abstinence. We also examine whether tracking differences predict the magnitude of DS-modulated reinstatement of fentanyl seeking after electric barrier-induced abstinence. METHODS: We used Pavlovian lever autoshaping (PLA) training to determine sign-, goal-, and intermediate tracking groups in male and female Sprague Dawley rats. We then trained rats in a DS model of intermittent fentanyl self-administration, and extinguished drug seeking by imposing an electric barrier of increasing intensity. We then measured the level of DS-modulated reinstatement in the presence of a reduced electric barrier intensity. RESULTS: We report that DS strongly modulate fentanyl seeking after electric barrier-induced abstinence. DS-modulation of fentanyl acquisition, electric barrier-induced abstinence, and reinstatement was similar for sign- and goal-tracking groups. CONCLUSIONS: Discriminative stimuli powerfully motivate opioid seeking, despite continued aversive consequences. Pavlovian approach differences do not predict the level of DS-modulated reinstatement to fentanyl seeking after conflict-induced abstinence.

Neural correlates of variations in event processing during learning in basolateral amygdala.

The discovery that dopamine neurons signal errors in reward prediction has demonstrated that concepts empirically derived from the study of animal behavior can be used to understand the neural implementation of reward learning. Yet the learning theory models linked to phasic dopamine activity treat attention to events such as cues and rewards as static quantities; other models, such as Pearce-Hall, propose that learning might be influenced by variations in processing of these events. A key feature of these accounts is that event processing is modulated by unsigned rather than signed reward prediction errors. Here we tested whether neural activity in rat basolateral amygdala conforms to this pattern by recording single units in a behavioral task in which rewards were unexpectedly delivered or omitted. We report that neural activity at the time of reward is providing an unsigned error signal with characteristics consistent with those postulated by these models. This neural signal increased immediately after a change in reward, and stronger firing was evident whether the value of the reward increased or decreased. Further, as predicted by these models, the change in firing developed over several trials as expectations for reward were repeatedly violated. This neural signal was correlated with faster orienting to predictive cues after changes in reward, and abolition of the signal by inactivation of basolateral amygdala disrupted this change in orienting and retarded learning in response to changes in reward. These results suggest that basolateral amygdala serves a critical function in attention for learning.

Inactivation of the central but not the basolateral nucleus of the amygdala disrupts learning in response to overexpectation of reward.

The amygdala is critical for associating predictive cues with primary rewarding and aversive outcomes. This is particularly evident in tasks in which information about expected outcomes is required for normal responding. Here we used a pavlovian overexpectation task to test whether outcome signaling by amygdala might also be necessary for changing those representations in the face of unexpected outcomes. Rats were trained to associate several different cues with a food reward. After learning, two of the cues were presented together, in compound, followed by the same reward. Before each compound training session, rats received infusions of 2,3-dioxo-6-nitro-1,2,3,4-tetrahydrobenzo[f]quinoxaline-7-sulfonamide or saline into either the basolateral (ABL) or central nucleus (CeN) of amygdala. We found that infusions into CeN abolished the normal decline in responding to the compounded cue in a later probe test, whereas infusions into ABL had no effect. These results are inconsistent with the proposal that signaling of information about expected outcomes by ABL contributes to learning, at least in this setting, and instead implicate the CeN in this process, perhaps attributable to the hypothesized involvement of this area in attention and variations in stimulus processing.

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards.

The dopamine system is thought to be involved in making decisions about reward. Here we recorded from the ventral tegmental area in rats learning to choose between differently delayed and sized rewards. As expected, the activity of many putative dopamine neurons reflected reward prediction errors, changing when the value of the reward increased or decreased unexpectedly. During learning, neural responses to reward in these neurons waned and responses to cues that predicted reward emerged. Notably, this cue-evoked activity varied with size and delay. Moreover, when rats were given a choice between two differently valued outcomes, the activity of the neurons initially reflected the more valuable option, even when it was not subsequently selected.

Cocaine-induced decision-making deficits are mediated by miscoding in basolateral amygdala.

Addicts and drug-experienced animals have decision-making deficits in reversal-learning tasks and more complex 'gambling' variants. Here we show evidence that these deficits are mediated by persistent encoding of outdated associative information in the basolateral amygdala. Cue-selective neurons in the basolateral amygdala, recorded in cocaine-treated rats, failed to change cue preference during reversal learning. Further, the presence of these neurons was critical to the expression of the reversal-learning deficit in the cocaine-treated rats.

Choose your path: Divergent basolateral amygdala efferents differentially mediate incentive motivation, flexibility and decision-making.

To survive in a complex environment, individuals form associations between environmental stimuli and rewards to organize and optimize reward seeking behaviors. The basolateral amygdala (BLA) uses these learned associations to inform decision-making processes. In this review, we describe functional projections between BLA and its cortical and striatal targets that promote learning and motivational processes central to decision-making. Specifically, we compare and contrast divergent projections from the BLA to the orbitofrontal (OFC) and to the nucleus accumbens (NAc) and examine the roles of these pathways in associative learning, value-guided decision-making, choice behaviors, as well as cue and context-driven drug seeking. Finally, we consider how these projections are involved in disorders of motivation, with a focus on Substance Use Disorder.

Corrigendum: Role of BNST CRFR1 Receptors in Incubation of Fentanyl Seeking.

[This corrects the article DOI: 10.3389/fnbeh.2020.00153.].

All that glitters ... dissociating attention and outcome expectancy from prediction errors signals.

Initially reported in dopamine neurons, neural correlates of prediction errors have now been shown in a variety of areas, including orbitofrontal cortex, ventral striatum, and amygdala. Yet changes in neural activity to an outcome or cues that precede it can reflect other processes. We review the recent literature and show that although activity in dopamine neurons appears to signal prediction errors, similar activity in orbitofrontal cortex, basolateral amygdala, and ventral striatum does not. Instead, increased firing in basolateral amygdala to unexpected outcomes likely reflects attention, whereas activity in orbitofrontal cortex and ventral striatum is unaffected by prior expectations and may provide information on outcome expectancy. These results have important implications for how these areas interact to facilitate learning and guide behavior.