Optogenetic Inhibition of Rat Anterior Cingulate Cortex Impairs the Ability to Initiate and Stay on Task.

Our prior research has identified neural correlates of cognitive control in the anterior cingulate cortex (ACC), leading us to hypothesize that the ACC is necessary for increasing attention as rats flexibly learn new contingencies during a complex reward-guided decision-making task. Here, we tested this hypothesis by using optogenetics to transiently inhibit the ACC, while rats of either sex performed the same two-choice task. ACC inhibition had a profound impact on behavior that extended beyond deficits in attention during learning when expected outcomes were uncertain. We found that ACC inactivation slowed and reduced the number of trials rats initiated and impaired both their accuracy and their ability to complete sessions. Furthermore, drift-diffusion model analysis suggested that free-choice performance and evidence accumulation (i.e., reduced drift rates) were degraded during initial learning-leading to weaker associations that were more easily overridden in later trial blocks (i.e., stronger bias). Together, these results suggest that in addition to attention-related functions, the ACC contributes to the ability to initiate trials and generally stay on task.

Orbitofrontal State Representations Are Related to Choice Adaptations and Reward Predictions.

Animals can categorize the environment into "states," defined by unique sets of available action-outcome contingencies in different contexts. Doing so helps them choose appropriate actions and make accurate outcome predictions when in each given state. State maps have been hypothesized to be held in the orbitofrontal cortex (OFC), an area implicated in decision-making and encoding information about outcome predictions. Here we recorded neural activity in OFC in 6 male rats to test state representations. Rats were trained on an odor-guided choice task consisting of five trial blocks containing distinct sets of action-outcome contingencies, constituting states, with unsignaled transitions between them. OFC neural ensembles were analyzed using decoding algorithms. Results indicate that the vast majority of OFC neurons contributed to representations of the current state at any point in time, independent of odor cues and reward delivery, even at the level of individual neurons. Across state transitions, these representations gradually integrated evidence for the new state; the rate at which this integration happened in the prechoice part of the trial was related to how quickly the rats' choices adapted to the new state. Finally, OFC representations of outcome predictions, often thought to be the primary function of OFC, were dependent on the accuracy of OFC state representations.SIGNIFICANCE STATEMENT A prominent hypothesis proposes that orbitofrontal cortex (OFC) tracks current location in a "cognitive map" of state space. Here we tested this idea in detail by analyzing neural activity recorded in OFC of rats performing a task consisting of a series of states, each defined by a set of available action-outcome contingencies. Results show that most OFC neurons contribute to state representations and that these representations are related to the rats' decision-making and OFC reward predictions. These findings suggest new interpretations of emotional dysregulation in pathologies, such as addiction, which have long been known to be related to OFC dysfunction.

Prior Cocaine Use Alters the Normal Evolution of Information Coding in Striatal Ensembles during Value-Guided Decision-Making.

Substance use disorders (SUDs) are characterized by maladaptive behavior. The ability to properly adjust behavior according to changes in environmental contingencies necessitates the interlacing of existing memories with updated information. This can be achieved by assigning learning in different contexts to compartmentalized "states." Though not often framed this way, the maladaptive behavior observed in individuals with SUDs may result from a failure to properly encode states because of drug-induced neural alterations. Previous studies found that the dorsomedial striatum (DMS) is important for behavioral flexibility and state encoding, suggesting the DMS may be an important substrate for these effects. Here, we recorded DMS neural activity in cocaine-experienced male rats during a decision-making task where blocks of trials represented distinct states to probe whether the encoding of state and state-related information is affected by prior drug exposure. We found that DMS medium spiny neurons (MSNs) and fast-spiking interneurons (FSIs) encoded such information and that prior cocaine experience disrupted the evolution of representations both within trials and across recording sessions. Specifically, DMS MSNs and FSIs from cocaine-experienced rats demonstrated higher classification accuracy of trial-specific rules, defined by response direction and value, compared with those drawn from sucrose-experienced rats, and these overly strengthened trial-type representations were related to slower switching behavior and reaction times. These data show that prior cocaine experience paradoxically increases the encoding of state-specific information and rules in the DMS and suggest a model in which abnormally specific and persistent representation of rules throughout trials in DMS slows value-based decision-making in well trained subjects.SIGNIFICANCE STATEMENT Substance use disorders (SUDs) may result from a failure to properly encode rules guiding situationally appropriate behavior. The dorsomedial striatum (DMS) is thought to be important for such behavioral flexibility and encoding that defines the situation or "state." This suggests that the DMS may be an important substrate for the maladaptive behavior observed in SUDs. In the current study, we show that prior cocaine experience results in over-encoding of state-specific information and rules in the DMS, which may impair normal adaptive decision-making in the task, akin to what is observed in SUDs.

An Integrated Model of Action Selection: Distinct Modes of Cortical Control of Striatal Decision Making.

Making decisions in environments with few choice options is easy. We select the action that results in the most valued outcome. Making decisions in more complex environments, where the same action can produce different outcomes in different conditions, is much harder. In such circumstances, we propose that accurate action selection relies on top-down control from the prelimbic and orbitofrontal cortices over striatal activity through distinct thalamostriatal circuits. We suggest that the prelimbic cortex exerts direct influence over medium spiny neurons in the dorsomedial striatum to represent the state space relevant to the current environment. Conversely, the orbitofrontal cortex is argued to track a subject's position within that state space, likely through modulation of cholinergic interneurons.

Orbitofrontal neurons signal reward predictions, not reward prediction errors.

Neurons in the orbitofrontal cortex (OFC) fire in anticipation of and during rewards. Such firing has been suggested to encode reward predictions and to account in some way for the role of this area in adaptive behavior and learning. However, it has also been reported that neural activity in OFC reflects reward prediction errors, which might drive learning directly. Here we tested this question by analyzing the firing of OFC neurons recorded in an odor discrimination task in which rats were trained to sample odor cues and respond left or right on each trial for reward. Neurons were recorded across blocks of trials in which we switched either the number or the flavor of the reward delivered in each well. Previously we have described how neurons in this dataset fired to the predictive cues (Stalnaker et al., 2014); here we focused on the firing in anticipation of and just after delivery of each drop of reward, looking specifically for differences in firing based on whether the reward number or flavor was unexpected or expected. Unlike dopamine neurons recorded in this setting, which exhibited phasic error-like responses after surprising changes in either reward number or reward flavor (Takahashi et al., 2017), OFC neurons showed no such error correlates and instead fired in a way that reflected reward predictions.

Cholinergic Interneurons Use Orbitofrontal Input to Track Beliefs about Current State.

UNLABELLED: When conditions change, organisms need to learn about the changed conditions without interfering with what they already know. To do so, they can assign the new learning to a new "state" and the old learning to a previous state. This state assignment is fundamental to behavioral flexibility. Cholinergic interneurons (CINs) in the dorsomedial striatum (DMS) are necessary for associative information to be compartmentalized in this way, but the mechanism by which they do so is unknown. Here we addressed this question by recording putative CINs from the DMS in rats performing a task consisting of a series of trial blocks, or states, that required the recall and application of contradictory associative information. We found that individual CINs in the DMS represented the current state throughout each trial. These state correlates were not observed in dorsolateral striatal CINs recorded in the same rats. Notably, DMS CIN ensembles tracked rats' beliefs about the current state such that, when states were miscoded, rats tended to make suboptimal choices reflecting the miscoding. State information held by the DMS CINs also depended completely on the orbitofrontal cortex, an area that has been proposed to signal environmental states. These results suggest that CINs set the stage for recalling associative information relevant to the current environment by maintaining a real-time representation of the current state. Such a role has novel implications for understanding the neural basis of a variety of psychiatric diseases, such as addiction or anxiety disorders, in which patients generalize inappropriately (or fail to generalize) between different environments. SIGNIFICANCE STATEMENT: Striatal cholinergic interneurons (CINs) are thought to be identical to tonically active neurons. These neurons have long been thought to have an important influence on striatal processing during reward-related learning. Recently, a more specific function for striatal CINs has been suggested, which is that they are necessary for striatal learning to be compartmentalized into different states as the state of the environment changes. Here we report that putative CINs appear to track rats' beliefs about which environmental state is current. We further show that this property of CINs depends on orbitofrontal cortex input and is correlated with choices made by rats. These findings could provide new insight into neuropsychiatric diseases that involve improper generalization between different contexts.

A new perspective on the role of the orbitofrontal cortex in adaptive behaviour.

The orbitofrontal cortex (OFC) is crucial for changing established behaviour in the face of unexpected outcomes. This function has been attributed to the role of the OFC in response inhibition or to the idea that the OFC is a rapidly flexible associative-learning area. However, recent data contradict these accounts, and instead suggest that the OFC is crucial for signalling outcome expectancies. We suggest that this function--signalling of expected outcomes--can also explain the crucial role of the OFC in changing behaviour in the face of unexpected outcomes.

Reward prediction error signaling in posterior dorsomedial striatum is action specific.

Neural correlates of reward prediction errors (RPEs) have been found in dorsal striatum. Such signals may be important for updating associative action representations within striatum. In order that the appropriate representations can be updated, it might be important for the RPE signal to be specific for the action that led to that error. However, RPEs signaled by midbrain dopamine neurons, which project heavily to striatum, are not action-specific. Here we tested whether RPE-like activity in dorsal striatum is action-specific; we recorded single-unit activity in posterior dorsomedial and dorsolateral striatum as rats performed a task in which the reward predictions associated with two different actions were repeatedly violated, thereby eliciting RPEs. We separately analyzed fast firing neurons (FFNs) and phasically firing neurons (total n = 1076). Only among FFNs recorded in posterior dorsomedial striatum did we find a population with RPE-like characteristics (19 of all 196 FFNs, 10%). This population showed a phasic increase in activity during unexpected rewards, a phasic decrease in activity during unexpected omission of rewards, and a phasic increase in activity during cues when they predicted high-value reward. However, unlike a classical RPE signal, this signal was linked to the action that elicited the prediction error, in that neurons tended to signal RPEs only after their anti-preferred action. This action-specific RPE-like signal could provide a mechanism for updating specific associative action representations in posterior dorsomedial striatum.

Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model.

Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.

Neuroscience: Action history and the orbitofrontal cortex.

Decisions are often made in the absence of instructive cues, based instead on memories of previous actions and outcomes. A new study sheds light on how orbitofrontal cortex tracks action history to adjust actions over time.

Basolateral amygdala lesions abolish orbitofrontal-dependent reversal impairments.

Damage to orbitofrontal cortex (OFC) has long been associated with deficits in reversal learning. OFC damage also causes inflexible associative encoding in basolateral amygdala (ABL) during reversal learning. Here we provide a critical test of the hypothesis that the reversal deficit in OFC-lesioned rats is caused by this inflexible encoding in ABL. Rats with bilateral neurotoxic lesions of OFC, ABL, or both areas were tested on a series of two-odor go/no-go discrimination problems, followed by two serial reversals of the final problem. As expected, all groups acquired the initial problems at the same rate, and rats with OFC lesions were slower to acquire the reversals than sham controls. This impairment was abolished by accompanying ABL lesions, while ABL lesions alone had no effect on reversal learning. These results are consistent with the hypothesis that OFC facilitates cognitive flexibility by promoting updating of associative encoding in downstream brain areas.

Cocaine-induced decision-making deficits are mediated by miscoding in basolateral amygdala.

Addicts and drug-experienced animals have decision-making deficits in reversal-learning tasks and more complex 'gambling' variants. Here we show evidence that these deficits are mediated by persistent encoding of outdated associative information in the basolateral amygdala. Cue-selective neurons in the basolateral amygdala, recorded in cocaine-treated rats, failed to change cue preference during reversal learning. Further, the presence of these neurons was critical to the expression of the reversal-learning deficit in the cocaine-treated rats.

Neural substrates of cognitive inflexibility after chronic cocaine exposure.

Cognitive changes in addicts and animals exposed to addictive drugs have been extensively investigated over the past decades. One advantage of studying addiction using cognitive paradigms is that neural processing in addicts or drug-exposed animals can be compared to that in normal subjects. Tests of cognitive flexibility that measure the ability to change responding to a previously rewarded or punished stimulus are of potential interest in the study of addiction, because addiction can itself be viewed as an inability to change responding to stimuli previously associated with drug reward. One such test is reversal learning, which is impaired in cocaine addicts and animals that have chronically self-administered or been exposed to cocaine. A circuit including orbitofrontal cortex, basolateral amygdala and striatum subserves reversal learning. In rats that have been previously exposed to cocaine, neurons in these regions show selective and distinct changes in how they encode information during reversal learning. These changes suggest that in these rats, orbitofrontal cortex loses the ability to signal expected outcomes, and basolateral amygdala becomes inflexible in its encoding of cue significance. These changes could explain cocaine-induced impairments to cognitive flexibility and may have theoretical importance in addiction.

Coping behavior causes asymmetric changes in neuronal activation in the prefrontal cortex and amygdala.

When faced with an inescapable stressor, animals may engage in 'coping' behaviors, such as chewing inedible objects, that attenuate some physiological responses to the stressor. Previous evidence indicates that dopamine neurotransmission in the right prefrontal cortex is modulated by coping processes. Here we tested whether medial prefrontal cortical (mPFC) neuronal activation, as measured by Fos-immunoreactivity (Fos-ir), was altered in rats chewing inedible objects during exposure to novelty stress. We found that chewing caused an increase in Fos-ir that was selective for the right hemisphere of the mPFC along with a decrease in Fos-ir that was selective for the right central nucleus of the amygdala (CeA), a region that may regulate dopamine neurotransmission in mPFC. These observations suggest that coping during stress engages mPFC and CeA neuronal activity asymmetrically.

Expectancy-Related Changes in Dopaminergic Error Signals Are Impaired by Cocaine Self-Administration.

Addiction is a disorder of behavioral control and learning. While this may reflect pre-existing propensities, drug use also clearly contributes by causing changes in outcome processing in prefrontal and striatal regions. This altered processing is associated with behavioral deficits, including changes in learning. These areas provide critical input to midbrain dopamine neurons regarding expected outcomes, suggesting that effects on learning may result from changes in dopaminergic error signaling. Here, we show that dopamine neurons recorded in rats that had self-administered cocaine failed to suppress firing on omission of an expected reward and exhibited lower amplitude and imprecisely timed increases in firing to an unexpected reward. Learning also appeared to have less of an effect on reward-evoked and cue-evoked firing in the cocaine-experienced rats. Overall, the changes are consistent with reduced fidelity of input regarding the expected outcomes, such as their size, timing, and overall value, because of cocaine use.

Dopamine neuron ensembles signal the content of sensory prediction errors.

Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in the activity of individual neurons or population averages. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

Rat Orbitofrontal Ensemble Activity Contains Multiplexed but Dissociable Representations of Value and Task Structure in an Odor Sequence Task.

The orbitofrontal cortex (OFC) has long been implicated in signaling information about expected outcomes to facilitate adaptive or flexible behavior. Current proposals focus on signaling of expected value versus the representation of a value-agnostic cognitive map of the task. While often suggested as mutually exclusive, these alternatives may represent extreme ends of a continuum determined by task complexity and experience. As learning proceeds, an initial, detailed cognitive map might be acquired, based largely on external information. With more experience, this hypothesized map can then be tailored to include relevant abstract hidden cognitive constructs. The map would default to an expected value in situations where other attributes are largely irrelevant, but, in richer tasks, a more detailed structure might continue to be represented, at least where relevant to behavior. Here, we examined this by recording single-unit activity from the OFC in rats navigating an odor sequence task analogous to a spatial maze. The odor sequences provided a mappable state space, with 24 unique "positions" defined by sensory information, likelihood of reward, or both. Consistent with the hypothesis that the OFC represents a cognitive map tailored to the subjects' intentions or plans, we found a close correspondence between how subjects were using the sequences and the neural representations of the sequences in OFC ensembles. Multiplexed with this value-invariant representation of the task, we also found a representation of the expected value at each location. Thus, the value and task structure co-existed as dissociable components of the neural code in OFC.

Orbitofrontal cortex, decision-making and drug addiction.

The orbitofrontal cortex, as a part of prefrontal cortex, is implicated in executive function. However, within this broad region, the orbitofrontal cortex is distinguished by its unique pattern of connections with crucial subcortical associative learning nodes, such as basolateral amygdala and nucleus accumbens. By virtue of these connections, the orbitofrontal cortex is uniquely positioned to use associative information to project into the future, and to use the value of perceived or expected outcomes to guide decisions. This review will discuss recent evidence that supports this proposal and will examine evidence that loss of this signal, as the result of drug-induced changes in these brain circuits, might account for the maladaptive decision-making that characterizes drug addiction.

Withdrawal from cocaine self-administration produces long-lasting deficits in orbitofrontal-dependent reversal learning in rats.

Drug addicts make poor decisions. These decision-making deficits have been modeled in addicts and laboratory animals using reversal-learning tasks. However, persistent reversal-learning impairments have been shown in rats and monkeys only after noncontingent cocaine injections. Current thinking holds that to represent the human condition effectively, animal models of addiction must utilize self-administration procedures in which drug is earned contingently; thus, it remains unclear whether reversal-learning deficits caused by noncontingent cocaine exposure are relevant to addiction. To test whether reversal learning deficits are caused by contingent cocaine exposure, we trained rats to self-administer cocaine, assessed cue-induced cocaine seeking in extinction tests after 1 and 30 d of withdrawal, and then tested for reversal learning more than a month later. We found robust time-dependent increases in cue-induced cocaine seeking in the two extinction tests (incubation of craving) and severe reversal-learning impairments.

Reconciling the roles of orbitofrontal cortex in reversal learning and the encoding of outcome expectancies.

Damage to orbitofrontal cortex (OFC) has long been associated with decision-making deficits. Such deficits are epitomized by impairments in reversal learning. Historically, reversal learning deficits have been linked to a response inhibition function or to the rapid reversal of associative encoding in OFC neurons. However here we will suggest that OFC supports reversal learning not because its encoding is particularly flexible-indeed it actually is not-but rather because output from OFC is critical for flexible associative encoding downstream in basolateral amygdala (ABL). Consistent with this argument, we will show that reversal performance is actually inversely related to the flexibility of associative encoding in OFC (i.e., the better the reversal performance, the less flexible the encoding). Further, we will demonstrate that associative correlates in ABL are more flexible during reversal learning than in OFC, become less flexible after damage to OFC, and are required for the expression of the reversal deficit caused by OFC lesions. We will propose that OFC facilitates associative flexibility in downstream regions, such as ABL, for the same reason that it is critical for outcome-guided behavior in a variety of setting-namely that processing in OFC signals the value of expected outcomes. In addition to their role in guiding behavior, these outcome expectancies permit the rapid recognition of unexpected outcomes, thereby driving new learning.