Evolving schema representations in orbitofrontal ensembles during learning.

How do we learn about what to learn about? Specifically, how do the neural elements in our brain generalize what has been learned in one situation to recognize the common structure of-and speed learning in-other, similar situations? We know this happens because we become better at solving new problems-learning and deploying schemas1-5-through experience. However, we have little insight into this process. Here we show that using prior knowledge to facilitate learning is accompanied by the evolution of a neural schema in the orbitofrontal cortex. Single units were recorded from rats deploying a schema to learn a succession of odour-sequence problems. With learning, orbitofrontal cortex ensembles converged onto a low-dimensional neural code across both problems and subjects; this neural code represented the common structure of the problems and its evolution accelerated across their learning. These results demonstrate the formation and use of a schema in a prefrontal brain region to support a complex cognitive operation. Our results not only reveal a role for the orbitofrontal cortex in learning but also have implications for using ensemble analyses to tap into complex cognitive functions.

Spatial Representations in Rat Orbitofrontal Cortex.

The orbitofrontal cortex (OFC) and hippocampus share striking cognitive and functional similarities. As a result, both structures have been proposed to encode "cognitive maps" that provide useful scaffolds for planning complex behaviors. However, while this function has been exemplified by spatial coding in neurons of hippocampal regions-particularly place and grid cells-spatial representations in the OFC have been investigated far less. Here we sought to address this by recording OFC neurons from male rats engaged in an open-field foraging task like that originally developed to characterize place fields in rodent hippocampal neurons. Single-unit activity was recorded as rats searched for food pellets scattered randomly throughout a large enclosure. In some sessions, particular flavors of food occurred more frequently in particular parts of the enclosure; in others, only a single flavor was used. OFC neurons showed spatially localized firing fields in both conditions, and representations changed between flavored and unflavored foraging periods in a manner reminiscent of remapping in the hippocampus. Compared with hippocampal recordings taken under similar behavioral conditions, OFC spatial representations were less temporally reliable, and there was no significant evidence of grid tuning in OFC neurons. These data confirm that OFC neurons show spatial firing fields in a large, two-dimensional environment in a manner similar to hippocampus. Consistent with the focus of the OFC on biological meaning and goals, spatial coding was weaker than in hippocampus and influenced by outcome identity.SIGNIFICANCE STATEMENT The orbitofrontal cortex (OFC) and hippocampus have both been proposed to encode "cognitive maps" that provide useful scaffolds for planning complex behaviors. This function is exemplified by place and grid cells identified in hippocampus, the activity of which maps spatial environments. The current study directly demonstrates very similar, though not identical, spatial representatives in OFC neurons, confirming that OFC-like hippocampus-can represent a spatial map under the appropriate experimental conditions.

Brief, But Not Prolonged, Pauses in the Firing of Midbrain Dopamine Neurons Are Sufficient to Produce a Conditioned Inhibitor.

Prediction errors are critical for associative learning. In the brain, these errors are thought to be signaled, in part, by midbrain dopamine neurons. However, although there is substantial direct evidence that brief increases in the firing of these neurons can mimic positive prediction errors, there is less evidence that brief pauses mimic negative errors. Whereas pauses in the firing of midbrain dopamine neurons can substitute for missing negative prediction errors to drive extinction, it has been suggested that this effect might be attributable to changes in salience rather than the operation of this signal as a negative prediction error. Here we address this concern by showing that the same pattern of inhibition will create a cue able to meet the classic definition of a conditioned inhibitor by showing suppression of responding in a summation test and slower learning in a retardation test. Importantly, these classic criteria were designed to rule out explanations founded on attention or salience; thus the results cannot be explained in this manner. We also show that this pattern of behavior is not produced by a single, prolonged, ramped period of inhibition, suggesting that it is precisely timed, sudden change and not duration that conveys the teaching signal.SIGNIFICANCE STATEMENT Here we show that brief pauses in the firing of midbrain dopamine neurons are sufficient to produce a cue that meets the classic criteria defining a conditioned inhibitor, or a cue that predicts the omission of a reward. These criteria were developed to distinguish actual learning from salience or attentional effects; thus these results formally show that brief pauses in the firing of dopamine neurons can serve as key teaching signals in the brain. Interestingly, this was not true for gradual prolonged pauses, suggesting it is the dynamic change in firing that serves as the teaching signal.

Rethinking dopamine as generalized prediction error.

Midbrain dopamine neurons are commonly thought to report a reward prediction error (RPE), as hypothesized by reinforcement learning (RL) theory. While this theory has been highly successful, several lines of evidence suggest that dopamine activity also encodes sensory prediction errors unrelated to reward. Here, we develop a new theory of dopamine function that embraces a broader conceptualization of prediction errors. By signalling errors in both sensory and reward predictions, dopamine supports a form of RL that lies between model-based and model-free algorithms. This account remains consistent with current canon regarding the correspondence between dopamine transients and RPEs, while also accounting for new data suggesting a role for these signals in phenomena such as sensory preconditioning and identity unblocking, which ostensibly draw upon knowledge beyond reward predictions.

Neural Estimates of Imagined Outcomes in Basolateral Amygdala Depend on Orbitofrontal Cortex.

Reciprocal connections between the orbitofrontal cortex (OFC) and the basolateral nucleus of the amygdala (BLA) provide a critical circuit for guiding normal behavior when information about expected outcomes is required. Recently, we reported that outcome signaling by OFC neurons is also necessary for learning in the face of unexpected outcomes during a Pavlovian over-expectation task. Key to learning in this task is the ability to build on prior learning to infer or estimate an amount of reward never previously received. OFC was critical to this process. Notably, in parallel work, we found that BLA was not necessary for learning in this setting. This suggested a dissociation in which the BLA might be critical for acquiring information about the outcomes but not for subsequently using it to make novel predictions. Here we evaluated this hypothesis by recording single-unit activity from BLA in rats during the same Pavlovian over-expectation task used previously. We found that spiking activity recorded in BLA in control rats did reflect novel outcome estimates derived from the integration of prior learning, however consistent with a model in which this process occurs in the OFC, these correlates were entirely abolished by ipsilateral OFC lesions. These data indicate that this information about these novel predictions is represented in the BLA, supported via direct or indirect input from the OFC, even though it does not appear to be necessary for learning. SIGNIFICANCE STATEMENT: The basolateral nucleus of the amygdala (BLA) and the orbitofrontal cortex (OFC) are involved in behavior that depends on knowledge of impending outcomes. Recently, we found that only the OFC was necessary for using such information for learning in a Pavlovian over-expectation task. The current experiment was designed to search for neural correlates of this process in the BLA and, if present, to ask whether they would still be dependent on OFC input. We found that although spiking activity in BLA in control rats did reflect the novel outcome estimates underlying learning, these correlates were entirely abolished by OFC lesions.

Encoding and tracking of outcome-specific expectancy in the gustatory cortex of alert rats.

In natural conditions, gustatory stimuli are typically expected. Anticipatory and contextual cues provide information that allows animals to predict the availability and the identity of the substance to be ingested. Recording in alert rats trained to self-administer tastants following a go signal revealed that neurons in the primary gustatory cortex (GC) can respond to anticipatory cues. These experiments were optimized to demonstrate that even the most general form of expectation can activate neurons in GC, and did not provide indications on whether cues predicting different tastants could be encoded selectively by GC neurons. Here we recorded single-neuron activity in GC of rats engaged in a task where one auditory cue predicted sucrose, while another predicted quinine. We found that GC neurons respond differentially to the two cues. Cue-selective responses develop in parallel with learning. Comparison between cue and sucrose responses revealed that cues could trigger the activation of anticipatory representations. Additional experiments showed that an expectation of sucrose leads a subset of neurons to produce sucrose-like responses even when the tastant was omitted. Altogether, the data show that primary sensory cortices can encode for cues predicting different outcomes, and that specific expectations result in the activation of anticipatory representations.

Thalamic contribution to cortical processing of taste and expectation.

Taste-related information reaches the gustatory cortex (GC) through two routes: a thalamic and a limbic pathway. While evidence is accumulating on limbic-cortical interactions in taste, very little information is available on the function of the gustatory thalamus in shaping GC activity. Here we rely on behavioral electrophysiological techniques to study taste-evoked activity in GC before and after inactivation of the parvicellular portion of the ventroposteromedial nucleus of thalamus (VPMpc; i.e., the gustatory thalamus). Gustatory stimuli were presented to rats either alone or preceded by an anticipatory cue. The reliance on two different behavioral contexts allowed us to investigate how the VPMpc mediates GC responses to uncued tastants, cued tastants, and anticipatory cues. Inactivation of the thalamus resulted in a dramatic reduction of taste processing in GC. However, responses to anticipatory cues were unaffected by this manipulation. The use of a cue-taste association paradigm also allowed for the identification of two subpopulations of taste-specific neurons: those that responded to gustatory stimulation and to the cue (i.e., cue-and-taste) and those that responded to tastants only (i.e., taste-only). Analyses of these two populations revealed differences in response dynamics and connectivity with the VPMpc. The results provide novel evidence for the role of VPMpc in shaping GC activity and demonstrate a previously unknown association between responsiveness to behavioral events, temporal dynamics, and thalamic connectivity in GC.

The role of the lateral orbitofrontal cortex in creating cognitive maps.

We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.

Anterior cingulate neurons signal neutral cue pairings during sensory preconditioning.

Of all frontocortical subregions, the anterior cingulate cortex (ACC) has perhaps the most overlapping theories of function.1-3 Recording studies in rats, humans, and other primates have reported diverse neural responses that support many theories,4-12 yet nearly all these studies have in common tasks in which one event reliably predicts another. This leaves open the possibility that ACC represents associative pairing of events, independent of their overt biological significance. Sensory preconditioning13 provides an opportunity to test this. In the first phase, preconditioning, value-neutral sensory stimuli are paired (A→B). To test whether this was learned, subjects are given standard conditioning during which one of the previously neutral sensory cues is paired with a biologically meaningful outcome (B→outcome). During the final probe test, the neutral cue which was never paired with a biologically meaningful outcome is presented alone (A→) and will elicit a conditional response, suggesting that subjects had learned the associative structure during preconditioning and use that knowledge to infer presentation of the biologically relevant outcome (A→B→outcome). Inference-based responding demonstrates a fundamental property of model-based reasoning14,15 and requires learning of the associations between neutral stimuli before rewards are introduced.16-19 ACC neurons developed firing patterns that reflected the learning of sensory associations during preconditioning, even though no rewards were present. The strength of these correlates predicted rats' ability to later mobilize and use that associative information during the probe test. These results demonstrate that clear biological significance is not necessary to produce correlates of learning in ACC.

Calcium activity is a degraded estimate of spikes.

Recording action potentials extracellularly during behavior has led to fundamental discoveries regarding neural function-hippocampal neurons respond to locations in space,1 motor cortex neurons encode movement direction,2 and dopamine neurons signal reward prediction errors3-observations undergirding current theories of cognition,4 movement,5 and learning.6 Recently it has become possible to measure calcium flux, an internal cellular signal related to spiking. The ability to image calcium flux in anatomically7,8 or genetically9 identified neurons can extend our knowledge of neural circuit function by allowing activity to be monitored in specific cell types or projections, or in the same neurons across many days. However, while initial studies were grounded in prior unit recording work, it has become fashionable to assume that calcium is identical to spiking, even though the spike-to-fluorescence transformation is nonlinear, noisy, and unpredictable under real-world conditions.10 It remains an open question whether calcium provides a high-fidelity representation of single-unit activity in awake, behaving subjects. Here, we have addressed this question by recording both signals in the lateral orbitofrontal cortex (OFC) of rats during olfactory discrimination learning. Activity in the OFC during olfactory learning has been well-studied in humans,11,12,13,14 nonhuman primates,15,16 and rats,17,18,19,20,21 where it has been shown to signal information about both the sensory properties of odor cues and the rewards they predict. Our single-unit results replicated prior findings, whereas the calcium signal provided only a degraded estimate of the information available in the single-unit spiking, reflecting primarily reward value.

The orbitofrontal cartographer.

Theories of orbitofrontal cortex (OFC) function have evolved substantially over the last few decades. There is now a general consensus that the OFC is important for predicting aspects of future events and for using these predictions to guide behavior. Yet the precise content of these predictions and the degree to which OFC contributes to agency contingent upon them has become contentious, with several plausible theories advocating different answers to these questions. In this review we will focus on three of these ideas-the economic value, credit assignment, and cognitive map hypotheses-describing both their successes and failures. We will propose that these failures hint at a more nuanced and perhaps unique role for the OFC, particularly the lateral subdivision, in supporting the proposed functions when an underlying model or map of the causal structures in the environment must be constructed or updated. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Is the core function of orbitofrontal cortex to signal values or make predictions?

One dominant hypothesis about the function of the orbitofrontal cortex (OFC) is that the OFC signals the subjective values of possible outcomes to other brain areas for learning and decision making. This popular view generally neglects the fact that OFC is not necessary for simple value-based behavior (i.e., when values have been directly experienced). An alternative, emerging view suggests that OFC plays a more general role in representing structural information about the task or environment, derived from prior experience, and relevant to predicting behavioral outcomes, such as value. From this perspective, value signaling is simply one derivative of the core underlying function of OFC. New data in favor of both views have been accumulating rapidly. Here we review these new data in discussing the relative merits of these two ideas.

Prospective representations in rat orbitofrontal ensembles.

The orbitofrontal cortex (OFC) has been proposed to encode expected outcomes, which is thought to be important for outcome-directed behavior. However, such neural encoding can also often be explained by the recall of information about the recent past. To dissociate the retrospective and prospective aspects of encoding in the OFC, we designed a nonspatial, continuous, alternating odor-sequence task that mimicked a continuous T-maze. The task consisted of two alternating sequences of four odor-guided trials (2 sequences × 4 positions). In each trial, rats were asked to make a "go" or "no-go" action based on a fixed odor-reward contingency. Odors at both the first and last positions were distinct across the two sequences, such that they resembled unique paths in the past and future, respectively; odors at positions in between were the same and thus resembled a common path. We trained classifiers using neural activity to distinguish between either sequences or positions and asked whether the neural activity patterns in the common path were more like the ones in the past or the future. We found a proximal prospective code for sequence information as well as a distal perspective code for positional information, the latter of which was closely associated with rats' ability to predict future outcomes. This study demonstrates a behaviorally relevant predictive code in rat OFC. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Past experience shapes the neural circuits recruited for future learning.

Experimental research controls for past experience, yet prior experience influences how we learn. Here, we tested whether we could recruit a neural population that usually encodes rewards to encode aversive events. Specifically, we found that GABAergic neurons in the lateral hypothalamus (LH) were not involved in learning about fear in naïve rats. However, if these rats had prior experience with rewards, LH GABAergic neurons became important for learning about fear. Interestingly, inhibition of these neurons paradoxically enhanced learning about neutral sensory information, regardless of prior experience, suggesting that LH GABAergic neurons normally oppose learning about irrelevant information. These experiments suggest that prior experience shapes the neural circuits recruited for future learning in a highly specific manner, reopening the neural boundaries we have drawn for learning of particular types of information from work in naïve subjects.

Neuroscience: From Sensory Discrimination to Choice in Gustatory Cortex.

Sensory areas have been shown to be influenced by higher-order cognitive processes. Yet how do these top-down processes affect decisions? A recent study has revealed a dynamic evolution of neural activity from sensory discrimination to choice in rodent taste cortex.

Processing in Lateral Orbitofrontal Cortex Is Required to Estimate Subjective Preference during Initial, but Not Established, Economic Choice.

The orbitofrontal cortex (OFC) is proposed to be critical to economic decision making. Yet one can inactivate OFC without affecting well-practiced choices. One possible explanation of this lack of effect is that well-practiced decisions are codified into habits or configural-based policies not normally thought to require OFC. Here, we tested this idea by training rats to choose between different pellet pairs across a set of standard offers and then inactivating OFC subregions during choices between novel offers of previously experienced pairs or between novel pairs of previously experienced pellets. Contrary to expectations, controls performed as well on novel as experienced offers yet had difficulty initially estimating their subjective preference on novel pairs, difficulty exacerbated by lateral OFC inactivation. This pattern of results indicates that established economic choice reflects the use of an underlying model or goods space and that lateral OFC is only required for normal behavior when the established framework must incorporate new information.

Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors.

Reward-evoked dopamine transients are well established as prediction errors. However, the central tenet of temporal difference accounts-that similar transients evoked by reward-predictive cues also function as errors-remains untested. In the present communication we addressed this by showing that optogenetically shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results indicate that cue-evoked transients function as temporal-difference prediction errors rather than reward predictions.

Real-Time Value Integration during Economic Choice Is Regulated by Orbitofrontal Cortex.

Neural correlates implicate the orbitofrontal cortex (OFC) in value-based or economic decision making [1-3]. Yet inactivation of OFC in rats performing a rodent version of the standard economic choice task is without effect [4, 5], a finding more in accord with ideas that the OFC is primarily necessary for behavior when new information must be taken into account [6-9]. Neural activity in the OFC spontaneously updates to reflect new information, particularly about outcomes [10-16], and the OFC is necessary for adjustments to learned behavior only under these conditions [4, 16-26]. Here, we merge these two independent lines of research by inactivating lateral OFC during an economic choice that requires new information about the value of the predicted outcomes to be incorporated into an already established choice. Outcome value was changed by pre-feeding the rats one of two food options before testing. In control rats, this pre-feeding resulted in divergent changes in choice behavior that depended on the rats' prior preference for the pre-fed food. Optogenetic inactivation of the OFC disrupted this bi-directional effect of pre-feeding without affecting other measures that describe the underlying choice behavior. This finding unifies the role of the OFC in economic choice with its role in a host of other behaviors, causally demonstrating that the OFC is not necessary for economic choice per se-unless that choice incorporates new information about the outcomes.

Complementary Task Structure Representations in Hippocampus and Orbitofrontal Cortex during an Odor Sequence Task.

Both hippocampus (HPC) and orbitofrontal cortex (OFC) have been shown to be critical for behavioral tasks that require use of an internal model or cognitive map, composed of the states and the relationships between them, which define the current environment or task at hand. One general idea is that the HPC provides the cognitive map, which is then transformed by OFC to emphasize information of relevance to current goals. Our previous analysis of ensemble activity in OFC in rats performing an odor sequence task revealed a rich representation of behaviorally relevant task structure, consistent with this proposal. Here, we compared those data to recordings from single units in area CA1 of the HPC of rats performing the same task. Contrary to expectations that HPC ensembles would represent detailed, even incidental, information defining the full task space, we found that HPC ensembles-like those in OFC-failed to distinguish states when it was not behaviorally necessary. However, hippocampal ensembles were better than those in OFC at distinguishing task states in which prospective memory was necessary for future performance. These results suggest that, in familiar environments, the HPC and OFC may play complementary roles, with the OFC maintaining the subjects' current position on the cognitive map or state space, supported by HPC when memory demands are high.

Rat Orbitofrontal Ensemble Activity Contains Multiplexed but Dissociable Representations of Value and Task Structure in an Odor Sequence Task.

The orbitofrontal cortex (OFC) has long been implicated in signaling information about expected outcomes to facilitate adaptive or flexible behavior. Current proposals focus on signaling of expected value versus the representation of a value-agnostic cognitive map of the task. While often suggested as mutually exclusive, these alternatives may represent extreme ends of a continuum determined by task complexity and experience. As learning proceeds, an initial, detailed cognitive map might be acquired, based largely on external information. With more experience, this hypothesized map can then be tailored to include relevant abstract hidden cognitive constructs. The map would default to an expected value in situations where other attributes are largely irrelevant, but, in richer tasks, a more detailed structure might continue to be represented, at least where relevant to behavior. Here, we examined this by recording single-unit activity from the OFC in rats navigating an odor sequence task analogous to a spatial maze. The odor sequences provided a mappable state space, with 24 unique "positions" defined by sensory information, likelihood of reward, or both. Consistent with the hypothesis that the OFC represents a cognitive map tailored to the subjects' intentions or plans, we found a close correspondence between how subjects were using the sequences and the neural representations of the sequences in OFC ensembles. Multiplexed with this value-invariant representation of the task, we also found a representation of the expected value at each location. Thus, the value and task structure co-existed as dissociable components of the neural code in OFC.