Dopamine Release in the Nucleus Accumbens Core Encodes the General Excitatory Components of Learning.

Dopamine release in the nucleus accumbens core (NAcC) is generally considered to be a proxy for phasic firing of dopamine neurons in the ventral tegmental area (VTADA). Thus, dopamine release in NAcC is hypothesized to reflect a unitary role in reward prediction error signalling. However, recent studies revealed more diverse roles of dopamine neurons, which support an emerging idea that dopamine regulates learning differently in distinct circuits. To understand whether the NAcC might regulate a unique component of learning, we recorded dopamine release in NAcC while male rats performed a backward conditioning task where a reward is followed by a cue. We used this task because we can delineate different components of learning, which include sensory-specific inhibitory and general excitatory components. Further, we have shown that VTADA neurons are necessary for both the specific and general components of backward associations. Here, we found that dopamine release in NAcC increased to the reward across learning, while reducing to the cue that followed as it became more expected. This mirrors the dopamine prediction error signal seen during forward conditioning and cannot be accounted for temporal-difference reinforcement learning (TDRL). Subsequent tests allowed us to dissociate these learning components and revealed that dopamine release in NAcC reflects the general excitatory component of backward associations, but not their sensory-specific component. These results emphasize the importance of examining distinct functions of different dopamine projections in reinforcement learning.Significance Statement Dopamine regulates reinforcement learning. While it was previously believed that this system contributed to simple value assignment to reward cues, we now know dopamine plays increasingly diverse roles in reinforcement learning. How these diverse roles are achieved in distinct circuits is not fully understood. By using behavioural tasks that examine distinctive components of learning separately, we reveal that NAcC dopamine release contributes to a unique component of learning. Thus, the present study supports a distinct role of NAcC in reinforcement learning, consistent with the idea that different dopamine systems serve different learning functions. Examining the roles of different dopamine projections is an important approach to identify neuronal mechanisms underlying the reinforcement-learning deficits observed in schizophrenia and drug addiction.

Making Sense of the Multiplicity and Dynamics of Navigational Codes in the Brain.

Since the discovery of conspicuously spatially tuned neurons in the hippocampal formation over 50 years ago, characterizing which, where, and how neurons encode navigationally relevant variables has been a major thrust of navigational neuroscience. While much of this effort has centered on the hippocampal formation and functionally-adjacent structures, recent work suggests that spatial codes, in some form or another, can be found throughout the brain, even in areas traditionally associated with sensation, movement, and executive function. In this review, we highlight these unexpected results, draw insights from comparison of these codes across contexts, regions, and species, and finally suggest an avenue for future work to make sense of these diverse and dynamic navigational codes.

Spatial Representations in Rat Orbitofrontal Cortex.

The orbitofrontal cortex (OFC) and hippocampus share striking cognitive and functional similarities. As a result, both structures have been proposed to encode "cognitive maps" that provide useful scaffolds for planning complex behaviors. However, while this function has been exemplified by spatial coding in neurons of hippocampal regions-particularly place and grid cells-spatial representations in the OFC have been investigated far less. Here we sought to address this by recording OFC neurons from male rats engaged in an open-field foraging task like that originally developed to characterize place fields in rodent hippocampal neurons. Single-unit activity was recorded as rats searched for food pellets scattered randomly throughout a large enclosure. In some sessions, particular flavors of food occurred more frequently in particular parts of the enclosure; in others, only a single flavor was used. OFC neurons showed spatially localized firing fields in both conditions, and representations changed between flavored and unflavored foraging periods in a manner reminiscent of remapping in the hippocampus. Compared with hippocampal recordings taken under similar behavioral conditions, OFC spatial representations were less temporally reliable, and there was no significant evidence of grid tuning in OFC neurons. These data confirm that OFC neurons show spatial firing fields in a large, two-dimensional environment in a manner similar to hippocampus. Consistent with the focus of the OFC on biological meaning and goals, spatial coding was weaker than in hippocampus and influenced by outcome identity.SIGNIFICANCE STATEMENT The orbitofrontal cortex (OFC) and hippocampus have both been proposed to encode "cognitive maps" that provide useful scaffolds for planning complex behaviors. This function is exemplified by place and grid cells identified in hippocampus, the activity of which maps spatial environments. The current study directly demonstrates very similar, though not identical, spatial representatives in OFC neurons, confirming that OFC-like hippocampus-can represent a spatial map under the appropriate experimental conditions.

Prior Cocaine Use Alters the Normal Evolution of Information Coding in Striatal Ensembles during Value-Guided Decision-Making.

Substance use disorders (SUDs) are characterized by maladaptive behavior. The ability to properly adjust behavior according to changes in environmental contingencies necessitates the interlacing of existing memories with updated information. This can be achieved by assigning learning in different contexts to compartmentalized "states." Though not often framed this way, the maladaptive behavior observed in individuals with SUDs may result from a failure to properly encode states because of drug-induced neural alterations. Previous studies found that the dorsomedial striatum (DMS) is important for behavioral flexibility and state encoding, suggesting the DMS may be an important substrate for these effects. Here, we recorded DMS neural activity in cocaine-experienced male rats during a decision-making task where blocks of trials represented distinct states to probe whether the encoding of state and state-related information is affected by prior drug exposure. We found that DMS medium spiny neurons (MSNs) and fast-spiking interneurons (FSIs) encoded such information and that prior cocaine experience disrupted the evolution of representations both within trials and across recording sessions. Specifically, DMS MSNs and FSIs from cocaine-experienced rats demonstrated higher classification accuracy of trial-specific rules, defined by response direction and value, compared with those drawn from sucrose-experienced rats, and these overly strengthened trial-type representations were related to slower switching behavior and reaction times. These data show that prior cocaine experience paradoxically increases the encoding of state-specific information and rules in the DMS and suggest a model in which abnormally specific and persistent representation of rules throughout trials in DMS slows value-based decision-making in well trained subjects.SIGNIFICANCE STATEMENT Substance use disorders (SUDs) may result from a failure to properly encode rules guiding situationally appropriate behavior. The dorsomedial striatum (DMS) is thought to be important for such behavioral flexibility and encoding that defines the situation or "state." This suggests that the DMS may be an important substrate for the maladaptive behavior observed in SUDs. In the current study, we show that prior cocaine experience results in over-encoding of state-specific information and rules in the DMS, which may impair normal adaptive decision-making in the task, akin to what is observed in SUDs.

Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex.

The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.

Subjective costs drive overly patient foraging strategies in rats on an intertemporal foraging task.

Laboratory studies of decision making often take the form of two-alternative, forced-choice paradigms. In natural settings, however, many decision problems arise as stay/go choices. We designed a foraging task to test intertemporal decision making in rats via stay/go decisions. Subjects did not follow the rate-maximizing strategy of choosing only food items associated with short delays. Instead, rats were often willing to wait for surprisingly long periods, and consequently earned a lower rate of food intake than they might have by ignoring long-delay options. We tested whether foraging theory or delay discounting models predicted the behavior we observed but found that these models could not account for the strategies subjects selected. Subjects' behavior was well accounted for by a model that incorporated a cost for rejecting potential food items. Interestingly, subjects' cost sensitivity was proportional to environmental richness. These findings are at odds with traditional normative accounts of decision making but are consistent with retrospective considerations having a deleterious influence on decisions (as in the "sunk-cost" effect). More broadly, these findings highlight the utility of complementing existing assays of decision making with tasks that mimic more natural decision topologies.

The balance of forward and backward hippocampal sequences shifts across behavioral states.

Place cell firing patterns in the rat hippocampus are often organized as sequences. Sequences falling within cycles of the theta (6-10 Hz) local field potential (LFP) oscillation represent segments of ongoing behavioral trajectories. Sequences expressed during sharp wave ripple (SWR) complexes represent spatial trajectories through the environment, in both the same direction as actual trajectories (forward sequences) and in an ordering opposite that of behavior (backward sequences). Although hippocampal sequences could fulfill unique functional roles depending on the direction of the sequence and the animal's state when the sequence occurs, quantitative comparisons of sequence direction across behavioral and physiological states within the same experiment, employing consistent methodology, are lacking. Here, we used cross-correlation and Bayesian decoding to measure the direction of hippocampal sequences in rats during active behavior, awake rest and slow-wave sleep. During pretask sleep, few sequences were detected in either direction. Sequences within theta cycles during active behavior were overwhelmingly forward. Sequences during quiescent moments of behavior were both forward and backward, in equal proportion. During postbehavior sleep, sequences were again expressed in both directions, but significantly more forward than backward sequences were detected. The shift in the balance of sequence direction could reflect changing functional demands on the hippocampal network across behavioral and physiological states.

Sex differences in patch-leaving foraging decisions in rats.

The ubiquity, importance, and sophistication of foraging behavior makes it an ideal platform for studying naturalistic decision making in animals. We developed a spatial patch-foraging task for rats, in which subjects chose how long to remain in one foraging patch as the rate of food earnings steadily decreased. The cost of seeking out a new location was varied across sessions. The behavioral task was designed to mimic the structure of natural foraging problems, where distinct spatial locations are associated with different reward statistics, and decisions require navigation and movement through space. Male and female Long-Evans rats generally followed the predictions of theoretical models of foraging, albeit with a consistent tendency to persist with patches for too long compared to behavioral strategies that maximize food intake rate. The tendency to choose overly-long patch residence times was stronger in male rats. We also observed sex differences in locomotion as rats performed the task, but these differences in movement only partially accounted for the differences in patch residence durations observed between male and female rats. Together, these results suggest a nuanced relationship between movement, sex, and foraging decisions.

Sex differences in patch-leaving foraging decisions in rats.

The ubiquity, importance, and sophistication of foraging behavior makes it an ideal platform for studying naturalistic decision making in animals. We developed a spatial patch-foraging task for rats, in which subjects chose how long to remain in one foraging patch as the rate of food earnings steadily decreased. The cost of seeking out a new location was varied across sessions. The behavioral task was designed to mimic the structure of natural foraging problems, where distinct spatial locations are associated with different reward statistics, and decisions require navigation and movement through space. Male and female Long-Evans rats generally followed the predictions of theoretical models of foraging, albeit with a consistent tendency to persist with patches for too long compared to behavioral strategies that maximize food intake rate. The tendency to choose overly-long patch residence times was stronger in male rats. We also observed sex differences in locomotion as rats performed the task, but these differences in movement only partially accounted for the differences in patch residence durations observed between male and female rats. Together, these results suggest a nuanced relationship between movement, sex, and foraging decisions.

Theta oscillations in anterior cingulate cortex and orbitofrontal cortex differentially modulate accuracy and speed in flexible reward learning.

Flexible reward learning relies on frontal cortex, with substantial evidence indicating that anterior cingulate cortex (ACC) and orbitofrontal cortex (OFC) subregions play important roles. Recent studies in both rat and macaque suggest theta oscillations (5-10 Hz) may be a spectral signature that coordinates this learning. However, network-level interactions between ACC and OFC in flexible learning remain unclear. We investigated the learning of stimulus-reward associations using a combination of simultaneous in vivo electrophysiology in dorsal ACC and ventral OFC, partnered with bilateral inhibitory DREADDs in ACC. In freely behaving male and female rats and using a within-subject design, we examined accuracy and speed of response across distinct and precisely defined trial epochs during initial visual discrimination learning and subsequent reversal of stimulus-reward contingencies. Following ACC inhibition, there was a propensity for random responding in early reversal learning, with correct vs. incorrect trials distinguished only from OFC, not ACC, theta power differences in the reversal phase. ACC inhibition also hastened incorrect choices during reversal. This same pattern of change in accuracy and speed was not observed in viral control animals. Thus, characteristics of impaired reversal learning following ACC inhibition are poor deliberation and weak theta signaling of accuracy in this region. The present results also point to OFC theta oscillations as a prominent feature of reversal learning, unperturbed by ACC inhibition.

Quitting while you're ahead: Patch foraging and temporal cognition.

Theoretical models of foraging are based on the maximization of food intake rate. Remarkably, foragers often hew close to the predictions of rate maximization, except for a frequently observed bias to remain in patches for too long. By sticking with depleting options beyond the optimal patch residence time-a phenomenon referred to as overharvesting or overstaying-foragers miss out on food they could have earned had they sought a new option elsewhere. Here, we review potential causes of overstaying and consider the role that temporal cognition might play in this phenomenon. We first consider how an explicit, internal sense of time might inform foraging behaviors, and next examine patch-leaving choices from the perspective of intertemporal decision-making. Finally, we identify promising areas for future research that will provide a better understanding of how foraging decisions are made, and what factors drive the tendency to overharvest patches. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Changes in reward contingency modulate the trial-to-trial variability of hippocampal place cells.

Pyramidal cells in the rodent hippocampus often exhibit clear spatial tuning. Theories of hippocampal function suggest that these "place cells" implement multiple, independent neural representations of position (maps), based on different reference frames or environmental features. Consistent with the "multiple maps" theory, previous studies have shown that manipulating spatial factors related to task performance modulates the within-session variability (overdispersion) of cells in the hippocampus. However, the influence of changes in reward contingency on overdispersion has not been examined. To test this, we first trained rats to collect food from three feeders positioned around a circular track (task(1)). When subjects were proficient, the reward contingency was altered such that every other feeder delivered food (task(2)). We recorded ensembles of hippocampal neurons as rats performed both tasks. Place cell overdispersion was high during task(1) but decreased significantly during task(2), and this increased reliability could not be accounted for by changes in running speed or familiarity with the task. Intuitively, decreased variability might be expected to improve neural representations of position. To test this, we used Bayesian decoding of hippocampal spike trains to estimate subjects' location. Neither the amount of probability decoded to subjects' position (local probability) nor the difference between estimated position and true location (decoding accuracy) differed between tasks. However, we found that hippocampal ensembles were significantly more self-consistent during task(2) performance. These results suggest that changes in task demands can affect the firing statistics of hippocampal neurons, leading to changes in the properties of decoded neural representations.

KChIP4a regulates Kv4.2 channel trafficking through PKA phosphorylation.

Voltage-gated potassium (Kv) channels play important roles in regulating the excitability of myocytes and neurons. Kv4.2 is the primary alpha-subunit of the channel that produces the A-type K(+) current in CA1 pyramidal neurons of the hippocampus, which is critically involved in the regulation of dendritic excitability and plasticity. K(+) channel-interacting proteins, KChIPs (KChIP1-4), associate with the N-terminal of Kv4.2 and modulate the channel's biophysical properties, turnover rate and surface expression. In the present study, we investigated the role of Kv4.2 C-terminal PKA phosphorylation site S552 in the KChIP4a-mediated effects on Kv4.2 channel trafficking. We found that while interaction between Kv4.2 and KChIP4a does not require PKA phosphorylation of Kv4.2(S552), phosphorylation of this site is necessary for both enhanced stabilization and membrane expression of Kv4.2 channel complexes produced by KChIP4a. Enhanced surface expression and protein stability conferred by co-expression of Kv4.2 with other KChIP isoforms did not require PKA phosphorylation of Kv4.2 S552. Finally, we identify A-kinase anchoring proteins (AKAPs) as Kv4.2 binding partners, allowing for discrete local PKA signaling. These data demonstrate that PKA phosphorylation of Kv4.2 plays an important role in the trafficking of Kv4.2 through its specific interaction with KChIP4a.

A bidirectional corticoamygdala circuit for the encoding and retrieval of detailed reward memories.

Adaptive reward-related decision making often requires accurate and detailed representation of potential available rewards. Environmental reward-predictive stimuli can facilitate these representations, allowing one to infer which specific rewards might be available and choose accordingly. This process relies on encoded relationships between the cues and the sensory-specific details of the rewards they predict. Here, we interrogated the function of the basolateral amygdala (BLA) and its interaction with the lateral orbitofrontal cortex (lOFC) in the ability to learn such stimulus-outcome associations and use these memories to guide decision making. Using optical recording and inhibition approaches, Pavlovian cue-reward conditioning, and the outcome-selective Pavlovian-to-instrumental transfer (PIT) test in male rats, we found that the BLA is robustly activated at the time of stimulus-outcome learning and that this activity is necessary for sensory-specific stimulus-outcome memories to be encoded, so they can subsequently influence reward choices. Direct input from the lOFC was found to support the BLA in this function. Based on prior work, activity in BLA projections back to the lOFC was known to support the use of stimulus-outcome memories to influence decision making. By multiplexing optogenetic and chemogenetic inhibition we performed a serial circuit disconnection and found that the lOFC→BLA and BLA→lOFC pathways form a functional circuit regulating the encoding (lOFC→BLA) and subsequent use (BLA→lOFC) of the stimulus-dependent, sensory-specific reward memories that are critical for adaptive, appetitive decision making.

Can the VTA Come Out to Play? Only When the mPFC's Predictions Go Astray!

Confidence in perceptual decisions scales neural responses to violations in reward expectation. In this issue of Neuron, Lak et al. (2020) show that the medial prefrontal cortex in mice computes a confidence-dependent expectation signal that influences how dopamine neurons convey reward prediction errors to guide learning.

Processing in Lateral Orbitofrontal Cortex Is Required to Estimate Subjective Preference during Initial, but Not Established, Economic Choice.

The orbitofrontal cortex (OFC) is proposed to be critical to economic decision making. Yet one can inactivate OFC without affecting well-practiced choices. One possible explanation of this lack of effect is that well-practiced decisions are codified into habits or configural-based policies not normally thought to require OFC. Here, we tested this idea by training rats to choose between different pellet pairs across a set of standard offers and then inactivating OFC subregions during choices between novel offers of previously experienced pairs or between novel pairs of previously experienced pellets. Contrary to expectations, controls performed as well on novel as experienced offers yet had difficulty initially estimating their subjective preference on novel pairs, difficulty exacerbated by lateral OFC inactivation. This pattern of results indicates that established economic choice reflects the use of an underlying model or goods space and that lateral OFC is only required for normal behavior when the established framework must incorporate new information.

Protein kinase a mediates activity-dependent Kv4.2 channel trafficking.

The A-type potassium channel subunit Kv4.2 influences hippocampal function through regulation of dendritic excitability, and changes in Kv4.2 surface expression alter synaptic plasticity. Recent data from our laboratory demonstrate that EGFP (enhanced green fluorescent protein)-tagged Kv4.2 channels located in dendritic spines are internalized in an activity-dependent manner after synaptic stimulation and during chemically induced long-term potentiation. However, the molecular trigger for Kv4.2 internalization remains unknown. Here we examined the role of protein kinase A (PKA) in Kv4.2 activity-dependent trafficking. In hippocampal neurons, PKA activation with forskolin or 8-Br-cAMP induced Kv4.2 internalization from dendritic spines, whereas PKA inhibition with H89 prevented AMPA-induced internalization. Furthermore, introduction of a point mutation at the C-terminal PKA phosphorylation site of Kv4.2 (S552A) prevented the AMPA-induced internalization of Kv4.2. Together, these data demonstrate that Kv4.2 activity-dependent internalization requires PKA phosphorylation of Kv4.2 at serine 522.

Complementary Task Structure Representations in Hippocampus and Orbitofrontal Cortex during an Odor Sequence Task.

Both hippocampus (HPC) and orbitofrontal cortex (OFC) have been shown to be critical for behavioral tasks that require use of an internal model or cognitive map, composed of the states and the relationships between them, which define the current environment or task at hand. One general idea is that the HPC provides the cognitive map, which is then transformed by OFC to emphasize information of relevance to current goals. Our previous analysis of ensemble activity in OFC in rats performing an odor sequence task revealed a rich representation of behaviorally relevant task structure, consistent with this proposal. Here, we compared those data to recordings from single units in area CA1 of the HPC of rats performing the same task. Contrary to expectations that HPC ensembles would represent detailed, even incidental, information defining the full task space, we found that HPC ensembles-like those in OFC-failed to distinguish states when it was not behaviorally necessary. However, hippocampal ensembles were better than those in OFC at distinguishing task states in which prospective memory was necessary for future performance. These results suggest that, in familiar environments, the HPC and OFC may play complementary roles, with the OFC maintaining the subjects' current position on the cognitive map or state space, supported by HPC when memory demands are high.

Rat Orbitofrontal Ensemble Activity Contains Multiplexed but Dissociable Representations of Value and Task Structure in an Odor Sequence Task.

The orbitofrontal cortex (OFC) has long been implicated in signaling information about expected outcomes to facilitate adaptive or flexible behavior. Current proposals focus on signaling of expected value versus the representation of a value-agnostic cognitive map of the task. While often suggested as mutually exclusive, these alternatives may represent extreme ends of a continuum determined by task complexity and experience. As learning proceeds, an initial, detailed cognitive map might be acquired, based largely on external information. With more experience, this hypothesized map can then be tailored to include relevant abstract hidden cognitive constructs. The map would default to an expected value in situations where other attributes are largely irrelevant, but, in richer tasks, a more detailed structure might continue to be represented, at least where relevant to behavior. Here, we examined this by recording single-unit activity from the OFC in rats navigating an odor sequence task analogous to a spatial maze. The odor sequences provided a mappable state space, with 24 unique "positions" defined by sensory information, likelihood of reward, or both. Consistent with the hypothesis that the OFC represents a cognitive map tailored to the subjects' intentions or plans, we found a close correspondence between how subjects were using the sequences and the neural representations of the sequences in OFC ensembles. Multiplexed with this value-invariant representation of the task, we also found a representation of the expected value at each location. Thus, the value and task structure co-existed as dissociable components of the neural code in OFC.

Suppression of Ventral Hippocampal Output Impairs Integrated Orbitofrontal Encoding of Task Structure.

The hippocampus and orbitofrontal cortex (OFC) both make important contributions to decision making and other cognitive processes. However, despite anatomical links between the two, few studies have tested the importance of hippocampal-OFC interactions. Here, we recorded OFC neurons in rats performing a decision making task while suppressing activity in a key hippocampal output region, the ventral subiculum. OFC neurons encoded information about expected outcomes and rats' responses. With hippocampal output suppressed, rats were slower to adapt to changes in reward contingency, and OFC encoding of response information was strongly attenuated. In addition, ventral subiculum inactivation prevented OFC neurons from integrating information about features of outcomes to form holistic representations of the outcomes available in specific trial blocks. These data suggest that the hippocampus contributes to OFC encoding of both concrete, low-level features of expected outcomes, and abstract, inferred properties of the structure of the world, such as task state.