Paradoxical replay can protect contextual task representations from destructive interference when experience is unbalanced.

Experience replay is a powerful mechanism to learn efficiently from limited experience. Despite several decades of compelling experimental results, the factors that determine which experiences are selected for replay remain unclear. A particular challenge for current theories is that on tasks that feature unbalanced experience, rats paradoxically replay the less-experienced trajectory. To understand why, we simulated a feedforward neural network with two regimes: rich learning (structured representations tailored to task demands) and lazy learning (unstructured, task-agnostic representations). Rich, but not lazy, representations degraded following unbalanced experience, an effect that could be reversed with paradoxical replay. To test if this computational principle can account for the experimental data, we examined the relationship between paradoxical replay and learned task representations in the rat hippocampus. Strikingly, we found a strong association between the richness of learned task representations and the paradoxicality of replay. Taken together, these results suggest that paradoxical replay specifically serves to protect rich representations from the destructive effects of unbalanced experience, and more generally demonstrate a novel interaction between the nature of task representations and the function of replay in artificial and biological systems.

Behavioral and neural generalization: Hitting the right notes.

Successful generalization to novel stimuli is a core goal of learning and memory systems, but how do we do it? In this issue of Neuron, Miller et al.1 use a novel auditory structure learning task to reveal neural and behavioral signatures of generalization.

Temporal context and latent state inference in the hippocampal splitter signal.

The hippocampus is thought to enable the encoding and retrieval of ongoing experience, the organization of that experience into structured representations like contexts, maps, and schemas, and the use of these structures to plan for the future. A central goal is to understand what the core computations supporting these functions are, and how these computations are realized in the collective action of single neurons. A potential access point into this issue is provided by 'splitter cells', hippocampal neurons that fire differentially on the overlapping segment of trajectories that differ in their past and/or future. However, the literature on splitter cells has been fragmented and confusing, owing to differences in terminology, behavioral tasks, and analysis methods across studies. In this review, we synthesize consistent findings from this literature, establish a common set of terms, and translate between single-cell and ensemble perspectives. Most importantly, we examine the combined findings through the lens of two major theoretical ideas about hippocampal function: representation of temporal context and latent state inference. We find that unique signature properties of each of these models are necessary to account for the data, but neither theory, by itself, explains all of its features. Specifically, the temporal gradedness of the splitter signal is strong support for temporal context, but is hard to explain using state models, while its flexibility and task-dependence is naturally accounted for using state inference, but poses a challenge otherwise. These theories suggest a number of avenues for future work, and we believe their application to splitter cells is a timely and informative domain for testing and refining theoretical ideas about hippocampal function.

A consensus statement on detection of hippocampal sharp wave ripples and differentiation from other fast oscillations.

Decades of rodent research have established the role of hippocampal sharp wave ripples (SPW-Rs) in consolidating and guiding experience. More recently, intracranial recordings in humans have suggested their role in episodic and semantic memory. Yet, common standards for recording, detection, and reporting do not exist. Here, we outline the methodological challenges involved in detecting ripple events and offer practical recommendations to improve separation from other high-frequency oscillations. We argue that shared experimental, detection, and reporting standards will provide a solid foundation for future translational discovery.

Context coding in the mouse nucleus accumbens modulates motivationally relevant information.

Neural activity in the nucleus accumbens (NAc) is thought to track fundamentally value-centric quantities linked to reward and effort. However, the NAc also contributes to flexible behavior in ways that are difficult to explain based on value signals alone, raising the question of if and how nonvalue signals are encoded in NAc. We recorded NAc neural ensembles while head-fixed mice performed an odor-based biconditional discrimination task where an initial discrete cue modulated the behavioral significance of a subsequently presented reward-predictive cue. We extracted single-unit and population-level correlates related to the cues and found value-independent coding for the initial, context-setting cue. This context signal occupied a population-level coding space orthogonal to outcome-related representations and was predictive of subsequent behaviorally relevant responses to the reward-predictive cues. Together, these findings support a gating model for how the NAc contributes to behavioral flexibility and provide a novel population-level perspective from which to view NAc computations.

Between-subject prediction reveals a shared representational geometry in the rodent hippocampus.

The rodent hippocampus constructs statistically independent representations across environments ("global remapping") and assigns individual neuron firing fields to locations within an environment in an apparently random fashion, processes thought to contribute to the role of the hippocampus in episodic memory. This random mapping implies that it should be challenging to predict hippocampal encoding of a given experience in one subject based on the encoding of that same experience in another subject. Contrary to this prediction, we find that by constructing a common representational space across rats in which neural activity is aligned using geometric operations (rotation, reflection, and translation; "hyperalignment"), we can predict data of "right" trials (R) on a T-maze in a target rat based on (1) the "left" trials (L) of the target rat and (2) the relationship between L and R trials from a different source rat. These cross-subject predictions relied on ensemble activity patterns, including both firing rate and field location, and outperformed a number of control mappings, such as those based on permuted data that broke the relationship between L and R activity for individual neurons and those based solely on within-subject prediction. This work constitutes proof of principle for successful cross-subject prediction of ensemble activity patterns in the hippocampus and provides new insights in understanding how different experiences are structured, enabling further work identifying what aspects of experience encoding are shared versus unique to an individual.

A framework to identify structured behavioral patterns within rodent spatial trajectories.

Animal behavior is highly structured. Yet, structured behavioral patterns-or "statistical ethograms"-are not immediately apparent from the full spatiotemporal data that behavioral scientists usually collect. Here, we introduce a framework to quantitatively characterize rodent behavior during spatial (e.g., maze) navigation, in terms of movement building blocks or motor primitives. The hypothesis that we pursue is that rodent behavior is characterized by a small number of motor primitives, which are combined over time to produce open-ended movements. We assume motor primitives to be organized in terms of two sparsity principles: each movement is controlled using a limited subset of motor primitives (sparse superposition) and each primitive is active only for time-limited, time-contiguous portions of movements (sparse activity). We formalize this hypothesis using a sparse dictionary learning method, which we use to extract motor primitives from rodent position and velocity data collected during spatial navigation, and successively to reconstruct past trajectories and predict novel ones. Three main results validate our approach. First, rodent behavioral trajectories are robustly reconstructed from incomplete data, performing better than approaches based on standard dimensionality reduction methods, such as principal component analysis, or single sparsity. Second, the motor primitives extracted during one experimental session generalize and afford the accurate reconstruction of rodent behavior across successive experimental sessions in the same or in modified mazes. Third, in our approach the number of motor primitives associated with each maze correlates with independent measures of maze complexity, hence showing that our formalism is sensitive to essential aspects of task structure. The framework introduced here can be used by behavioral scientists and neuroscientists as an aid for behavioral and neural data analysis. Indeed, the extracted motor primitives enable the quantitative characterization of the complexity and similarity between different mazes and behavioral patterns across multiple trials (i.e., habit formation). We provide example uses of this computational framework, showing how it can be used to identify behavioural effects of maze complexity, analyze stereotyped behavior, classify behavioral choices and predict place and grid cell displacement in novel environments.

Progress and issues in second-order analysis of hippocampal replay.

Patterns of neural activity that occur spontaneously during sharp-wave ripple (SWR) events in the hippocampus are thought to play an important role in memory formation, consolidation and retrieval. Typical studies examining the content of SWRs seek to determine whether the identity and/or temporal order of cell firing is different from chance. Such 'first-order' analyses are focused on a single time point and template (map), and have been used to show, for instance, the existence of preplay. The major methodological challenge in first-order analyses is the construction and interpretation of different chance distributions. By contrast, 'second-order' analyses involve a comparison of SWR content between different time points, and/or between different templates. Typical second-order questions include tests of experience-dependence (replay) that compare SWR content before and after experience, and comparisons or replay between different arms of a maze. Such questions entail additional methodological challenges that can lead to biases in results and associated interpretations. We provide an inventory of analysis challenges for second-order questions about SWR content, and suggest ways of preventing, identifying and addressing possible analysis biases. Given evolving interest in understanding SWR content in more complex experimental scenarios and across different time scales, we expect these issues to become increasingly pervasive. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

Complementary Control over Habits and Behavioral Vigor by Phasic Activity in the Dorsolateral Striatum.

Despite clear evidence linking the basal ganglia to the control of outcome insensitivity (i.e., habit) and behavioral vigor (i.e., its behavioral speed/fluidity), it remains unclear whether or how these functions relate to one another. Here, using male Long-Evans rats in response-based and cue-based maze-running tasks, we demonstrate that phasic dorsolateral striatum (DLS) activity occurring at the onset of a learned behavior regulates how vigorous and habitual it is. In a response-based task, brief optogenetic excitation at the onset of runs decreased run duration and the occurrence of deliberative behaviors, whereas midrun stimulation carried little effect. Outcome devaluation showed these runs to be habitual. DLS inhibition at run start did not produce robust effects on behavior until after outcome devaluation. At that time, when the DLS was plausibly most critically required for performance (i.e., habitual), inhibition reduced performance vigor measures and caused a dramatic loss of habitual responding (i.e., animals quit the task). In a second cue-based "beacon" task requiring behavior initiation at the start of the run and again in the middle of the run, DLS excitation at both time points could improve the vigor of runs. Postdevaluation testing showed behavior on the beacon task to be habitual as well. This pattern of results suggests that one role for phasic DLS activity at behavior initiation is to promote the execution of the behavior in a vigorous and habitual fashion by a diverse set of measures.SIGNIFICANCE STATEMENT Our research expands the literature twofold. First, we find that features of a habitual behavior that are typically studied separately (i.e., maze response performance, deliberation movements, running vigor, and outcome insensitivity) are quite closely linked together. Second, efforts have been made to understand "what" the dorsolateral striatum (DLS) does for habitual behavior, and our research provides a key set of results showing "when" it is important (i.e., at behavior initiation). By showing such dramatic control over habits by DLS activity in a phasic time window, plausible real-world applications could involve more informed DLS perturbations to curb intractably problematic habits.

Reward revaluation biases hippocampal replay content away from the preferred outcome.

The rodent hippocampus spontaneously generates bursts of neural activity (replay) that can depict spatial trajectories to reward locations, suggesting a role in model-based behavioral control. A largely separate literature emphasizes reward revaluation as the litmus test for such control, yet the content of hippocampal replay under revaluation conditions is unknown. We examined the content of awake replay events following motivational shifts between hunger and thirst. On a T-maze offering free choice between food and water outcomes, rats shifted their behavior toward the restricted outcome, but replay content was shifted away from the restricted outcome. This effect preceded experience on the task each day and did not reverse with experience. These results demonstrate that replay content is not limited to reflecting recent experience or trajectories toward the preferred goal and suggest a role for motivational states in determining replay content.

Gamma Oscillations in the Rat Ventral Striatum Originate in the Piriform Cortex.

Local field potentials (LFPs) recorded from the human and rodent ventral striatum (vStr) exhibit prominent, behaviorally relevant gamma-band oscillations. These oscillations are related to local spiking activity and transiently synchronize with anatomically related areas, suggesting a possible role in organizing vStr activity. However, the origin of vStr gamma is unknown. We recorded vStr gamma oscillations across a 1.4 mm2 grid spanned by 64 recording electrodes as male rats rested and foraged for rewards, revealing a highly consistent power gradient originating in the adjacent piriform cortex. Phase differences across the vStr were consistently small (<15°) and current source density analysis further confirmed the absence of local sink-source pairs in the vStr. Reversible occlusions of the ipsilateral (but not contralateral) nostril, known to abolish gamma oscillations in the piriform cortex, strongly reduced vStr gamma power and the occurrence of transient gamma-band events. These results imply that local circuitry is not a major contributor to gamma oscillations in the vStr LFP and that piriform cortex is an important driver of gamma-band oscillations in the vStr and associated limbic areas.SIGNIFICANCE STATEMENT The ventral striatum (vStr) is an area of anatomical convergence in circuits underlying motivated behavior, but it remains unclear how its inputs from different sources interact. A major proposal about how neural circuits may switch dynamically between convergent inputs is through temporal organization reflected in local field potential (LFP) oscillations. Our results show that, in the rat, the mechanisms controlling gamma-band oscillations in the vStr LFP are primarily located in the in the adjacent piriform cortex rather than in the vStr itself, providing a novel interpretation of previous rodent work on gamma oscillations in the vStr and related circuits and an important consideration for future work seeking to use oscillations in these areas as biomarkers for behavioral and neurological disorders.

Internally generated hippocampal sequences as a vantage point to probe future-oriented cognition.

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences (IGSs), expressed during two different network states: theta sequences, which repeat and reset at the ∼8 Hz theta rhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakeful rest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in a fragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferential process that samples a policy from the animal's generative model, supported by hippocampus-specific priors. The same inference affords different cognitive functions when the animal is in distinct dynamical modes, associated with specific functional networks. Theta sequences arise when inference is coupled to the animal's action-perception cycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise when the animal is decoupled from the action-perception cycle and may support offline cognitive processing, such as memory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empirical bases of this proposal in relation to rodent studies and highlight how the proposed computational principles can shed light on the mechanisms of future-oriented cognition in humans.

The Head-Direction Signal Plays a Functional Role as a Neural Compass during Navigation.

The rat limbic system contains head direction (HD) cells that fire according to heading in the horizontal plane, and these cells are thought to provide animals with an internal compass. Previous work has found that HD cell tuning correlates with behavior on navigational tasks, but a direct, causal link between HD cells and navigation has not been demonstrated. Here, we show that pathway-specific optogenetic inhibition of the nucleus prepositus caused HD cells to become directionally unstable under dark conditions without affecting the animals' locomotion. Then, using the same technique, we found that this decoupling of the HD signal in the absence of visual cues caused the animals to make directional homing errors and that the magnitude and direction of these errors were in a range that corresponded to the degree of instability observed in the HD signal. These results provide evidence that the HD signal plays a causal role as a neural compass in navigation.

Optimizing for generalization in the decoding of internally generated activity in the hippocampus.

The decoding of a sensory or motor variable from neural activity benefits from a known ground truth against which decoding performance can be compared. In contrast, the decoding of covert, cognitive neural activity, such as occurs in memory recall or planning, typically cannot be compared to a known ground truth. As a result, it is unclear how decoders of such internally generated activity should be configured in practice. We suggest that if the true code for covert activity is unknown, decoders should be optimized for generalization performance using cross-validation. Using ensemble recording data from hippocampal place cells, we show that this cross-validation approach results in different decoding error, different optimal decoding parameters, and different distributions of error across the decoded variable space. In addition, we show that a minor modification to the commonly used Bayesian decoding procedure, which enables the use of spike density functions, results in substantially lower decoding errors. These results have implications for the interpretation of covert neural activity, and suggest easy-to-implement changes to commonly used procedures across domains, with applications to hippocampal place cells in particular. © 2017 Wiley Periodicals, Inc.

Low- and high-gamma oscillations deviate in opposite directions from zero-phase synchrony in the limbic corticostriatal loop.

The loop structure of cortico-striatal anatomy in principle enables both descending (cortico-striatal) and ascending (striato-cortical) influences, but the factors that regulate the flow of information in these loops are not known. We report that low- and high-gamma oscillations (∼50 and ∼80 Hz, respectively) in the local field potential of freely moving rats are highly synchronous between the infralimbic region of the medial prefrontal cortex (mPFC) and the ventral striatum (vStr). Strikingly, high-gamma oscillations in mPFC preceded those in vStr, whereas low-gamma oscillations in mPFC lagged those in vStr, with short (∼1 ms) time lags. These systematic deviations from zero-phase synchrony were consistent across measures based on amplitude cross-correlation and phase slopes and were robustly maintained between behavioral states and different individual subjects. Furthermore, low- and high-gamma oscillations were associated with distinct ensemble spiking patterns in vStr, even when controlling for overt behavioral differences and slow changes in neural activity. These results imply that neural activity in vStr and mPFC is tightly coupled at the gamma timescale and raise the intriguing possibility that frequency-specific deviations from this coupling may signal transient leader-follower switches.

Ventral striatal gamma oscillations are highly variable from trial to trial, and are dominated by behavioural state, and only weakly influenced by outcome value.

The human and rodent ventral striatal local field potentials show striking oscillations in the gamma band (~ 40-100 Hz), which have been linked to aspects of behaviour such as reward anticipation and delivery, movement initiation, learning from feedback, and decision-making. These oscillations show a rich temporal organization, whose relationship with behavioural variables is not well understood. Here, we show that, in rats performing a conditioned approach task, low-gamma and high-gamma oscillations during an immobile reward anticipation epoch were largely insensitive to outcome value, even though rats distinguished behaviourally between different outcomes, and single units encoded outcome value. Behaviour was highly stereotyped, yet we observed large variability from trial to trial in the occurrence and timing of these oscillations. Furthermore, higher-order features such as high-gamma power leading low-gamma power, and phase-amplitude coupling to lower-frequency bands, were only marginally modulated by outcome value. Moreover, these patterns closely resembled those found during off-task rest periods in which no rewards could be earned. These observations suggest a new interpretation of ventral striatal gamma oscillations as reflecting a default or resting state, with only minor and highly variable modulation by specific task-related variables.

Internally generated sequences in learning and executing goal-directed behavior.

A network of brain structures including hippocampus (HC), prefrontal cortex, and striatum controls goal-directed behavior and decision making. However, the neural mechanisms underlying these functions are unknown. Here, we review the role of 'internally generated sequences': structured, multi-neuron firing patterns in the network that are not confined to signaling the current state or location of an agent, but are generated on the basis of internal brain dynamics. Neurophysiological studies suggest that such sequences fulfill functions in memory consolidation, augmentation of representations, internal simulation, and recombination of acquired information. Using computational modeling, we propose that internally generated sequences may be productively considered a component of goal-directed decision systems, implementing a sampling-based inference engine that optimizes goal acquisition at multiple timescales of on-line choice, action control, and learning.

Adaptive properties of differential learning rates for positive and negative outcomes.

The concept of the reward prediction error-the difference between reward obtained and reward predicted-continues to be a focal point for much theoretical and experimental work in psychology, cognitive science, and neuroscience. Models that rely on reward prediction errors typically assume a single learning rate for positive and negative prediction errors. However, behavioral data indicate that better-than-expected and worse-than-expected outcomes often do not have symmetric impacts on learning and decision-making. Furthermore, distinct circuits within cortico-striatal loops appear to support learning from positive and negative prediction errors, respectively. Such differential learning rates would be expected to lead to biased reward predictions and therefore suboptimal choice performance. Contrary to this intuition, we show that on static "bandit" choice tasks, differential learning rates can be adaptive. This occurs because asymmetric learning enables a better separation of learned reward probabilities. We show analytically how the optimal learning rate asymmetry depends on the reward distribution and implement a biologically plausible algorithm that adapts the balance of positive and negative learning rates from experience. These results suggest specific adaptive advantages for separate, differential learning rates in simple reinforcement learning settings and provide a novel, normative perspective on the interpretation of associated neural data.

Risk-responsive orbitofrontal neurons track acquired salience.

Decision making is impacted by uncertainty and risk (i.e., variance). Activity in the orbitofrontal cortex, an area implicated in decision making, covaries with these quantities. However, this activity could reflect the heightened salience of situations in which multiple outcomes-reward and reward omission-are expected. To resolve these accounts, rats were trained to respond to cues predicting 100%, 67%, 33%, or 0% reward. Consistent with prior reports, some orbitofrontal neurons fired differently in anticipation of uncertain (33% and 67%) versus certain (100% and 0%) reward. However, over 90% of these neurons also fired differently prior to 100% versus 0% reward (or baseline) or prior to 33% versus 67% reward. These responses are inconsistent with risk but fit well with the representation of acquired salience linked to the sum of cue-outcome and cue-no-outcome associative strengths. These results expand our understanding of how the orbitofrontal cortex might regulate learning and behavior.