The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation.

The hippocampal-entorhinal system is important for spatial and relational memory tasks. We formally link these domains, provide a mechanistic understanding of the hippocampal role in generalization, and offer unifying principles underlying many entorhinal and hippocampal cell types. We propose medial entorhinal cells form a basis describing structural knowledge, and hippocampal cells link this basis with sensory representations. Adopting these principles, we introduce the Tolman-Eichenbaum machine (TEM). After learning, TEM entorhinal cells display diverse properties resembling apparently bespoke spatial responses, such as grid, band, border, and object-vector cells. TEM hippocampal cells include place and landmark cells that remap between environments. Crucially, TEM also aligns with empirically recorded representations in complex non-spatial tasks. TEM also generates predictions that hippocampal remapping is not random as previously believed; rather, structural knowledge is preserved across environments. We confirm this structural transfer over remapping in simultaneously recorded place and grid cells.

Predictive Representations: Building Blocks of Intelligence.

Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This review integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation and its generalizations, which have been widely applied as both engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

Vector-based navigation using grid-like representations in artificial agents.

Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go1,2. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning3-5 failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex 6 . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space7,8 and is critical for integrating self-motion (path integration)6,7,9 and planning direct trajectories to goals (vector-based navigation)7,10,11. Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types 12 . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation7,10,11, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.

Choice of method of place cell classification determines the population of cells identified.

Place cells, spatially responsive hippocampal cells, provide the neural substrate supporting navigation and spatial memory. Historically most studies of these neurons have used electrophysiological recordings from implanted electrodes but optical methods, measuring intracellular calcium, are becoming increasingly common. Several methods have been proposed as a means to identify place cells based on their calcium activity but there is no common standard and it is unclear how reliable different approaches are. Here we tested four methods that have previously been applied to two-photon hippocampal imaging or electrophysiological data, using both model datasets and real imaging data. These methods use different parameters to identify place cells, including the peak activity in the place field, compared to other locations (the Peak method); the stability of cells' activity over repeated traversals of an environment (Stability method); a combination of these parameters with the size of the place field (Combination method); and the spatial information held by the cells (Information method). The methods performed differently from each other on both model and real data. In real datasets, vastly different numbers of place cells were identified using the four methods, with little overlap between the populations identified as place cells. Therefore, choice of place cell detection method dramatically affects the number and properties of identified cells. Ultimately, we recommend the Peak method be used in future studies to identify place cell populations, as this method is robust to moderate variations in place field within a session, and makes no inherent assumptions about the spatial information in place fields, unless there is an explicit theoretical reason for detecting cells with more narrowly defined properties.

Grid cell symmetry is shaped by environmental geometry.

Grid cells represent an animal's location by firing in multiple fields arranged in a striking hexagonal array. Such an impressive and constant regularity prompted suggestions that grid cells represent a universal and environmental-invariant metric for navigation. Originally the properties of grid patterns were believed to be independent of the shape of the environment and this notion has dominated almost all theoretical grid cell models. However, several studies indicate that environmental boundaries influence grid firing, though the strength, nature and longevity of this effect is unclear. Here we show that grid orientation, scale, symmetry and homogeneity are strongly and permanently affected by environmental geometry. We found that grid patterns orient to the walls of polarized enclosures such as squares, but not circles. Furthermore, the hexagonal grid symmetry is permanently broken in highly polarized environments such as trapezoids, the pattern being more elliptical and less homogeneous. Our results provide compelling evidence for the idea that environmental boundaries compete with the internal organization of the grid cell system to drive grid firing. Notably, grid cell activity is more local than previously thought and as a consequence cannot provide a universal spatial metric in all environments.

Neurobiological successor features for spatial navigation.

The hippocampus has long been observed to encode a representation of an animal's position in space. Recent evidence suggests that the nature of this representation is somewhat predictive and can be modeled by learning a successor representation (SR) between distinct positions in an environment. However, this discretization of space is subjective making it difficult to formulate predictions about how some environmental manipulations should impact the hippocampal representation. Here, we present a model of place and grid cell firing as a consequence of learning a SR from a basis set of known neurobiological features-boundary vector cells (BVCs). The model describes place cell firing as the successor features of the SR, with grid cells forming a low-dimensional representation of these successor features. We show that the place and grid cells generated using the BVC-SR model provide a good account of biological data for a variety of environmental manipulations, including dimensional stretches, barrier insertions, and the influence of environmental geometry on the hippocampal representation of space.

Efficient neural decoding of self-location with a deep recurrent network.

Place cells in the mammalian hippocampus signal self-location with sparse spatially stable firing fields. Based on observation of place cell activity it is possible to accurately decode an animal's location. The precision of this decoding sets a lower bound for the amount of information that the hippocampal population conveys about the location of the animal. In this work we use a novel recurrent neural network (RNN) decoder to infer the location of freely moving rats from single unit hippocampal recordings. RNNs are biologically plausible models of neural circuits that learn to incorporate relevant temporal context without the need to make complicated assumptions about the use of prior information to predict the current state. When decoding animal position from spike counts in 1D and 2D-environments, we show that the RNN consistently outperforms a standard Bayesian approach with either flat priors or with memory. In addition, we also conducted a set of sensitivity analysis on the RNN decoder to determine which neurons and sections of firing fields were the most influential. We found that the application of RNNs to neural data allowed flexible integration of temporal context, yielding improved accuracy relative to the more commonly used Bayesian approaches and opens new avenues for exploration of the neural code.

Evidence for grid cells in a human memory network.

Grid cells in the entorhinal cortex of freely moving rats provide a strikingly periodic representation of self-location which is indicative of very specific computational mechanisms. However, the existence of grid cells in humans and their distribution throughout the brain are unknown. Here we show that the preferred firing directions of directionally modulated grid cells in rat entorhinal cortex are aligned with the grids, and that the spatial organization of grid-cell firing is more strongly apparent at faster than slower running speeds. Because the grids are also aligned with each other, we predicted a macroscopic signal visible to functional magnetic resonance imaging (fMRI) in humans. We then looked for this signal as participants explored a virtual reality environment, mimicking the rats' foraging task: fMRI activation and adaptation showing a speed-modulated six-fold rotational symmetry in running direction. The signal was found in a network of entorhinal/subicular, posterior and medial parietal, lateral temporal and medial prefrontal areas. The effect was strongest in right entorhinal cortex, and the coherence of the directional signal across entorhinal cortex correlated with spatial memory performance. Our study illustrates the potential power of combining single-unit electrophysiology with fMRI in systems neuroscience. Our results provide evidence for grid-cell-like representations in humans, and implicate a specific type of neural representation in a network of regions which supports spatial cognition and also autobiographical memory.

Grid cell firing patterns signal environmental novelty by expansion.

The hippocampal formation plays key roles in representing an animal's location and in detecting environmental novelty to create or update those representations. However, the mechanisms behind this latter function are unclear. Here, we show that environmental novelty causes the spatial firing patterns of grid cells to expand in scale and reduce in regularity, reverting to their familiar scale as the environment becomes familiar. Simultaneously recorded place cell firing fields remapped and showed a smaller, temporary expansion. Grid expansion provides a potential mechanism for novelty signaling and may enhance the formation of new hippocampal representations, whereas the subsequent slow reduction in scale provides a potential familiarity signal.

RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments.

Generating synthetic locomotory and neural data is a useful yet cumbersome step commonly required to study theoretical models of the brain's role in spatial navigation. This process can be time consuming and, without a common framework, makes it difficult to reproduce or compare studies which each generate test data in different ways. In response, we present RatInABox, an open-source Python toolkit designed to model realistic rodent locomotion and generate synthetic neural data from spatially modulated cell types. This software provides users with (i) the ability to construct one- or two-dimensional environments with configurable barriers and visual cues, (ii) a physically realistic random motion model fitted to experimental data, (iii) rapid online calculation of neural data for many of the known self-location or velocity selective cell types in the hippocampal formation (including place cells, grid cells, boundary vector cells, head direction cells) and (iv) a framework for constructing custom cell types, multi-layer network models and data- or policy-controlled motion trajectories. The motion and neural models are spatially and temporally continuous as well as topographically sensitive to boundary conditions and walls. We demonstrate that out-of-the-box parameter settings replicate many aspects of rodent foraging behaviour such as velocity statistics and the tendency of rodents to over-explore walls. Numerous tutorial scripts are provided, including examples where RatInABox is used for decoding position from neural data or to solve a navigational reinforcement learning task. We hope this tool will significantly streamline computational research into the brain's role in navigation.

The brain is a complex system made up of over 100 billion neurons that interact to give rise to all sorts of behaviours. To understand how neural interactions enable distinct behaviours, neuroscientists often build computational models that can reproduce some of the interactions and behaviours observed in the brain. Unfortunately, good computational models can be hard to build, and it can be wasteful for different groups of scientists to each write their own software to model a similar system. Instead, it is more effective for scientists to share their code so that different models can be quickly built from an identical set of core elements. These toolkits should be well made, free and easy to use. One of the largest fields within neuroscience and machine learning concerns navigation: how does an organism ­ or an artificial agent ­ know where they are and how to get where they are going next? Scientists have identified many different types of neurons in the brain that are important for navigation. For example, 'place cells' fire whenever the animal is at a specific location, and 'head direction cells' fire when the animal's head is pointed in a particular direction. These and other neurons interact to support navigational behaviours. Despite the importance of navigation, no single computational toolkit existed to model these behaviours and neural circuits. To fill this gap, George et al. developed RatInABox, a toolkit that contains the building blocks needed to study the brain's role in navigation. One module, called the 'Environment', contains code for making arenas of arbitrary shapes. A second module contains code describing how organisms or 'Agents' move around the arena and interact with walls, objects, and other agents. A final module, called 'Neurons', contains code that reproduces the reponse patterns of well-known cell types involved in navigation. This module also has code for more generic, trainable neurons that can be used to model how machines and organisms learn. Environments, Agents and Neurons can be combined and modified in many ways, allowing users to rapidly construct complex models and generate artificial datasets. A diversity of tutorials, including how the package can be used for reinforcement learning (the study of how agents learn optimal motions) are provided. RatInABox will benefit many researchers interested in neuroscience and machine learning. It is particularly well positioned to bridge the gap between these two fields and drive a more brain-inspired approach to machine learning. RatInABox's userbase is fast growing, and it is quickly becoming one of the core computational tools used by scientists to understand the brain and navigation. Additionally, its ease of use and visual clarity means that it can be used as an accessible teaching tool for learning about spatial representations and navigation.

Environment geometry alters subiculum boundary vector cell receptive fields in adulthood and early development.

Boundaries to movement form a specific class of landmark information used for navigation: Boundary Vector Cells (BVCs) are neurons which encode an animal's location as a vector displacement from boundaries. Here we characterise the prevalence and spatial tuning of subiculum BVCs in adult and developing male rats, and investigate the relationship between BVC spatial firing and boundary geometry. BVC directional tunings align with environment walls in squares, but are uniformly distributed in circles, demonstrating that environmental geometry alters BVC receptive fields. Inserted barriers uncover both excitatory and inhibitory components to BVC receptive fields, demonstrating that inhibitory inputs contribute to BVC field formation. During post-natal development, subiculum BVCs mature slowly, contrasting with the earlier maturation of boundary-responsive cells in upstream Entorhinal Cortex. However, Subiculum and Entorhinal BVC receptive fields are altered by boundary geometry as early as tested, suggesting this is an inherent feature of the hippocampal representation of space.

Theta-band phase locking during encoding leads to coordinated entorhinal-hippocampal replay.

Precisely timed interactions between hippocampal and cortical neurons during replay epochs are thought to support learning. Indeed, research has shown that replay is associated with heightened hippocampal-cortical synchrony. Yet many caveats remain in our understanding. Namely, it remains unclear how this offline synchrony comes about, whether it is specific to particular behavioral states, and how-if at all-it relates to learning. In this study, we sought to address these questions by analyzing coordination between CA1 cells and neurons of the deep layers of the medial entorhinal cortex (dMEC) while rats learned a novel spatial task. During movement, we found a subset of dMEC cells that were particularly locked to hippocampal LFP theta-band oscillations and that were preferentially coordinated with hippocampal replay during offline periods. Further, dMEC synchrony with CA1 replay peaked ∼10 ms after replay initiation in CA1, suggesting that the distributed replay reflects extra-hippocampal information propagation and is specific to "offline" periods. Finally, theta-modulated dMEC cells showed a striking experience-dependent increase in synchronization with hippocampal replay trajectories, mirroring the animals' acquisition of the novel task and coupling to the hippocampal local field. Together, these findings provide strong support for the hypothesis that synergistic hippocampal-cortical replay supports learning and highlights phase locking to hippocampal theta oscillations as a potential mechanism by which such cross-structural synchrony comes about.

Rapid learning of predictive maps with STDP and theta phase precession.

The predictive map hypothesis is a promising candidate principle for hippocampal function. A favoured formalisation of this hypothesis, called the successor representation, proposes that each place cell encodes the expected state occupancy of its target location in the near future. This predictive framework is supported by behavioural as well as electrophysiological evidence and has desirable consequences for both the generalisability and efficiency of reinforcement learning algorithms. However, it is unclear how the successor representation might be learnt in the brain. Error-driven temporal difference learning, commonly used to learn successor representations in artificial agents, is not known to be implemented in hippocampal networks. Instead, we demonstrate that spike-timing dependent plasticity (STDP), a form of Hebbian learning, acting on temporally compressed trajectories known as 'theta sweeps', is sufficient to rapidly learn a close approximation to the successor representation. The model is biologically plausible - it uses spiking neurons modulated by theta-band oscillations, diffuse and overlapping place cell-like state representations, and experimentally matched parameters. We show how this model maps onto known aspects of hippocampal circuitry and explains substantial variance in the temporal difference successor matrix, consequently giving rise to place cells that demonstrate experimentally observed successor representation-related phenomena including backwards expansion on a 1D track and elongation near walls in 2D. Finally, our model provides insight into the observed topographical ordering of place field sizes along the dorsal-ventral axis by showing this is necessary to prevent the detrimental mixing of larger place fields, which encode longer timescale successor representations, with more fine-grained predictions of spatial location.

State transitions in the statistically stable place cell population correspond to rate of perceptual change.

The hippocampus occupies a central role in mammalian navigation and memory. Yet an understanding of the rules that govern the statistics and granularity of the spatial code, as well as its interactions with perceptual stimuli, is lacking. We analyzed CA1 place cell activity recorded while rats foraged in different large-scale environments. We found that place cell activity was subject to an unexpected but precise homeostasis-the distribution of activity in the population as a whole being constant at all locations within and between environments. Using a virtual reconstruction of the largest environment, we showed that the rate of transition through this statistically stable population matches the rate of change in the animals' visual scene. Thus, place fields near boundaries were small but numerous, while in the environment's interior, they were larger but more dispersed. These results indicate that hippocampal spatial activity is governed by a small number of simple laws and, in particular, suggest the presence of an information-theoretic bound imposed by perception on the fidelity of the spatial memory system.

Ripple band phase precession of place cell firing during replay.

Neuronal "replay," in which place cell firing during rest recapitulates recently experienced trajectories, is thought to mediate the transmission of information from hippocampus to neocortex, but the mechanism for this transmission is unknown. Here, we show that replay uses a phase code to represent spatial trajectories by the phase of firing relative to the 150- to 250-Hz "ripple" oscillations that accompany replay events. This phase code is analogous to the theta phase precession of place cell firing during navigation, in which place cells fire at progressively earlier phases of the 6- to 12-Hz theta oscillation as their place field is traversed, providing information about self-location that is additional to the rate code and a necessary precursor of replay. Thus, during replay, each ripple cycle contains a "forward sweep" of decoded locations along the recapitulated trajectory. Our results indicate a novel encoding of trajectory information during replay and implicates phase coding as a general mechanism by which the hippocampus transmits experienced and replayed sequential information to downstream targets.

Spatial goal coding in the hippocampal formation.

The mammalian hippocampal formation contains several distinct populations of neurons involved in representing self-position and orientation. These neurons, which include place, grid, head direction, and boundary cells, are thought to collectively instantiate cognitive maps supporting flexible navigation. However, to flexibly navigate, it is necessary to also maintain internal representations of goal locations, such that goal-directed routes can be planned and executed. Although it has remained unclear how the mammalian brain represents goal locations, multiple neural candidates have recently been uncovered during different phases of navigation. For example, during planning, sequential activation of spatial cells may enable simulation of future routes toward the goal. During travel, modulation of spatial cells by the prospective route, or by distance and direction to the goal, may allow maintenance of route and goal-location information, supporting navigation on an ongoing basis. As the goal is approached, an increased activation of spatial cells may enable the goal location to become distinctly represented within cognitive maps, aiding goal localization. Lastly, after arrival at the goal, sequential activation of spatial cells may represent the just-taken route, enabling route learning and evaluation. Here, we review and synthesize these and other evidence for goal coding in mammalian brains, relate the experimental findings to predictions from computational models, and discuss outstanding questions and future challenges.

Predictive maps in rats and humans for spatial navigation.

Much of our understanding of navigation comes from the study of individual species, often with specific tasks tailored to those species. Here, we provide a novel experimental and analytic framework integrating across humans, rats, and simulated reinforcement learning (RL) agents to interrogate the dynamics of behavior during spatial navigation. We developed a novel open-field navigation task ("Tartarus maze") requiring dynamic adaptation (shortcuts and detours) to frequently changing obstructions on the path to a hidden goal. Humans and rats were remarkably similar in their trajectories. Both species showed the greatest similarity to RL agents utilizing a "successor representation," which creates a predictive map. Humans also displayed trajectory features similar to model-based RL agents, which implemented an optimal tree-search planning procedure. Our results help refine models seeking to explain mammalian navigation in dynamic environments and highlight the utility of modeling the behavior of different species to uncover the shared mechanisms that support behavior.

Experience-dependent rescaling of entorhinal grids.

The firing pattern of entorhinal 'grid cells' is thought to provide an intrinsic metric for space. We report a strong experience-dependent environmental influence: the spatial scales of the grids (which are aligned and have fixed relative sizes within each animal) vary parametrically with changes to a familiar environment's size and shape. Thus grid scale reflects an interaction between intrinsic, path-integrative calculation of location and learned associations to the external environment.

Interpreting wide-band neural activity using convolutional neural networks.

Rapid progress in technologies such as calcium imaging and electrophysiology has seen a dramatic increase in the size and extent of neural recordings. Even so, interpretation of this data requires considerable knowledge about the nature of the representation and often depends on manual operations. Decoding provides a means to infer the information content of such recordings but typically requires highly processed data and prior knowledge of the encoding scheme. Here, we developed a deep-learning framework able to decode sensory and behavioral variables directly from wide-band neural data. The network requires little user input and generalizes across stimuli, behaviors, brain regions, and recording techniques. Once trained, it can be analyzed to determine elements of the neural code that are informative about a given variable. We validated this approach using electrophysiological and calcium-imaging data from rodent auditory cortex and hippocampus as well as human electrocorticography (ECoG) data. We show successful decoding of finger movement, auditory stimuli, and spatial behaviors - including a novel representation of head direction - from raw neural activity.