Vector-based navigation using grid-like representations in artificial agents.

Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go1,2. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning3-5 failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex 6 . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space7,8 and is critical for integrating self-motion (path integration)6,7,9 and planning direct trajectories to goals (vector-based navigation)7,10,11. Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types 12 . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation7,10,11, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.

Correction: An Unexpected Sequence of Events: Mismatch Detection in the Human Hippocampus.

[This corrects the article DOI: 10.1371/journal.pbio.0040424.].

Human-level control through deep reinforcement learning.

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

Overcoming catastrophic forgetting in neural networks.

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

Prefrontal Dynamics Associated with Efficient Detours and Shortcuts: A Combined Functional Magnetic Resonance Imaging and Magnetoencenphalography Study.

Central to the concept of the "cognitive map" is that it confers behavioral flexibility, allowing animals to take efficient detours, exploit shortcuts, and avoid alluring, but unhelpful, paths. The neural underpinnings of such naturalistic and flexible behavior remain unclear. In two neuroimaging experiments, we tested human participants on their ability to navigate to a set of goal locations in a virtual desert island riven by lava, which occasionally spread to block selected paths (necessitating detours) or receded to open new paths (affording real shortcuts or false shortcuts to be avoided). Detours activated a network of frontal regions compared with shortcuts. Activity in the right dorsolateral PFC specifically increased when participants encountered tempting false shortcuts that led along suboptimal paths that needed to be differentiated from real shortcuts. We also report modulation in event-related fields and theta power in these situations, providing insight to the temporal evolution of response to encountering detours and shortcuts. These results help inform current models as to how the brain supports navigation and planning in dynamic environments.

Backtracking during navigation is correlated with enhanced anterior cingulate activity and suppression of alpha oscillations and the 'default-mode' network.

Successful navigation can require realizing the current path choice was a mistake and the best strategy is to retreat along the recent path: 'back-track'. Despite the wealth of studies on the neural correlates of navigation little is known about backtracking. To explore the neural underpinnings of backtracking we tested humans during functional magnetic resonance imaging on their ability to navigate to a set of goal locations in a virtual desert island riven by lava which constrained the paths that could be taken. We found that on a subset of trials, participants spontaneously chose to backtrack and that the majority of these choices were optimal. During backtracking, activity increased in frontal regions and the dorsal anterior cingulate cortex, while activity was suppressed in regions associated with the core default-mode network. Using the same task, magnetoencephalography and a separate group of participants, we found that power in the alpha band was significantly decreased immediately prior to such backtracking events. These results highlight the importance for navigation of brain networks previously identified in processing internally-generated errors and that such error-detection responses may involve shifting the brain from default-mode states to aid successful spatial orientation.

Semantic representations in the temporal pole predict false memories.

Recent advances in neuroscience have given us unprecedented insight into the neural mechanisms of false memory, showing that artificial memories can be inserted into the memory cells of the hippocampus in a way that is indistinguishable from true memories. However, this alone is not enough to explain how false memories can arise naturally in the course of our daily lives. Cognitive psychology has demonstrated that many instances of false memory, both in the laboratory and the real world, can be attributed to semantic interference. Whereas previous studies have found that a diverse set of regions show some involvement in semantic false memory, none have revealed the nature of the semantic representations underpinning the phenomenon. Here we use fMRI with representational similarity analysis to search for a neural code consistent with semantic false memory. We find clear evidence that false memories emerge from a similarity-based neural code in the temporal pole, a region that has been called the "semantic hub" of the brain. We further show that each individual has a partially unique semantic code within the temporal pole, and this unique code can predict idiosyncratic patterns of memory errors. Finally, we show that the same neural code can also predict variation in true-memory performance, consistent with an adaptive perspective on false memory. Taken together, our findings reveal the underlying structure of neural representations of semantic knowledge, and how this semantic structure can both enhance and distort our memories.

Strong Evidence for Pattern Separation in Human Dentate Gyrus.

UNLABELLED: The hippocampus is proposed to be critical in distinguishing between similar experiences by performing pattern separation computations that create orthogonalized representations for related episodes. Previous neuroimaging studies have provided indirect evidence that the dentate gyrus (DG) and CA3 hippocampal subregions support pattern separation by inferring the nature of underlying representations from the observation of novelty signals. Here, we use ultra-high-resolution fMRI at 7 T and multivariate pattern analysis to provide compelling evidence that the DG subregion specifically sustains representations of similar scenes that are less overlapping than in other hippocampal (e.g., CA3) and medial temporal lobe regions (e.g., entorhinal cortex). Further, we provide evidence that novelty signals within the DG are stimulus specific rather than generic in nature. Our study, in providing a mechanistic link between novelty signals and the underlying representations, constitutes the first demonstration that the human DG performs pattern separation. SIGNIFICANCE STATEMENT: A fundamental property of an episodic memory system is the ability to minimize interference between similar episodes. The dentate gyrus (DG) subregion of the hippocampus is widely viewed to realize this function through a computation referred to as pattern separation, which creates distinct nonoverlapping neural codes for individual events. Here, we leveraged 7 T fMRI to test the hypothesis that this region supports pattern separation. Our results demonstrate that the DG supports representations of similar scenes that are less overlapping than those in neighboring subregions. The current study therefore is the first to offer compelling evidence that the human DG supports pattern separation by obtaining critical empirical data at the representational level: the level where this computation is defined.

Context-specific activation of hippocampus and SN/VTA by reward is related to enhanced long-term memory for embedded objects.

Animal studies indicate that hippocampal representations of environmental context modulate reward-related processing in the substantia nigra and ventral tegmental area (SN/VTA), a major origin of dopamine in the brain. Using functional magnetic resonance imaging (fMRI) in humans, we investigated the neural specificity of context-reward associations under conditions where the presence of perceptually similar neutral contexts imposed high demands on a putative hippocampal function, pattern separation. The design also allowed us to investigate how contextual reward enhances long-term memory for embedded neutral objects. SN/VTA activity underpinned specific context-reward associations in the face of perceptual similarity. A reward-related enhancement of long-term memory was restricted to the condition where the rewarding and the neutral contexts were perceptually similar, and in turn was linked to co-activation of the hippocampus (subfield DG/CA3) and SN/VTA. Thus, an ability of contextual reward to enhance memory for focal objects is closely linked to context-related engagement of hippocampal-SN/VTA circuitry.

Damage to the Ventromedial Prefrontal Cortex Impairs Learning from Observed Outcomes.

Individuals learn both from the outcomes of their own internally generated actions ("experiential learning") and from the observation of the consequences of externally generated actions ("observational learning"). While neuroscience research has focused principally on the neural mechanisms by which brain structures such as the ventromedial prefrontal cortex (vmPFC) support experiential learning, relatively less is known regarding how learning proceeds through passive observation. We explored the necessity of the vmPFC for observational learning by testing a group of patients with damage to the vmPFC as well as demographically matched normal comparison and brain-damaged comparison groups--and a single patient with bilateral dorsal prefrontal damage--using several value-learning tasks that required learning from direct experience, observational learning, or both. We found a specific impairment in observational learning in patients with vmPFC damage manifest in the reduced influence of previously observed rewards on current choices, despite a relatively intact capacity for experiential learning. The current study provides evidence that the vmPFC plays a critical role in observational learning, suggests that there are dissociable neural circuits for experiential and observational learning, and offers an important new extension of how the vmPFC contributes to learning and memory.

Frontoparietal representations of task context support the flexible control of goal-directed cognition.

Cognitive control allows stimulus-response processing to be aligned with internal goals and is thus central to intelligent, purposeful behavior. Control is thought to depend in part on the active representation of task information in prefrontal cortex (PFC), which provides a source of contextual bias on perception, decision making, and action. In the present study, we investigated the organization, influences, and consequences of context representation as human subjects performed a cued sorting task that required them to flexibly judge the relationship between pairs of multivalent stimuli. Using a connectivity-based parcellation of PFC and multivariate decoding analyses, we determined that context is specifically and transiently represented in a region spanning the inferior frontal sulcus during context-dependent decision making. We also found strong evidence that decision context is represented within the intraparietal sulcus, an area previously shown to be functionally networked with the inferior frontal sulcus at rest and during task performance. Rule-guided allocation of attention to different stimulus dimensions produced discriminable patterns of activation in visual cortex, providing a signature of top-down bias over perception. Furthermore, demands on cognitive control arising from the task structure modulated context representation, which was found to be strongest after a shift in task rules. When context representation in frontoparietal areas increased in strength, as measured by the discriminability of high-dimensional activation patterns, the bias on attended stimulus features was enhanced. These results provide novel evidence that illuminates the mechanisms by which humans flexibly guide behavior in complex environments.

Ventromedial prefrontal cortex drives hippocampal theta oscillations induced by mismatch computations.

Detecting environmental change is fundamental for adaptive behavior in an uncertain world. Previous work indicates the hippocampus supports the generation of novelty signals via implementation of a match-mismatch detector that signals when an incoming sensory input violates expectations based on past experience. While existing work has emphasized the particular contribution of the hippocampus, here we ask which other brain structures also contribute to match-mismatch detection. Furthermore, we leverage the fine-grained temporal resolution of magnetoencephalography (MEG) to investigate whether mismatch computations are spectrally confined to the theta range, based on the prominence of this range of oscillations in models of hippocampal function. By recording MEG activity while human subjects perform a task that incorporates conditions of match-mismatch novelty we show that mismatch signals are confined to the theta band and are expressed in both the hippocampus and ventromedial prefrontal cortex (vmPFC). Effective connectivity analyses (dynamic causal modeling) show that the hippocampus and vmPFC work as a functional circuit during mismatch detection. Surprisingly, our results suggest that the vmPFC drives the hippocampus during the generation and processing of mismatch signals. Our findings provide new evidence that the hippocampal-vmPFC circuit is engaged during novelty processing, which has implications for emerging theories regarding the role of vmPFC in memory.

Schema-driven facilitation of new hierarchy learning in the transitive inference paradigm.

Prior knowledge, in the form of a mental schema or framework, is viewed to facilitate the learning of new information in a range of experimental and everyday scenarios. Despite rising interest in the cognitive and neural mechanisms underlying schema-driven facilitation of new learning, few paradigms have been developed to examine this issue in humans. Here we develop a multiphase experimental scenario aimed at characterizing schema-based effects in the context of a paradigm that has been very widely used across species, the transitive inference task. We show that an associative schema, comprised of prior knowledge of the rank positions of familiar items in the hierarchy, has a marked effect on transitivity performance and the development of relational knowledge of the hierarchy that cannot be accounted for by more general changes in task strategy. Further, we show that participants are capable of deploying prior knowledge to successful effect under surprising conditions (i.e., when corrective feedback is totally absent), but only when the associative schema is robust. Finally, our results provide insights into the cognitive mechanisms underlying such schema-driven effects, and suggest that new hierarchy learning in the transitive inference task can occur through a contextual transfer mechanism that exploits the structure of associative experiences.

Language models, like humans, show content effects on reasoning tasks.

reasoning is a key ability for an intelligent system. Large language models (LMs) achieve above-chance performance on abstract reasoning tasks but exhibit many imperfections. However, human abstract reasoning is also imperfect. Human reasoning is affected by our real-world knowledge and beliefs, and shows notable "content effects"; humans reason more reliably when the semantic content of a problem supports the correct logical inferences. These content-entangled reasoning patterns are central to debates about the fundamental nature of human intelligence. Here, we investigate whether language models-whose prior expectations capture some aspects of human knowledge-similarly mix content into their answers to logic problems. We explored this question across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task. We evaluate state of the art LMs, as well as humans, and find that the LMs reflect many of the same qualitative human patterns on these tasks-like humans, models answer more accurately when the semantic content of a task supports the logical inferences. These parallels are reflected in accuracy patterns, and in some lower-level features like the relationship between LM confidence over possible answers and human response times. However, in some cases the humans and models behave differently-particularly on the Wason task, where humans perform much worse than large models, and exhibit a distinct error pattern. Our findings have implications for understanding possible contributors to these human cognitive effects, as well as the factors that influence language model performance.

Transitivity performance, relational hierarchy knowledge and awareness: results of an instructional framing manipulation.

The transitive inference (TI) paradigm has been widely used to examine the role of the hippocampus in generalization. Here we consider a surprising feature of experimental findings in this task: the relatively poor transitivity performance and levels of hierarchy knowledge achieved by adult human subjects. We focused on the influence of the task instructions on participants' subsequent performance--a single-word framing manipulation which either specified the relation between items as transitive (i.e., OLD-FRAME: choose which item is "older") or left it ambiguous (i.e., NO-FRAME: choose which item is "correct"). We show a marked but highly specific effect of manipulating prior knowledge through instruction: transitivity performance and levels of relational hierarchy knowledge were enhanced, but premise performance unchanged. Further, we show that hierarchy recall accuracy, but not conventional awareness scores, was a significant predictor of inferential performance across the entire group of participants. The current study has four main implications: first, our findings establish the importance of the task instructions, and prior knowledge, in the TI paradigm--suggesting that they influence the size of the overall hypothesis space (e.g., to favor a linear hierarchical structure over other possibilities in the OLD-FRAME). Second, the dissociable effects of the instructional frame on premise and inference performance provide evidence for the operation of distinct underlying mechanisms (i.e., an associative mechanism vs. relational hierarchy knowledge). Third, our findings suggest that a detailed measurement of hierarchy recall accuracy may be a more sensitive index of relational hierarchy knowledge, than conventional awareness score--and should be used in future studies investigating links between awareness and inferential performance. Finally, our study motivates an experimental setting that ensures robust hierarchy learning across participants--therefore facilitating study of the neural mechanisms underlying the learning and representation of linear hierarchies.

Double dissociation between hippocampal and parahippocampal responses to object-background context and scene novelty.

Several recent models of medial temporal lobe (MTL) function have proposed that the parahippocampal cortex processes context information, the perirhinal cortex processes item information, and the hippocampus binds together items and contexts. While evidence for a clear functional distinction between the perirhinal cortex and other regions within the MTL has been well supported, it has been less clear whether such a dissociation exists between the hippocampus and parahippocampal cortex. In the current study, we use a novel approach applying a functional magnetic resonance imaging adaptation paradigm to address these issues. During scanning, human subjects performed an incidental target detection task while viewing trial-unique sequentially presented pairs of natural scenes, each containing a single prominent object. We observed a striking double dissociation between the hippocampus and parahippocampal cortex, with the former showing a selective sensitivity to changes in the spatial relationship between objects and their background context and the latter engaged only by scene novelty. Our findings provide compelling support for the hypothesis that rapid item-context binding is a function of the hippocampus, rather than the parahippocampal cortex, with the former acting to detect relational novelty of this nature through its function as a match-mismatch detector.

Decoding representations of scenes in the medial temporal lobes.

Recent theoretical perspectives have suggested that the function of the human hippocampus, like its rodent counterpart, may be best characterized in terms of its information processing capacities. In this study, we use a combination of high-resolution functional magnetic resonance imaging, multivariate pattern analysis, and a simple decision making task, to test specific hypotheses concerning the role of the medial temporal lobe (MTL) in scene processing. We observed that while information that enabled two highly similar scenes to be distinguished was widely distributed throughout the MTL, more distinct scene representations were present in the hippocampus, consistent with its role in performing pattern separation. As well as viewing the two similar scenes, during scanning participants also viewed morphed scenes that spanned a continuum between the original two scenes. We found that patterns of hippocampal activity during morph trials, even when perceptual inputs were held entirely constant (i.e., in 50% morph trials), showed a robust relationship with participants' choices in the decision task. Our findings provide evidence for a specific computational role for the hippocampus in sustaining detailed representations of complex scenes, and shed new light on how the information processing capacities of the hippocampus may influence the decision making process.

The dynamics of hippocampal activation during encoding of overlapping sequences.

Sequence disambiguation, the process by which overlapping sequences are kept separate, has been proposed to underlie a wide range of memory capacities supported by the hippocampus, including episodic memory and spatial navigation. We used functional magnetic resonance imaging (fMRI) to explore the dynamic pattern of hippocampal activation during the encoding of sequences of faces. Activation in right posterior hippocampus, only during the encoding of overlapping sequences but not nonoverlapping sequences, was found to correlate robustly with a subject-specific behavioral index of sequence learning. Moreover, our data indicate that hippocampal activation in response to elements common to both sequences in the overlapping sequence pair, may be particularly important for accurate sequence encoding and retrieval. Together, these findings support the conclusion that the human hippocampus is involved in the earliest stage of sequence disambiguation, when memory representations are in the process of being created, and provide empirical support for contemporary computational models of hippocampal function.

The neurobiology of reference-dependent value computation.

A key focus of current research in neuroeconomics concerns how the human brain computes value. Although, value has generally been viewed as an absolute measure (e.g., expected value, reward magnitude), much evidence suggests that value is more often computed with respect to a changing reference point, rather than in isolation. Here, we present the results of a study aimed to dissociate brain regions involved in reference-independent (i.e., "absolute") value computations, from those involved in value computations relative to a reference point. During functional magnetic resonance imaging, subjects acted as buyers and sellers during a market exchange of lottery tickets. At a behavioral level, we demonstrate that subjects systematically accorded a higher value to objects they owned relative to those they did not, an effect that results from a shift in reference point (i.e., status quo bias or endowment effect). Our results show that activity in orbitofrontal cortex and dorsal striatum track parameters such as the expected value of lottery tickets indicating the computation of reference-independent value. In contrast, activity in ventral striatum indexed the degree to which stated prices, at a within-subjects and between-subjects level, were distorted with respect to a reference point. The findings speak to the neurobiological underpinnings of reference dependency during real market value computations.