Learning in deep neural networks and brains with similarity-weighted interleaved learning.

Understanding how the brain learns throughout a lifetime remains a long-standing challenge. In artificial neural networks (ANNs), incorporating novel information too rapidly results in catastrophic interference, i.e., abrupt loss of previously acquired knowledge. Complementary Learning Systems Theory (CLST) suggests that new memories can be gradually integrated into the neocortex by interleaving new memories with existing knowledge. This approach, however, has been assumed to require interleaving all existing knowledge every time something new is learned, which is implausible because it is time-consuming and requires a large amount of data. We show that deep, nonlinear ANNs can learn new information by interleaving only a subset of old items that share substantial representational similarity with the new information. By using such similarity-weighted interleaved learning (SWIL), ANNs can learn new information rapidly with a similar accuracy level and minimal interference, while using a much smaller number of old items presented per epoch (fast and data-efficient). SWIL is shown to work with various standard classification datasets (Fashion-MNIST, CIFAR10, and CIFAR100), deep neural network architectures, and in sequential learning frameworks. We show that data efficiency and speedup in learning new items are increased roughly proportionally to the number of nonoverlapping classes stored in the network, which implies an enormous possible speedup in human brains, which encode a high number of separate categories. Finally, we propose a theoretical model of how SWIL might be implemented in the brain.

Finding the Pattern: On-Line Extraction of Spatial Structure During Virtual Navigation.

While navigating the world, we pick up on patterns of where things tend to appear. According to theories of memory and studies of animal behavior, knowledge of these patterns emerges gradually over days or weeks via consolidation of individual navigation episodes. Here, we discovered that navigation patterns can also be extracted on-line, prior to the opportunity for off-line consolidation, as a result of rapid statistical learning. Thirty human participants navigated a virtual water maze in which platform locations were drawn from a spatial distribution. Within a single session, participants increasingly navigated through the mean of the distribution. This behavior was better simulated by random walks from a model that had only an explicit representation of the current mean, compared with a model that had only memory for the individual platform locations. These results suggest that participants rapidly summarized the underlying spatial distribution and used this statistical knowledge to guide future navigation.

Richness of information about novel words influences how episodic and semantic memory networks interact during lexicalization.

The complementary learning systems account of declarative memory suggests two distinct memory networks, a fast-mapping, episodic system involving the hippocampus, and a slower semantic memory system distributed across the neocortex in which new information is gradually integrated with existing representations. In this study, we investigated the extent to which these two networks are involved in the integration of novel words into the lexicon after extensive learning, and how the involvement of these networks changes after 24h. In particular, we explored whether having richer information at encoding influences the lexicalization trajectory. We trained participants with two sets of novel words, one where exposure was only to the words' phonological forms (the form-only condition), and one where pictures of unfamiliar objects were associated with the words' phonological forms (the picture-associated condition). A behavioral measure of lexical competition (indexing lexicalization) indicated stronger competition effects for the form-only words. Imaging (fMRI) results revealed greater involvement of phonological lexical processing areas immediately after training in the form-only condition, suggesting that tight connections were formed between novel words and existing lexical entries already at encoding. Retrieval of picture-associated novel words involved the episodic/hippocampal memory system more extensively. Although lexicalization was weaker in the picture-associated condition, overall memory strength was greater when tested after a 24hour delay, probably due to the availability of both episodic and lexical memory networks to aid retrieval. It appears that, during lexicalization of a novel word, the relative involvement of different memory networks differs according to the richness of the information about that word available at encoding.

A complementary learning systems model of how sleep moderates retrieval practice effects.

While many theories assume that sleep is critical in stabilizing and strengthening memories, our recent behavioral study (Liu & Ranganath, 2021, Psychonomic Bulletin & Review, 28[6], 2035-2044) suggests that sleep does not simply stabilize memories. Instead, it plays a more complex role, integrating information across two temporally distinct learning episodes. In the current study, we simulated the results of Liu and Ranganath (2021) using our biologically plausible computational model, TEACH, developed based on the complementary learning systems (CLS) framework. Our model suggests that when memories are activated during sleep, the reduced influence of temporal context establishes connections across temporally separated events through mutual training between the hippocampus and neocortex. In addition to providing a compelling mechanistic explanation for the selective effect of sleep, this model offers new examples of the diverse ways in which the cortex and hippocampus can interact during learning.

Bridging Neuroscience and AI: Environmental Enrichment as a Model for Forward Knowledge Transfer.

Continual learning (CL) refers to an agent's capability to learn from a continuous stream of data and transfer knowledge without forgetting old information. One crucial aspect of CL is forward transfer, i.e., improved and faster learning on a new task by leveraging information from prior knowledge. While this ability comes naturally to biological brains, it poses a significant challenge for artificial intelligence (AI). Here, we suggest that environmental enrichment (EE) can be used as a biological model for studying forward transfer, inspiring human-like AI development. EE refers to animal studies that enhance cognitive, social, motor, and sensory stimulation and is a model for what, in humans, is referred to as 'cognitive reserve'. Enriched animals show significant improvement in learning speed and performance on new tasks, typically exhibiting forward transfer. We explore anatomical, molecular, and neuronal changes post-EE and discuss how artificial neural networks (ANNs) can be used to predict neural computation changes after enriched experiences. Finally, we provide a synergistic way of combining neuroscience and AI research that paves the path toward developing AI capable of rapid and efficient new task learning.

Learning new vocabulary during childhood: effects of semantic training on lexical consolidation and integration.

Research suggests that word learning is an extended process, with offline consolidation crucial for the strengthening of new lexical representations and their integration with existing lexical knowledge (as measured by engagement in lexical competition). This supports a dual memory systems account, in which new information is initially sparsely encoded separately from existing knowledge and integrated with long-term memory over time. However, previous studies of this type exploited unnatural learning contexts, involving fictitious words in the absence of word meaning. In this study, 5- to 9-year-old children learned real science words (e.g., hippocampus) with or without semantic information. Children in both groups were slower to detect pauses in familiar competitor words (e.g., hippopotamus) relative to control words 24h after training but not immediately, confirming that offline consolidation is required before new words are integrated with the lexicon and engage in lexical competition. Children recalled more new words 24h after training than immediately (with similar improvements shown for the recall and recognition of new word meanings); however, children who were exposed to the meanings during training showed further improvements in recall after 1 week and outperformed children who were not exposed to meanings. These findings support the dual memory systems account of vocabulary acquisition and suggest that the association of a new phonological form with semantic information is critical for the development of stable lexical representations.

Is word learning capacity restored after a daytime nap?

Sleep is thought to be involved in the consolidation of new memories encoded during the day, as proposed by complementary learning systems accounts of memory. Other theories suggest that sleep's role in memory is not restricted to consolidation. The synaptic homeostasis hypothesis proposes that new learning is implemented in the brain through strengthening synaptic connections, a biologically costly process that gradually saturates encoding capacity during wake. During slow-wave sleep, synaptic strength is renormalized, thus restoring memory encoding ability. While the role of sleep in memory consolidation has been extensively documented, few human studies have explored the impact of sleep in restoring encoding ability, and none have looked at learning beyond episodic memory. In this registered report we test the predictions made by the complementary learning systems accounts and the synaptic homeostasis hypothesis regarding adult participants' ability to learn new words, and to integrate these words with existing knowledge. Participants took a polysomnographically-monitored daytime nap or remained awake prior to learning a set of new spoken words. Shortly after learning, and again on the following day, we measured participants' episodic memory for new words. We also assessed the degree to which newly learned words engage in competition with existing words. We predicted that sleep before encoding would result in better episodic memory for the words, and facilitate the overnight integration of new words with existing words. Based on existing literature and theory we further predicted that this restorative function is associated with slow-wave and sleep spindle activity. Our pre-registered analyses did not find a significant benefit of napping prior to encoding on word learning or integration. Exploratory analyses using a more sensitive measure of recall accuracy demonstrated significantly better performance in the nap condition compared to the no-nap condition in the immediate test. At the delayed test there was no longer a significant benefit of the nap. Of note, we found no significant effect of slow-wave activity prior to encoding on episodic memory or integration of newly learned words into the mental lexicon. However, we found that greater levels of Stage 2 sleep spindles were significantly associated with greater improvements in lexical competition from the immediate to the delayed test. Therefore, our results demonstrate some support for theories that implicate sleep spindles in restoring encoding capacity.

A model of bi-directional interactions between complementary learning systems for memory consolidation of sequential experiences.

The standard theory of memory consolidation posits a dual-store memory system: a fast-learning fast-decaying hippocampus that transfers memories to slow-learning long-term cortical storage. Hippocampal lesions interrupt this transfer, so recent memories are more likely to be lost than more remote memories. Existing models of memory consolidation that simulate this temporally graded retrograde amnesia operate only on static patterns or unitary variables as memories and study only one-way interaction from the hippocampus to the cortex. However, the mechanisms underlying the consolidation of episodes, which are sequential in nature and comprise multiple events, are not well-understood. The representation of learning for sequential experiences in the cortical-hippocampal network as a self-consistent dynamical system is not sufficiently addressed in prior models. Further, there is evidence for a bi-directional interaction between the two memory systems during offline periods, whereby the reactivation of waking neural patterns originating in the cortex triggers time-compressed sequential replays in the hippocampus, which in turn drive the consolidation of the pertinent sequence in the cortex. We have developed a computational model of memory encoding, consolidation, and recall for storing temporal sequences that explores the dynamics of this bi-directional interaction and time-compressed replays in four simulation experiments, providing novel insights into whether hippocampal learning needs to be suppressed for stable memory consolidation and into how new and old memories compete for limited replay opportunities during offline periods. The salience of experienced events, based on factors such as recency and frequency of use, is shown to have considerable impact on memory consolidation because it biases the relative probability that a particular event will be cued in the cortex during offline periods. In the presence of hippocampal learning during sleep, our model predicts that the fast-forgetting hippocampus can continually refresh the memory traces of a given episodic sequence if there are no competing experiences to be replayed.

Remembering the pattern: A longitudinal case study on statistical learning in spatial navigation and memory consolidation.

Distinct brain systems are thought to support statistical learning over different timescales. Regularities encountered during online perceptual experience can be acquired rapidly by the hippocampus. Further processing during offline consolidation can establish these regularities gradually in cortical regions, including the medial prefrontal cortex (mPFC). These mechanisms of statistical learning may be critical during spatial navigation, for which knowledge of the structure of an environment can facilitate future behavior. Rapid acquisition and prolonged retention of regularities have been investigated in isolation, but how they interact in the context of spatial navigation is unknown. We had the rare opportunity to study the brain systems underlying both rapid and gradual timescales of statistical learning using intracranial electroencephalography (iEEG) longitudinally in the same patient over a period of three weeks. As hypothesized, spatial patterns were represented in the hippocampus but not mPFC for up to one week after statistical learning and then represented in the mPFC but not hippocampus two and three weeks after statistical learning. Taken together, these findings suggest that the hippocampus may contribute to the initial extraction of regularities prior to cortical consolidation.

Complementary Structure-Learning Neural Networks for Relational Reasoning.

The neural mechanisms supporting flexible relational inferences, especially in novel situations, are a major focus of current research. In the complementary learning systems framework, pattern separation in the hippocampus allows rapid learning in novel environments, while slower learning in neocortex accumulates small weight changes to extract systematic structure from well-learned environments. In this work, we adapt this framework to a task from a recent fMRI experiment where novel transitive inferences must be made according to implicit relational structure. We show that computational models capturing the basic cognitive properties of these two systems can explain relational transitive inferences in both familiar and novel environments, and reproduce key phenomena observed in the fMRI experiment.

Evidence of hippocampal learning in human infants.

The hippocampus is essential for human memory.1 The protracted maturation of memory capacities from infancy through early childhood2-4 is thus often attributed to hippocampal immaturity.5-7 The hippocampus of human infants has been characterized in terms of anatomy,8,9 but its function has never been tested directly because of technical challenges.10,11 Here, we use recently developed methods for task-based fMRI in awake human infants12 to test the hypothesis that the infant hippocampus supports statistical learning.13-15 Hippocampal activity increased with exposure to visual sequences of objects when the temporal order contained regularities to be learned, compared to when the order was random. Despite the hippocampus doubling in anatomical volume across infancy, learning-related functional activity bore no relationship to age. This suggests that the hippocampus is recruited for statistical learning at the youngest ages in our sample, around 3 months. Within the hippocampus, statistical learning was clearer in anterior than posterior divisions. This is consistent with the theory that statistical learning occurs in the monosynaptic pathway,16 which is more strongly represented in the anterior hippocampus.17,18 The monosynaptic pathway develops earlier than the trisynaptic pathway, which is linked to episodic memory,19,20 raising the possibility that the infant hippocampus participates in statistical learning before it forms durable memories. Beyond the hippocampus, the medial prefrontal cortex showed statistical learning, consistent with its role in adult memory integration21 and generalization.22 These results suggest that the hippocampus supports the vital ability of infants to extract the structure of their environment through experience.

A complementary learning systems approach to temporal difference learning.

Complementary Learning Systems (CLS) theory suggests that the brain uses a 'neocortical' and a 'hippocampal' learning system to achieve complex behaviour. These two systems are complementary in that the 'neocortical' system relies on slow learning of distributed representations while the 'hippocampal' system relies on fast learning of pattern-separated representations. Both of these systems project to the striatum, which is a key neural structure in the brain's implementation of Reinforcement Learning (RL). Current deep RL approaches share similarities with a 'neocortical' system because they slowly learn distributed representations through backpropagation in Deep Neural Networks (DNNs). An ongoing criticism of such approaches is that they are data inefficient and lack flexibility. CLS theory suggests that the addition of a 'hippocampal' system could address these criticisms. In the present study we propose a novel algorithm known as Complementary Temporal Difference Learning (CTDL), which combines a DNN with a Self-Organizing Map (SOM) to obtain the benefits of both a 'neocortical' and a 'hippocampal' system. Key features of CTDL include the use of Temporal Difference (TD) error to update a SOM and the combination of a SOM and DNN to calculate action values. We evaluate CTDL on Grid World, Cart-Pole and Continuous Mountain Car tasks and show several benefits over the classic Deep Q-Network (DQN) approach. These results demonstrate (1) the utility of complementary learning systems for the evaluation of actions, (2) that the TD error signal is a useful form of communication between the two systems and (3) that our approach extends to both discrete and continuous state and action spaces.

Integration of new information in memory: new insights from a complementary learning systems perspective.

According to complementary learning systems theory, integrating new memories into the neocortex of the brain without interfering with what is already known depends on a gradual learning process, interleaving new items with previously learned items. However, empirical studies show that information consistent with prior knowledge can sometimes be integrated very quickly. We use artificial neural networks with properties like those we attribute to the neocortex to develop an understanding of the role of consistency with prior knowledge in putatively neocortex-like learning systems, providing new insights into when integration will be fast or slow and how integration might be made more efficient when the items to be learned are hierarchically structured. The work relies on deep linear networks that capture the qualitative aspects of the learning dynamics of the more complex nonlinear networks used in previous work. The time course of learning in these networks can be linked to the hierarchical structure in the training data, captured mathematically as a set of dimensions that correspond to the branches in the hierarchy. In this context, a new item to be learned can be characterized as having aspects that project onto previously known dimensions, and others that require adding a new branch/dimension. The projection onto the known dimensions can be learned rapidly without interleaving, but learning the new dimension requires gradual interleaved learning. When a new item only overlaps with items within one branch of a hierarchy, interleaving can focus on the previously known items within this branch, resulting in faster integration with less interleaving overall. The discussion considers how the brain might exploit these facts to make learning more efficient and highlights predictions about what aspects of new information might be hard or easy to learn. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

The prevalence and importance of statistical learning in human cognition and behavior.

Statistical learning, the ability to extract regularities from the environment over time, has become a topic of burgeoning interest. Its influence on behavior, spanning infancy to adulthood, has been demonstrated across a range of tasks, both those labeled as tests of statistical learning and those from other learning domains that predated statistical learning research or that are not typically considered in the context of that literature. Given this pervasive role in human cognition, statistical learning has the potential to reconcile seemingly distinct learning phenomena and may be an under-appreciated but important contributor to a wide range of human behaviors that are studied as unrelated processes, such as episodic memory and spatial navigation.

The role of complementary learning systems in learning and consolidation in a quasi-regular domain.

We examine the role of off-line memory consolidation processes in the learning and retention of a new quasi-regular linguistic system similar to the English past tense. Quasi-regular systems are characterized by a dominance of systematic, regular forms (e.g., walk-walked, jump-jumped) alongside a smaller number of high frequency irregulars (e.g., sit-sat, go-went), and are found across many cognitive domains, from spelling-sound mappings to inflectional morphology to semantic cognition. Participants were trained on the novel morphological system using an artificial language paradigm, and then tested after different delays. Based on a complementary systems account of memory, we predicted that irregular forms would show stronger off-line changes due to consolidation processes. Across two experiments, participants were tested either immediately after learning, 12 h later with or without sleep, or 24 h later. Testing involved generalization of the morphological patterns to previously unseen words (both experiments) as well as recall of the trained words (Experiment 2). In generalization, participants showed 'default' regularization across a range of novel forms, as well as irregularization for previously unseen items that were similar to unique high-frequency irregular trained forms. Both patterns of performance remained stable across the delays. Generalizations involving competing tendencies to regularize and irregularize were balanced between the two immediately after learning. Crucially, at both 12-h delays the tendency to irregularize in these cases was strengthened, with further strengthening after 24 h. Consolidated knowledge of both regular and irregular trained items contributed significantly to generalization performance, with evidence of strengthening of irregular forms and weakening of regular forms. We interpret these findings in the context of a complementary systems model, and discuss how maintenance, strengthening, and forgetting of the new memories across sleep and wake can play a role in acquiring quasi-regular systems.

Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization.

Artificial autonomous agents and robots interacting in complex environments are required to continually acquire and fine-tune knowledge over sustained periods of time. The ability to learn from continuous streams of information is referred to as lifelong learning and represents a long-standing challenge for neural network models due to catastrophic forgetting in which novel sensory experience interferes with existing representations and leads to abrupt decreases in the performance on previously acquired knowledge. Computational models of lifelong learning typically alleviate catastrophic forgetting in experimental scenarios with given datasets of static images and limited complexity, thereby differing significantly from the conditions artificial agents are exposed to. In more natural settings, sequential information may become progressively available over time and access to previous experience may be restricted. Therefore, specialized neural network mechanisms are required that adapt to novel sequential experience while preventing disruptive interference with existing representations. In this paper, we propose a dual-memory self-organizing architecture for lifelong learning scenarios. The architecture comprises two growing recurrent networks with the complementary tasks of learning object instances (episodic memory) and categories (semantic memory). Both growing networks can expand in response to novel sensory experience: the episodic memory learns fine-grained spatiotemporal representations of object instances in an unsupervised fashion while the semantic memory uses task-relevant signals to regulate structural plasticity levels and develop more compact representations from episodic experience. For the consolidation of knowledge in the absence of external sensory input, the episodic memory periodically replays trajectories of neural reactivations. We evaluate the proposed model on the CORe50 benchmark dataset for continuous object recognition, showing that we significantly outperform current methods of lifelong learning in three different incremental learning scenarios.

Dual-memory neural networks for modeling cognitive activities of humans via wearable sensors.

Wearable devices, such as smart glasses and watches, allow for continuous recording of everyday life in a real world over an extended period of time or lifelong. This possibility helps better understand the cognitive behavior of humans in real life as well as build human-aware intelligent agents for practical purposes. However, modeling the human cognitive activity from wearable-sensor data stream is challenging because learning new information often results in loss of previously acquired information, causing a problem known as catastrophic forgetting. Here we propose a deep-learning neural network architecture that resolves the catastrophic forgetting problem. Based on the neurocognitive theory of the complementary learning systems of the neocortex and hippocampus, we introduce a dual memory architecture (DMA) that, on one hand, slowly acquires the structured knowledge representations and, on the other hand, rapidly learns the specifics of individual experiences. The DMA system learns continuously through incremental feature adaptation and weight transfer. We evaluate the performance on two real-life datasets, the CIFAR-10 image-stream dataset and the 46-day Lifelog dataset collected from Google Glass, showing that the proposed model outperforms other online learning methods.

Eye-tracking the time-course of novel word learning and lexical competition in adults and children.

Lexical competition is a hallmark of proficient, automatic word recognition. Previous research suggests that there is a delay before a new spoken word becomes engaged in this process, with sleep playing an important role. However, data from one method - the visual world paradigm - consistently show competition without a delay. We trained 42 adults and 40 children (aged 7-8) on novel word-object pairings, and employed this paradigm to measure the time-course of lexical competition. Fixations to novel objects upon hearing existing words (e.g., looks to the novel object biscal upon hearing "click on the biscuit") were compared to fixations on untrained objects. Novel word-object pairings learned immediately before testing and those learned the previous day exhibited significant competition effects, with stronger competition for the previous day pairings for children but not adults. Crucially, this competition effect was significantly smaller for novel than existing competitors (e.g., looks to candy upon hearing "click on the candle"), suggesting that novel items may not compete for recognition like fully-fledged lexical items, even after 24h. Explicit memory (cued recall) was superior for words learned the day before testing, particularly for children; this effect (but not the lexical competition effects) correlated with sleep-spindle density. Together, the results suggest that different aspects of new word learning follow different time courses: visual world competition effects can emerge swiftly, but are qualitatively different from those observed with established words, and are less reliant upon sleep. Furthermore, the findings fit with the view that word learning earlier in development is boosted by sleep to a greater degree.

Oscillations and Episodic Memory: Addressing the Synchronization/Desynchronization Conundrum.

Brain oscillations are one of the core mechanisms underlying episodic memory. However, while some studies highlight the role of synchronized oscillatory activity, others highlight the role of desynchronized activity. We here describe a framework to resolve this conundrum and integrate these two opposing oscillatory behaviors. Specifically, we argue that the synchronization and desynchronization reflect a division of labor between a hippocampal and a neocortical system, respectively. We describe a novel oscillatory framework that integrates synchronization and desynchronization mechanisms to explain how the two systems interact in the service of episodic memory.

Stroop effects from newly learned color words: effects of memory consolidation and episodic context.

The Stroop task is an excellent tool to test whether reading a word automatically activates its associated meaning, and it has been widely used in mono- and bilingual contexts. Despite of its ubiquity, the task has not yet been employed to test the automaticity of recently established word-concept links in novel-word-learning studies, under strict experimental control of learning and testing conditions. In three experiments, we thus paired novel words with native language (German) color words via lexical association and subsequently tested these words in a manual version of the Stroop task. Two crucial findings emerged: When novel word Stroop trials appeared intermixed among native-word trials, the novel-word Stroop effect was observed immediately after the learning phase. If no native color words were present in a Stroop block, the novel-word Stroop effect only emerged 24 h later. These results suggest that the automatic availability of a novel word's meaning depends either on supportive context from the learning episode and/or on sufficient time for memory consolidation. We discuss how these results can be reconciled with the complementary learning systems account of word learning.