Generative replay underlies compositional inference in the hippocampal-prefrontal circuit.

Human reasoning depends on reusing pieces of information by putting them together in new ways. However, very little is known about how compositional computation is implemented in the brain. Here, we ask participants to solve a series of problems that each require constructing a whole from a set of elements. With fMRI, we find that representations of novel constructed objects in the frontal cortex and hippocampus are relational and compositional. With MEG, we find that replay assembles elements into compounds, with each replay sequence constituting a hypothesis about a possible configuration of elements. The content of sequences evolves as participants solve each puzzle, progressing from predictable to uncertain elements and gradually converging on the correct configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry and a role for generative replay in compositional inference and hypothesis testing.

Human Replay Spontaneously Reorganizes Experience.

Knowledge abstracted from previous experiences can be transferred to aid new learning. Here, we asked whether such abstract knowledge immediately guides the replay of new experiences. We first trained participants on a rule defining an ordering of objects and then presented a novel set of objects in a scrambled order. Across two studies, we observed that representations of these novel objects were reactivated during a subsequent rest. As in rodents, human "replay" events occurred in sequences accelerated in time, compared to actual experience, and reversed their direction after a reward. Notably, replay did not simply recapitulate visual experience, but followed instead a sequence implied by learned abstract knowledge. Furthermore, each replay contained more than sensory representations of the relevant objects. A sensory code of object representations was preceded 50 ms by a code factorized into sequence position and sequence identity. We argue that this factorized representation facilitates the generalization of a previously learned structure to new objects.

Multiple-Timescale Representations of Space: Linking Memory to Navigation.

When navigating through space, we must maintain a representation of our position in real time; when recalling a past episode, a memory can come back in a flash. Interestingly, the brain's spatial representation system, including the hippocampus, supports these two distinct timescale functions. How are neural representations of space used in the service of both real-world navigation and internal mnemonic processes? Recent progress has identified sequences of hippocampal place cells, evolving at multiple timescales in accordance with either navigational behaviors or internal oscillations, that underlie these functions. We review experimental findings on experience-dependent modulation of these sequential representations and consider how they link real-world navigation to time-compressed memories. We further discuss recent work suggesting the prevalence of these sequences beyond hippocampus and propose that these multiple-timescale mechanisms may represent a general algorithm for organizing cell assemblies, potentially unifying the dual roles of the spatial representation system in memory and navigation.

CA2: A Highly Connected Intrahippocampal Relay.

Although Lorente de No' recognized the anatomical distinction of the hippocampal Cornu Ammonis (CA) 2 region, it had, until recently, been assigned no unique function. Its location between the key players of the circuit, CA3 and CA1, which along with the entorhinal cortex and dentate gyrus compose the classic trisynaptic circuit, further distracted research interest. However, the connectivity of CA2 pyramidal cells, together with unique patterns of gene expression, hints at a much larger contribution to hippocampal information processing than has been ascribed. Here we review recent advances that have identified new roles for CA2 in hippocampal centric processing, together with specialized functions in social memory and, potentially, as a broadcaster of novelty. These new data, together with CA2's role in disease, justify a closer look at how this small region exerts its influence and how it might best be exploited to understand and treat disease-related circuit dysfunctions.

Repetition dynamically and rapidly increases cortical, but not hippocampal, offline reactivation.

No sooner is an experience over than its neural representation begins to be transformed through memory reactivation during offline periods. The lion's share of prior research has focused on understanding offline reactivation within the hippocampus. However, it is hypothesized that consolidation processes involve offline reactivation in cortical regions as well as coordinated reactivation in the hippocampus and cortex. Using fMRI, we presented novel and repeated paired associates to participants during encoding and measured offline memory reactivation for those events during an immediate post-encoding rest period. post-encoding reactivation frequency of repeated and once-presented events did not differ in the hippocampus. However, offline reactivation in widespread cortical regions and hippocampal-cortical coordinated reactivation were significantly enhanced for repeated events. These results provide evidence that repetition might facilitate the distribution of memory representations across cortical networks, a hallmark of systems-level consolidation. Interestingly, we found that offline reactivation frequency in both hippocampus and cortex explained variance in behavioral success on an immediate associative recognition test for the once-presented information, potentially indicating a role of offline reactivation in maintaining these novel, weaker, memories. Together, our findings highlight that endogenous offline reactivation can be robustly and significantly modulated by study repetition.

scEVOLVE: cell-type incremental annotation without forgetting for single-cell RNA-seq data.

The evolution in single-cell RNA sequencing (scRNA-seq) technology has opened a new avenue for researchers to inspect cellular heterogeneity with single-cell precision. One crucial aspect of this technology is cell-type annotation, which is fundamental for any subsequent analysis in single-cell data mining. Recently, the scientific community has seen a surge in the development of automatic annotation methods aimed at this task. However, these methods generally operate at a steady-state total cell-type capacity, significantly restricting the cell annotation systems'capacity for continuous knowledge acquisition. Furthermore, creating a unified scRNA-seq annotation system remains challenged by the need to progressively expand its understanding of ever-increasing cell-type concepts derived from a continuous data stream. In response to these challenges, this paper presents a novel and challenging setting for annotation, namely cell-type incremental annotation. This concept is designed to perpetually enhance cell-type knowledge, gleaned from continuously incoming data. This task encounters difficulty with data stream samples that can only be observed once, leading to catastrophic forgetting. To address this problem, we introduce our breakthrough methodology termed scEVOLVE, an incremental annotation method. This innovative approach is built upon the methodology of contrastive sample replay combined with the fundamental principle of partition confidence maximization. Specifically, we initially retain and replay sections of the old data in each subsequent training phase, then establish a unique prototypical learning objective to mitigate the cell-type imbalance problem, as an alternative to using cross-entropy. To effectively emulate a model that trains concurrently with complete data, we introduce a cell-type decorrelation strategy that efficiently scatters feature representations of each cell type uniformly. We constructed the scEVOLVE framework with simplicity and ease of integration into most deep softmax-based single-cell annotation methods. Thorough experiments conducted on a range of meticulously constructed benchmarks consistently prove that our methodology can incrementally learn numerous cell types over an extended period, outperforming other strategies that fail quickly. As far as our knowledge extends, this is the first attempt to propose and formulate an end-to-end algorithm framework to address this new, practical task. Additionally, scEVOLVE, coded in Python using the Pytorch machine-learning library, is freely accessible at https://github.com/aimeeyaoyao/scEVOLVE.

Distinct replay signatures for prospective decision-making and memory preservation.

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts.

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

Interictal epileptiform discharges affect memory in an Alzheimer's disease mouse model.

Interictal epileptiform discharges (IEDs) are transient abnormal electrophysiological events commonly observed in epilepsy patients but are also present in other neurological diseases, such as Alzheimer's disease (AD). Understanding the role IEDs have on the hippocampal circuit is important for our understanding of the cognitive deficits seen in epilepsy and AD. We characterize and compare the IEDs of human epilepsy patients from microwire hippocampal recording with those of AD transgenic mice with implanted multilayer hippocampal silicon probes. Both the local field potential features and firing patterns of pyramidal cells and interneurons were similar in the mouse and human. We found that as IEDs emerged from the CA3-1 circuits, they recruited pyramidal cells and silenced interneurons, followed by post-IED suppression. IEDs suppressed the incidence and altered the properties of physiological sharp-wave ripples, altered their physiological properties, and interfered with the replay of place field sequences in a maze. In addition, IEDs in AD mice inversely correlated with daily memory performance. Together, our work implies that IEDs may present a common and epilepsy-independent phenomenon in neurodegenerative diseases that perturbs hippocampal-cortical communication and interferes with memory.

Memory Reactivation during Sleep Does Not Act Holistically on Object Memory.

Memory reactivation during sleep is thought to facilitate memory consolidation. Most sleep reactivation research has examined how reactivation of specific facts, objects, and associations benefits their overall retention. However, our memories are not unitary, and not all features of a memory persist in tandem over time. Instead, our memories are transformed, with some features strengthened and others weakened. Does sleep reactivation drive memory transformation? We leveraged the Targeted Memory Reactivation technique in an object category learning paradigm to examine this question. Participants (20 female, 14 male) learned three categories of novel objects, where each object had unique, distinguishing features as well as features shared with other members of its category. We used a real-time EEG protocol to cue the reactivation of these objects during sleep at moments optimized to generate reactivation events. We found that reactivation improved memory for distinguishing features while worsening memory for shared features, suggesting a differentiation process. The results indicate that sleep reactivation does not act holistically on object memories, instead supporting a transformation where some features are enhanced over others.

Replay Comes of Age.

Hippocampal place cells take part in sequenced patterns of reactivation after behavioral experience, known as replay. Since replay was first reported, nearly 20 years ago, many new results have been found, necessitating revision of the original interpretations. We review some of these results with a focus on the phenomenology of replay.

Brain-wide interactions during hippocampal sharp wave ripples.

During periods of disengagement from the environment, transient population bursts, known as sharp wave ripples (SPW-Rs), occur sporadically. While numerous experiments have characterized the bidirectional relationship between SPW-Rs and activity in chosen brain areas, the topographic relationship between different segments of the hippocampus and brain-wide target areas has not been studied at high temporal and spatial resolution. Yet, such knowledge is necessary to infer the direction of communication. We analyzed two publicly available datasets with simultaneous high-density silicon probe recordings from across the mouse forebrain. We found that SPW-Rs coincide with a transient brain-wide increase in functional connectivity. In addition, we show that the diversity in SPW-R features, such as their incidence, magnitude, and intrahippocampal topography in the septotemporal axis, are correlated with slower excitability fluctuations in cortical and subcortical areas. Further, variations in SPW-R features correlated with the timing, sign, and magnitude of downstream responses with large-amplitude SPW-Rs followed by transient silence in extrahippocampal structures. Our findings expand on previous results and demonstrate that the activity patterns in extrahippocampal structures depend both on the intrahippocampal topographic origin and magnitude of hippocampal SPW-Rs.

Neural waves and computation in a neural net model II: Data-like structures and the dynamics of episodic memory.

The computational resources of a neuromorphic network model introduced earlier were investigated in the first paper of this series. It was argued that a form of ubiquitous spontaneous local convolution enabled logical gate-like neural motifs to form into hierarchical feed-forward structures of the Hubel-Wiesel type. Here we investigate concomitant data-like structures and their dynamic rôle in memory formation, retrieval, and replay. The mechanisms give rise to the need for general inhibitory sculpting, and the simulation of the replay of episodic memories, well known in humans and recently observed in rats. Other consequences include explanations of such findings as the directional flows of neural waves in memory formation and retrieval, visual anomalies and memory deficits in schizophrenia, and the operation of GABA agonist drugs in suppressing episodic memories. We put forward the hypothesis that all neural logical operations and feature extractions are of the convolutional hierarchical type described here and in the earlier paper, and exemplified by the Hubel-Wiesel model of the visual cortex, but that in more general cases the precise geometric layering might be obscured and so far undetected.

Optimal STI controls for HIV patients based on an efficient deep Q learning method.

We investigate an efficient computational tool to suggest useful treatment regimens for people infected with the human immunodeficiency virus (HIV). Structured treatment interruption (STI) is a regimen in which therapeutic drugs are periodically administered and withdrawn to give patients relief from an arduous drug therapy. Numerous studies have been conducted to find better STI treatment strategies using various computational tools with mathematical models of HIV infection. In this paper, we leverage a modified version of the double deep Q network with prioritized experience replay to improve the performance of classic deep learning algorithms. Numerical simulation results show that our methodology produces significantly more optimal cost values for shorter treatment periods compared to other recent studies. Furthermore, our proposed algorithm performs well in one-day segment scenarios, whereas previous studies only reported results for five-day segment scenarios.

Replay of incidentally encoded novel odors in the rat.

Although events are not always known to be important when they occur, people can remember details about such incidentally encoded information using episodic memory. Sheridan et al. (2024) argued that rats replayed episodic memories of incidentally encoded information in an unexpected assessment of memory. In one task, rats reported the third-last item in an explicitly encoded list of trial-unique odors. In a second task, rats foraged in a radial maze in the absence of odors. On a critical test, rats foraged in the maze, but scented lids covered the food. Next, memory of the third-last odor was assessed. The rats correctly answered the unexpected question. Because the odors used in the critical test were the same as those used during training, automatically encoding odors for the purpose of taking an upcoming test of memory (stimulus generalization) may have been encouraged. Here, we provided an opportunity for incidental encoding of novel odors. Previously trained rats foraged in the radial maze with entirely novel odors covering the food. Next, memory of the third-last odor was assessed. The rats correctly answered the unexpected question. High accuracy when confronted with novel odors provides evidence that the rats did not automatically encode odors for the purpose of taking an upcoming test, ruling out stimulus generalization. We conclude that rats encode multiple pieces of putatively unimportant information, and later replayed a stream of novel episodic memories when that information was needed to solve an unexpected problem.

Validation of a rodent model of episodic memory replay.

Vivid episodic memories in humans have been described as the replay of the flow of past events in sequential order. Recently, Panoz-Brown et al. Current Biology, 28, 1628-1634, (2018) developed an olfactory memory task in which rats were presented with a list of trial-unique odors in an encoding context; next, in a distinctive memory assessment context, the rats were rewarded for choosing the second to last item from the list while avoiding other items from the list. In a different memory assessment context, the fourth to last item was rewarded. According to the episodic memory replay hypothesis, the rat remembers the list items and searches these items to find the item at the targeted locations in the list. However, events presented sequentially differ in memory trace strength, allowing a rat to use the relative familiarity of the memory traces, instead of episodic memory replay, to solve the task. Here, we directly manipulated memory trace strength by manipulating the odor intensity of target odors in both the list presentation and memory assessment. The rats relied on episodic memory replay to solve the memory assessment in conditions in which reliance on memory trace strength is ruled out. We conclude that rats are able to replay episodic memories.

Awake targeted memory reactivation doesn't work.

Memories are pliable and can be biased by post-encoding information. In targeted memory reactivation (TMR) studies, participants encode information then sleep, during which time sounds or scents that were previously associated with the encoded images are re-presented in an effort to trigger reactivation of the associated memory traces. Upon subsequent testing, memory for reactivated items is often enhanced. Is sleep essential for this process? The literature on awake TMR is small and findings are mixed. Here, we asked English-speaking adults to learn Japanese vocabulary words. During a subsequent active rest phase, participants played Tetris while sound cues associated with the vocabulary words were presented. Results showed that when memories were reactivated, they were either disrupted (Experiment 1) or unaffected (Experiments 2, 3). These findings indicate that awake TMR is not beneficial, and may actually impair subsequent memory. These findings have important implications for research on memory consolidation and reactivation.

Prioritized experience replays on a hippocampal predictive map for learning.

Hippocampal cells are central to spatial and predictive representations, and experience replays by place cells are crucial for learning and memory. Nonetheless, how hippocampal replay patterns dynamically change during the learning process remains to be elucidated. Here, we designed a spatial task in which rats learned a new behavioral trajectory for reward. We found that as rats updated their behavioral strategies for a novel salient location, hippocampal cell ensembles increased theta-sequences and sharp wave ripple-associated synchronous spikes that preferentially replayed salient locations and reward-related contexts in reverse order. The directionality and contents of the replays progressively varied with learning, including an optimized path that had never been exploited by the animals, suggesting prioritized replays of significant experiences on a predictive map. Online feedback blockade of sharp wave ripples during a learning process inhibited stabilizing optimized behavior. These results implicate learning-associated experience replays that act to learn and reinforce specific behavioral strategies.

Enhancing Stability and Performance in Mobile Robot Path Planning with PMR-Dueling DQN Algorithm.

Path planning for mobile robots in complex circumstances is still a challenging issue. This work introduces an improved deep reinforcement learning strategy for robot navigation that combines dueling architecture, Prioritized Experience Replay, and shaped Rewards. In a grid world and two Gazebo simulation environments with static and dynamic obstacles, the Dueling Deep Q-Network with Modified Rewards and Prioritized Experience Replay (PMR-Dueling DQN) algorithm is compared against Q-learning, DQN, and DDQN in terms of path optimality, collision avoidance, and learning speed. To encourage the best routes, the shaped Reward function takes into account target direction, obstacle avoidance, and distance. Prioritized replay concentrates training on important events while a dueling architecture separates value and advantage learning. The results show that the PMR-Dueling DQN has greatly increased convergence speed, stability, and overall performance across conditions. In both grid world and Gazebo environments the PMR-Dueling DQN achieved higher cumulative rewards. The combination of deep reinforcement learning with reward design, network architecture, and experience replay enables the PMR-Dueling DQN to surpass traditional approaches for robot path planning in complex environments.

HRP-OG: Online Learning with Generative Feature Replay for Hypertension Risk Prediction in a Nonstationary Environment.

Hypertension is a major risk factor for many serious diseases. With the aging population and lifestyle changes, the incidence of hypertension continues to rise, imposing a significant medical cost burden on patients and severely affecting their quality of life. Early intervention can greatly reduce the prevalence of hypertension. Research on hypertension early warning models based on electronic health records (EHRs) is an important and effective method for achieving early hypertension warning. However, limited by the scarcity and imbalance of multivisit records, and the nonstationary characteristics of hypertension features, it is difficult to predict the probability of hypertension prevalence in a patient effectively. Therefore, this study proposes an online hypertension monitoring model (HRP-OG) based on reinforcement learning and generative feature replay. It transforms the hypertension prediction problem into a sequential decision problem, achieving risk prediction of hypertension for patients using multivisit records. Sensors embedded in medical devices and wearables continuously capture real-time physiological data such as blood pressure, heart rate, and activity levels, which are integrated into the EHR. The fit between the samples generated by the generator and the real visit data is evaluated using maximum likelihood estimation, which can reduce the adversarial discrepancy between the feature space of hypertension and incoming incremental data, and the model is updated online based on real-time data using generative feature replay. The incorporation of sensor data ensures that the model adapts dynamically to changes in the condition of patients, facilitating timely interventions. In this study, the publicly available MIMIC-III data are used for validation, and the experimental results demonstrate that compared to existing advanced methods, HRP-OG can effectively improve the accuracy of hypertension risk prediction for few-shot multivisit record in nonstationary environments.