Closed-Loop Slow-Wave tACS Improves Sleep-Dependent Long-Term Memory Generalization by Modulating Endogenous Oscillations.

Benefits in long-term memory retention and generalization have been shown to be related to sleep-dependent processes, which correlate with neural oscillations as measured by changes in electric potential. The specificity and causal role of these oscillations, however, are still poorly understood. Here, we investigated the potential for augmenting endogenous slow-wave (SW) oscillations in humans with closed-loop transcranial alternating current stimulation (tACS) with an aim toward enhancing the consolidation of recent experiences into long-term memory. Sixteen (three female) participants were trained presleep on a target detection task identifying targets hidden in complex visual scenes. During post-training sleep, closed-loop SW detection and stimulation were used to deliver tACS matching the phase and frequency of the dominant oscillation in the range of 0.5-1.2 Hz. Changes in performance were assessed the following day using test images that were identical to the training ("repeated"), and images generated from training scenes but with novel viewpoints ("generalized"). Results showed that active SW tACS during sleep enhanced the postsleep versus presleep target detection accuracy for the generalized images compared with sham nights, while no significant change was found for repeated images. Using a frequency-agnostic clustering approach sensitive to stimulation-induced spectral power changes in scalp EEG, this behavioral enhancement significantly correlated with both a poststimulation increase and a subsequent decrease in measured spectral power within the SW band, which in turn showed increased coupling with spindle amplitude. These results suggest that augmenting endogenous SW oscillations can enhance consolidation by specifically improving generalization over recognition or cued recall.SIGNIFICANCE STATEMENT This human study demonstrates the use of a closed-loop noninvasive brain stimulation method to enhance endogenous neural oscillations during sleep with the effect of improving consolidation of recent experiences into long-term memory. Here we show that transient slow oscillatory transcranial alternating current stimulation (tACS) triggered by endogenous slow oscillations and matching their frequency and phase can increase slow-wave power and coupling with spindles. Further, this increase correlates with overnight improvements in generalization of recent training to facilitate performance in a target detection task. We also provide novel evidence for a tACS-induced refractory period following the tACS-induced increase. Here slow-wave power is temporarily reduced relative to sham stimulation, which nonetheless maintains a positive relationship with behavioral improvements.

Classification aided analysis of oscillatory signatures in controlled retrieval.

Control processes are critical for both facilitating and suppressing memory retrieval, but these processes are not well understood. The current work, inspired by a similar fMRI design (Detre et al., in press), used a modified Think/No-Think(TNT) paradigm to investigate the neural signatures of volition over enhancing and suppressing memory retrieval. Previous studies have shown memory enhancement when well-learned stimulus pairs are restudied in cued recall ("Recall or think of studied pair item"), and degradation when restudied with cued suppression ("Avoid thinking of studied pair item"). We used category-based (faces vs. scenes) multivariate classification of electroencephalography signals to determine if individual target items were successfully retrieved or suppressed. A logistic regression based on classifier output determined that retrieval activation during the cued recall/suppression period was a predictor for subsequent memory. Labeling trials with this internal measure, as opposed to their nominal Think vs. No-Think condition, revealed the classic TNT pattern of enhanced memory for successful cued-retrieval and degraded memory for cued-suppression. This classification process enabled a more selective investigation into the time-frequency signatures of control over retrieval. Comparing controlled retrieval vs. controlled suppression, results showed more prominent Theta oscillations (3 to 8 Hz) in controlled retrieval. Beta oscillations (12 to 30 Hz) were involved in high levels of both controlled retrieval and suppression, suggesting it may have a more general control-related role. These results suggest unique roles for these frequency bands in retrieval processes.

Theta coordinated error-driven learning in the hippocampus.

The learning mechanism in the hippocampus has almost universally been assumed to be Hebbian in nature, where individual neurons in an engram join together with synaptic weight increases to support facilitated recall of memories later. However, it is also widely known that Hebbian learning mechanisms impose significant capacity constraints, and are generally less computationally powerful than learning mechanisms that take advantage of error signals. We show that the differential phase relationships of hippocampal subfields within the overall theta rhythm enable a powerful form of error-driven learning, which results in significantly greater capacity, as shown in computer simulations. In one phase of the theta cycle, the bidirectional connectivity between CA1 and entorhinal cortex can be trained in an error-driven fashion to learn to effectively encode the cortical inputs in a compact and sparse form over CA1. In a subsequent portion of the theta cycle, the system attempts to recall an existing memory, via the pathway from entorhinal cortex to CA3 and CA1. Finally the full theta cycle completes when a strong target encoding representation of the current input is imposed onto the CA1 via direct projections from entorhinal cortex. The difference between this target encoding and the attempted recall of the same representation on CA1 constitutes an error signal that can drive the learning of CA3 to CA1 synapses. This CA3 to CA1 pathway is critical for enabling full reinstatement of recalled hippocampal memories out in cortex. Taken together, these new learning dynamics enable a much more robust, high-capacity model of hippocampal learning than was available previously under the classical Hebbian model.

Closed-Loop tACS Delivered during Slow-Wave Sleep Reduces Retroactive Interference on a Paired-Associates Learning Task.

Previous studies have found a benefit of closed-loop transcranial alternating current stimulation (CL-tACS) matched to ongoing slow-wave oscillations (SWO) during sleep on memory consolidation for words in a paired associates task (PAT). Here, we examined the effects of CL-tACS in a retroactive interference PAT (ri-PAT) paradigm, where additional stimuli were presented to increase interference and reduce memory performance. Thirty-one participants were tested on a PAT before sleep, and CL-tACS was applied over the right and left DLPFC (F3 and F4) vs. mastoids for five cycles after detection of the onset of each discrete event of SWO during sleep. Participants were awoken the following morning, learned a new PAT list, and then were tested on the original list. There was a significant effect of stimulation condition (p = 0.04297; Cohen's d = 0.768), where verum stimulation resulted in reduced retroactive interference compared with sham and a significant interaction of encoding strength and stimulation condition (p = 0.03591). Planned simple effects testing within levels of encoding revealed a significant effect of stimulation only for low-encoders (p = 0.0066; Cohen's d = 1.075) but not high-encoders. We demonstrate here for the first time that CL-tACS during sleep can enhance the protective benefits on retroactive interference in participants who have lower encoding aptitude.

A domain-agnostic approach for characterization of lifelong learning systems.

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of (1) Continuous Learning, (2) Transfer and Adaptation, and (3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

Evidence for area CA1 as a match/mismatch detector: a high-resolution fMRI study of the human hippocampus.

The hippocampus is proposed to switch between memory encoding and retrieval by continually computing the overlap between what is expected and what is encountered. Central to this hypothesis is that area CA1 performs this calculation. However, empirical evidence for this is lacking. To test the theoretical role of area CA1 in match/mismatch detection, we had subjects study complex stimuli and then, during high-resolution fMRI scanning, make memory judgments about probes that either matched or mismatched expectations. More than any other hippocampal subfield, area CA1 displayed responses consistent with a match/mismatch detector. Specifically, the responses in area CA1 tracked the total number of changes present in the probe. Additionally, area CA1 was sensitive to both behaviorally relevant and irrelevant changes, a key feature of an automatic comparator. These results are consistent with, and provide the first evidence in humans for, the theoretically important role of area CA1 as a match/mismatch detector.

Context meta-reinforcement learning via neuromodulation.

Meta-reinforcement learning (meta-RL) algorithms enable agents to adapt quickly to tasks from few samples in dynamic environments. Such a feat is achieved through dynamic representations in an agent's policy network (obtained via reasoning about task context, model parameter updates, or both). However, obtaining rich dynamic representations for fast adaptation beyond simple benchmark problems is challenging due to the burden placed on the policy network to accommodate different policies. This paper addresses the challenge by introducing neuromodulation as a modular component to augment a standard policy network that regulates neuronal activities in order to produce efficient dynamic representations for task adaptation. The proposed extension to the policy network is evaluated across multiple discrete and continuous control environments of increasing complexity. To prove the generality and benefits of the extension in meta-RL, the neuromodulated network was applied to two state-of-the-art meta-RL algorithms (CAVIA and PEARL). The result demonstrates that meta-RL augmented with neuromodulation produces significantly better result and richer dynamic representations in comparison to the baselines.

Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture.

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

Identifying objects impairs knowledge of other objects: a relearning explanation for the neural repetition effect.

Different items in long-term knowledge are stored in the neocortex as partially overlapping representations that can be altered slightly with usage. This encoding scheme affords well-documented benefits, but potential costs have not been well explored. Here we use functional magnetic resonance imaging (fMRI), neurocomputational modeling, and electrophysiological measures to show that strengthening some visual object representations not only enhances the subsequent ability to identify those (repeated) objects-an effect long known as repetition priming-but also impairs the ability to identify other (non-repeated) objects-a new effect labeled antipriming. As a result, the non-repeated objects elicit increased neural activity likely for the purpose of reestablishing their previously weakened representations. These results suggest a novel reevaluation of the ubiquitously observed repetition effect on neural activity, and they indicate that maintenance relearning may be a crucial aspect of preserving overlapping neural representations of visual objects in long-term memory.

Transcranial Current Stimulation During Sleep Facilitates Insight into Temporal Rules, but does not Consolidate Memories of Individual Sequential Experiences.

Slow-wave sleep (SWS) is known to contribute to memory consolidation, likely through the reactivation of previously encoded waking experiences. Contemporary studies demonstrate that when auditory or olfactory stimulation is administered during memory encoding and then reapplied during SWS, memory consolidation can be enhanced, an effect that is believed to rely on targeted memory reactivation (TMR) induced by the sensory stimulation. Here, we show that transcranial current stimulations (tCS) during sleep can also be used to induce TMR, resulting in the facilitation of high-level cognitive processes. Participants were exposed to repeating sequences in a realistic 3D immersive environment while being stimulated with particular tCS patterns. A subset of these tCS patterns was then reapplied during sleep stages N2 and SWS coupled to slow oscillations in a closed-loop manner. We found that in contrast to our initial hypothesis, performance for the sequences corresponding to the reapplied tCS patterns was no better than for other sequences that received stimulations only during wake or not at all. In contrast, we found that the more stimulations participants received overnight, the more likely they were to detect temporal regularities governing the learned sequences the following morning, with tCS-induced beta power modulations during sleep mediating this effect.

One-Shot Tagging During Wake and Cueing During Sleep With Spatiotemporal Patterns of Transcranial Electrical Stimulation Can Boost Long-Term Metamemory of Individual Episodes in Humans.

Targeted memory reactivation (TMR) during slow-wave oscillations (SWOs) in sleep has been demonstrated with sensory cues to achieve about 5-12% improvement in post-nap memory performance on simple laboratory tasks. But prior work has not yet addressed the one-shot aspect of episodic memory acquisition, or dealt with the presence of interference from ambient environmental cues in real-world settings. Further, TMR with sensory cues may not be scalable to the multitude of experiences over one's lifetime. We designed a novel non-invasive non-sensory paradigm that tags one-shot experiences of minute-long naturalistic episodes in immersive virtual reality (VR) with unique spatiotemporal amplitude-modulated patterns (STAMPs) of transcranial electrical stimulation (tES). In particular, we demonstrated that these STAMPs can be re-applied as brief pulses during SWOs in sleep to achieve about 10-20% improvement in the metamemory of targeted episodes compared to the control episodes at 48 hours after initial viewing. We found that STAMPs can not only facilitate but also impair metamemory for the targeted episodes based on an interaction between pre-sleep metamemory and the number of STAMP applications during sleep. Overnight metamemory improvements were mediated by spectral power increases following the offset of STAMPs in the slow-spindle band (8-12 Hz) for left temporal areas in the scalp electroencephalography (EEG) during sleep. These results prescribe an optimal strategy to leverage STAMPs for boosting metamemory and suggest that real-world episodic memories can be modulated in a targeted manner even with coarser, non-invasive spatiotemporal stimulation.

The Benefits of Closed-Loop Transcranial Alternating Current Stimulation on Subjective Sleep Quality.

BACKGROUND: Poor sleep quality is a common complaint, affecting over one third of people in the United States. While sleep quality is thought to be related to slow-wave sleep (SWS), there has been little investigation to address whether modulating slow-wave oscillations (SWOs) that characterize SWS could impact sleep quality. Here we examined whether closed-loop transcranial alternating current stimulation (CL-tACS) applied during sleep impacts sleep quality and efficiency. METHODS: CL-tACS was used in 21 participants delivered at the same frequency and in phase with endogenous SWOs during sleep. Sleep quality was assessed in the morning following either verum or sham control stimulation during sleep, with order counterbalanced within participants. RESULTS: Higher sleep quality and efficiency were found after verum stimulation nights compared to control. The largest effects on sleep quality were found immediately following an adaptation night in the laboratory for which sleep quality was reduced. CONCLUSIONS: Applying CL-tACS at the same frequency and phase as endogenous SWOs may offer a novel method to improve subjective sleep quality after a night with poor quality sleep. CL-tACS might be helpful for increasing sleep quality and efficiency in otherwise healthy people, and in patients with clinical disorders that involve sleep deficits.

Dose-Dependent Effects of Closed-Loop tACS Delivered During Slow-Wave Oscillations on Memory Consolidation.

Sleep is critically important to consolidate information learned throughout the day. Slow-wave sleep (SWS) serves to consolidate declarative memories, a process previously modulated with open-loop non-invasive electrical stimulation, though not always effectively. These failures to replicate could be explained by the fact that stimulation has only been performed in open-loop, as opposed to closed-loop where phase and frequency of the endogenous slow-wave oscillations (SWOs) are matched for optimal timing. The current study investigated the effects of closed-loop transcranial Alternating Current Stimulation (tACS) targeting SWOs during sleep on memory consolidation. 21 participants took part in a three-night, counterbalanced, randomized, single-blind, within-subjects study, investigating performance changes (correct rate and F1 score) on images in a target detection task over 24 h. During sleep, 1.5 mA closed-loop tACS was delivered in phase over electrodes at F3 and F4 and 180° out of phase over electrodes at bilateral mastoids at the frequency (range 0.5-1.2 Hz) and phase of ongoing SWOs for a duration of 5 cycles in each discrete event throughout the night. Data were analyzed in a repeated measures ANOVA framework, and results show that verum stimulation improved post-sleep performance specifically on generalized versions of images used in training at both morning and afternoon tests compared to sham, suggesting the facilitation of schematization of information, but not of rote, veridical recall. We also found a surprising inverted U-shaped dose effect of sleep tACS, which is interpreted in terms of tACS-induced faciliatory and subsequent refractory dynamics of SWO power in scalp EEG. This is the first study showing a selective modulation of long-term memory generalization using a novel closed-loop tACS approach, which holds great potential for both healthy and neuropsychiatric populations.

Thalamic pathways underlying prefrontal cortex-medial temporal lobe oscillatory interactions.

As focus shifts to large-scale network interactions involved in memory, it is becoming increasingly clear that oscillatory dynamics are critically involved. A number of studies have shown a negative correlation between memory retrieval in alpha and beta power, and a positive correlation between retrieval and theta power. In this opinion article, we suggest three thalamic sub-regions responsible for the coordination of oscillatory activity and the facilitation of memory processes. Specifically, the medial dorsal nucleus is related to changes in beta synchrony, the pulvinar is responsible for alpha synchrony, and the anterior thalamus is related to theta synchrony. These pathways may be modulated via frontal control, and changes in oscillations could be used to track the engagement of underlying memory systems.

Complementary learning systems.

This paper reviews the fate of the central ideas behind the complementary learning systems (CLS) framework as originally articulated in McClelland, McNaughton, and O'Reilly (1995). This framework explains why the brain requires two differentially specialized learning and memory systems, and it nicely specifies their central properties (i.e., the hippocampus as a sparse, pattern-separated system for rapidly learning episodic memories, and the neocortex as a distributed, overlapping system for gradually integrating across episodes to extract latent semantic structure). We review the application of the CLS framework to a range of important topics, including the following: the basic neural processes of hippocampal memory encoding and recall, conjunctive encoding, human recognition memory, consolidation of initial hippocampal learning in cortex, dynamic modulation of encoding versus recall, and the synergistic interactions between hippocampus and neocortex. Overall, the CLS framework remains a vital theoretical force in the field, with the empirical data over the past 15 years generally confirming its key principles.

Enhanced brain correlations during rest are related to memory for recent experiences.

Long-term storage of episodic memories is hypothesized to result from the off-line transfer of information from the hippocampus to neocortex, allowing a hippocampal-independent cortical representation to emerge. However, off-line hippocampal-cortical interactions have not been demonstrated to be linked with long-term memory. Here, using functional magnetic resonance imaging, we examined if hippocampal-cortical BOLD correlations during rest following an associative encoding task are related to later associative memory performance. Our data show enhanced functional connectivity between the hippocampus and a portion of the lateral occipital complex (LO) during rest following a task with high subsequent memory compared to pretask baseline resting connectivity. This effect is not seen during rest following a task with poor subsequent memory. Furthermore, the magnitude of hippocampal-LO correlations during posttask rest predicts individual differences in later associative memory. These results demonstrate the importance of postexperience resting brain correlations for memory for recent experiences.