Top-down task goals induce the retrieval state.

Engaging the retrieval state (Tulving, 1983) impacts processing and behavior (Long & Kuhl, 2019, 2021; Smith, Moore, & Long, 2022), but the extent to which top-down factors - explicit instructions and goals -vs. bottom-up factors - stimulus properties such as repetition and similarity - jointly or independently induce the retrieval state is unclear. Identifying the impact of bottom-up and top-down factors on re-trieval state engagement is critical for understanding how control of task-relevant vs. task-irrelevant brain states influence cognition. We conducted between-subjects recognition memory tasks on male and fe-male human participants in which we varied test phase goals. We recorded scalp electroencephalography and used an independently validated mnemonic state classifier (Long, 2023) to measure retrieval state engagement as a function of top-down task goals (recognize old vs. detect new items) and bottom-up stimulus repetition (hits vs. correct rejections). We find that whereas the retrieval state is engaged for hits regardless of top-down goals, the retrieval state is only engaged during correct rejections when the top- down goal is to recognize old items. Furthermore, retrieval state engagement is greater for low compared to high confidence hits when the task goal is to recognize old items. Together, these results suggest that top-down demands to recognize old items induce the retrieval state independent from bottom-up factors, potentially reflecting the recruitment of internal attention to enable access of a stored representation.Significance Statement Both top-down goals and automatic bottom-up influences may lead us into a retrieval brain state -awhole- brain pattern of activity that supports our ability to remember the past. Here we tested the extent to which top-down vs. bottom-up factors independently influence the retrieval state by manipulating participants' goals and stimulus repetition during a memory test. We find that in response to the top-down goal to recognize old items, the retrieval state is engaged for both old and new probes, suggesting that top- down and bottom-up factors independently engage the retrieval state. Our interpretation is that top-down demands recruit internal attention in service of the attempt to access a stored representation.

Semantic associations restore neural encoding mechanisms.

Lapses in attention can negatively impact later memory of an experience. Attention and encoding resources are thought to decline as more experiences are encountered in succession, accounting for the primacy effect in which memory is better for items encountered early compared to late in a study list. However, accessing prior knowledge during study can facilitate subsequent memory, suggesting a potential avenue to counteract this decline. Here, we investigated the extent to which semantic associations-shared meaning between experiences-can counteract declines in encoding resources. Our hypothesis is that semantic associations restore neural encoding mechanisms, which in turn improves memory. We recorded scalp electroencephalography (EEG) while male and female human participants performed a delayed free recall task. Half of the items from late in each study list were semantically associated with an item presented earlier in the list. We find that semantic associations improve memory specifically for late list items and selectively modulate the neural signals engaged during the study of late list items. Relative to other recalled items, late list items that are subsequently semantically clustered-recalled consecutively with their semantic associate-elicit increased high-frequency activity and decreased low-frequency activity, a hallmark of successful encoding. Our findings demonstrate that semantic associations restore neural encoding mechanisms and improve later memory. More broadly, these findings suggest that prior knowledge modulates the orientation of attention to influence encoding mechanisms.

Temporal Context Modulates Encoding and Retrieval of Overlapping Events.

Overlap between events can lead to interference because of a trade-off between encoding the present event and retrieving the past event. Temporal context information, "when" something occurred, a defining feature of episodic memory, can cue retrieval of a past event. However, the influence of temporal overlap, or proximity in time, on the mechanisms of interference is unclear. Here, by identifying brain states using scalp EEG from male and female human subjects, we show the extent to which temporal overlap promotes interference and induces retrieval. In this experiment, subjects were explicitly directed to either encode the present event or retrieve a past, overlapping event while perceptual input was held constant. We find that the degree of temporal overlap between events leads to selective interference. Specifically, greater temporal overlap between two events leads to impaired memory for the past event selectively when the top-down goal is to encode the present event. Using pattern classification analyses to measure neural evidence for a retrieval state, we find that greater temporal overlap leads to automatic retrieval of a past event, independent of top-down goals. Critically, the retrieval evidence we observe likely reflects a general retrieval mode, rather than retrieval success or effort. Collectively, our findings provide insight into the role of temporal overlap on interference and memory formation.SIGNIFICANCE STATEMENT When a present event overlaps with an event from the past, this leads to a trade-off between the tendency to encode the present event versus retrieve the past event. Here we show that, when two events are experienced nearby in time, the memory system is biased toward a retrieval state and that subsequent memory for the past event is impaired. These findings suggest an influence of bottom-up temporal factors on both interference and the trade-off between memory states.

Spatiotemporal Dynamics of Memory Encoding and Memory Retrieval States.

Memory encoding and memory retrieval are neurally distinct brain states that can be differentiated on the basis of cortical network activity. However, it is unclear whether sustained engagement of one network or fluctuations between multiple networks give rise to these memory states. The spatiotemporal dynamics of memory states may have important implications for memory behavior and cognition; however, measuring temporally resolved signals of cortical networks poses a challenge. Here, we recorded scalp electroencephalography from participants performing a mnemonic state task in which they were biased toward memory encoding or retrieval. We performed a microstate analysis to measure the temporal dynamics of cortical networks throughout this mnemonic state task. We find that Microstate E, a putative analog of the default mode network, shows temporally sustained dissociations between memory encoding and retrieval, with greater engagement during retrieve compared with encode trials. We further show that decreased engagement of Microstate E is a general property of encoding, rather than a reflection of retrieval suppression. Thus, memory success, as well as cognition more broadly, may be influenced by the ability to engage or disengage Microstate E in a goal-dependent manner.

Bottom-Up and Top-Down Factors Differentially Influence Stimulus Representations Across Large-Scale Attentional Networks.

Visual attention is thought to be supported by three large-scale frontoparietal networks: the frontoparietal control network (FPCN), the dorsal attention network (DAN), and the ventral attention network (VAN). The traditional view is that these networks support visual attention by biasing and evaluating sensory representations in visual cortical regions. However, recent evidence suggests that frontoparietal regions actively represent perceptual stimuli. Here, we assessed how perceptual stimuli are represented across large-scale frontoparietal and visual networks. Specifically, we tested whether representations of stimulus features across these networks are differentially sensitive to bottom-up and top-down factors. In a pair of pattern-based fMRI studies, male and female human subjects made perceptual decisions about face images that varied along two independent dimensions: gender and affect. Across studies, we interrupted bottom-up visual input using backward masks. Within studies, we manipulated which stimulus features were goal relevant (i.e., whether gender or affect was relevant) and task switching (i.e., whether the goal on the current trial matched the goal on the prior trial). We found that stimulus features could be reliably decoded from all four networks and, importantly, that subregions within each attentional network maintained coherent representations. Critically, the different attentional manipulations (interruption, goal relevance, and task switching) differentially influenced feature representations across networks. Whereas visual interruption had a relatively greater influence on representations in visual regions, goal relevance and task switching had a relatively greater influence on representations in frontoparietal networks. Therefore, large-scale brain networks can be dissociated according to how attention influences the feature representations that they maintain.SIGNIFICANCE STATEMENT Visual attention is supported by multiple frontoparietal attentional networks. However, it remains unclear how stimulus features are represented within these networks and how they are influenced by attention. Here, we assessed feature representations in four large-scale networks using a perceptual decision-making paradigm in which we manipulated top-down and bottom-up factors. We found that top-down manipulations such as goal relevance and task switching modulated feature representations in attentional networks, whereas bottom-up manipulations such as interruption of visual processing had a relatively stronger influence on feature representations in visual regions. Together, these findings indicate that attentional networks actively represent stimulus features and that representations within different large-scale networks are influenced by different forms of attention.

Decoding the tradeoff between encoding and retrieval to predict memory for overlapping events.

When new events overlap with past events, there is a natural tradeoff between encoding the new event and retrieving the past event. Given the ubiquity of overlap among memories, this tradeoff between memory encoding and retrieval is of central importance to computational models of episodic memory (O'Reilly & McClelland 1994; Hasselmo 2005). However, prior studies have not directly linked neural markers of encoding/retrieval tradeoffs to behavioral measures of how overlapping events are remembered. Here, by decoding patterns of scalp electroencephalography (EEG) from male and female human subjects, we show that tradeoffs between encoding and retrieval states are reflected in distributed patterns of neural activity and, critically, these neural tradeoffs predict how overlapping events will later be remembered. Namely, new events that overlapped with past events were more likely to be subsequently remembered if neural patterns were biased toward a memory encoding state-or, conversely, away from a retrieval state. Additionally, we show that neural markers of encoding vs. retrieval states are surprisingly independent from previously-described EEG predictors of subsequent memory. Instead, we demonstrate that previously-described EEG predictors of subsequent memory are better explained by task engagement than by memory encoding, per se. Collectively, our findings provide important insight into how the memory system balances memory encoding and retrieval states and, more generally, into the neural mechanisms that support successful memory formation.

Hippocampal contributions to serial-order memory.

Our memories form a record not only of our experiences, but also of their temporal structure. Although memory for the temporal structure of experience likely relies on multiple neural systems, numerous studies have implicated the hippocampus in the encoding and retrieval of temporal information. This review evaluates the literature on hippocampal contributions to human serial-order memory from the perspective of three cognitive theories: associative chaining theory, positional-coding theory and retrieved-context theory. Evaluating neural findings through the lens of cognitive theories enables us to draw more incisive conclusions about the relations between brain and behavior.

Hippocampal Mismatch Signals Are Modulated by the Strength of Neural Predictions and Their Similarity to Outcomes.

The hippocampus is thought to compare predicted events with current perceptual input, generating a mismatch signal when predictions are violated. However, most prior studies have only inferred when predictions occur without measuring them directly. Moreover, an important but unresolved question is whether hippocampal mismatch signals are modulated by the degree to which predictions differ from outcomes. Here, we conducted a human fMRI study in which subjects repeatedly studied various word-picture pairs, learning to predict particular pictures (outcomes) from the words (cues). After initial learning, a subset of cues was paired with a novel, unexpected outcome, whereas other cues continued to predict the same outcome. Critically, when outcomes changed, the new outcome was either "near" to the predicted outcome (same visual category as the predicted picture) or "far" from the predicted outcome (different visual category). Using multivoxel pattern analysis, we indexed cue-evoked reactivation (prediction) within neocortical areas and related these trial-by-trial measures of prediction strength to univariate hippocampal responses to the outcomes. We found that prediction strength positively modulated hippocampal responses to unexpected outcomes, particularly when unexpected outcomes were close, but not identical, to the prediction. Hippocampal responses to unexpected outcomes were also associated with a tradeoff in performance during a subsequent memory test: relatively faster retrieval of new (updated) associations, but relatively slower retrieval of the original (older) associations. Together, these results indicate that hippocampal mismatch signals reflect a comparison between active predictions and current outcomes and that these signals are most robust when predictions are similar, but not identical, to outcomes. SIGNIFICANCE STATEMENT: Although the hippocampus is widely thought to signal "mismatches" between memory-based predictions and outcomes, previous research has not linked hippocampal mismatch signals directly to neural measures of prediction strength. Here, we show that hippocampal mismatch signals increase as a function of the strength of predictions in neocortical regions. This increase in hippocampal mismatch signals was particularly robust when outcomes were similar, but not identical, to predictions. These results indicate that hippocampal mismatch signals are driven by both the active generation of predictions and the similarity between predictions and outcomes.

Successful memory formation is driven by contextual encoding in the core memory network.

To understand how memories are successfully formed, scientists have compared neural activity during the encoding of subsequently remembered and forgotten items. Though this approach has elucidated a network of brain regions involved in memory encoding, this method cannot distinguish broad, non-specific signals from memory specific encoding processes, such as associative encoding. Associative encoding, which is a key mechanism of learning, can be seen in the tendency of participants to successively recall, or cluster, study neighbors. We assessed the electrophysiological correlates of associative processing by comparing intracranially recorded EEG activity during the encoding of items that were subsequently recalled and clustered; recalled and not clustered; or not recalled. We found that high frequency activity (HFA) in left prefrontal cortex, left temporal cortex and hippocampus increased during the encoding of subsequently recalled items. Critically, the magnitude of this effect was largest for those recalled items that were also subsequently clustered. HFA temporally dissociated across regions, with increases in left prefrontal cortex preceding those in hippocampus. Furthermore, late hippocampal HFA positively correlated with behavioral measures of clustering. These results suggest that associative processes linking items to their spatiotemporal context underlie the traditionally observed subsequent memory effect and support successful memory formation.

Ventral striatum and the evaluation of memory retrieval strategies.

Adaptive memory retrieval requires mechanisms of cognitive control that facilitate the recovery of goal-relevant information. Frontoparietal systems are known to support control of memory retrieval. However, the mechanisms by which the brain acquires, evaluates, and adapts retrieval strategies remain unknown. Here, we provide evidence that ventral striatal activation tracks the success of a retrieval strategy and correlates with subsequent reliance on that strategy. Human participants were scanned with fMRI while performing a lexical decision task. A rule was provided that indicated the likely semantic category of a target word given the category of a preceding prime. Reliance on the rule improved decision-making, as estimated within a drift diffusion framework. Ventral striatal activation tracked the benefit that relying on the rule had on decision-making. Moreover, activation in ventral striatum correlated with a participant's subsequent reliance on the rule. Taken together, these results support a role for ventral striatum in learning and evaluating declarative retrieval strategies.

Subsequent memory effect in intracranial and scalp EEG.

Successful memory encoding is marked by increases in 30-100Hz gamma-band activity in a broad network of brain regions. Activity in the 3-8Hz theta band has also been shown to modulate memory encoding, but this effect has been found to vary in direction across studies. Because of the diversity in memory tasks, and in recording and data-analytic methods, our knowledge of the theta frequency modulations remains limited. The difference in the directionality of these theta effects could arise from a distinction between global cortical and deeper subcortical effects. To address this issue, we examined the spectral correlates of successful memory encoding using intracranial EEG recordings in neurosurgical patients and scalp EEG recordings in healthy controls. We found significant theta (3-8Hz) power modulations (both increases and decreases) and high gamma (44-100Hz) power increases in both samples of participants. These results suggest that (1) there are two separate theta mechanisms supporting memory success, a broad theta decrease present across both the cortex and hippocampus as well as a theta power increase in the frontal cortex, (2) scalp EEG is capable of resolving high frequency gamma activity, and (3) iEEG theta effects are likely not the result of epileptic pathology.

Human intracranial high-frequency activity maps episodic memory formation in space and time.

Noninvasive neuroimaging studies have revealed a network of brain regions that activate during human memory encoding; however, the relative timing of such activations remains unknown. Here we used intracranially recorded high-frequency activity (HFA) to first identify regions that activate during successful encoding. Then, we leveraged the high-temporal precision of HFA to investigate the timing of such activations. We found that memory encoding invokes two spatiotemporally distinct activations: early increases in HFA that involve the ventral visual pathway as well as the medial temporal lobe and late increases in HFA that involve the left inferior frontal gyrus, left posterior parietal cortex, and left ventrolateral temporal cortex. We speculate that these activations reflect higher-order visual processing and top-down modulation of attention/semantic information, respectively.

The persistence of memory: prior memory responses modulate behavior and brain state engagement.

Memory brain states may influence how we experience an event. Memory encoding and retrieval constitute neurally dissociable brain states that individuals can selectively engage based on top-down goals. To the extent that memory states linger in time - as suggested by prior behavioral work - memory states may influence not only the current experience, but also subsequent stimuli and judgments. Thus lingering memory states may have broad influences on cognition, yet this account has not been directly tested utilizing neural measures of memory states. Here we address this gap by testing the hypothesis that memory brain states are modulated by memory judgments, and that these brain states persist for several hundred milliseconds. We recorded scalp electroencephalography (EEG) while participants completed a recognition memory task. We used an independently validated multivariate mnemonic state classifier to assess memory state engagement. We replicate prior behavioral findings; however, our neural findings run counter to the predictions made on the basis of the behavioral data. Surprisingly, we find that prior responses modulate current memory state engagement on the basis of response congruency. That is, we find strong engagement of the retrieval state on incongruent trials - when a target is preceded by a correct rejection of a lure and when a lure is preceded by successful recognition of a target. These findings indicate that cortical brain states are influenced by prior judgments and suggest that a non-mnemonic, internal attention state may be recruited to in the face of changing demands in a dynamic environment.

Top-down task goals induce the retrieval state.

Engaging the retrieval state (Tulving, 1983) impacts processing and behavior (Long & Kuhl, 2019, 2021; Smith, Moore, & Long, 2022), but the extent to which top-down factors - explicit instructions and goals - vs. bottom-up factors - stimulus properties such as repetition and similarity - jointly or independently induce the retrieval state is unclear. Identifying the impact of bottom-up and top-down factors on retrieval state engagement is critical for understanding how control of task-relevant vs. task-irrelevant brain states influence cognition. We conducted between-subjects recognition memory tasks on male and female human participants in which we varied test phase goals. We recorded scalp electroencephalography and used an independently validated mnemonic state classifier (Long, 2023) to measure retrieval state engagement as a function of top-down task goals (recognize old vs. detect new items) and bottom-up stimulus repetition (hits vs. correct rejections). We find that whereas the retrieval state is engaged for hits regardless of top-down goals, the retrieval state is only engaged during correct rejections when the top-down goal is to recognize old items. Furthermore, retrieval state engagement is greater for low compared to high confidence hits when the task goal is to recognize old items. Together, these results suggest that top-down demands to recognize old items induce the retrieval state independent from bottom-up factors, potentially reflecting the recruitment of internal attention to enable access of a stored representation. Significance Statement: Both top-down goals and automatic bottom-up influences may lead us into a retrieval brain state - a whole-brain pattern of activity that supports our ability to remember the past. Here we tested the extent to which top-down vs. bottom-up factors independently influence the retrieval state by manipulating participants' goals and stimulus repetition during a memory test. We find that in response to the top-down goal to recognize old items, the retrieval state is engaged for both old and new probes, suggesting that top-down and bottom-up factors independently engage the retrieval state. Our interpretation is that top-down demands recruit internal attention in service of the attempt to access a stored representation.

Response-locked theta dissociations reveal potential feedback signal following successful retrieval.

Successful memory retrieval relies on memory processes to access an internal representation and decision processes to evaluate and respond to the accessed representation, both of which are supported by fluctuations in theta (4-8Hz) activity. However, the extent to which decision making processes are engaged following a memory response is unclear. Here, we recorded scalp electroencephalography (EEG) while human participants performed a recognition memory task. We focused on response-locked data, allowing us to investigate the processes that occur prior to and following a memory response. We replicate previous work and find that prior to a memory response theta power is greater for identification of previously studied items (hits) relative to rejection of novel lures (correct rejections; CRs). Following the memory response, the theta power dissociation 'flips' whereby theta power is greater for CRs relative to hits. We find that the post-response 'flip' is more robust for hits that are committed quickly, potentially reflecting a positive feedback signal for strongly remembered experiences. Our findings suggest that there are potentially distinct processes occurring before and after a memory response that are modulated by successful memory retrieval.

The intersection of the retrieval state and internal attention.

Large-scale brain states or distributed patterns of brain activity modulate downstream processing and behavior. Sustained attention and memory retrieval states impact subsequent memory, yet how these states relate to one another is unclear. I hypothesize that internal attention is a central process of the retrieval state. The alternative is that the retrieval state specifically reflects a controlled, episodic retrieval mode, engaged only when intentionally accessing events situated within a spatiotemporal context. To test my hypothesis, I developed a mnemonic state classifier independently trained to measure retrieval state evidence and applied this classifier to a spatial attention task. I find that retrieval state evidence increases during delay and response intervals when participants are maintaining spatial information. Critically, retrieval state evidence is positively related to the amount of maintained spatial location information and predicts target detection reaction times. Together, these findings support the hypothesis that internal attention is a central process of the retrieval state.

Separable prefrontal cortex contributions to free recall.

In everyday life, we often must remember the past in the absence of helpful cues in the environment. In these cases, the brain directs retrieval by relying on internally maintained cues and strategies. Free recall is a widely used behavioral paradigm for studying retrieval with minimal cue support. During free recall, individuals often recall semantically related items consecutively--an effect termed semantic clustering--and previous studies have sought to understand clustering to gain leverage on the basic mechanisms supporting strategic recall. Successful recall and semantic clustering depend on the prefrontal cortex (PFC). However, as a result of methodological limitations, few functional magnetic resonance imaging (fMRI) studies have assessed the neural mechanisms at encoding that support subsequent recall, and none have tested the event-related correlates of recall itself. Thus, it remains open whether one or several frontal control mechanisms operate during encoding and recall. Here, we applied a recently developed method (Oztekin et al., 2010) to assess event-related fMRI signal changes during free recall. During encoding, dorsolateral prefrontal cortex (DLPFC) activation was predictive of subsequent semantic clustering. In contrast, subregions of ventrolateral prefrontal cortex (VLPFC) were predictive of subsequent recall, whether clustered or nonclustered, and were inversely associated with clustering during recall. These results suggest that DLPFC supports relational processes at encoding that are sufficient to produce category clustering effects during recall. Conversely, controlled retrieval mechanisms supported by VLPFC support item-specific search during recall.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States.

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).1516 Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

Optimizing design efficiency of free recall events for FMRI.

Free recall is a fundamental paradigm for studying memory retrieval in the context of minimal cue support. Accordingly, free recall has been extensively studied using behavioral methods. However, the neural mechanisms that support free recall have not been fully investigated due to technical challenges associated with probing individual recall events with neuroimaging methods. Of particular concern is the extent to which the uncontrolled latencies associated with recall events can confer sufficient design efficiency to permit neural activation for individual conditions to be distinguished. The present study sought to rigorously assess the feasibility of testing individual free recall events with fMRI. We used both theoretically and empirically derived free recall latency distributions to generate simulated fMRI data sets and assessed design efficiency across a range of parameters that describe free recall performance and fMRI designs. In addition, two fMRI experiments empirically assessed whether differential neural activation in visual cortex at onsets determined by true free recall performance across different conditions can be resolved. Collectively, these results specify the design and performance parameters that can provide comparable efficiency between free recall designs and more traditional jittered event-related designs. These findings suggest that assessing BOLD response during free recall using fMRI is feasible, under certain conditions, and can serve as a powerful tool in understanding the neural bases of memory search and overt retrieval.

When the Memory System Gets Ahead of Itself.

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.