The neural representations underlying asymmetric cross-modal prediction of words.

Cross-modal prediction serves a crucial adaptive role in the multisensory world, yet the neural mechanisms underlying this prediction are poorly understood. The present study addressed this important question by combining a novel audiovisual sequence memory task, functional magnetic resonance imaging (fMRI), and multivariate neural representational analyses. Our behavioral results revealed a reliable asymmetric cross-modal predictive effect, with a stronger prediction from visual to auditory (VA) modality than auditory to visual (AV) modality. Mirroring the behavioral pattern, we found the superior parietal lobe (SPL) showed higher pattern similarity for VA than AV pairs, and the strength of the predictive coding in the SPL was positively correlated with the behavioral predictive effect in the VA condition. Representational connectivity analyses further revealed that the SPL mediated the neural pathway from the visual to the auditory cortex in the VA condition but was not involved in the auditory to visual cortex pathway in the AV condition. Direct neural pathways within the unimodal regions were found for the visual-to-visual and auditory-to-auditory predictions. Together, these results provide novel insights into the neural mechanisms underlying cross-modal sequence prediction.

Theta EEG neurofeedback promotes early consolidation of real life-like episodic memory.

Theta oscillations are believed to coordinate neuronal activity related to human cognition, especially for memory functions. Theta power during learning and retrieval has been found to correlate with memory performance success. Additionally, up-regulating theta oscillations during a post-encoding epoch crucial for memory consolidation was previously shown to benefit long-term memory for acquired motor sequences, pictures, and object-location associations. However, it remains to be determined whether such effects would be found for more ecological aspects of long-term episodic memory. Therefore, the current study assessed neurofeedback-based theta upregulation effects on movie memory. After viewing a 15-minute silent, narrative movie, participants engaged in neurofeedback-based theta/beta up-regulation, neurofeedback beta/theta up-regulation as an active control condition, or an unrelated passive control task. Memory was tested three times: once immediately after watching the movie (as baseline); 24 hours thereafter; and once again 1 week later. Memory performance 1 week after encoding was significantly enhanced in the theta/beta up-regulation group compared with the other groups. Additionally, changes in neurofeedback theta/beta ratio from baseline EEG recordings correlated with long-term memory gains in retrieving the movie's content. These findings highlight the relationship between post-learning theta oscillations and the consolidation of episodic memory for a naturalistic event.

Negative recency effects in delayed recognition: Spacing, consolidation, and retrieval strategy processes.

While items learned immediately before testing are generally remembered better than prior items in a study list, in delayed testing this relationship is reversed, yielding a negative recency effect. To adjudicate between the strategic rehearsal and spacing accounts of this phenomenon, we examined performance of 169 participants on a delayed recognition test following multiple sessions requiring the study and immediate free recall testing of 16 lists of 16 words. This revealed a strong effect of the amount of spacing between initial study position and initial free recall position on the degree of negative recency, supporting the spacing account. Furthermore, these spacing effects were nonmonotonic, suggesting that they are mediated by consolidation processes. Additional analyses indicate that strategies and rehearsal opportunities may also contribute to the effects of within-list encoding position on subsequent long-term memory, but for recall more than for recognition.

Fractionating the episodic buffer.

The episodic buffer is a putative component of working memory proposed to account for several short-term memory functions, including unexpectedly preserved immediate prose recall by amnesic patients. Over the course of time, this component has increasingly become associated with binding functions. Considering recent findings regarding the performance of both memory-impaired and healthy individuals on the range of tasks purported to require the contribution of the episodic buffer, we suggest that it should be fractionated into two functional systems. One is a schematic store instantiated in brain areas responsible for conceptual and schema representations, which is likely to be hippocampus-independent, and preserved in the face of amnesia. In contrast, short-term maintenance of novel associative binding is likely to require the contribution of the hippocampus and may therefore not be functionally dissociable from long-term memory.

Multimodal Integration and Vividness in the Angular Gyrus During Episodic Encoding and Retrieval.

Much evidence suggests that the angular gyrus (AnG) is involved in episodic memory, but its precise role has yet to be determined. We examined two possible accounts within the same experimental paradigm: the "cortical binding of relational activity" (CoBRA) account (Shimamura, 2011), which suggests that the AnG acts as a convergence zone that binds multimodal episodic features, and the subjectivity account (Yazar et al., 2012), which implicates AnG involvement in subjective mnemonic experience (such as vividness or confidence). fMRI was used during both encoding and retrieval of paired associates. During study, female and male human participants memorized picture-pairs of common objects (in the unimodal task) or of an object-picture and an environmental sound (in the crossmodal task). At test, they performed a cued-recall task and further indicated the vividness of their memory. During retrieval, BOLD activation in the AnG was greatest for vividly remembered associates, consistent with the subjectivity account. During encoding, the same effect of vividness was found, but this was further modulated by task: greater activations were associated with subsequent recall in the crossmodal than the unimodal task. Therefore, encoding data suggest an additional role to the AnG in crossmodal integration, consistent with its role at retrieval proposed by CoBRA. These results resolve some of the puzzles in the literature and indicate that the AnG can play different roles during encoding and retrieval as determined by the cognitive demands posed by different mnemonic tasks.SIGNIFICANCE STATEMENT We offer new insights into the multiplicity of processes that are associated with angular gyrus (AnG) activation during encoding and retrieval of newly formed memories. We used fMRI while human participants learned and subsequently recalled pairs of objects presented to the same sensory modality or to different modalities. We were able to show that the AnG is involved when vivid memories are created and retrieved, as well as when encoded information is integrated across different sensory modalities. These findings provide novel evidence for the contribution of the AnG to our subjective experience of remembering alongside its role in integrative processes that promote subsequent memory.

The impact of retrieval suppression on conceptual implicit memory.

When people suppress retrieval of episodic memories, it can induce forgetting on later direct tests of memory for those events. Recent reports indicate that suppressing retrieval affects less conscious, unintentional retrieval of unwanted memories as well, at least on perceptually-oriented indirect tests. In the current study we examined how suppressing retrieval affects conceptual implicit memory for the suppressed content, using a category verification task. Participants studied cue-target words pairs in which the targets were exemplars of 22 semantic categories, such as vegetables or occupations. They then repeatedly retrieved or suppressed the targets in response to the cues for some of those pairs. Afterwards, they were exposed to the targets intermixed with novel items, one at a time, and asked to verify the membership of each of the words in a semantic category, as quickly as possible. Judgment response times to studied words were faster than to unstudied exemplars, reflecting repetition priming, as has been previously observed. Strikingly, the beneficial effects of prior exposure on response time were eliminated for targets that had been suppressed. Follow-up explicit memory tests also demonstrated that retrieval suppression continued to disrupt episodic recall for the items that had been just been re-exposed on the category verification test. These findings support the contention that the effects of retrieval suppression are not limited to episodic memory, but also affect indirect expressions of those memories on conceptually oriented tests, raising the possibility that underlying semantic representations of suppressed content are affected.

Spared perception of object geometry and object components after hippocampal damage.

We tested the proposal that medial temporal lobe (MTL) structures support not just memory but also high-level object perception. In one task, participants decided whether a line drawing could represent an object in three-dimensional space and, in another task, they saw the components of an object and decided what object could be formed if the components were assembled. Patients with hippocampal lesions were intact, indicating that the hippocampus is not needed for perceiving the structural coherence of objects or appreciating the relations among object parts. Patients with large MTL lesions were moderately impaired, likely due to damage outside the MTL.

Behavioral reconsolidation interference with episodic memory within-subjects is elusive.

In studies of behavioral reconsolidation interference, reactivation of a consolidated memory using some form of reminder is followed by the presentation of new information that can cause interference with that memory. Under these conditions, the interference not only impairs retrieval by indirect processes such as cue interference, but supposedly disrupts the original memory trace directly. Almost all studies of behavioral reconsolidation interference in episodic memory in humans have employed between-subjects paradigms, and deduced reminder effects from intrusion errors. Such studies might introduce confounds arising, for example, from differences in retrieval strategies engendered by the pre-test treatments. We therefore set out to examine whether behavioral reconsolidation interference in episodic memory might be demonstrated within-subjects and by direct memory strength rather than intrusion errors. In three separate experiments, we attempted to disrupt reconsolidation of episodic object-picture memory using a reminder + retroactive interference manipulation. We applied the manipulation over three consecutive days, using a forced-choice recognition test without intrusions from interfering learning, keeping all other study and test parameters constant. No effects of reminder-potentiated interference were observed for measures of accuracy, response times, subjective expressions of recollection, or levels of confidence, as substantiated by Bayesian analyses. These results highlight the difficulty of observing clear behavioral reconsolidation interference effects within-subjects in human episodic memory, and provide some indications of what might be boundary conditions for its demonstration.

Enhancing early consolidation of human episodic memory by theta EEG neurofeedback.

Consolidation of newly formed memories is readily disrupted, but can it be enhanced? Given the prominent role of hippocampal theta oscillations in memory formation and retrieval, we hypothesized that upregulating theta power during early stages of consolidation might benefit memory stability and persistence. We used EEG neurofeedback to enable participants to selectively increase theta power in their EEG spectra following episodic memory encoding, while other participants engaged in low beta-focused neurofeedback or passively viewed a neutral nature movie. Free recall assessments immediately following the interventions, 24h later and 7d later all indicated benefit to memory of theta neurofeedback, relative to low beta neurofeedback or passive movie-viewing control conditions. The degree of benefit to memory was correlated with the extent of theta power modulation, but not with other spectral changes. Theta enhancement may provide optimal conditions for stabilization of new hippocampus-dependent memories.

Associative recognition processes are modulated by modality relations.

Although memory of episodic associations is generally considered to be recollective in nature, it has been suggested that when stimuli are experienced as a unit, familiarity-related processes might contribute to their subsequent associative recognition. Furthermore, intradomain associations are believed to be unitized more readily than interdomain associations. To assess these claims, we tested associative recognition following two types of pair associate learning. In the unimodal task, stimulus pairs were pictures of common objects, whereas in the cross-modal task, stimulus pairs consisted of an object picture and an unrelated environmental sound. At test, participants discriminated intact from recombined pairs while ERPs were recorded. In the unimodal task only, associative recognition was accompanied by a robust frontal deflection reminiscent of a component commonly interpreted as related to familiarity processes. In contrast, ERP correlates of associative recognition observed at more posterior sites, akin to a component that has been related to recollection, were apparent in both tasks. These findings indicate that retrieval of unimodal associations can be supported by familiarity-related processes that are dissociable from recollective processes required for the retrieval of cross-modal associations.

The time course of episodic associative retrieval: electrophysiological correlates of cued recall of unimodal and crossmodal pair-associate learning.

Little is known about the time course of processes supporting episodic cued recall. To examine these processes, we recorded event-related scalp electrical potentials during episodic cued recall following pair-associate learning of unimodal object-picture pairs and crossmodal object-picture and sound pairs. Successful cued recall of unimodal associates was characterized by markedly early scalp potential differences over frontal areas, while cued recall of both unimodal and crossmodal associates were reflected by subsequent differences recorded over frontal and parietal areas. Notably, unimodal cued recall success divergences over frontal areas were apparent in a time window generally assumed to reflect the operation of familiarity but not recollection processes, raising the possibility that retrieval success effects in that temporal window may reflect additional mnemonic processes beyond familiarity. Furthermore, parietal scalp potential recall success differences, which did not distinguish between crossmodal and unimodal tasks, seemingly support attentional or buffer accounts of posterior parietal mnemonic function but appear to constrain signal accumulation, expectation, or representational accounts.

Temporal texture of associative encoding modulates recall processes.

Binding aspects of an experience that are distributed over time is an important element of episodic memory. In the current study, we examined how the temporal complexity of an experience may govern the processes required for its retrieval. We recorded event-related potentials during episodic cued recall following pair associate learning of concurrently and sequentially presented object-picture pairs. Cued recall success effects over anterior and posterior areas were apparent in several time windows. In anterior locations, these recall success effects were similar for concurrently and sequentially encoded pairs. However, in posterior sites clustered over parietal scalp the effect was larger for the retrieval of sequentially encoded pairs. We suggest that anterior aspects of the mid-latency recall success effects may reflect working-with-memory operations or direct access recall processes, while more posterior aspects reflect recollective processes which are required for retrieval of episodes of greater temporal complexity.

Associative recognition processes are modulated by the semantic unitizability of memoranda.

Although memory of episodic associations is generally considered to be recollective in nature, it has been suggested that when stimuli are experienced as a unit, familiarity processes might contribute to their subsequent associative recognition. To investigate the effect of semantic relatedness during episodic encoding on the processes of retrieval of associative information, we had participants interactively encode pairs of object pictures, vertically arranged so as to suggest a functional or configural relationship between them. Half the pairs were independently judged to be of related objects (e.g., a lamp over a table) and half of unrelated objects (e.g., a key-ring over an apple). At test, participants discriminated between intact, recombined, and new pairs while event related potentials (ERPs) were recorded. In an early ERP marker of retrieval success generally associated with familiarity processes, differences related to associative memory only emerged for related pairs, while differences associated with item memory emerged for both related and unrelated pairs. In contrast, in a later ERP effect associated with recollection, differences related to associative memory emerged for both related and unrelated pairs. These findings may indicate that retrieval of episodic associations formed between two semantically related visual stimuli can be supported by familiarity-related processes.

Frontal midline theta transcranial alternating current stimulation enhances early consolidation of episodic memory.

Evidence implicating theta rhythms in declarative memory encoding and retrieval, together with the notion that both retrieval and consolidation involve memory reinstatement or replay, suggests that post-learning theta rhythm modulation can promote early consolidation of newly formed memories. Building on earlier work employing theta neurofeedback, we examined whether theta-frequency transcranial alternating stimulation (tACS) can engender effective consolidation of newly formed episodic memories, compared with beta frequency stimulation or sham control conditions. We compared midline frontal and posterior parietal theta stimulation montages and examined whether benefits to memory of theta upregulation are attributable to consolidation rather than to retrieval processes by using a washout period to eliminate tACS after-effects between stimulation and memory assessment. Four groups of participants viewed object pictures followed by a free recall test during three study-test cycles. They then engaged in tACS (frontal theta montage/parietal theta montage/frontal beta montage/sham) for a period of 20 min, followed by a 2-h break. Free recall assessments were conducted after the break, 24 h later, and 7 days later. Frontal midline theta-tACS induced significant off-line retrieval gains at all assessment time points relative to all other conditions. This indicates that theta upregulation provides optimal conditions for the consolidation of episodic memory, independent of mental-state strategies.

Lesion-behavior mapping indicates a strategic role for parietal substrates of associative memory.

Numerous neuroimaging studies indicate that ventral parietal cortex (VPC), especially angular gyrus, plays an important role in episodic memory. However, the nature of the mnemonic processes supported by this region is far from clear. We previously found that stroke lesions in VPC and lateral temporal cortex caused deficits in cued recall of unimodal word pairs and picture pairs, and cross-modal picture-sound pairs, with larger deficits in the cross-modal task. However, those findings leave open the question whether those regions' integrity is necessary for maintenance of associative representations, or for strategic processes required for their recall. We addressed this question using associative recognition versions of those tasks. We additionally manipulated semantic relatedness of the associated memoranda, to assess VPC's involvement in semantic processing in the context of episodic memory. We analyzed performance of 62 first-event, sub-acute phase stroke patients (31 right- and 31 left-hemisphere damage) relative to 65 healthy participants, and employed voxel-based lesion-behavior mapping (VLBM) to identify task-relevant structures. Patients displayed greater false associative recognition of semantically related compared to unrelated recombined pairs. VLBM analysis implicated right lateral temporo-parietal regions in associative recognition deficits in the cross-modal pairs task, specifically for related recombined and new pairs, seemingly because of difficulty overcoming semantic relatedness bias effects on episodic discrimination. In contrast, damage to ventral parietal and lateral temporal cortex was not implicated in memory for unrelated memoranda. We interpret this pattern of lesion-behavior effects as indicating lateral temporo-parietal cortex involvement in strategic, rather than representational, roles in episodic associative memory.

Does this ring a bell? Music-cued retrieval of semantic knowledge and metamemory judgments.

Failed knowledge recall attempts are sometimes accompanied by a strong feeling of imminent success, giving rise to a "tip-of-the-tongue" (TOT) experience. Similar to successful retrieval (i.e., the Know state, K), a TOT commences with strong cue familiarity but involves only partial retrieval of related information. We sought to characterize the cognitive processes and temporal dynamics of these retrieval states and to extend the applicability of previous findings about TOT to the auditory modality. Participants heard 3-sec initial segments of popular songs and were asked to recall their names. EEG was recorded while participants indicated their retrieval state via button press. Stimulus-locked analyses revealed a significant early left fronto-central difference between TOT and K, at 300-550 msec postcue onset. Post hoc analysis revealed that, in this time window, TOT also differed from DK (Don't Know) responses, which themselves were similar to the K responses. This finding indicates that neural processes, which may reflect strategy selection, ease of semantic processing, familiarity-related processes, or conflict monitoring, are indicative of the fate of our knowledge judgments long before we actually execute them.

Optimizing the social utility of judicial punishment: An evolutionary biology and neuroscience perspective.

Punishment as a response to impairment of individual or group welfare may be found not only among humans but also among a wide range of social animals. In some cases, acts of punishment serve to increase social cooperation among conspecifics. Such phenomena motivate the search for the biological foundations of punishment among humans. Of special interest are cases of pro-social punishment of individuals harming others. Behavioral studies have shown that in economic games people punish exploiters even at a cost to their own welfare. Additionally, neuroimaging studies have reported activity during the planning of such punishment in brain areas involved in the anticipation of reward. Such findings hint that there is an evolutionarily honed basic drive to punish social offenders. I argue that the transfer of punishment authority from the individual to the group requires that social offenders be punished as a public good, even if such punishment is not effective as retribution or deterrent. Furthermore, the social need for punishment of offenders has implications for alternatives to incarceration, publicity of punishment, and judicial structure.

Associative recognition memory for identity, spatial and temporal relations in healthy aging.

The ability to generate associative representations and to retrieve them from long-term episodic memory generally declines in healthy aging. However, it is unclear whether healthy aging has differential effects on associative memory for identity, spatial configuration, and temporal order relationships. In the current study, we assessed how healthy aging impacts on associative memory for identity, spatial, or temporal relationships between pairs of visual objects via discrimination of intact and rearranged pairs. Accuracy and response time performance of healthy older adults (aged 65-80) were compared with young adults (ages 19-30). Age-related declines in associative memory were observed equally for all types of associations, but these declines differed by associative status: aging most strongly affected ability to discriminate rearranged pairs. These results suggest that associative memory for identity, spatial, and temporal relationships are equally affected by healthy aging, and may all depend on a shared set of basic associative mechanisms.

Behavioural reconsolidation interference not observed in a within-subjects design.

Studies of reconsolidation interference posit that reactivation of a previously consolidated memory via a reminder brings it into an active, labile state, leaving it open for potential manipulation. If interfered with, this may disrupt the original memory trace. While evidence for pharmacological reconsolidation interference is widespread, it remains unclear whether behavioural interference using the presentation of competing information can engender it, especially in declarative memory. Almost all previous studies in this area have employed between-subjects designs, in which there are potential confounds, such as different retrieval strategies for the multiple conditions. In the current studies, within-subjects paradigms were applied to test the effects of reconsolidation interference on associative recognition and free recall. In Experiment 1, participants engaged in pair-associate learning of unrelated object pictures on Day 1, and after a reminder, interference, reminder + interference, or no manipulation (control) on Day 2, were tested on associative recognition of these pairs on Day 3. In Experiments 2 and 3, memoranda were short stories studied on Day 1. On Day 2, stories were assigned to either control, reminder, interference by alternative stories, or reminder + interference conditions. On Day 3 participants recalled the Day 1 stories, and answered yes/no recognition questions. Reminders improved subsequent memory, while interference was effective in reducing retrieval in differing degrees across the experiments. Importantly, the reminder + interference condition was no more effective in impairing retrieval than the interference-alone condition, contrary to the prediction of the behavioural reconsolidation-interference approach.

Lesion-behaviour mapping reveals multifactorial neurocognitive processes in recognition memory for unfamiliar faces.

Face recognition abilities, which play a critical role in social interactions, involve face processing and identifying familiar faces, but also remembering one-off encounters with previously unfamiliar faces. Previous functional imaging and lesion studies have found evidence for temporal, frontal, and parietal contributions to episodic recognition memory for previously unfamiliar faces. However, the functional contributions of these regions remain unclear. We, therefore, conducted a systematic group analysis of this memory function using lesion-behavior mapping. 95 first-event stroke patients (53 with right- and 42 with left-hemisphere damage) in the sub-acute phase performed the Wechsler Memory Scale (WMS-III) face recognition memory subtest. We analyzed their performance relative to 75 healthy controls, using signal detection measures. To identify brain lesions specifically implicated in face recognition deficits, we used voxel-based lesion-behavior mapping (VLBM; an analysis comparing the performance of participants with and without damage affecting a given voxel). Behavioral analysis disclosed a pronounced impairment in the performance of patients with right hemisphere damage. Frontal damage was associated with an increased amount of false alarms (i.e., failed rejection of new face items) and overly liberal criterion setting, without affecting the recognition of studied faces. In contrast, parietal damage was associated with impaired recognition of studied faces, which was more pronounced in immediate than in delayed retrieval. These findings suggest the existence of multifactorial neurocognitive processes in recognition memory for unfamiliar faces.