Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface.

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Reactivation of prior responses drives serial dependence and stabilizes working memory representations.

Serial dependence has shown seemingly contradictory effects on visual perception and working memory. While serial dependence promotes perpetual and mnemonic stability, it biases behavioral reports towards prior information. The neural mechanisms that drive both biasing and adaptive stabilizing effects are not well understood. We proposed and tested a reactivation and integration mechanism that can account for these contradictory effects. We used multivariate pattern analyses of EEG data (26 human participants, 17 females, 9 males) to examine the reactivation of prior reported orientation during the delay period of a visual working memory task. The reactivation strength of prior reports, but not prior sensory items, was predictive of the magnitude of serial-dependency biases. These reactivated representations integrated with the representation of the current memory item and improved the ability to decode the current contents of memory. Overall, our data provide convergent evidence suggesting that prior reports in a visual working memory task are reactivated on the subsequent trial and become integrated with current memory representations. This similarity-dependent reactivation mechanism drives both report biasing and stabilization effects attributed to serial-dependence in working memory.Significance Statement A primary objective of working memory is to differentiate the present from the past. However, in a continuous visual experience, recent past information often remains relevant for processing subsequent information. How can working memory leverage relevant past information to aid perception and memory while avoiding unwanted influence from irrelevant past information? Here, we show that maintaining new information in working memory can selectively reactivate and assimilate similar information from the recent past. This reactivation biases memory for present items towards the past, which may not always be desirable, but it also enhances the fidelity and precision of these memories, which is certainly an adaptive feature.

Suppressing the Maintenance of Information in Working Memory Alters Long-term Memory Traces.

The sensory recruitment hypothesis conceptualizes information in working memory as being activated representations of information in long-term memory. Accordingly, changes made to an item in working memory would be expected to influence its subsequent retention. Here, we tested the hypothesis that suppressing information from working memory, which can reduce short-term access to that information, may also alter its long-term neural representation. We obtained fMRI data (n = 25; 13 female / 12 male participants) while participants completed a working memory removal task with scene images as stimuli, followed by a final surprise recognition test of the examined items. We applied a multivariate pattern analysis to the data to quantify the engagement of suppression on each trial, to track the contents of working memory during suppression, and to assess representational changes afterward. Our analysis confirms previous reports that suppression of information in working memory involves focused attention to target and remove unwanted information. Furthermore, our findings provide new evidence that even a single dose of suppression of an item in working memory can (if engaged with sufficient strength) produce lasting changes in its neural representation, particularly weakening the unique, item-specific features, which leads to forgetting. Our study sheds light on the underlying mechanisms that contribute to the suppression of unwanted thoughts and highlights the dynamic interplay between working memory and long-term memory.

Neural Systems Underlying the Implementation of Working Memory Removal Operations.

Recently, multi-voxel pattern analysis has verified that information can be removed from working memory (WM) via three distinct operations replacement, suppression, or clearing compared to information being maintained ( Kim et al., 2020). While univariate analyses and classifier importance maps in Kim et al. (2020) identified brain regions that contribute to these operations, they did not elucidate whether these regions represent the operations similarly or uniquely. Using Leiden-community-detection on a sample of 55 humans (17 male), we identified four brain networks, each of which has a unique configuration of multi-voxel activity patterns by which it represents these WM operations. The visual network (VN) shows similar multi-voxel patterns for maintain and replace, which are highly dissimilar from suppress and clear, suggesting this network differentiates whether an item is held in WM or not. The somatomotor network (SMN) shows a distinct multi-voxel pattern for clear relative to the other operations, indicating the uniqueness of this operation. The default mode network (DMN) has distinct patterns for suppress and clear, but these two operations are more similar to each other than to maintain and replace, a pattern intermediate to that of the VN and SMN. The frontoparietal control network (FPCN) displays distinct multi-voxel patterns for each of the four operations, suggesting that this network likely plays an important role in implementing these WM operations. These results indicate that the operations involved in removing information from WM can be performed in parallel by distinct brain networks, each of which has a particular configuration by which they represent these operations.

The effect of biomechanical features on classification of dual-task gait.

Early detection of Alzheimer's Disease and Related Disorders (ADRD) has been a focus of research with the hope that early intervention may improve clinical outcomes. The manifestation of motor impairment in early stages of ADRD has led to the inclusion of gait assessments including spatiotemporal parameters in clinical evaluations. This study aims to determine the effect of adding kinetic and kinematic gait features to classification of different levels of cognitive load in healthy individuals. A dual-task paradigm was used to simulate cognitive impairment in 40 healthy adults, with single-task walking trials representing normal, healthy gait. The Paced Auditory Serial Addition Task was administered at two different inter-stimulus intervals representing two levels of cognitive load in dual-task gait. We predicted that a richer dataset would improve classification accuracy relative to spatiotemporal parameters. Repeated Measures ANOVA showed significant changes in 15 different gait features across all three levels of cognitive load. We used three supervised machine learning algorithms to classify data points using a series of different gait feature sets with performance based on the area under the curve (AUC). Classification yielded 0.778 AUC across all three conditions (0.889 AUC Single vs. Dual) using kinematic and spatiotemporal features compared to 0.724 AUC using spatiotemporal features only (0.792 AUC Single vs. Dual). These data suggest that additional kinematic parameters improve classification performance. However, the benefit of measuring a wider set of parameters compared to their cost needs consideration. Further work will lead to a clinically viable ADRD detection classifier.

Bend but don't break: Prioritization protects working memory from displacement but leaves it vulnerable to distortion from distraction.

Perceptual distraction distorts visual working memory representations. Previous research has shown that memory responses are systematically biased towards passively viewed visual distractors that are similar to the memoranda. However, it remains unclear whether the prioritization of one working memory representation over another reduces the impact of perceptual distractors. We designed a study with five different types of visual distraction that varied in engagement and found evidence for both subtle distortions and catastrophic failures of memory. Importantly, prioritization protected working memories from catastrophic loss (fewer "swap errors") but rendered them more vulnerable to distortion (greater attractive "biases" towards the distractor). Our findings demonstrate that prioritization does not simply protect working memory from any and all interference, but rather it reduces the likelihood of catastrophic disruption from perceptual distraction at the cost of an increased likelihood of distortion.

Rumination burdens the updating of working memory.

Working memory is a vital, but capacity-limited, cognitive instrument that requires frequent updating as our goals and environment change. Individuals diagnosed with depression have a reduced capacity compared with the general population, as they have a propensity to fixate on negative information, even when it is not relevant for the task at hand. Here we investigated how characteristics of psychiatric illnesses, such as rumination, affect a person's ability to efficiently update emotional information in mind. We used both neutral and negative pictures of scenes in a working memory updating task that required participants to occasionally replace items held in mind during a brief delay period. Participants were presented with a probe item at the end of each trial and asked to report whether that item was in their current memory set. Responses were slowest and least accurate for images that had been replaced (i.e., "lures"), indicating there was some difficulty in successfully updating working memory in this paradigm. Participants who have both a high propensity to ruminate and a low working memory capacity were significantly more likely to false alarm to these lures. While emotional valence did not impact accuracy for these participants, their false alarms were faster for negative stimuli compared with neutral stimuli, indicating that task-irrelevant emotional information was more difficult to remove from working memory. These results demonstrate how rumination impairs goal-directed behavior by obscuring the boundary between relevant and irrelevant information in working memory.

Prioritization sharpens working memories but does not protect them from distraction.

Perceptual distraction distorts visual working memory representations. Previous research has shown that memory responses are systematically biased toward visual distractors that are similar to the memoranda. However, it remains unclear whether the prioritization of one working memory representation over another reduces the impact of perceptual distractors. In five behavioral experiments, we used different forms of retrospective cues (indicating the likelihood of testing each item and/or the reward for responding correctly to each item) to manipulate the prioritization of items in working memory before visual distraction. We examined the effects of distraction with nonparametric analyses and a novel distractor intrusion model. We found that memory responses were more precise (lower absolute response errors and stronger memory signals) for items that were prioritized. However, these prioritized items were not immune to distraction, and their memory responses were biased toward the visual distractors to the same degree as were unprioritized items. Our findings demonstrate that the benefits associated with prioritization in working memory do not include protection from distraction biases. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Pattern analysis of neuroimaging data reveals novel insights on threat learning and extinction in humans.

Several decades of rodent neurobiology research have identified a network of brain regions that support Pavlovian threat conditioning and extinction, focused predominately on the amygdala, hippocampus, and medial prefrontal cortex (mPFC). Surprisingly, functional magnetic resonance imaging (fMRI) studies have shown inconsistent evidence for these regions while humans undergo threat conditioning and extinction. In this review, we suggest that translational neuroimaging efforts have been hindered by reliance on traditional univariate analysis of fMRI. Whereas univariate analyses average activity across voxels in a given region, multivariate pattern analyses (MVPA) leverage the information present in spatial patterns of activity. MVPA therefore provides a more sensitive analysis tool to translate rodent neurobiology to human neuroimaging. We review human fMRI studies using MVPA that successfully bridge rodent models of amygdala, hippocampus, and mPFC function during Pavlovian learning. We also highlight clinical applications of these information-sensitive multivariate analyses. In sum, we advocate that the field should consider adopting a variety of multivariate approaches to help bridge cutting-edge research on the neuroscience of threat and anxiety.

Rewarded Extinction Increases Amygdalar Connectivity and Stabilizes Long-Term Memory Traces in the vmPFC.

Neurobiological evidence in rodents indicates that threat extinction incorporates reward neurocircuitry. Consequently, incorporating reward associations with an extinction memory may be an effective strategy to persistently attenuate threat responses. Moreover, while there is considerable research on the short-term effects of extinction strategies in humans, the long-term effects of extinction are rarely considered. In a within-subjects fMRI study with both female and male participants, we compared counterconditioning (CC; a form of rewarded-extinction) to standard extinction at recent (24 h) and remote (approximately one month) retrieval tests. Relative to standard extinction, rewarded extinction diminished 24-h relapse of arousal and threat expectancy, and reduced activity in brain regions associated with the appraisal and expression of threat (e.g., thalamus, insula, periaqueductal gray). The retrieval of reward-associated extinction memory was accompanied by functional connectivity between the amygdala and the ventral striatum, whereas the retrieval of standard-extinction memories was associated with connectivity between the amygdala and ventromedial prefrontal cortex (vmPFC). One month later, the retrieval of both standard-extinction and rewarded-extinction was associated with amygdala-vmPFC connectivity. However, only rewarded extinction created a stable memory trace in the vmPFC, identified through overlapping multivariate patterns of fMRI activity from extinction to 24-h and one-month retrieval. These findings provide new evidence that reward may generate a more stable and enduring memory trace of attenuated threat in humans.SIGNIFICANCE STATEMENT Prevalent treatments for pathologic fear and anxiety are based on the principles of Pavlovian extinction. Unfortunately, extinction forms weak memories that only temporarily inhibit the retrieval of threat associations. Thus, to increase the translational relevance of extinction research, it is critical to investigate whether extinction can be augmented to form a more enduring memory, especially after long intervals. Here, we used a multiday fMRI paradigm in humans to compare the short-term and long-term neurobehavioral effects of aversive-to-appetitive counterconditioning (CC), a form of augmented extinction. Our results provide novel evidence that including an appetitive stimulus during extinction can reduce short-term threat relapse and stabilize the memory trace of extinction in the ventromedial prefrontal cortex (vmPFC), for at least one month after learning.

Distinct monitoring strategies underlie costs and performance in prospective memory.

Prospective memory (PM) describes the ability to remember to perform goal-relevant actions at an appropriate time in the future amid concurrent demands. A key contributor to PM performance is thought to be the effortful monitoring of the environment for PM-related cues, a process whose existence is typically inferred from a behavioral interference measure of reaction times. This measure, referred to as "PM costs," is an informative but indirect proxy for monitoring, and it may not be sufficient to understand PM behaviors in all situations. In this study, we asked participants to perform a visual search task with arrows that varied in difficulty while concurrently performing a delayed-recognition PM task with pictures of faces and scenes. To gain a precise measurement of monitoring behavior, we used eye-tracking to record fixations to all task-relevant stimuli and related these fixation measures to both PM costs and PM accuracy. We found that PM costs reflected dissociable monitoring strategies: higher costs were associated with early and frequent monitoring while lower costs were associated with delayed and infrequent monitoring. Moreover, the link between fixations and PM costs varied with cognitive load, and the inclusion of fixation data yielded better predictions of PM accuracy than using PM costs alone. This study demonstrates the benefit of eye-tracking to disentangle the nature of PM costs and more precisely describe strategies involved in prospective remembering.

Working Memory Swap Errors Have Identifiable Neural Representations.

Working memory is an essential component of cognition that facilitates goal-directed behavior. Famously, it is severely limited and performance suffers when memory load exceeds an individual's capacity. Modeling of visual working memory responses has identified two likely types of errors: guesses and swaps. Swap errors may arise from a misbinding between the features of different items. Alternatively, these errors could arise from memory noise in the feature dimension used for cueing a to-be-tested memory item, resulting in the wrong item being selected. Finally, it is possible that so-called swap errors actually reflect informed guessing, which could occur at the time of a cue, or alternatively, at the time of the response. Here, we combined behavioral response modeling and fMRI pattern analysis to test the hypothesis that swap errors involve the active maintenance of an incorrect memory item. After the encoding of six spatial locations, a retro-cue indicated which location would be tested after memory retention. On accurate trials, we could reconstruct a memory representation of the cued location in both early visual cortex and intraparietal sulcus. On swap error trials identified with mixture modeling, we were able to reconstruct a representation of the swapped location, but not of the cued location, suggesting the maintenance of the incorrect memory item before response. Moreover, participants subjectively responded with some level of confidence, rather than complete guessing, on a majority of swap error trials. Together, these results suggest that swap errors are not mere response-phase guesses, but instead result from failures of selection in working memory, contextual binding errors, or informed guesses, which produce active maintenance of incorrect memory representations.

Estimating Intrinsic Manifold Dimensionality to Classify Task-Related Information in Human and Non-Human Primate Data.

Feature selection, or dimensionality reduction, has become a standard step in reducing large-scale neural datasets into usable signals for brain-machine interface and neurofeedback decoders. Current techniques in fMRI data reduce the number of voxels (features) by performing statistics on individual voxels or using traditional techniques that utilize linear combinations of features (e.g., principal component analysis (PCA)). However, these methods often do not account for the cross-correlations found across voxels and do not sufficiently reduce the feature space to support efficient real-time feedback. To overcome these limitations, we propose using factor analysis on fMRI data. This technique has become increasingly popular for extracting a minimal number of latent features to explain high-dimensional data in non-human primates (NHPs). Here, we demonstrate these methods in both NHP and human data. In NHP subjects (n=2), we reduced the number of features to an average of 26.86% and 14.86% of the total feature space to build our multinomial classifier. In one NHP subject, the average accuracy of classifying eight target locations over 64 sessions was 62.43% (+/-6.19%) compared to a PCA-based classifier with 60.26% (+/-6.02%). In healthy fMRI subjects, we reduced the feature space to an average of 0.33% of the initial space. Group average (n=5) accuracy of FA-based category classification was 74.33% (+/- 4.91%) compared to a PCA-based classifier with 68.42% (+/-4.79%). FA-based classifiers can maintain the performance fidelity observed with PCA-based decoders. Importantly, FA-based methods allow researchers to address specific hypotheses about how underlying neural activity relates to behavior.

Neural reinstatement reveals divided organization of fear and extinction memories in the human brain.

Neurobiological research in rodents has revealed that competing experiences of fear and extinction are stored as distinct memory traces in the brain. This divided organization is adaptive for mitigating overgeneralization of fear to related stimuli that are learned to be safe while also maintaining threat associations for unsafe stimuli. The mechanisms involved in organizing these competing memories in the human brain remain unclear. Here, we used a hybrid form of Pavlovian conditioning with an episodic memory component to identify overlapping multivariate patterns of fMRI activity associated with the formation and retrieval of fear versus extinction. In healthy adults, distinct regions of the medial prefrontal cortex (PFC) and hippocampus showed selective reactivation of fear versus extinction memories based on the temporal context in which these memories were encoded. This dissociation was absent in participants with posttraumatic stress disorder (PTSD) symptoms. The divided neural organization of fear and extinction may support flexible retrieval of context-appropriate emotional memories, while their disorganization may promote overgeneralization and increased fear relapse in affective disorders.

Multi-scale neural decoding and analysis.

Objective. Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.Approach. We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.Main results. We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.Significance. This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

Emotional learning retroactively enhances item memory but distorts source attribution.

An adaptive memory system should prioritize information surrounding a powerful learning event that may prove useful for predicting future meaningful events. The behavioral tagging hypothesis provides a mechanistic framework to interpret how weak experiences persist as durable memories through temporal association with a strong experience. Memories are composed of multiple elements, and different mnemonic aspects of the same experience may be uniquely affected by mechanisms that retroactively modulate a weakly encoded memory. Here, we investigated how emotional learning affects item and source memory for related events encoded close in time. Participants encoded trial-unique category exemplars before, during, and after Pavlovian fear conditioning. Selective retroactive enhancements in 24-h item memory were accompanied by a bias to misattribute items to the temporal context of fear conditioning. The strength of this source memory bias correlated with participants' retroactive item memory enhancement, and source misattribution to the emotional context predicted whether items were remembered overall. In the framework of behavioral tagging: Memory attribution was biased to the temporal context of the stronger event that provided the putative source of memory stabilization for the weaker event. We additionally found that fear conditioning selectively and retroactively enhanced stimulus typicality ratings for related items, and that stimulus typicality also predicted overall item memory. Collectively, these results provide new evidence that items related to emotional learning are misattributed to the temporal context of the emotional event and judged to be more representative of their semantic category. Both processes may facilitate memory retrieval for related events encoded close in time.

Separation of item and context in item-method directed forgetting.

Contextual information plays a critical role in directed forgetting (DF) of lists of items, whereas DF of individual items has been primarily associated with item-level processing. This study was designed to investigate whether context processing also contributes to the forgetting of individual items. Participants first viewed a series of words, with task-irrelevant scene images (used as "context tags") interspersed between them. Later, these words reappeared without the scenes and were followed by an instruction to remember or forget that word. Multivariate pattern analyses of fMRI data revealed that the reactivation of context information associated with the studied words (i.e., scene-related activity) was greater whereas the item-related information diminished after a forget instruction compared to a remember instruction. Critically, we found the magnitude of the separation between item information and context information predicted successful forgetting. These results suggest that the unbinding of an item from its context may support the intention to forget, and more generally they establish that contextual processing indeed contributes to item-method DF.

Differential neural plasticity of individual fingers revealed by fMRI neurofeedback.

Previous work has shown that functional magnetic resonance imaging (fMRI) activity patterns associated with individual fingers can be shifted by temporary impairment of the hand. Here, we investigated whether these neural activity patterns could be modulated endogenously and whether any behavioral changes result from this modulation. We used decoded neurofeedback in healthy individuals to encourage participants to shift the neural activity pattern in sensorimotor cortex of the middle finger toward the index finger, and the ring finger toward the little finger. We first mapped the neural activity patterns for all fingers of the right hand in an fMRI pattern localizer session. Then, in three subsequent neurofeedback sessions, participants were rewarded after middle/ring finger presses according to their activity pattern overlap during each trial. A force-sensitive keyboard was used to ensure that participants were not altering their physical finger coordination patterns. We found evidence that participants could learn to shift the activity pattern of the ring finger but not of the middle finger. Increased variability of these activity patterns during the localizer session was associated with the ability of participants to modulate them using neurofeedback. Participants also showed an increased preference for the ring finger but not for the middle finger in a postneurofeedback motor task. Our results show that neural activity and behaviors associated with the ring finger are more readily modulated than those associated with the middle finger. These results have broader implications for rehabilitation of individual finger movements, which may be limited or enhanced by individual finger plasticity after neurological injury.NEW & NOTEWORTHY It may be possible to remobilize fingers after neurological injury by altering neural activity patterns. Toward this end, we examined whether finger-related neural activity patterns could be modified in healthy individuals without physical intervention, using fMRI neurofeedback. Our findings show that greater variability of neural patterns at baseline predicted a participant's ability to successfully shift these patterns. Because neural variability is common in individuals poststroke, this illustrates a potential clinical benefit of this procedure.

Towards a common template for neural reinforcement of finger individuation.

The inability to individuate finger movements is a common impairment following stroke. Conventional physical therapy ignores underlying neural changes with recovery, leaving it unclear why sensorimotor function often remains impaired. Functional MRI neurofeedback can monitor neural activity and reinforce it towards a healthy template to restore function. However, identifying an individualized training template may not be possible depending on the severity of impairment. In this study, we investigated the use of functional alignment of brain data across healthy participants to create an idealized neural template to be used as a training target for new participants. We employed multi-voxel pattern analyses to assess the prediction accuracy and robustness to missing data of pre-trained functional templates corresponding to individual finger presses. We found a significant improvement in classification accuracy (p < 0.001) of individual finger presses when group data was aligned based on function (88%) rather than anatomy (46%). Importantly, we found no significant drop in performance when aligning a new participant to a pre-established template as compared to including this new participant in the creation of a new template. These results indicate that functionally aligned templates could provide an effective surrogate training target for patients following neurological injury.

Distraction in Visual Working Memory: Resistance is Not Futile.

Over half a century of research focused on understanding how working memory is capacity constrained has overshadowed the fact that it is also remarkably resistant to interference. Protecting goal-relevant information from distraction is a cornerstone of cognitive function that involves a multifaceted collection of control processes and storage mechanisms. Here, we discuss recent advances in cognitive psychology and neuroscience that have produced new insights into the nature of visual working memory and its ability to resist distraction. We propose that distraction resistance should be an explicit component in any model of working memory and that understanding its behavioral and neural correlates is essential for building a comprehensive understanding of real-world memory function.