A Neural Decision Signal during Internal Sampling from Working Memory in Humans.

How humans transform sensory information into decisions that steer purposeful behavior is a central question in psychology and neuroscience that is traditionally investigated during the sampling of external environmental signals. The decision-making framework of gradual information sampling toward a decision has also been proposed to apply when sampling internal sensory evidence from working memory. However, neural evidence for this proposal remains scarce. Here we show (using scalp EEG in male and female human volunteers) that sampling internal visual representations from working memory elicits a scalp EEG potential associated with gradual evidence accumulation-the central parietal positivity. Consistent with an evolving decision process, we show how this signal (1) scales with the time participants require to reach a decision about the cued memory content and (2) is amplified when having to decide among multiple contents in working memory. These results bring the electrophysiology of decision-making into the domain of working memory and suggest that variability in memory-guided behavior may be driven (at least in part) by variations in the sampling of our inner mental contents.

Boolean map and object reconcile as the unit of visual working memory.

The unit of visual working memory is a fundamental issue under debate in the fields of cognitive psychology and neuroscience, with some traditional research suggesting that it is an object, while other recent studies demonstrating that a Boolean map offers a better account. The controversy surrounding the unit of visual working memory often centers on the representation of objects consist of same dimensional features (e.g. bicolor objects). For 2 colors in a bicolor object, some behavioral studies have suggested that they need to be represented by separate units, while some other studies using electrophysiological measures have found that they can be represented within a single unit. This apparent conflict hints that Boolean map and object may reconcile as the unit of visual working memory. Adopting the contralateral delay activity as an electrophysiological marker of visual working memory, experiments 1 and 2 consistently found that the contralateral delay activity amplitude for memorizing bicolor circles at P7/P8 conformed the Boolean map-based storage throughout the whole maintenance, while the contralateral delay activity amplitude at P3/P4 just conformed the object-based storage during the early period. It suggests though Boolean map got stronger supporting evidence than object, they 2 may coexist as the unit of visual working memory.

Intrinsic functional networks for distinct sources of error in visual working memory.

Visual working memory (VWM) is a core cognitive function wherein visual information is stored and manipulated over short periods. Response errors in VWM tasks arise from the imprecise memory of target items, swaps between targets and nontargets, and random guesses. However, it remains unclear whether these types of errors are underpinned by distinct neural networks. To answer this question, we recruited 80 healthy adults to perform delayed estimation tasks and acquired their resting-state functional magnetic resonance imaging scans. The tasks required participants to reproduce the memorized visual feature along continuous scales, which, combined with mixture distribution modeling, allowed us to estimate the measures of memory precision, swap errors, and random guesses. Intrinsic functional connectivity within and between different networks, identified using a hierarchical clustering approach, was estimated for each participant. Our analyses revealed that higher memory precision was associated with increased connectivity within a frontal-opercular network, as well as between the dorsal attention network and an angular-gyrus-cerebellar network. We also found that coupling between the frontoparietal control network and the cingulo-opercular network contributes to both memory precision and random guesses. Our findings demonstrate that distinct sources of variability in VWM performance are underpinned by different yet partially overlapping intrinsic functional networks.

Prospection of Potential Actions during Visual Working Memory Starts Early, Is Flexible, and Predicts Behavior.

For visual working memory to serve upcoming behavior, it is crucial that we prepare for the potential use of working-memory contents ahead of time. Recent studies have demonstrated how the prospection and planning for an upcoming manual action starts early after visual encoding, and occurs alongside visual retention. Here, we address whether such "output planning" in visual working memory flexibly adapts to different visual-motor mappings, and occurs even when an upcoming action will only potentially become relevant for behavior. Human participants (female and male) performed a visual-motor working memory task in which they remembered one or two colored oriented bars for later (potential) use. We linked, and counterbalanced, the tilt of the visual items to specific manual responses. This allowed us to track planning of upcoming behavior through contralateral attenuation of ß band activity, a canonical motor-cortical EEG signature of manual-action planning. The results revealed how action encoding and subsequent planning alongside visual working memory (1) reflect anticipated task demands rather than specific visual-motor mappings, (2) occur even for actions that will only potentially become relevant for behavior, and (3) are associated with faster performance for the encoded item, at the expense of performance to other working-memory content. This reveals how the potential prospective use of visual working memory content is flexibly planned early on, with consequences for the speed of memory-guided behavior.SIGNIFICANCE STATEMENT It is increasingly studied how visual working memory helps us to prepare for the future, in addition to how it helps us to hold onto the past. Recent studies have demonstrated that the planning of prospective actions occurs alongside encoding and retention in working memory. We show that such early "output planning" flexibly adapts to varying visual-motor mappings, occurs both for certain and potential actions, and predicts ensuing working-memory guided behavior. These results highlight the flexible and future-oriented nature of visual working memory, and provide insight into the neural basis of the anticipatory dynamics that translate visual representations into adaptive upcoming behavior.

Ocular working memory signals are flexible to behavioral priority and subjective imagery strength.

The pupillary light response was long considered a brainstem reflex, outside of cognitive influence. However, newer findings indicate that pupil dilation (and eye movements) can reflect content held "in mind" with working memory (WM). These findings may reshape understanding of ocular and WM mechanisms, but it is unclear whether the signals are artifactual or functional to WM. Here, we ask whether peripheral and oculomotor WM signals are sensitive to the task-relevance or "attentional state" of WM content. During eye-tracking, human participants saw both dark and bright WM stimuli, then were retroactively cued to the item that would most likely be tested. Critically, we manipulated the attentional priority among items by varying the cue reliability across blocks. We confirmed previous findings that remembering darker items is associated with larger pupils (vs. brighter), and that gaze is biased toward cued item locations. Moreover, we discovered that pupil and eye movement responses were influenced differently by WM item relevance. Feature-specific pupillary effects emerged only for highly prioritized WM items but were eliminated when cues were less reliable, and pupil effects also increased with self-reported visual imagery strength. Conversely, gaze position consistently veered toward the cued item location, regardless of cue reliability. However, biased microsaccades occurred at a higher frequency when cues were more reliable, though only during a limited post-cue time window. Therefore, peripheral sensorimotor processing is sensitive to the task-relevance or functional state of internal WM content, but pupillary and eye movement WM signals show distinct profiles. These results highlight a potential role for early visual processing in maintaining multiple WM content dimensions.NEW & NOTEWORTHY Here, we found that working memory (WM)-driven ocular inflections-feature-specific pupillary and saccadic biases-were muted for memory items that were less behaviorally relevant. This work illustrates that functionally informative goal signals may extend as early as the sensorimotor periphery, that pupil size may be under more fine-grained control than originally thought, and that ocular signals carry multiple dimensions of cognitively relevant information.

Investigating the effects of perceptual complexity versus conceptual meaning on the object benefit in visual working memory.

Previous research has demonstrated greater visual working memory (VWM) performance for real-world objects compared with simple features. Greater amplitudes of the contralateral delay activity (CDA)-a sustained event-related potential measured during the delay period of a VWM task-have also been noted for meaningful stimuli, despite being thought of as a neural marker of a fixed working memory capacity. The current study aimed to elucidate the factors underlying improved memory performance for real-world objects by isolating the relative contributions of perceptual complexity (i.e., number of visual features) and conceptual meaning (i.e., availability of semantic, meaningful features). Participants (N = 22) performed a lateralized VWM task to test their memory of intact real-world objects, scrambled real-world objects and colours. The CDA was measured during both encoding and WM retention intervals (600-1000 ms and 1300-1700 ms poststimulus onset, respectively), and behavioural performance was estimated by using d' (memory strength in a two-alternative forced choice task). Behavioural results revealed significantly better performance within-subjects for real-world objects relative to scrambled objects and colours, with no difference between colours and scrambled objects. The amplitude of the CDA was also largest for intact real-world objects, with no difference in magnitude for scrambled objects and colours, during working memory maintenance. However, during memory encoding, both the colours and intact real-world objects had significantly greater amplitudes than scrambled objects and were comparable in magnitude. Overall, findings suggest that conceptual meaning (semantics) supports the memory benefit for real-world objects.

ERPs and alpha oscillations track the encoding and maintenance of object-based representations in visual working memory.

When memorizing an integrated object such as a Kanizsa figure, the completion of parts into a coherent whole is attained by grouping processes which render a whole-object representation in visual working memory (VWM). The present study measured event-related potentials (ERPs) and oscillatory amplitudes to track these processes of encoding and representing multiple features of an object in VWM. To this end, a change detection task was performed, which required observers to memorize both the orientations and colors of six "pacman" items while inducing configurations of the pacmen that systematically varied in terms of their grouping strength. The results revealed an effect of object configuration in VWM despite physically constant visual input: change detection for both orientation and color features was more accurate with increased grouping strength. At the electrophysiological level, the lateralized ERPs and alpha activity mirrored this behavioral pattern. Perception of the orientation features gave rise to the encoding of a grouped object as reflected by the amplitudes of the Ppc. The grouped object structure, in turn, modulated attention to both orientation and color features as indicated by the enhanced N1pc and N2pc. Finally, during item retention, the representation of individual objects and the concurrent allocation of attention to these memorized objects were modulated by grouping, as reflected by variations in the CDA amplitude and a concurrent lateralized alpha suppression, respectively. These results indicate that memorizing multiple features of grouped, to-be-integrated objects involves multiple, sequential stages of processing, providing support for a hierarchical model of object representations in VWM.

Strategic control of location and ordinal context in visual working memory.

Working memory (WM) requires encoding stimulus identity and context (e.g. where or when stimuli were encountered). To explore the neural bases of the strategic control of context binding in WM, we acquired fMRI while subjects performed delayed recognition of 3 orientation patches presented serially and at different locations. The recognition probe was an orientation patch with a superimposed digit, and pretrial instructions directed subjects to respond according to its location ("location-relevant"), to the ordinal position corresponding to its digit ("order-relevant"), or to just its orientation (relative to all three samples; "context-irrelevant"). Delay period signal in PPC was greater for context-relevant than for "context-irrelevant" trials, and multivariate decoding revealed strong sensitivity to context binding requirements (relevant vs. "irrelevant") and to context domain ("location-" vs. "order-relevant") in both occipital cortex and PPC. At recognition, multivariate inverted encoding modeling revealed markedly different patterns in these 2 regions, suggesting different context-processing functions. In occipital cortex, an active representation of the location of each of the 3 samples was reinstated regardless of the trial type. The pattern in PPC, by contrast, suggested a trial type-dependent filtering of sample information. These results indicate that PPC exerts strategic control over the representation of stimulus context in visual WM.

Consistent social information perceived in animated backgrounds improves ensemble perception of facial expressions.

Observers can rapidly extract the mean emotion from a set of faces with remarkable precision, known as ensemble coding. Previous studies have demonstrated that matched physical backgrounds improve the precision of ongoing ensemble tasks. However, it remains unknown whether this facilitation effect still occurs when matched social information is perceived from the backgrounds. In two experiments, participants decided whether the test face in the retrieving phase appeared more disgusted or neutral than the mean emotion of the face set in the encoding phase. Both phases were paired with task-irrelevant animated backgrounds, which included either the forward movement trajectory carrying the "cooperatively chasing" information, or the backward movement trajectory conveying no such chasing information. The backgrounds in the encoding and retrieving phases were either mismatched (i.e., forward and backward replays of the same trajectory), or matched (i.e., two identical forward movement trajectories in Experiment 1, or two different forward movement trajectories in Experiment 2). Participants in both experiments showed higher ensemble precisions and better discrimination sensitivities when backgrounds matched. The findings suggest that consistent social information perceived from memory-related context exerts a context-matching facilitation effect on ensemble coding and, more importantly, this effect is independent of consistent physical information.

Unravelling the object-based nature of visual working memory: insight from pointers.

Visual working memory (VWM) plays a crucial role in temporarily storing and processing visual information, but the nature of stored representations and their interaction with new inputs has long been unclear. The pointer system refers to how VWM links new sensory inputs to stored information using specific cues. This study aimed to investigate whether the pointer system is based on spatial, feature-based, or object-based cues by employing the repetition benefit effect, where memory performance improves with repeated memory items. Across three experiments, we manipulated spatial positions, shapes, and colors as pointer cues to determine how these features affect VWM consolidation and updating. The results showed that while spatial location serves as a strong pointer cue, shape and color features can also effectively reestablish object correspondence in VWM. These findings support the view that the pointer system in VWM is flexible and object-based, utilizing various feature cues to maintain memory continuity. This study provides new insights into how VWM connects new inputs with stored information through the pointer system.

Selectively maintaining an object's feature in visual working memory: A comparison between highly discriminable and fine-grained features.

Selectively maintaining information is an essential function of visual working memory (VWM). Recent VWM studies have mainly focused on selective maintenance of objects, leaving the mechanisms of selectively maintaining an object's feature in VWM unknown. Based on the interactive model of perception and VWM, we hypothesized that there are distinct selective maintenance mechanisms for objects containing fine-grained features versus objects containing highly discriminable features. To test this hypothesis, we first required participants to memorize a dual-feature object (colored simple shapes vs. colored polygons), and informed them about the target feature via a retro-cue. Then a visual search task was added to examine the fate of the irrelevant feature. The selective maintenance of an object's feature predicted that the irrelevant feature should be removed from the active state of VWM and should not capture attention when presented as a distractor in the visual search task. We found that irrelevant simple shapes impaired performance in the visual search task (Experiment 1). However, irrelevant polygons did not affect visual search performance (Experiment 2), and this could not be explained by decay of polygons (Experiment 3) or by polygons not capturing attention (Experiment 4). These findings suggest that VWM adopts dissociable mechanisms to selectively maintain an object's feature, depending on the feature's perceptual characteristics.

Comparing memory capacity across stimuli requires maximally dissimilar foils: Using deep convolutional neural networks to understand visual working memory capacity for real-world objects.

The capacity of visual working and visual long-term memory plays a critical role in theories of cognitive architecture and the relationship between memory and other cognitive systems. Here, we argue that before asking the question of how capacity varies across different stimuli or what the upper bound of capacity is for a given memory system, it is necessary to establish a methodology that allows a fair comparison between distinct stimulus sets and conditions. One of the most important factors determining performance in a memory task is target/foil dissimilarity. We argue that only by maximizing the dissimilarity of the target and foil in each stimulus set can we provide a fair basis for memory comparisons between stimuli. In the current work we focus on a way to pick such foils objectively for complex, meaningful real-world objects by using deep convolutional neural networks, and we validate this using both memory tests and similarity metrics. Using this method, we then provide evidence that there is a greater capacity for real-world objects relative to simple colors in visual working memory; critically, we also show that this difference can be reduced or eliminated when non-comparable foils are used, potentially explaining why previous work has not always found such a difference. Our study thus demonstrates that working memory capacity depends on the type of information that is remembered and that assessing capacity depends critically on foil dissimilarity, especially when comparing memory performance and other cognitive systems across different stimulus sets.

Conceptual masking disrupts change-detection performance.

The present study investigated the effects of long-term knowledge on backward masking interference in visual working memory (VWM) by varying the similarity of mask stimuli along categorical dimensions. To-be-remembered items and masks were taken from categories controlled for perceptual distinctiveness and distinctiveness in kinds (e.g., there are many kinds of cars and few kinds of coffee mugs). Participants completed a change-detection task in which the memory array consisted of exemplars from either a similar or distinctive category, followed by a mask array of items from the same category (conceptually similar versus conceptually distinct categories), a different category, or no mask. The results over two experiments showed greater interference from conceptually similar masks as compared with the other conditions across stimulus onset asynchrony (SOA) conditions, suggesting masking with conceptually similar categories leads to more interference even when masks are shown well after the stimulus. These results have important implications for both the nature and time course of long-term conceptual knowledge influencing VWM, particularly when using complex real-world objects.

Modifiable dementia risk factors associated with objective and subjective cognition.

INTRODUCTION: Early detection of both objective and subjective cognitive impairment is important. Subjective complaints in healthy individuals can precede objective deficits. However, the differential associations of objective and subjective cognition with modifiable dementia risk factors are unclear. METHODS: We gathered a large cross-sectional sample (N = 3327, age 18 to 84) via a smartphone app and quantified the associations of 13 risk factors with subjective memory problems and three objective measures of executive function (visual working memory, cognitive flexibility, model-based planning). RESULTS: Depression, socioeconomic status, hearing handicap, loneliness, education, smoking, tinnitus, little exercise, small social network, stroke, diabetes, and hypertension were all associated with impairments in at least one cognitive measure. Subjective memory had the strongest link to most factors; these associations persisted after controlling for depression. Age mostly did not moderate these associations. DISCUSSION: Subjective cognition was more sensitive to self-report risk factors than objective cognition. Smartphones could facilitate detecting the earliest cognitive impairments. HIGHLIGHTS: Smartphone assessments of cognition were sensitive to dementia risk factors. Subjective cognition had stronger links to most factors than did objective cognition. These associations were not fully explained by depression. These associations were largely consistent across the lifespan.

Changes in behavioral priority influence the accessibility of working memory content.

Evolving behavioral goals require the existence of selection mechanisms that prioritize task-relevant working memory (WM) content for action. Selecting an item stored in WM is known to blunt and/or reverse information loss in stimulus-specific representations of that item reconstructed from human brain activity, but extant studies have focused on all-or-none circumstances that allow or disallow an agent to select one of several items stored in WM. Conversely, behavioral studies suggest that humans can flexibly assign different levels of priority to different items stored in WM, but how doing so influences neural representations of WM content is unclear. One possibility is that assigning different levels of priority to items in WM influences the quality of those representations, resulting in more robust neural representations of high- vs. low-priority WM content. A second - and non-exclusive - possibility is that asymmetries in behavioral priority influence how rapidly neural representations of high- vs. low-priority WM content can be selected and reported. We tested these possibilities in two experiments by decoding high- and low-priority WM content from EEG recordings obtained while human volunteers performed a retrospectively cued WM task. Probabilistic changes in the behavioral relevance of a remembered item had no effect on our ability to decode it from EEG signals; instead, these changes influenced the latency at which above-chance decoding performance was reached. Thus, our results indicate that probabilistic changes in the behavioral relevance of WM content influence the ease with which memories can be selected independently of their strength.

Cross-modal interactions at the audiovisual cocktail-party revealed by behavior, ERPs, and neural oscillations.

Theories of attention argue that objects are the units of attentional selection. In real-word environments such objects can contain visual and auditory features. To understand how mechanisms of selective attention operate in multisensory environments, in this pre-registered study, we created an audiovisual cocktail-party situation, in which two speakers (left and right of fixation) simultaneously articulated brief numerals. In three separate blocks, informative auditory speech was presented (a) alone or paired with (b) congruent or (c) uninformative visual speech. In all blocks, subjects localized a pre-defined numeral. While audiovisual-congruent and uninformative speech improved response times and speed of information uptake according to diffusion modeling, an ERP analysis revealed that this did not coincide with enhanced attentional engagement. Yet, consistent with object-based attentional selection, the deployment of auditory spatial attention (N2ac) was accompanied by visuo-spatial attentional orienting (N2pc) irrespective of the informational content of visual speech. Notably, an N2pc component was absent in the auditory-only condition, demonstrating that a sound-induced shift of visuo-spatial attention relies on the availability of audio-visual features evolving coherently in time. Additional exploratory analyses revealed cross-modal interactions in working memory and modulations of cognitive control.

White matter microstructure is associated with the precision of visual working memory.

Visual working memory is critical for goal-directed behavior as it maintains continuity between previous and current visual input. Functional neuroimaging studies have shown that visual working memory relies on communication between distributed brain regions, which implies an important role for long-range white matter connections in visual working memory performance. Here, we characterized the relationship between the microstructure of white matter association tracts and the precision of visual working memory representations. To that purpose, we devised a delayed estimation task which required participants to reproduce visual features along a continuous scale. A sample of 80 healthy adults performed the task and underwent diffusion-weighted MRI. We applied mixture distribution modelling to quantify the precision of working memory representations, swap errors, and guess rates, all of which contribute to observed responses. Latent components of microstructural properties in sets of anatomical tracts were identified by principal component analysis. We found an interdependency between fibre coherence in the bilateral superior longitudinal fasciculus (SLF) I, SLF II, and SLF III, on one hand, and the bilateral inferior fronto-occipital fasciculus (IFOF), on the other, in mediating the precision of visual working memory in a functionally specific manner. We also found that individual differences in axonal density in a network comprising the bilateral inferior longitudinal fasciculus (ILF) and SLF III and right SLF II, in combination with a supporting network located elsewhere in the brain, form a common system for visual working memory to modulate response precision, swap errors, and random guess rates.

Egocentric cues influence the allocentric spatial memory of object configurations for memory-guided actions.

Allocentric and egocentric reference frames are used to code the spatial position of action targets in reference to objects in the environment, i.e., relative to landmarks (allocentric), or the observer (egocentric). Previous research investigated reference frames in isolation, for example, by shifting landmarks relative to the target and asking participants to reach to the remembered target location. Systematic reaching errors were found in the direction of the landmark shift and used as a proxy for allocentric spatial coding. Here, we examined the interaction of both allocentric and egocentric reference frames by shifting the landmarks as well as the observer. We asked participants to encode a three-dimensional configuration of balls and to reproduce this configuration from memory after a short delay followed by a landmark or an observer shift. We also manipulated the number of landmarks to test its effect on the use of allocentric and egocentric reference frames. We found that participants were less accurate when reproducing the configuration of balls after an observer shift, which was reflected in larger configurational errors. In addition, an increase in the number of landmarks led to a stronger reliance on allocentric cues and a weaker contribution of egocentric cues. In sum, our results highlight the important role of egocentric cues for allocentric spatial coding in the context of memory-guided actions.NEW & NOTEWORTHY Objects in our environment are coded relative to each other (allocentrically) and are thought to serve as independent and reliable cues (landmarks) in the context of unreliable egocentric signals. Contrary to this assumption, we demonstrate that egocentric cues alter the allocentric spatial memory, which could reflect recently discovered interactions between allocentric and egocentric neural processing pathways. Furthermore, additional landmarks lead to a higher contribution of allocentric and a lower contribution of egocentric cues.

Learning-Induced Plasticity Enhances the Capacity of Visual Working Memory.

Visual working memory (VWM) is limited in capacity, though memorizing meaningful objects may refine this limitation. However, meaningful and meaningless stimuli typically differ perceptually, and objects' associations with meaning are usually already established outside the laboratory, potentially confounding experimental findings. Here, in two experiments with young adults (N = 45 and N = 20), we controlled for these influences by having observers actively learn associations of (for them) initially meaningless stimuli: Chinese characters, half of which were consistently paired with pictures of animals or everyday objects in a learning phase. This phase was preceded and followed by a (pre- and postlearning) change-detection task to assess VWM performance. The results revealed that short-term retention was enhanced after learning, particularly for meaning-associated characters, although participants did not quite reach the accuracy level attained by native Chinese observers (young adults, N = 20). These results thus provide direct experimental evidence that participants' VWM of objects is boosted by them having acquired a long-term-memory association with meaning.

Cerebellar Transcranial Magnetic Stimulation (TMS) Impairs Visual Working Memory.

An increasing body of evidence points to the involvement of the cerebellum in cognition. Specifically, previous studies have shown that the superior and inferior portions of the cerebellum are involved in different verbal working memory (WM) mechanisms as part of two separate cerebro-cerebellar loops for articulatory rehearsal and phonological storage mechanisms. In comparison, our understanding of the involvement of the cerebellum in visual WM remains limited. We have previously shown that performance in verbal WM is disrupted by single-pulse transcranial magnetic stimulation (TMS) of the right superior cerebellum. The present study aimed to expand on this notion by exploring whether the inferior cerebellum is similarly involved in visual WM. Here, we used fMRI-guided, double-pulse TMS to probe the necessity of left superior and left inferior cerebellum in visual WM. We first conducted an fMRI localizer using the Sternberg visual WM task, which yielded targets in left superior and inferior cerebellum. Subsequently, TMS stimulation of these regions at the end of the encoding phase resulted in decreased accuracy in the visual WM task. Differences in the visual WM deficits caused by stimulation of superior and inferior left cerebellum raise the possibility that these regions are involved in different stages of visual WM.