Just-in-Time Encoding Into Visual Working Memory Is Contingent Upon Constant Availability of External Information.

Humans maintain an intricate balance between storing information in visual working memory (VWM) and just-in-time sampling of the external world, rooted in a trade-off between the cost of maintaining items in VWM versus retrieving information as it is needed. Previous studies have consistently shown that one prerequisite of just-in-time sampling is a high degree of availability of external information, and that introducing a delay before being able to access information led participants to rely less on the external world and more on VWM. However, these studies manipulated availability in such a manner that the cost of sampling was stable and predictable. It is yet unclear whether participants become less reliant on external information when it is more difficult to factor in the cost of sampling that information. In two experiments, participants copied an example layout from the left to the right side of the screen. In Experiment 1, intermittent occlusion of the example layout led participants to attempt to encode more items per inspection than when the layout was constantly available, but this did not consistently result in more correct placements. However, these findings could potentially be explained by inherent differences in how long the example layout could be viewed. Therefore in Experiment 2, the example layout only became available after a gaze-contingent delay, which could be constant or variable. Here, the introduction of any delay led to increased VWM load compared to no delay, although the degree of variability in the delay did not alter behaviour. These results reaffirm that the nature of when we engage VWM is dynamical, and suggest that any disruption to the continuous availability of external information is the main driver of increased VWM usage relative to whether availability is predictable or not.

The impact of acute asymmetric hearing loss on multisensory integration.

Humans have the remarkable ability to integrate information from different senses, which greatly facilitates the detection, localization and identification of events in the environment. About 466 million people worldwide suffer from hearing loss. Yet, the impact of hearing loss on how the senses work together is rarely investigated. Here, we investigate how a common sensory impairment, asymmetric conductive hearing loss (AHL), alters the way our senses interact by examining human orienting behaviour with normal hearing (NH) and acute AHL. This type of hearing loss disrupts auditory localization. We hypothesized that this creates a conflict between auditory and visual spatial estimates and alters how auditory and visual inputs are integrated to facilitate multisensory spatial perception. We analysed the spatial and temporal properties of saccades to auditory, visual and audiovisual stimuli before and after plugging the right ear of participants. Both spatial and temporal aspects of multisensory integration were affected by AHL. Compared with NH, AHL caused participants to make slow, inaccurate and unprecise saccades towards auditory targets. Surprisingly, increased weight on visual input resulted in accurate audiovisual localization with AHL. This came at a cost: saccade latencies for audiovisual targets increased significantly. The larger the auditory localization errors, the less participants were able to benefit from audiovisual integration in terms of saccade latency. Our results indicate that observers immediately change sensory weights to effectively deal with acute AHL and preserve audiovisual accuracy in a way that cannot be fully explained by statistical models of optimal cue integration.

Action consequences guide the use of visual working memory.

Visual working memory (VWM) is a store for temporary maintenance of visual information. It is often disregarded, though, that information is typically stored to enable actions. Therefore, the context of these actions is of great importance for how VWM is used. Here, we questioned whether the severity of the consequence of an action might affect how well information is memorized, and how cautiously it is utilized. We employed an (online) copying task, in which participants reproduced an example display comprised of six items in a grid, using a pool of items. Crucially, we manipulated the severity of penalties: participants had to wait 5 (high) or 0.5 (low error cost) s after an error. Additionally, we manipulated the accessibility of task-relevant information (a well-studied manipulation in this paradigm): participants had to wait 5 (high) or 0.5 (low sampling cost) s to inspect the example. Our results show that with higher error cost the number of inspections remained comparable, but the number of errors decreased. Furthermore, they show that with higher sampling cost the number of inspections halved, and the number of errors increased. Thus, more severe action consequences increase the reluctance to act on uncertain information in VWM, but do not lead to more attempts to store information in VWM. We conclude that, in contrast to the effect of the accessibility of information, action consequences do not affect how well information is memorized, but affect how cautiously this stored information is utilized. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The Intensity of Internal and External Attention Assessed with Pupillometry.

Not only is visual attention shifted to objects in the external world, attention can also be directed to objects in memory. We have recently shown that pupil size indexes how strongly items are attended externally, which was reflected in more precise encoding into visual working memory. Using a retro-cuing paradigm, we here replicated this finding by showing that stronger pupil constrictions during encoding were reflective of the depth of encoding. Importantly, we extend this previous work by showing that pupil size also revealed the intensity of internal attention toward content stored in visual working memory. Specifically, pupil dilation during the prioritization of one among multiple internally stored representations predicted the precision of the prioritized item. Furthermore, the dynamics of the pupillary responses revealed that the intensity of internal and external attention independently determined the precision of internalized visual representations. Our results show that both internal and external attention are not all-or-none processes, but should rather be thought of as continuous resources that can be deployed at varying intensities. The employed pupillometric approach allows to unravel the intricate interplay between internal and external attention and their effects on visual working memory.

Steady-state Visual Evoked Potentials Reveal Dynamic (Re)allocation of Spatial Attention during Maintenance and Utilization of Visual Working Memory.

Visual working memory (VWM) allows storing goal-relevant information to guide future behavior. Prior work suggests that VWM is spatially organized and relies on spatial attention directed toward locations at which memory items were encoded, even if location is task-irrelevant. Importantly, attention often needs to be dynamically redistributed between locations, for example, in preparation for an upcoming probe. Very little is known about how attentional resources are distributed between multiple locations during a VWM task and even less about the dynamic changes governing such attentional shifts over time. This is largely due to the inability to use behavioral outcomes to reveal fast dynamic changes within trials. We here demonstrated that EEG steady-state visual evoked potentials (SSVEPs) successfully track the dynamic allocation of spatial attention during a VWM task. Participants were presented with to-be-memorized gratings and distractors at two distinct locations, tagged with flickering discs. This allowed us to dynamically track attention allocated to memory and distractor items via their coupling with space by quantifying the amplitude and coherence of SSVEP responses in the EEG signal to flickering stimuli at the former memory and distractor locations. SSVEP responses did not differ between memory and distractor locations during early maintenance. However, shortly before probe comparison, we observed a decrease in SSVEP coherence over distractor locations indicative of a reallocation of spatial attentional resources. RTs were shorter when preceded by stronger decreases in SSVEP coherence at distractor locations, likely reflecting attentional shifts from the distractor to the probe or memory location. We demonstrate that SSVEPs can inform about dynamic processes in VWM, even if location does not have to be reported by participants. This finding not only supports the notion of a spatially organized VWM but also reveals that SSVEPs betray a dynamic prioritization process of working memory items and locations over time that is directly predictive of memory performance.

Memory reports are biased by all relevant contents of working memory.

Sensory input is inherently noisy while the world is inherently predictable. When multiple observations of the same object are available, integration of the available information necessarily increases the reliability of a world estimate. Optimal integration of multiple instances of sensory evidence has already been demonstrated during multisensory perception but could benefit unimodal perception as well. In the present study 330 participants observed a sequence of four orientations and were cued to report one of them. Reports were biased by all simultaneously memorized items that were similar and relevant to the target item, weighted by their reliability (signal-to-noise ratio). Orientations presented before and presented after the target biased report, demonstrating that the bias emerges in memory and not (exclusively) during perception or encoding. Only attended, task-relevant items biased report. We suggest that these results reflect how the visual system integrates information that is sampled from the same object at consecutive timepoints to promote perceptual stability and behavioural effectiveness in a dynamic world. We suggest that similar response biases, such as serial dependence, might be instances of a more general mechanism of working memory averaging. Data is available at https://osf.io/embcf/ .

Zooming in on abnormal local and global processing biases after stroke: Frequency, lateralization, and associations with cognitive functions.

OBJECTIVES: The 'attentional spotlight' can be adjusted depending on the task requirements, resulting in processing information at either the local or global level. Stroke can lead to local or global processing biases, or the inability to simultaneously attend both levels. In this study, we assessed the (1) prevalence of abnormal local and global biases following stroke, (2) differences between left- and right-sided brain damaged patients, and (3) relations between local and global interference, the ability to attend local and global levels simultaneously, and lateralized attention, search organization, search speed, visuo-construction, executive functioning, and verbal (working) memory. METHODS: Stroke patients admitted for inpatient rehabilitation completed directed (N = 192 total; N = 46 left-sided/N = 48 right-sided lesion) and divided (N = 258 total; N = 67 left-sided/N = 66 right-sided lesion) local-global processing tasks, as well as a conventional neuropsychological assessment. Processing biases and interference effects were separately computed for directed and divided tasks. RESULTS: On the local-global tasks, 7.8-10.9% of patients showed an abnormal local bias and 6.3-8.3% an abnormal global bias for directed attention, and 5.4-10.1% an abnormal local bias and 6.6-15.9% an abnormal global bias for divided attention. There was no significant difference between patients with left- and right-sided brain damage. There was a moderate positive relation between local interference and search speed, and a small positive relation between global interference and neglect. CONCLUSIONS: Abnormal local and global biases can occur after stroke and might relate to a range of cognitive functions. A specific bias might require a different approach in assessment, psycho-education, and treatment.

When the central integrator disintegrates: A review of the role of the thalamus in cognition and dementia.

The thalamus is a complex neural structure with numerous anatomical subdivisions and intricate connectivity patterns. In recent decades, the traditional view of the thalamus as a relay station and "gateway to the cortex" has expanded in recognition of its role as a central integrator of inputs from sensory systems, cortex, basal ganglia, limbic systems, brain stem nuclei, and cerebellum. As such, the thalamus is critical for numerous aspects of human cognition, mood, and behavior, as well as serving sensory processing and motor functions. Thalamus pathology is an important contributor to cognitive and functional decline, and it might be argued that the thalamus has been somewhat overlooked as an important player in dementia. In this review, we provide a comprehensive overview of thalamus anatomy and function, with an emphasis on human cognition and behavior, and discuss emerging insights on the role of thalamus pathology in dementia.

Revealing visual working memory operations with pupillometry: Encoding, maintenance, and prioritization.

Pupillary dynamics reflect effects of distinct and important operations of visual working memory: encoding, maintenance, and prioritization. Here, we review how pupil size predicts memory performance and how it provides novel insights into the mechanisms of each operation. Visual information must first be encoded into working memory with sufficient precision. The depth of this encoding process couples to arousal-linked baseline pupil size as well as a pupil constriction response before and after stimulus onset, respectively. Subsequently, the encoded information is maintained over time to ensure it is not lost. Pupil dilation reflects the effortful maintenance of information, wherein storing more items is accompanied by larger dilations. Lastly, the most task-relevant information is prioritized to guide upcoming behavior, which is reflected in yet another dilatory component. Moreover, activated content in memory can be pupillometrically probed directly by tagging visual information with distinct luminance levels. Through this luminance-tagging mechanism, pupil light responses reveal whether dark or bright items receive more attention during encoding and prioritization. Together, conceptualizing pupil responses as a sum of distinct components over time reveals insights into operations of visual working memory. From this viewpoint, pupillometry is a promising avenue to study the most vital operations through which visual working memory works. This article is categorized under: Psychology > Attention Psychology > Memory Psychology > Theory and Methods.

Open-DPSM: An open-source toolkit for modeling pupil size changes to dynamic visual inputs.

Pupil size change is a widely adopted, sensitive indicator for sensory and cognitive processes. However, the interpretation of these changes is complicated by the influence of multiple low-level effects, such as brightness or contrast changes, posing challenges to applying pupillometry outside of extremely controlled settings. Building on and extending previous models, we here introduce Open Dynamic Pupil Size Modeling (Open-DPSM), an open-source toolkit to model pupil size changes to dynamically changing visual inputs using a convolution approach. Open-DPSM incorporates three key steps: (1) Modeling pupillary responses to both luminance and contrast changes; (2) Weighing of the distinct contributions of visual events across the visual field on pupil size change; and (3) Incorporating gaze-contingent visual event extraction and modeling. These steps improve the prediction of pupil size changes beyond the here-evaluated benchmarks. Open-DPSM provides Python functions, as well as a graphical user interface (GUI), enabling the extension of its applications to versatile scenarios and adaptations to individualized needs. By obtaining a predicted pupil trace using video and eye-tracking data, users can mitigate the effects of low-level features by subtracting the predicted trace or assess the efficacy of the low-level feature manipulations a priori by comparing estimated traces across conditions.

Taking a Popular Science Approach to the Field of Attention.

Over the past few years, I have been actively engaged in informing the public, including policy makers and teachers, about the latest findings in attention research. Despite certain challenges in meeting expectations of both the public and peers, engaging with the public and sharing scientific knowledge not only inspired new research avenues but also highlighted the importance of supporting popular science activities within academia.

More (corrective) consecutive saccades after a lesion to the posterior parietal cortex.

To reach a target, primary saccades (S1s) are often followed by (corrective) consecutive saccades (S2, and potentially S3, S4, S5), which are based on retinal and extraretinal feedback. Processing these extraretinal signals was found to be significantly impaired by lesions to the posterior parietal cortex (PPC). Recent studies, however, added a more nuanced view to the role of the PPC, where patients with PPC lesions still used extraretinal signals for S2s and perceptual judgements (Fabius et al., 2020; Rath-Wilson & Guitton, 2015). Hence, it seems that a PPC lesion is not disrupting extraretinal processing per se. Yet, a lesion might still result in less reliable processing of extraretinal signals. Here, we investigated whether this lower reliability manifests as decreased or delayed S2 initiation. Patients with PPC lesions (n = 7) and controls (n = 26) performed a prosaccade task where the target either remained visible or was removed after S1 onset. When S1 is removed, accurate S2s (corrections of S1 error) rely solely on extraretinal signals. We analysed S2 quantity and timing using linear mixed-effects modelling and additive hazards analyses. Patients demonstrated slower S1 execution and lower S1 amplitudes than controls, but their S2s still compensated the S1 undershoot, also when they only relied on extraretinal information. Surprisingly, patients showed an increased amount of S2s. This deviation from control behaviour can be seen as suboptimal, but given the decreased accuracy of the primary saccade, it could be optimal for patients to employ more (corrective) consecutive saccades to overcome this inaccuracy.

Don't hide the instruction manual: A dynamic trade-off between using internal and external templates during visual search.

Visual search is typically studied by requiring participants to memorize a template initially, for which they subsequently search in a crowded display. Search in daily life, however, often involves templates that remain accessible externally, and may therefore be (re)attended for just-in-time encoding or to refresh internal template representations. Here, we show that participants indeed use external templates during search when given the chance. This behavior was observed during both simple and complex search, scaled with task difficulty, and was associated with improved performance. Furthermore, we show that participants used external sampling not only to offload memory, but also as a means of verifying whether the template was remembered correctly at the end of trials. We conclude that the external world may not only provide the challenge (e.g., distractors), but may dynamically ease search. These results argue for extensions of state-of-the-art models of search, because external sampling seems to be used frequently, in at least two ways and is actually beneficial for task performance. Our findings support a model of visual working memory that emphasizes a resource-efficient trade-off between storing and (re)attending external information.

Eye Movements as Proxy for Visual Working Memory Usage: Increased Reliance on the External World in Korsakoff Syndrome.

In the assessment of visual working memory, estimating the maximum capacity is currently the gold standard. However, traditional tasks disregard that information generally remains available in the external world. Only when to-be-used information is not readily accessible, memory is taxed. Otherwise, people sample information from the environment as a form of cognitive offloading. To investigate how memory deficits impact the trade-off between sampling externally or storing internally, we compared gaze behaviour of individuals with Korsakoff amnesia (n = 24, age range 47-74 years) and healthy controls (n = 27, age range 40-81 years) on a copy task that provoked different strategies by having information freely accessible (facilitating sampling) or introducing a gaze-contingent waiting time (provoking storing). Indeed, patients sampled more often and longer, compared to controls. When sampling became time-consuming, controls reduced sampling and memorised more. Patients also showed reduced and longer sampling in this condition, suggesting an attempt at memorisation. Importantly, however, patients sampled disproportionately more often than controls, whilst accuracy dropped. This finding suggests that amnesia patients sample frequently and do not fully compensate for increased sampling costs by memorising more at once. In other words, Korsakoff amnesia resulted in a heavy reliance on the world as 'external memory'.

Differential aspects of attention predict the depth of visual working memory encoding: Evidence from pupillometry.

What determines how much one encodes into visual working memory? Traditionally, encoding depth is considered to be indexed by spatiotemporal properties of gaze, such as gaze position and dwell time. Although these properties inform about where and how long one looks, they do not necessarily inform about the current arousal state or how strongly attention is deployed to facilitate encoding. Here, we found that two types of pupillary dynamics predict how much information is encoded during a copy task. The task involved encoding a spatial pattern of multiple items for later reproduction. Results showed that smaller baseline pupil sizes preceding and stronger pupil orienting responses during encoding predicted that more information was encoded into visual working memory. Additionally, we show that pupil size reflects not only how much but also how precisely material is encoded. We argue that a smaller pupil size preceding encoding is related to increased exploitation, whereas larger pupil constrictions signal stronger attentional (re)orienting to the to-be-encoded pattern. Our findings support the notion that the depth of visual working memory encoding is the integrative outcome of differential aspects of attention: how alert one is, how much attention one deploys, and how long it is deployed. Together, these factors determine how much information is encoded into visual working memory.

A matter of availability: sharper tuning for memorized than for perceived stimulus features.

Our visual environment is relatively stable over time. An optimized visual system could capitalize on this by devoting less representational resources to objects that are physically present. The vividness of subjective experience, however, suggests that externally available (perceived) information is more strongly represented in neural signals than memorized information. To distinguish between these opposing predictions, we use EEG multivariate pattern analysis to quantify the representational strength of task-relevant features in anticipation of a change-detection task. Perceptual availability was manipulated between experimental blocks by either keeping the stimulus available on the screen during a 2-s delay period (perception) or removing it shortly after its initial presentation (memory). We find that task-relevant (attended) memorized features are more strongly represented than irrelevant (unattended) features. More importantly, we find that task-relevant features evoke significantly weaker representations when they are perceptually available compared with when they are unavailable. These findings demonstrate that, contrary to what subjective experience suggests, vividly perceived stimuli elicit weaker neural representations (in terms of detectable multivariate information) than the same stimuli maintained in visual working memory. We hypothesize that an efficient visual system spends little of its limited resources on the internal representation of information that is externally available anyway.

Bilateral increase in MEG planar gradients prior to saccade onset.

Every time we move our eyes, the retinal locations of objects change. To distinguish the changes caused by eye movements from actual external motion of the objects, the visual system is thought to anticipate the consequences of eye movements (saccades). Single neuron recordings have indeed demonstrated changes in receptive fields before saccade onset. Although some EEG studies with human participants have also demonstrated a pre-saccadic increased potential over the hemisphere that will process a stimulus after a saccade, results have been mixed. Here, we used magnetoencephalography to investigate the timing and lateralization of visually evoked planar gradients before saccade onset. We modelled the gradients from trials with both a saccade and a stimulus as the linear combination of the gradients from two conditions with either only a saccade or only a stimulus. We reasoned that any residual gradients in the condition with both a saccade and a stimulus must be uniquely linked to visually-evoked neural activity before a saccade. We observed a widespread increase in residual planar gradients. Interestingly, this increase was bilateral, showing activity both contralateral and ipsilateral to the stimulus, i.e. over the hemisphere that would process the stimulus after saccade offset. This pattern of results is consistent with predictive pre-saccadic changes involving both the current and the future receptive fields involved in processing an attended object, well before the start of the eye movement. The active, sensorimotor coupling of vision and the oculomotor system may underlie the seamless subjective experience of stable and continuous perception.

Mountains of memory in a sea of uncertainty: Sampling the external world despite useful information in visual working memory.

A large part of research on visual working memory (VWM) has traditionally focused on estimating its maximum capacity. Yet, humans rarely need to load up their VWM maximally during natural behavior, since visual information often remains accessible in the external world. Recent work, using paradigms that take into account the accessibility of information in the outside world, has indeed shown that observers utilize only one or two items in VWM before sampling from the external world again. One straightforward interpretation of this finding is that, in daily behavior, much fewer items are memorized than the typically reported capacity limits. Here, we first investigate whether this lower reliance on VWM when information is externally accessible might instead reflect resampling before VWM is actually depleted. To this aim we devised an online task, in which participants copied a model (six items in a 4x4 grid; always accessible) in an adjacent empty 4x4 grid. A key aspect of our paradigm is that we (unpredictably) interrupted participants just before inspection of the model with a 2-alternative-forced-choice (2-AFC) question, probing their VWM content. Critically, we observed above-chance performance on probes appearing just before model inspection. This finding shows that the external world was resampled, despite VWM still containing relevant information. We then asked whether increasing the cost of sampling causes participants to load up more information in VWM or, alternatively, to squeeze out more information from VWM (at the cost of making more errors). To manipulate the cost of resampling, we made it more difficult (specifically, more time-consuming) to access the model. We show that with increased cost of accessing the model (which lead to fewer, but longer model inspections), participants could place more items correctly immediately after sampling, and they kept attempting to place items for longer after their first error. These findings demonstrate that participants both encoded more information in VWM and made attempts to squeeze out more information from VWM when sampling became more costly. We argue that human observers constantly evaluate how certain they are of their VWM contents, and only use that VWM content of which their certainty exceeds a context-dependent "action threshold". This threshold, in turn, depends on the trade-off between the cost of resampling and the benefits of making an action. We argue that considering the interplay between the available VWM contents and a context-dependent action threshold, is key for reconciling the traditional VWM literature with VWM use in our day-to-day behavior.

Prioritization of emotional faces is not driven by emotional content.

Emotional faces have prioritized access to visual awareness. However, studies concerned with what expressions are prioritized most are inconsistent and the source of prioritization remains elusive. Here we tested the predictive value of spatial frequency-based image-features and emotional content, the sub-part of the image content that signals the emotional expression of the actor in the image as opposed to the image content irrelevant for the emotional expression, for prioritization for awareness. Participants reported which of two faces (displaying a combination of angry, happy, and neutral expressions), that were temporarily suppressed from awareness, was perceived first. Even though the results show that happy expressions were prioritized for awareness, this prioritization was driven by the contrast energy of the images. In fact, emotional content could not predict prioritization at all. Our findings show that the source of prioritization for awareness is not the information carrying the emotional content. We argue that the methods used here, or similar approaches, should become standard practice to break the chain of inconsistent findings regarding emotional superiority effects that have been part of the field for decades.

After-image formation by adaptation to dynamic color gradients.

The eye's retinotopic exposure to an adapter typically produces an after-image. For example, an observer who fixates a red adapter on a gray background will see an illusory cyan after-image after removing the adapter. The after-image's content, like its color or intensity, gives insight into mechanisms responsible for adaptation and processing of a specific feature. To facilitate adaptation, vision scientists traditionally present stable, unchanging adapters for prolonged durations. How adaptation affects perception when features (e.g., color) dynamically change over time is not understood. To investigate adaptation to a dynamically changing feature, participants viewed a colored patch that changed from a color to gray, following either a direct or curved path through the (roughly) equiluminant color plane of CIE LAB space. We varied the speed and curvature of color changes across trials and experiments. Results showed that dynamic adapters produce after-images, vivid enough to be reported by the majority of participants. An after-image consisted of a color complementary to the average of the adapter's colors with a small bias towards more recent rather than initial adapter colors. The modelling of the reported after-image colors further confirmed that adaptation rapidly instigates and gradually dissipates. A second experiment replicated these results and further showed that the probability of observing an after-image diminishes only slightly when the adapter displays transient (stepwise, abrupt) color transitions. We conclude from the results that the visual system can adapt to dynamic colors, to a degree that is robust to the potential interference of transient changes in adapter content.