Just-in-Time Encoding Into Visual Working Memory Is Contingent Upon Constant Availability of External Information.

Humans maintain an intricate balance between storing information in visual working memory (VWM) and just-in-time sampling of the external world, rooted in a trade-off between the cost of maintaining items in VWM versus retrieving information as it is needed. Previous studies have consistently shown that one prerequisite of just-in-time sampling is a high degree of availability of external information, and that introducing a delay before being able to access information led participants to rely less on the external world and more on VWM. However, these studies manipulated availability in such a manner that the cost of sampling was stable and predictable. It is yet unclear whether participants become less reliant on external information when it is more difficult to factor in the cost of sampling that information. In two experiments, participants copied an example layout from the left to the right side of the screen. In Experiment 1, intermittent occlusion of the example layout led participants to attempt to encode more items per inspection than when the layout was constantly available, but this did not consistently result in more correct placements. However, these findings could potentially be explained by inherent differences in how long the example layout could be viewed. Therefore in Experiment 2, the example layout only became available after a gaze-contingent delay, which could be constant or variable. Here, the introduction of any delay led to increased VWM load compared to no delay, although the degree of variability in the delay did not alter behaviour. These results reaffirm that the nature of when we engage VWM is dynamical, and suggest that any disruption to the continuous availability of external information is the main driver of increased VWM usage relative to whether availability is predictable or not.

Gaze data of 4243 participants shows link between leftward and superior attention biases and age.

Healthy individuals typically show more attention to the left than to the right (known as pseudoneglect), and to the upper than to the lower visual field (known as altitudinal pseudoneglect). These biases are thought to reflect asymmetries in neural processes. Attention biases have been used to investigate how these neural asymmetries change with age. However, inconsistent results have been reported regarding the presence and direction of age-related effects on horizontal and vertical attention biases. The observed inconsistencies may be due to insensitive measures and small sample sizes, that usually only feature extreme age groups. We investigated whether spatial attention biases, as indexed by gaze position during free viewing of a single image, are influenced by age. We analysed free-viewing data from 4,243 participants aged 5-65 years and found that attention biases shifted to the right and superior directions with increasing age. These findings are consistent with the idea of developing cerebral asymmetries with age and support the hypothesis of the origin of the leftward bias. Age modulations were found only for the first seven fixations, corresponding to the time window in which an absolute leftward bias in free viewing was previously observed. We interpret this as evidence that the horizontal and vertical attention biases are primarily present when orienting attention to a novel stimulus - and that age modulations of attention orienting are not global modulations of spatial attention. Taken together, our results suggest that attention orienting may be modulated by age and that cortical asymmetries may change with age.

Pupil dilation reveals the intensity of touch.

Touch is important for many aspects of our daily activities. One of the most important tactile characteristics is its perceived intensity. However, quantifying the intensity of perceived tactile stimulation is not always possible using overt responses. Here, we show that pupil responses can objectively index the intensity of tactile stimulation in the absence of overt participant responses. In Experiment 1 (n = 32), we stimulated three reportedly differentially sensitive body locations (finger, forearm, and calf) with a single tap of a tactor while tracking pupil responses. Tactile stimulation resulted in greater pupil dilation than a baseline without stimulation. Furthermore, pupils dilated more for the more sensitive location (finger) than for the less sensitive location (forearm and calf). In Experiment 2 (n = 20) we extended these findings by manipulating the intensity of the stimulation with three different intensities, here a short vibration, always at the little finger. Again, pupils dilated more when being stimulated at higher intensities as compared to lower intensities. In summary, pupils dilated more for more sensitive parts of the body at constant stimulation intensity and for more intense stimulation at constant location. Taken together, the results show that the intensity of perceived tactile stimulation can be objectively measured with pupil responses - and that such responses are a versatile marker for touch research. Our findings may pave the way for previously impossible objective tests of tactile sensitivity, for example in minimally conscious state patients.

Steady-state Visual Evoked Potentials Reveal Dynamic (Re)allocation of Spatial Attention during Maintenance and Utilization of Visual Working Memory.

Visual working memory (VWM) allows storing goal-relevant information to guide future behavior. Prior work suggests that VWM is spatially organized and relies on spatial attention directed toward locations at which memory items were encoded, even if location is task-irrelevant. Importantly, attention often needs to be dynamically redistributed between locations, for example, in preparation for an upcoming probe. Very little is known about how attentional resources are distributed between multiple locations during a VWM task and even less about the dynamic changes governing such attentional shifts over time. This is largely due to the inability to use behavioral outcomes to reveal fast dynamic changes within trials. We here demonstrated that EEG steady-state visual evoked potentials (SSVEPs) successfully track the dynamic allocation of spatial attention during a VWM task. Participants were presented with to-be-memorized gratings and distractors at two distinct locations, tagged with flickering discs. This allowed us to dynamically track attention allocated to memory and distractor items via their coupling with space by quantifying the amplitude and coherence of SSVEP responses in the EEG signal to flickering stimuli at the former memory and distractor locations. SSVEP responses did not differ between memory and distractor locations during early maintenance. However, shortly before probe comparison, we observed a decrease in SSVEP coherence over distractor locations indicative of a reallocation of spatial attentional resources. RTs were shorter when preceded by stronger decreases in SSVEP coherence at distractor locations, likely reflecting attentional shifts from the distractor to the probe or memory location. We demonstrate that SSVEPs can inform about dynamic processes in VWM, even if location does not have to be reported by participants. This finding not only supports the notion of a spatially organized VWM but also reveals that SSVEPs betray a dynamic prioritization process of working memory items and locations over time that is directly predictive of memory performance.

The Intensity of Internal and External Attention Assessed with Pupillometry.

Not only is visual attention shifted to objects in the external world, attention can also be directed to objects in memory. We have recently shown that pupil size indexes how strongly items are attended externally, which was reflected in more precise encoding into visual working memory. Using a retro-cuing paradigm, we here replicated this finding by showing that stronger pupil constrictions during encoding were reflective of the depth of encoding. Importantly, we extend this previous work by showing that pupil size also revealed the intensity of internal attention toward content stored in visual working memory. Specifically, pupil dilation during the prioritization of one among multiple internally stored representations predicted the precision of the prioritized item. Furthermore, the dynamics of the pupillary responses revealed that the intensity of internal and external attention independently determined the precision of internalized visual representations. Our results show that both internal and external attention are not all-or-none processes, but should rather be thought of as continuous resources that can be deployed at varying intensities. The employed pupillometric approach allows to unravel the intricate interplay between internal and external attention and their effects on visual working memory.

Revealing visual working memory operations with pupillometry: Encoding, maintenance, and prioritization.

Pupillary dynamics reflect effects of distinct and important operations of visual working memory: encoding, maintenance, and prioritization. Here, we review how pupil size predicts memory performance and how it provides novel insights into the mechanisms of each operation. Visual information must first be encoded into working memory with sufficient precision. The depth of this encoding process couples to arousal-linked baseline pupil size as well as a pupil constriction response before and after stimulus onset, respectively. Subsequently, the encoded information is maintained over time to ensure it is not lost. Pupil dilation reflects the effortful maintenance of information, wherein storing more items is accompanied by larger dilations. Lastly, the most task-relevant information is prioritized to guide upcoming behavior, which is reflected in yet another dilatory component. Moreover, activated content in memory can be pupillometrically probed directly by tagging visual information with distinct luminance levels. Through this luminance-tagging mechanism, pupil light responses reveal whether dark or bright items receive more attention during encoding and prioritization. Together, conceptualizing pupil responses as a sum of distinct components over time reveals insights into operations of visual working memory. From this viewpoint, pupillometry is a promising avenue to study the most vital operations through which visual working memory works. This article is categorized under: Psychology > Attention Psychology > Memory Psychology > Theory and Methods.

Open-DPSM: An open-source toolkit for modeling pupil size changes to dynamic visual inputs.

Pupil size change is a widely adopted, sensitive indicator for sensory and cognitive processes. However, the interpretation of these changes is complicated by the influence of multiple low-level effects, such as brightness or contrast changes, posing challenges to applying pupillometry outside of extremely controlled settings. Building on and extending previous models, we here introduce Open Dynamic Pupil Size Modeling (Open-DPSM), an open-source toolkit to model pupil size changes to dynamically changing visual inputs using a convolution approach. Open-DPSM incorporates three key steps: (1) Modeling pupillary responses to both luminance and contrast changes; (2) Weighing of the distinct contributions of visual events across the visual field on pupil size change; and (3) Incorporating gaze-contingent visual event extraction and modeling. These steps improve the prediction of pupil size changes beyond the here-evaluated benchmarks. Open-DPSM provides Python functions, as well as a graphical user interface (GUI), enabling the extension of its applications to versatile scenarios and adaptations to individualized needs. By obtaining a predicted pupil trace using video and eye-tracking data, users can mitigate the effects of low-level features by subtracting the predicted trace or assess the efficacy of the low-level feature manipulations a priori by comparing estimated traces across conditions.

Uncovering the (un)attended: Pupil light responses index persistent biases of spatial attention in neglect.

Visuospatial neglect is a frequent and disabling disorder, mostly after stroke, that presents in impaired awareness to stimuli on one side of space. Neglect causes disability and functional dependence, even long after the injury. Improving measurements of the core attentional deficit might hold the key for better understanding of the condition and development of treatment. We present a rapid, pupillometry-based method that assesses automatic biases in (covert) attention, without requiring behavioral responses. We exploit the phenomenon that pupil light responses scale with the degree of covert attention to stimuli, and thereby reveal what draws (no) attention. Participants with left-sided neglect after right-sided lesions following stroke (n = 5), participants with hemianopia/quadrantanopia following stroke (n = 11), and controls (n = 22) were presented with two vertical bars, one of which was white and one of which was black, while fixating the center. We varied which brightness was left and right, respectively across trials. In line with the hypotheses, participants with neglect demonstrated biased pupil light responses to the brightness on the right side. Participants with hemianopia showed similar biases to intact parts of the visual field, whilst controls exhibited no bias. Together, this demonstrates that the pupil light response can reveal not only visual, but also attentional deficits. Strikingly, our pupillometry-based bias estimates were not in agreement with neuropsychological paper-and-pencil assessments conducted on the same day, but were with those administered in an earlier phase post-stroke. Potentially, we pick up on persistent biases in the covert attentional system that participants increasingly compensate for in classical neuropsychological tasks and everyday life. The here proposed method may not only find clinical application, but also advance theory and aid the development of successful restoration therapies by introducing a precise, longitudinally valid, and objective measurement that might not be affected by compensation.

Don't hide the instruction manual: A dynamic trade-off between using internal and external templates during visual search.

Visual search is typically studied by requiring participants to memorize a template initially, for which they subsequently search in a crowded display. Search in daily life, however, often involves templates that remain accessible externally, and may therefore be (re)attended for just-in-time encoding or to refresh internal template representations. Here, we show that participants indeed use external templates during search when given the chance. This behavior was observed during both simple and complex search, scaled with task difficulty, and was associated with improved performance. Furthermore, we show that participants used external sampling not only to offload memory, but also as a means of verifying whether the template was remembered correctly at the end of trials. We conclude that the external world may not only provide the challenge (e.g., distractors), but may dynamically ease search. These results argue for extensions of state-of-the-art models of search, because external sampling seems to be used frequently, in at least two ways and is actually beneficial for task performance. Our findings support a model of visual working memory that emphasizes a resource-efficient trade-off between storing and (re)attending external information.

Differential aspects of attention predict the depth of visual working memory encoding: Evidence from pupillometry.

What determines how much one encodes into visual working memory? Traditionally, encoding depth is considered to be indexed by spatiotemporal properties of gaze, such as gaze position and dwell time. Although these properties inform about where and how long one looks, they do not necessarily inform about the current arousal state or how strongly attention is deployed to facilitate encoding. Here, we found that two types of pupillary dynamics predict how much information is encoded during a copy task. The task involved encoding a spatial pattern of multiple items for later reproduction. Results showed that smaller baseline pupil sizes preceding and stronger pupil orienting responses during encoding predicted that more information was encoded into visual working memory. Additionally, we show that pupil size reflects not only how much but also how precisely material is encoded. We argue that a smaller pupil size preceding encoding is related to increased exploitation, whereas larger pupil constrictions signal stronger attentional (re)orienting to the to-be-encoded pattern. Our findings support the notion that the depth of visual working memory encoding is the integrative outcome of differential aspects of attention: how alert one is, how much attention one deploys, and how long it is deployed. Together, these factors determine how much information is encoded into visual working memory.

The Costs of Paying Overt and Covert Attention Assessed With Pupillometry.

Attention can be shifted with or without an accompanying saccade (i.e., overtly or covertly, respectively). Thus far, it is unknown how cognitively costly these shifts are, yet such quantification is necessary to understand how and when attention is deployed overtly or covertly. In our first experiment (N = 24 adults), we used pupillometry to show that shifting attention overtly is more costly than shifting attention covertly, likely because planning saccades is more complex. We pose that these differential costs will, in part, determine whether attention is shifted overtly or covertly in a given context. A subsequent experiment (N = 24 adults) showed that relatively complex oblique saccades are more costly than relatively simple saccades in horizontal or vertical directions. This provides a possible explanation for the cardinal-direction bias of saccades. The utility of a cost perspective as presented here is vital to furthering our understanding of the multitude of decisions involved in processing and interacting with the external world efficiently.

Seeing the Forrest through the trees: Oculomotor metrics are linked to heart rate.

Fluctuations in a person's arousal accompany mental states such as drowsiness, mental effort, or motivation, and have a profound effect on task performance. Here, we investigated the link between two central instances affected by arousal levels, heart rate and eye movements. In contrast to heart rate, eye movements can be inferred remotely and unobtrusively, and there is evidence that oculomotor metrics (i.e., fixations and saccades) are indicators for aspects of arousal going hand in hand with changes in mental effort, motivation, or task type. Gaze data and heart rate of 14 participants during film viewing were used in Random Forest models, the results of which show that blink rate and duration, and the movement aspect of oculomotor metrics (i.e., velocities and amplitudes) link to heart rate-more so than the amount or duration of fixations and saccades. We discuss that eye movements are not only linked to heart rate, but they may both be similarly influenced by the common underlying arousal system. These findings provide new pathways for the remote measurement of arousal, and its link to psychophysiological features.

Pupillometry as an integrated readout of distinct attentional networks.

The course of pupillary constriction and dilation provides an easy-to-access, inexpensive, and noninvasive readout of brain activity. We propose a new taxonomy of factors affecting the pupil and link these to associated neural underpinnings in an ascending hierarchy. In addition to two well-established low-level factors (light level and focal distance), we suggest two further intermediate-level factors, alerting and orienting, and a higher-level factor, executive functioning. Alerting, orienting, and executive functioning - including their respective underlying neural circuitries - overlap with the three principal attentional networks, making pupil size an integrated readout of distinct states of attention. As a now widespread technique, pupillometry is ready to provide meaningful applications and constitutes a viable part of the psychophysiological toolbox.

The orienting response drives pseudoneglect-Evidence from an objective pupillometric method.

Spatial attention is generally slightly biased leftward ("pseudoneglect"), a phenomenon typically assessed with paper-and-pencil tasks, limited by the requirement of explicit responses and the inability to assess on a subsecond timescale. Pseudoneglect is often stable within experiments, but differs vastly between investigations and is sometimes directed to the left, sometimes to the right. To date, no exhaustive explanation to this phenomenon has been provided. Here, we objectively assessed lateralized attention over time, exploiting the phenomenon that changes in the pupil reflect the allocation of attention in space. Pupil sizes of 41 healthy participants fixating the center were influenced stronger by the differential background luminance of the left side compared to the right side of the visual display. These differences were mainly driven by visual information in the periphery. Differences in pupil sizes positively related with greyscales scores. Time-based analyses within trials show strongest effects early on. With increasing trial number (not time), the initial leftward bias shifted central in pupillometry-based and greyscales measures. This suggests that the orienting response determines the degree of attention bias. In our amplification hypothesis we pose that the quality of pseudoneglect (i.e., the direction) is determined by higher order factors such as hemispheric imbalances, whereas the quantity (i.e., the degree) is determined by the orienting network. This account might explain numerous-previously thought opposing-findings. We here show how pupil light responses reveal pseudoneglect, in a next step, this might allow clinical diagnosis of hemispatial neglect.

Irissometry: Effects of Pupil Size on Iris Elasticity Measured With Video-Based Feature Tracking.

Purpose: It is unclear how the iris deforms during changes in pupil size. Here, we report an application of a multi-feature iris tracking method, which we call irissometry, to investigate how the iris deforms and affects the eye position signal as a function of pupil size. Methods: To evoke pupillary responses, we repeatedly presented visual and auditory stimuli to healthy participants while we additionally recorded their right eye with a macro lens-equipped camera. We tracked changes in iris surface structure between the pupil and sclera border (limbus) by calculating local densities (distance between feature points) across evenly spaced annular iris regions. Results: The time analysis of densities showed that the inner regions of the iris stretched more strongly as compared with the outer regions of the iris during pupil constrictions. The pattern of iris densities across eccentricities and pupil size showed highly similar patterns across participants, highlighting the robustness of this elastic property. Importantly, iris-based eye position detection led to more stable signals than pupil-based detection. Conclusions: The iris regions near the pupil appear to be more elastic than the outer regions near the sclera. This elastic property explains the instability of the pupil border and the related position errors induced by eye movement and pupil size in pupil-based eye-tracking. Tracking features in the iris produce more robust eye position signals. We expect that irissometry may pave the way to novel eye trackers and diagnostic tools in ophthalmology.

Increasing pupil size is associated with improved detection performance in the periphery.

Visible light enters our body via the pupil. By changing its size, the pupil shapes visual input. Small apertures increase the resolution of high spatial frequencies, thus allowing discrimination of fine details. Large apertures, in contrast, provide a better signal-to-noise ratio, because more light can enter the eye. This should lead to better detection performance of peripheral stimuli. Experiment 1 shows that the effect can reliably be demonstrated even in a less controlled online setting. In Experiment 2, pupil size was measured in a laboratory using an eye tracker. The findings replicate findings showing that large pupils provide an advantage for peripheral detection of faint stimuli. Moreover, not only pupil size during information intake in the current trial n, but also its interaction with pupil size preceding information intake, i.e., in trial n-1, predicted performance. This suggests that in addition to absolute pupil size, the extent of pupillary change provides a mechanism to modulate perceptual functions. The results are discussed in terms of low-level sensory as well as higher-level arousal-driven changes in stimulus processing.

Direct voluntary control of pupil constriction and dilation: Exploratory evidence from pupillometry, optometry, skin conductance, perception, and functional MRI.

We present a single case who can change pupil size on command with dilation of pupil diameter of around 0.8 mm, and constriction of around 2.4 mm. Using modern pupillometric and optometric techniques in combination with measuring electrodermal activity, various indirect mechanisms possibly mediating this phenomenon were tested: accommodation, brightness, increases in arousal by increased mental effort. None of these behavioral tests could support an indirect strategy as the mode of action, although it seems plausible that the case could have learned to gain control over the pupillary response by decoupling pupil size changes from accommodation and vergence in the near triad: Even at maximal accommodation, the case voluntarily constricted his pupil without changing vergence and could improve visual acuity by >6 diopters. Using task-based functional magnetic resonance imaging we found involvement of brain regions generating and mediating volitional impulses. Changes of the left pupil size were associated with increased activation of parts of the left dorsolateral prefrontal cortex, adjacent premotor areas, and supplementary motor area. It still remains open where these neural signals enter the final pathway, either innervating the pupil's dilator directly, or more indirectly by inhibiting the parasympathetically innervated antagonistic sphincter, and vice versa for constriction. To conclude, so far none of potential - conscious or unconscious - indirect strategies, may it be accommodative or vergence efforts or mental efforts and imaginations, could be observed or inferred to be fully responsible, suggesting direct voluntary control of pupil size in the present case.

Decoding binary decisions under differential target probabilities from pupil dilation: A random forest approach.

Although our pupils slightly dilate when we look at an intended target, they do not when we look at irrelevant distractors. This finding suggests that it may be possible to decode the intention of an observer, understood as the outcome of implicit covert binary decisions, from the pupillary dynamics over time. However, few previous works have investigated the feasibility of this approach and the few that did, did not control for possible confounds such as motor-execution, changes in brightness, or target and distractor probability. We report on our efforts to decode intentions from pupil dilation obtained under strict experimental control on a single trial basis using a machine learning approach. The basis for our analyses are data of 69 participants who looked at letters that needed to be selected with stimulus probabilities that varied systematically in a blockwise manner (n = 19,417 trials). We confirm earlier findings that pupil dilation is indicative of intentions and show that these can be decoded with a classification performance of up to 76% area under the curve for receiver operating characteristic curves if targets are rarer than distractors. To better understand which characteristics of the pupillary signal are most informative, we finally compare relative feature importances. The first derivative of pupil size changes was found to be most relevant, allowing us to decode intention within only about 800 ms of trial onset. Taken together, our results provide credible insights into the potential of decoding intentions from pupil dilation and may soon form the basis for new applications in visual search, gaze-based interaction, or human-robot interaction.

Decision Making and Oddball Effects on Pupil Size: Evidence for a Sequential Process.

In our physical environment as well as in many experimental paradigms, we need to decide whether an occurring stimulus is relevant to us or not; further, stimuli have uneven probabilities to emerge. Both, decision making and the difference between rare and frequent stimuli (oddball effect) are described to affect pupil dilation. Surprisingly though, conjoint systematic pupillometric investigations into both factors are still rare. In two experiments, both factors as well as their interplay were investigated. Participants completed a sequential letter matching task. In this task, stimulus probability and letter matching (decision making) were manipulated independently. As dependent variables, pupil dilation and reaction time were assessed. Results suggest a clearly larger pupil dilation for target than for distractor letters, even when targets were frequent and distractors rare. When considering the data structure best, no main effect of stimulus probability was found, instead, oddball effects only emerged when stimuli were goal-relevant to participants. The results are discussed in the light of common theoretical concepts of decision making and stimulus probability. Finally, relating theories of each factor, we propose an integrated framework for effects of decision making and stimulus features on pupil dilation. We assume a sequential mechanism during which incoming stimuli are decided upon regarding their goal relevance and, about 200 ms later, relevant stimuli are appraised regarding their value.

Get Ready for Being Chauffeured : Passenger's Preferences and Trust While Being Driven by Human and Automation.

OBJECTIVE: We investigated passenger's trust and preferences using subjective, qualitative, and psychophysiological measures while being driven either by human or automation in a field study and a driving simulator experiment. BACKGROUND: The passenger's perspective has largely been neglected in autonomous driving research, although the change of roles from an active driver to a passive passenger is incontrovertible. Investigations of passenger's appraisals on self-driving vehicles often seem convoluted with active manual driving experiences instead of comparisons with being driven by humans. METHOD: We conducted an exploratory field study using an autonomous research vehicle (N = 11) and a follow-up experimental driving simulation (N = 24). Participants were driven on the same course by a human and an autonomous agent sitting on a passenger seat. Skin conductance, trust, and qualitative characteristics of the perceived driving situation were assessed. In addition, the effect of driving style (defensive vs. sporty) was evaluated in the simulator. RESULTS: Both investigations revealed a close relation between subjective trust ratings and skin conductance, with increased trust and by trend reduced arousal for human compared with automation in control. Even though driving behavior was equivalent in the simulator when being driven by human and automation, passengers most preferred and trusted the human-defensive driver. CONCLUSION: Individual preferences for driving style and human or autonomous vehicle control influence trust and subjective driving characterizations. APPLICATION: The findings are applicable in human-automation research, reminding to not neglect subjective attributions and psychophysiological reactions as a result of ascribed control duties in relation to specific execution characteristics.