Emphasizing speed or accuracy in an eye-tracking version of the Trail-Making-Test: Towards experimental diagnostics for decomposing executive functions.

The Trail-Making-Test (TMT) is one of the most widely used neuropsychological tests for assessing executive functions, the brain functions underlying cognitively controlled thought and action. Obtaining a number of test scores at once, the TMT allows to characterize an assortment of executive functions efficiently. Critically, however, as most test scores are derived from test completion times, the scores only provide a summary measure of various cognitive control processes. To address this problem, we extended the TMT in two ways. First, using a computerized eye-tracking version of the TMT, we added specific eye movement measures that deliver a richer set of data with a higher degree of cognitive process specificity. Second, we included an experimental manipulation of a fundamental executive function, namely participants' ability to emphasize speed or accuracy in task performance. Our study of healthy participants showed that eye movement measures differed between TMT conditions that are usually compared to assess the cognitive control process of alternating between task sets for action control. This demonstrates that eye movement measures are indeed sensitive to executive functions implicated in the TMT. Crucially, comparing performance under cognitive control sets of speed vs. accuracy emphasis revealed which test scores primarily varied due to this manipulation (e.g., trial duration, number of fixations), and which were still more sensitive to other differences between individuals (e.g., fixation duration, saccade amplitude). This provided an experimental construct validation of the test scores by distinguishing scores primarily reflecting the executive function of emphasizing speed vs. accuracy and those independent from it. In sum, both the inclusion of eye movement measures and of the experimental manipulation of executive functions in the TMT enabled a more specific interpretation of the TMT in terms of cognitive functions and mechanisms, which offers more precise diagnoses in clinical applications and basic research.

Working memory and active sampling of the environment: Medial temporal contributions.

Working memory (WM) refers to the ability to maintain and actively process information-either derived from perception or long-term memory (LTM)-for intelligent thought and action. This chapter focuses on the contributions of the temporal lobe, particularly medial temporal lobe (MTL) to WM. First, neuropsychological evidence for the involvement of MTL in WM maintenance is reviewed, arguing for a crucial role in the case of retaining complex relational bindings between memorized features. Next, MTL contributions at the level of neural mechanisms are covered-with a focus on WM encoding and maintenance, including interactions with ventral temporal cortex. Among WM use processes, we focus on active sampling of environmental information, a key input source to capacity-limited WM. MTL contributions to the bidirectional relationship between active sampling and memory are highlighted-WM control of active sampling and sampling as a way of selecting input to WM. Memory-based sampling studies relying on scene and object inspection, visual-based exploration behavior (e.g., vicarious behavior), and memory-guided visual search are reviewed. The conclusion is that MTL serves an important function in the selection of information from perception and transfer from LTM to capacity-limited WM.

Action Selection and Execution in Everyday Activities: A Cognitive Robotics and Situation Model Perspective.

We examine the mechanisms required to handle everyday activities from the standpoint of cognitive robotics, distinguishing activities on the basis of complexity and transparency. Task complexity (simple or complex) reflects the intrinsic nature of a task, while task transparency (easy or difficult) reflects an agent's ability to identify a solution strategy in a given task. We show how the CRAM cognitive architecture allows a robot to carry out simple and complex activities such as laying a table for a meal and loading a dishwasher afterward. It achieves this by using generalized action plans that exploit reasoning with modular, composable knowledge chunks representing general knowledge to transform underdetermined everyday action requests into motion plans that successfully accomplish the required task. Noting that CRAM does not yet have the ability to deal with difficult activities, we leverage insights from the situation model perspective on the cognitive mechanisms underlying flexible context-sensitive behavior with a view to extending CRAM to overcome this deficit.

Differentiating Progressive Supranuclear Palsy and Parkinson's Disease With Head-Mounted Displays.

Background: Progressive supranuclear palsy (PSP) is a neurodegenerative disorder that, especially in the early stages of the disease, is clinically difficult to distinguish from Parkinson's disease (PD). Objective: This study aimed at assessing the use of eye-tracking in head-mounted displays (HMDs) for differentiating PSP and PD. Methods: Saccadic eye movements of 13 patients with PSP, 15 patients with PD, and a group of 16 healthy controls (HCs) were measured. To improve applicability in an inpatient setting and standardize the diagnosis, all the tests were conducted in a HMD. In addition, patients underwent atlas-based volumetric analysis of various brain regions based on high-resolution MRI. Results: Patients with PSP displayed unique abnormalities in vertical saccade velocity and saccade gain, while horizontal saccades were less affected. A novel diagnostic index was derived, multiplying the ratios of vertical to horizontal gain and velocity, allowing segregation of PSP from PD with high sensitivity (10/13, 77%) and specificity (14/15, 93%). As expected, patients with PSP as compared with patients with PD showed regional atrophy in midbrain volume, the midbrain plane, and the midbrain tegmentum plane. In addition, we found for the first time that oculomotor measures (vertical gain, velocity, and the diagnostic index) were correlated significantly to midbrain volume in the PSP group. Conclusions: Assessing eye movements in a HMD provides an easy to apply and highly standardized tool to differentiate PSP of patients from PD and HCs, which will aid in the diagnosis of PSP.

Oculomotor capture by search-irrelevant features in visual working memory: on the crucial role of target-distractor similarity.

When searching for varying targets in the environment, a target template has to be maintained in visual working memory (VWM). Recently, we showed that search-irrelevant features of a VWM template bias attention in an object-based manner, so that objects sharing such features with a VWM template capture the eyes involuntarily. Here, we investigated whether target-distractor similarity modulates capture strength. Participants saccaded to a target accompanied by a distractor. A single feature (e.g., shape) defined the target in each trial indicated by a cue, and the cue also varied in one irrelevant feature (e.g., color). The distractor matched the cue's irrelevant feature in half of the trials. Nine experiments showed that target-distractor similarity consistently influenced the degree of oculomotor capture. High target-distractor dissimilarity in the search-relevant feature reduced capture by the irrelevant feature (Experiments 1, 3, 6, 7). However, capture was reduced by high target-distractor similarity in the search-irrelevant feature (Experiments 1, 4, 5, 8). Strong oculomotor capture was observed if target-distractor similarity was reasonably low in the relevant and high in the irrelevant feature, irrespective of whether color or shape were relevant (Experiments 2 and 5). These findings argue for involuntary and object-based, top-down control by VWM templates, whereas its manifestation in oculomotor capture depends crucially on target-distractor similarity in relevant and irrelevant feature dimensions of the search object.

Task-Irrelevant Features in Visual Working Memory Influence Covert Attention: Evidence from a Partial Report Task.

Selecting a target based on a representation in visual working memory (VWM) affords biasing covert attention towards objects with memory-matching features. Recently, we showed that even task-irrelevant features of a VWM template bias attention. Specifically, when participants had to saccade to a cued shape, distractors sharing the cue's search-irrelevant color captured the eyes. While a saccade always aims at one target location, multiple locations can be attended covertly. Here, we investigated whether covert attention is captured similarly as the eyes. In our partial report task, each trial started with a shape-defined search cue, followed by a fixation cross. Next, two colored shapes, each including a letter, appeared left and right from fixation, followed by masks. The letter inside that shape matching the preceding cue had to be reported. In Experiment 1, either target, distractor, both, or no object matched the cue's irrelevant color. Target-letter reports were most frequent in target-match trials and least frequent in distractor-match trials. Irrelevant cue and target color never matched in Experiment 2. Still, participants reported the distractor more often to the target's disadvantage, when cue and distractor color matched. Thus, irrelevant features of a VWM template can influence covert attention in an involuntarily object-based manner when searching for trial-wise varying targets.

Neuropsychological assessment of visual selective attention and processing capacity with head-mounted displays.

OBJECTIVE: Neuropsychological patients often suffer from impairments in visual selective attention and processing capacity components. Their assessment demands a high standardization of testing conditions, which is difficult to achieve across institutions. Head-mounted displays (HMDs) provide a solution. These virtual reality devices cover the entire visual field in a shielded way and thus keep visual stimulation constant. For neuropsychological assessment with HMDs, sufficient reliability is required. We have previously demonstrated that an early developer version of an HMD can be used to reliably measure components of visual processing capacity. However, it is unclear whether this also holds for the assessment of components of visual selective attention. Moreover, it has yet to be established whether now commercially available HMDs are capable of reliable neuropsychological assessment. METHOD: We assessed the test-retest reliabilities of several components of visual selective attention and processing capacity of healthy subjects with the commercially available HTC Vive. Using an assessment procedure (combiTVA) derived from the theory of visual attention (TVA; Bundesen, 1990), we measured attentional selectivity, lateral bias, processing speed, visual working memory capacity, and the threshold of conscious perception. We compared the reliabilities of these components measured with the HTC Vive with those of a cathode ray tube (CRT) screen, the gold standard of visual presentation in the laboratory. RESULTS: Both devices provided comparable reliabilities. CONCLUSIONS: Thus, HMDs fulfill the requirement to replace standard screens. With their inherent visual standardization and portability, they offer unprecedented opportunities for neuropsychological assessment, such as computerized bedside testing and comparisons of test values across institutions. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Feature prediction across eye movements is location specific and based on retinotopic coordinates.

With each saccadic eye movement, internal object representations change their retinal position and spatial resolution. Recently, we suggested that the visual system deals with these saccade-induced changes by predicting visual features across saccades based on transsaccadic associations of peripheral and foveal input (Herwig & Schneider, 2014). Here we tested the specificity of feature prediction by asking (a) whether it is spatially restricted to the previous learning location or the saccade target location, and (b) whether it is based on retinotopic (eye-centered) or spatiotopic (world-centered) coordinates. In a preceding acquisition phase, objects systematically changed their spatial frequency during saccades. In the following test phases of two experiments, participants had to judge the frequency of briefly presented peripheral objects. These objects were presented either at the previous learning location or at new locations and were either the target of a saccadic eye movement or not (Experiment 1). Moreover, objects were presented either in the same or different retinotopic and spatiotopic coordinates (Experiment 2). Spatial frequency perception was biased toward previously associated foveal input indicating transsaccadic learning and feature prediction. Importantly, while this pattern was not bound to the saccade target location, it was seen only at the previous learning location in retinotopic coordinates, suggesting that feature prediction probably affects low- or mid-level perception.

Attentional competition across saccadic eye movements.

Human behavior is guided by visual object recognition. For being recognized, objects compete for limited attentional processing resources. The more objects compete, the lower is performance in recognizing each individual object. Here, we ask whether this competition is confined to eye fixations, periods of relatively stable gaze, or whether it extends from one fixation to the next, across saccadic eye movements. Participants made saccades to a peripheral saccade target. After the saccade, a letter was briefly presented within the saccade target and terminated by a mask. Object recognition of the letter was assessed as participants' report. Critically, either no, two, or four additional non-target objects appeared before the saccade. In Experiment 1, presaccadic non-targets were task-irrelevant and had no effects on postsaccadic object recognition. In Experiment 2, presaccadic non-targets were task-relevant and, here, postsaccadic object recognition deteriorated with increasing number of presaccadic non-targets. As suggested by Experiment 3 and a mathematical model, this effect was due to a slowing down but also a delayed start of visual processing after the saccade. Together, our findings show that objects compete for recognition across saccades, but only if they are task-relevant. This reveals an attentional mechanism of task-driven object recognition that is interlaced with active saccade-mediated vision.

Ultrahigh temporal resolution of visual presentation using gaming monitors and G-Sync.

Vision unfolds as an intricate pattern of information processing over time. Studying vision and visual cognition therefore requires precise manipulations of the timing of visual stimulus presentation. Although standard computer display technologies offer great accuracy and precision of visual presentation, their temporal resolution is limited. This limitation stems from the fact that the presentation of rendered stimuli has to wait until the next refresh of the computer screen. We present a novel method for presenting visual stimuli with ultrahigh temporal resolution (<1 ms) on newly available gaming monitors. The method capitalizes on the G-Sync technology, which allows for presenting stimuli as soon as they have been rendered by the computer's graphics card, without having to wait for the next screen refresh. We provide software implementations in the three programming languages C++, Python (using PsychoPy2), and Matlab (using Psychtoolbox3). For all implementations, we confirmed the ultrahigh temporal resolution of visual presentation with external measurements by using a photodiode. Moreover, a psychophysical experiment revealed that the ultrahigh temporal resolution impacts on human visual performance. Specifically, observers' object recognition performance improved over fine-grained increases of object presentation duration in a theoretically predicted way. Taken together, the present study shows that the G-Sync-based presentation method enables researchers to investigate visual processes whose data patterns were concealed by the low temporal resolution of previous technologies. Therefore, this new presentation method may be a valuable tool for experimental psychologists and neuroscientists studying vision and its temporal characteristics.

Involuntary top-down control by search-irrelevant features: Visual working memory biases attention in an object-based manner.

Many everyday tasks involve successive visual-search episodes with changing targets. Converging evidence suggests that these targets are retained in visual working memory (VWM) and bias attention from there. It is unknown whether all or only search-relevant features of a VWM template bias attention during search. Bias signals might be configured exclusively to task-relevant features so that only search-relevant features bias attention. Alternatively, VWM might maintain objects in the form of bound features. Then, all template features will bias attention in an object-based manner, so that biasing effects are ranked by feature relevance. Here, we investigated whether search-irrelevant VWM template features bias attention. Participants had to saccade to a target opposite a distractor. A colored cue depicted the target prior to each search trial. The target was predefined only by its identity, while its color was irrelevant. When target and cue matched not only in identity (search-relevant) but also in color (search-irrelevant), saccades went more often and faster directly to the target than without any color match (Experiment 1). When introducing a cue-distractor color match (Experiment 2), direct target saccades were most likely when target and cue matched in the search-irrelevant color and least likely in case of a cue-distractor color match. When cue and target were never colored the same (Experiment 3), cue-colored distractors still captured the eyes more often than different-colored distractors despite color being search-irrelevant. As participants were informed about the misleading color, the result argues against a strategical and voluntary usage of color. Instead, search-irrelevant features biased attention obligatorily arguing for involuntary top-down control by object-based VWM templates.

Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities.

Neuropsychological assessment of human visual processing capabilities strongly depends on visual testing conditions including room lighting, stimuli, and viewing-distance. This limits standardization, threatens reliability, and prevents the assessment of core visual functions such as visual processing speed. Increasingly available virtual reality devices allow to address these problems. One such device is the portable, light-weight, and easy-to-use Oculus Rift. It is head-mounted and covers the entire visual field, thereby shielding and standardizing the visual stimulation. A fundamental prerequisite to use Oculus Rift for neuropsychological assessment is sufficient test-retest reliability. Here, we compare the test-retest reliabilities of Bundesen's visual processing components (visual processing speed, threshold of conscious perception, capacity of visual working memory) as measured with Oculus Rift and a standard CRT computer screen. Our results show that Oculus Rift allows to measure the processing components as reliably as the standard CRT. This means that Oculus Rift is applicable for standardized and reliable assessment and diagnosis of elementary cognitive functions in laboratory and clinical settings. Oculus Rift thus provides the opportunity to compare visual processing components between individuals and institutions and to establish statistical norm distributions.

Episodic Short-Term Recognition Requires Encoding into Visual Working Memory: Evidence from Probe Recognition after Letter Report.

Human vision is organized in discrete processing episodes (e.g., eye fixations or task-steps). Object information must be transmitted across episodes to enable episodic short-term recognition: recognizing whether a current object has been seen in a previous episode. We ask whether episodic short-term recognition presupposes that objects have been encoded into capacity-limited visual working memory (VWM), which retains visual information for report. Alternatively, it could rely on the activation of visual features or categories that occurs before encoding into VWM. We assessed the dependence of episodic short-term recognition on VWM by a new paradigm combining letter report and probe recognition. Participants viewed displays of 10 letters and reported as many as possible after a retention interval (whole report). Next, participants viewed a probe letter and indicated whether it had been one of the 10 letters (probe recognition). In Experiment 1, probe recognition was more accurate for letters that had been encoded into VWM (reported letters) compared with non-encoded letters (non-reported letters). Interestingly, those letters that participants reported in their whole report had been near to one another within the letter displays. This suggests that the encoding into VWM proceeded in a spatially clustered manner. In Experiment 2, participants reported only one of 10 letters (partial report) and probes either referred to this letter, to letters that had been near to it, or far from it. Probe recognition was more accurate for near than for far letters, although none of these letters had to be reported. These findings indicate that episodic short-term recognition is constrained to a small number of simultaneously presented objects that have been encoded into VWM.

Breaking object correspondence across saccades impairs object recognition: The role of color and luminance.

Rapid saccadic eye movements bring the foveal region of the eye's retina onto objects for high-acuity vision. Saccades change the location and resolution of objects' retinal images. To perceive objects as visually stable across saccades, correspondence between the objects before and after the saccade must be established. We have previously shown that breaking object correspondence across the saccade causes a decrement in object recognition (Poth, Herwig, & Schneider, 2015). Color and luminance can establish object correspondence, but it is unknown how these surface features contribute to transsaccadic visual processing. Here, we investigated whether changing the surface features color-and-luminance and color alone across saccades impairs postsaccadic object recognition. Participants made saccades to peripheral objects, which either maintained or changed their surface features across the saccade. After the saccade, participants briefly viewed a letter within the saccade target object (terminated by a pattern mask). Postsaccadic object recognition was assessed as participants' accuracy in reporting the letter. Experiment A used the colors green and red with different luminances as surface features, Experiment B blue and yellow with approximately the same luminances. Changing the surface features across the saccade deteriorated postsaccadic object recognition in both experiments. These findings reveal a link between object recognition and object correspondence relying on the surface features colors and luminance, which is currently not addressed in theories of transsaccadic perception. We interpret the findings within a recent theory ascribing this link to visual attention (Schneider, 2013).

Anticipatory eye movements in sensorimotor actions: on the role of guiding fixations during learning.

During object-based sensorimotor tasks, humans look at target locations for subsequent hand actions. These anticipatory eye movements or guiding fixations seem to be necessary for a successful performance. By practicing such a sensorimotor task, humans become faster and perform fewer guiding fixations (Foerster and Schneider, In Prep; Foerster et al. in J Vis 11(7):9:1-16, 2011). We aimed at clarifying whether this decrease in guiding fixations is the cause or effect of faster task completion time. Participants may learn to use less visual input (fewer fixations) allowing shorter completion times. Alternatively, participants may speed up their hand movements (e.g., more efficient motor control) leaving less time for visual intake. The latter would imply that the number of fixations is directly connected to task speed. We investigated the relationship between the number of fixations and task speed in a computerized version of the number connection task (Foerster and Schneider in Ann N Y Acad Sci 2015. doi: 10.1111/nyas.12729 ). Eye movements were recorded while participants clicked in ascending order on nine numbered circles. In 90 learning trials, they clicked the sequence with a constant spatial configuration as fast as possible. In the subsequent experimental phase, they should perform 30 trials again under high-speed instruction and 30 trials under slow-speed instruction. During slow-speed instruction, fixation rates were lower with longer fixation durations and more fixations were performed than during high-speed instruction. The results suggest that the number of fixations depends on both the need for visual intake and task completion time. It seems that the decrease in anticipatory eye movements through sensorimotor learning is at the same time a result and a cause of faster task performance.

A "blanking effect" for surface features: Transsaccadic spatial-frequency discrimination is improved by postsaccadic blanking.

Although saccadic eye movements occur frequently­about three or four times a second­humans are astonishingly blind to transsaccadic changes. Locational displacements of the saccade target of up to 2 deg of visual angle, and even large changes of a visual scene, can go unnoticed. For a long time, this insensitivity was ascribed to deficits in transsaccadic memory: Only a coarse, (spatially) imprecise representation would be retained across a saccade. This assumption was contradicted by Deubel's and Schneider's (Behavioral and Brain Sciences 17:259-260, 1994) striking finding that locational discrimination performance across a saccade is greatly improved by inserting a short postsaccadic blank. Surprisingly, the question of whether blanking effects occur also for other forms of transsaccadic changes (i.e., surface-feature changes) has been widely ignored. We tested this question by means of a transsaccadic change in spatial frequency. Postsaccadic blanking facilitated spatial-frequency discrimination, but to a smaller amount than the usual blanking effects obtained with locational displacements. This finding bears important implications for models of visual stability and transsaccadic memory.

When circles become triangular: how transsaccadic predictions shape the perception of shape.

Human vision is characterized by a consistent pattern of saccadic eye movements. With each saccade, internal object representations change their retinal position and spatial resolution. This raises the question as to how peripheral perception is affected by imminent saccadic eye movements. Here, we suggest that saccades are accompanied by a prediction of their perceptual consequences (i.e., the foveation of the target object). Accordingly, peripheral perception should be biased toward previously associated foveal input. In this study, we first exposed participants to an altered visual stimulation where one object systematically changed its shape during saccades. Subsequently, participants had to judge the shape of briefly presented peripheral saccade targets. The results showed that targets were perceived as less curved for objects that previously changed from more circular in the periphery to more triangular in the fovea. Similarly, shapes were perceived as more curved for objects that previously changed from triangular to circular. Thus, peripheral perception seems to depend not solely on the current input but also on memorized experiences, enabling predictions about the perceptual consequences of saccadic eye movements.

Expectation violations in sensorimotor sequences: shifting from LTM-based attentional selection to visual search.

Long-term memory (LTM) delivers important control signals for attentional selection. LTM expectations have an important role in guiding the task-driven sequence of covert attention and gaze shifts, especially in well-practiced multistep sensorimotor actions. What happens when LTM expectations are disconfirmed? Does a sensory-based visual-search mode of attentional selection replace the LTM-based mode? What happens when prior LTM expectations become valid again? We investigated these questions in a computerized version of the number-connection test. Participants clicked on spatially distributed numbered shapes in ascending order while gaze was recorded. Sixty trials were performed with a constant spatial arrangement. In 20 consecutive trials, either numbers, shapes, both, or no features switched position. In 20 reversion trials, participants worked on the original arrangement. Only the sequence-affecting number switches elicited slower clicking, visual search-like scanning, and lower eye-hand synchrony. The effects were neither limited to the exchanged numbers nor to the corresponding actions. Thus, expectation violations in a well-learned sensorimotor sequence cause a regression from LTM-based attentional selection to visual search beyond deviant-related actions and locations. Effects lasted for several trials and reappeared during reversion.