Perceptual learning across saccades: Feature but not location specific.

Perceptual learning is the ability to enhance perception through practice. The hallmark of perceptual learning is its specificity for the trained location and stimulus features, such as orientation. For example, training in discriminating a grating's orientation improves performance only at the trained location but not in other untrained locations. Perceptual learning has mostly been studied using stimuli presented briefly while observers maintained gaze at one location. However, in everyday life, stimuli are actively explored through eye movements, which results in successive projections of the same stimulus at different retinal locations. Here, we studied perceptual learning of orientation discrimination across saccades. Observers were trained to saccade to a peripheral grating and to discriminate its orientation change that occurred during the saccade. The results showed that training led to transsaccadic perceptual learning (TPL) and performance improvements which did not generalize to an untrained orientation. Remarkably, however, for the trained orientation, we found a complete transfer of TPL to the untrained location in the opposite hemifield suggesting high flexibility of reference frame encoding in TPL. Three control experiments in which participants were trained without saccades did not show such transfer, confirming that the location transfer was contingent upon eye movements. Moreover, performance at the trained location, but not at the untrained location, was also improved in an untrained fixation task. Our results suggest that TPL has both, a location-specific component that occurs before the eye movement and a saccade-related component that involves location generalization.

Objects guide human gaze behavior in dynamic real-world scenes.

The complexity of natural scenes makes it challenging to experimentally study the mechanisms behind human gaze behavior when viewing dynamic environments. Historically, eye movements were believed to be driven primarily by space-based attention towards locations with salient features. Increasing evidence suggests, however, that visual attention does not select locations with high saliency but operates on attentional units given by the objects in the scene. We present a new computational framework to investigate the importance of objects for attentional guidance. This framework is designed to simulate realistic scanpaths for dynamic real-world scenes, including saccade timing and smooth pursuit behavior. Individual model components are based on psychophysically uncovered mechanisms of visual attention and saccadic decision-making. All mechanisms are implemented in a modular fashion with a small number of well-interpretable parameters. To systematically analyze the importance of objects in guiding gaze behavior, we implemented five different models within this framework: two purely spatial models, where one is based on low-level saliency and one on high-level saliency, two object-based models, with one incorporating low-level saliency for each object and the other one not using any saliency information, and a mixed model with object-based attention and selection but space-based inhibition of return. We optimized each model's parameters to reproduce the saccade amplitude and fixation duration distributions of human scanpaths using evolutionary algorithms. We compared model performance with respect to spatial and temporal fixation behavior, including the proportion of fixations exploring the background, as well as detecting, inspecting, and returning to objects. A model with object-based attention and inhibition, which uses saliency information to prioritize between objects for saccadic selection, leads to scanpath statistics with the highest similarity to the human data. This demonstrates that scanpath models benefit from object-based attention and selection, suggesting that object-level attentional units play an important role in guiding attentional processing.

Oculomotor freezing indicates conscious detection free of decision bias.

The appearance of a salient stimulus rapidly and automatically inhibits saccadic eye movements. Curiously, this "oculomotor freezing" response is triggered only by stimuli that the observer reports seeing. It remains unknown, however, whether oculomotor freezing is linked to the observer's sensory experience or their decision that a stimulus was present. To dissociate between these possibilities, we manipulated decision criterion via monetary payoffs and stimulus probability in a detection task. These manipulations greatly shifted observers' decision criteria but did not affect the degree to which microsaccades were inhibited by stimulus presence. Moreover, the link between oculomotor freezing and explicit reports of stimulus presence was stronger when the criterion was conservative rather than liberal. We conclude that the sensory threshold for oculomotor freezing is independent of decision bias. Provided that conscious experience is also unaffected by such bias, oculomotor freezing is an implicit indicator of sensory awareness.NEW & NOTEWORTHY Sometimes a visual stimulus reaches awareness, and sometimes it does not. To understand why, we need objective, bias-free measures of awareness. We discovered that a reflexive freezing of small eye movements indicates when an observer detects a stimulus. Furthermore, when we biased observers' decisions to report seeing the stimulus, the oculomotor response was unaltered. This suggests that the threshold for conscious perception is independent of the decision criterion and is revealed by oculomotor freezing.

Oculomotor corollary discharge signaling is related to repetitive behavior in children with autism spectrum disorder.

Corollary discharge (CD) signals are "copies" of motor signals sent to sensory regions that allow animals to adjust sensory consequences of self-generated actions. Autism spectrum disorder (ASD) is characterized by sensory and motor deficits, which may be underpinned by altered CD signaling. We evaluated oculomotor CD using the blanking task, which measures the influence of saccades on visual perception, in 30 children with ASD and 35 typically developing (TD) children. Participants were instructed to make a saccade to a visual target. Upon saccade initiation, the presaccadic target disappeared and reappeared to the left or right of the original position. Participants indicated the direction of the jump. With intact CD, participants can make accurate perceptual judgements. Otherwise, participants may use saccade landing site as a proxy of the presaccadic target and use it to inform perception. We used multilevel modeling to examine the influence of saccade landing site on trans-saccadic perceptual judgements. We found that, compared with TD participants, children with ASD were more sensitive to target displacement and less reliant on saccade landing site when spatial uncertainty of the post-saccadic target was high. This pattern was driven by ASD participants with less severe restricted and repetitive behaviors. These results suggest a relationship between altered CD signaling and core ASD symptoms.

Moving fast and seeing slow? The visual consequences of vigorous movement.

In active agents, sensory and motor processes form an inevitable bond. This wedding is particularly striking for saccadic eye movements - the prime target of Shadmehr and Ahmed's thesis - which impose frequent changes on the retinal image. Changes in movement vigor (latency and speed), therefore, will need to be accompanied by changes in visual and attentional processes. We argue that the mechanisms that control movement vigor may also enable vision to attune to changes in movement kinematics.

Structural Thalamofrontal Hypoconnectivity Is Related to Oculomotor Corollary Discharge Dysfunction in Schizophrenia.

By predicting sensory consequences of actions, humans can distinguish self-generated sensory inputs from those that are elicited externally. This is one mechanism by which we achieve a subjective sense of agency over our actions. Corollary discharge (CD) signals-"copies" of motor signals sent to sensory areas-permit such predictions, and CD abnormalities are a hypothesized mechanism for the agency disruptions in schizophrenia that characterize a subset of symptoms. Indeed, behavioral evidence of altered CD, including in the oculomotor system, has been observed in schizophrenia patients. A pathway projecting from the superior colliculus to the frontal eye fields (FEFs) via the mediodorsal thalamus (MD) conveys oculomotor CD associated with saccadic eye movements in nonhuman primates. This animal work provides a promising translational framework in which to investigate CD abnormalities in clinical populations. In the current study, we examined whether structural connectivity of this MD-FEF pathway relates to oculomotor CD functioning in schizophrenia. Twenty-two schizophrenia patients and 24 healthy control participants of both sexes underwent diffusion tensor imaging, and a large subset performed a trans-saccadic perceptual task that yields measures of CD. Using probabilistic tractography, we identified anatomical connections between FEF and MD and extracted indices of microstructural integrity. Patients exhibited compromised microstructural integrity in the MD-FEF pathway, which was correlated with greater oculomotor CD abnormalities and more severe psychotic symptoms. These data reinforce the role of the MD-FEF pathway in transmitting oculomotor CD signals and suggest that disturbances in this pathway may relate to psychotic symptom manifestation in patients.SIGNIFICANCE STATEMENT People with schizophrenia sometimes experience abnormalities in a sense of agency, which may stem from abnormal sensory predictions about their own actions. Consistent with this notion, the current study found reduced structural connectivity in patients with schizophrenia in a specific brain pathway found to transmit such sensorimotor prediction signals in nonhuman primates. Reduced structural connectivity was correlated with behavioral evidence for impaired sensorimotor predictions and psychotic symptoms.

A generative learning model for saccade adaptation.

Plasticity in the oculomotor system ensures that saccadic eye movements reliably meet their visual goals-to bring regions of interest into foveal, high-acuity vision. Here, we present a comprehensive description of sensorimotor learning in saccades. We induced continuous adaptation of saccade amplitudes using a double-step paradigm, in which participants saccade to a peripheral target stimulus, which then undergoes a surreptitious, intra-saccadic shift (ISS) as the eyes are in flight. In our experiments, the ISS followed a systematic variation, increasing or decreasing from one saccade to the next as a sinusoidal function of the trial number. Over a large range of frequencies, we confirm that adaptation gain shows (1) a periodic response, reflecting the frequency of the ISS with a delay of a number of trials, and (2) a simultaneous drift towards lower saccade gains. We then show that state-space-based linear time-invariant systems (LTIS) represent suitable generative models for this evolution of saccade gain over time. This state-equation algorithm computes the prediction of an internal (or hidden state-) variable by learning from recent feedback errors, and it can be compared to experimentally observed adaptation gain. The algorithm also includes a forgetting rate that quantifies per-trial leaks in the adaptation gain, as well as a systematic, non-error-based bias. Finally, we study how the parameters of the generative models depend on features of the ISS. Driven by a sinusoidal disturbance, the state-equation admits an exact analytical solution that expresses the parameters of the phenomenological description as functions of those of the generative model. Together with statistical model selection criteria, we use these correspondences to characterize and refine the structure of compatible state-equation models. We discuss the relation of these findings to established results and suggest that they may guide further design of experimental research across domains of sensorimotor adaptation.

Intra-saccadic motion streaks as cues to linking object locations across saccades.

When visual objects shift rapidly across the retina, they produce motion blur. Intra-saccadic visual signals, caused incessantly by our own saccades, are thought to be eliminated at early stages of visual processing. Here we investigate whether they are still available to the visual system and could-in principle-be used as cues for localizing objects as they change locations on the retina. Using a high-speed projection system, we developed a trans-saccadic identification task in which brief but continuous intra-saccadic object motion was key to successful performance. Observers made a saccade to a target stimulus that moved rapidly either up or down, strictly during the eye movement. Just as the target reached its final position, an identical distractor stimulus appeared on the opposite side, resulting in a display of two identical stimuli upon saccade landing. Observers had to identify the original target using the only available clue: the target's intra-saccadic movement. In an additional replay condition, we presented the observers' own intra-saccadic retinal stimulus trajectories during fixation. Compared to the replay condition, task performance was impaired during saccades but recovered fully when a post-saccadic blank was introduced. Reverse regression analyses and confirmatory experiments showed that performance increased markedly when targets had long movement durations, low spatial frequencies, and orientations parallel to their retinal trajectory-features that promote intra-saccadic motion streaks. Although the potential functional role of intra-saccadic visual signals is still unclear, our results suggest that they could provide cues to tracking objects that rapidly change locations across saccades.

Bold moves: Inevitable saccadic selection in visual short-term memory.

Selection for visual short-term memory (vstm) provides a basis for many cognitive functions. Saccadic eye movements sway this selection in favor of stimuli previously seen at locations congruent with their target. In three experiments, we provide converging evidence that this saccadic selection is implemented as a fundamental, inevitable selection process, rather than a top-down strategy. In particular, benefits for congruent over incongruent items were largely constant across set sizes ranging from two to eight items (Experiment 1), showing that saccadic selection imposes priorities on vstm irrespective of memory load and is effective even when only few representations need to be maintained. Moreover, a decrement in performance for incongruent items occurred reliably, whether the congruent location contained a task-relevant item or an irrelevant noise patch (Experiment 2). Finally, saccadic selection was immune to a strong manipulation of the observer's attentional priorities (Experiment 3). Given the prevalence of saccades in natural vision, our results demonstrate a fundamental and ecologically relevant selection mechanism for vstm: Saccades systematically eliminate information seen at non-target locations, while information at the saccade target remains available to recall. This simple heuristic is effective in the absence of informative cues and may incapacitate voluntary selection mechanisms that are incongruent with ongoing movement plans.

A comparison of the temporal and spatial properties of trans-saccadic perceptual recalibration and saccadic adaptation.

Repeated exposure to a consistent trans-saccadic step in the position of the saccadic target reliably produces a change of saccadic gain, a well-established trans-saccadic motor learning phenomenon known as saccadic adaptation. Trans-saccadic changes can also produce perceptual effects. Specifically, a systematic increase or decrease in the size of the object that is being foveated changes the perceptually equivalent size between fovea and periphery. Previous studies have shown that this recalibration of perceived size can be established within a few dozen trials, persists overnight, and generalizes across hemifields. In the current study, we use a novel adjustment paradigm to characterize both temporally and spatially the learning process that subtends this form of recalibration, and directly compare its properties to those of saccadic adaptation. We observed that sinusoidal oscillations in the amplitude of the trans-saccadic change produce sinusoidal oscillations in the reported peripheral size, with a lag of under 10 trials. This is qualitatively similar to what has been observed in the case of saccadic adaptation. We also tested whether learning is generalized to the mirror location on the opposite hemifield for both size recalibration and saccade adaptation. Here the results were markedly different, showing almost complete generalization for recalibration and no generalization for saccadic adaptation. We conclude that perceptual and visuomotor consequences of trans-saccadic changes rely on learning mechanisms that are distinct but develop on similar time scales.

An adaptive algorithm for fast and reliable online saccade detection.

To investigate visual perception around the time of eye movements, vision scientists manipulate stimuli contingent upon the onset of a saccade. For these experimental paradigms, timing is especially crucial, because saccade offset imposes a deadline on the display change. Although efficient online saccade detection can greatly improve timing, most algorithms rely on spatial-boundary techniques or absolute-velocity thresholds, which both suffer from weaknesses: late detections and false alarms, respectively. We propose an adaptive, velocity-based algorithm for online saccade detection that surpasses both standard techniques in speed and accuracy and allows the user to freely define the detection criteria. Inspired by the Engbert-Kliegl algorithm for microsaccade detection, our algorithm computes two-dimensional velocity thresholds from variance in the preceding fixation samples, while compensating for noisy or missing data samples. An optional direction criterion limits detection to the instructed saccade direction, further increasing robustness. We validated the algorithm by simulating its performance on a large saccade dataset and found that high detection accuracy (false-alarm rates of < 1%) could be achieved with detection latencies of only 3 ms. High accuracy was maintained even under simulated high-noise conditions. To demonstrate that purely intrasaccadic presentations are technically feasible, we devised an experimental test in which a Gabor patch drifted at saccadic peak velocities. Whereas this stimulus was invisible when presented during fixation, observers reliably detected it during saccades. Photodiode measurements verified that-including all system delays-the stimuli were physically displayed on average 20 ms after saccade onset. Thus, the proposed algorithm provides a valuable tool for gaze-contingent paradigms.

Motion Extrapolation for Eye Movements Predicts Perceived Motion-Induced Position Shifts.

Transmission delays in the nervous system pose challenges for the accurate localization of moving objects as the brain must rely on outdated information to determine their position in space. Acting effectively in the present requires that the brain compensates not only for the time lost in the transmission and processing of sensory information, but also for the expected time that will be spent preparing and executing motor programs. Failure to account for these delays will result in the mislocalization and mistargeting of moving objects. In the visuomotor system, where sensory and motor processes are tightly coupled, this predicts that the perceived position of an object should be related to the latency of saccadic eye movements aimed at it. Here we use the flash-grab effect, a mislocalization of briefly flashed stimuli in the direction of a reversing moving background, to induce shifts of perceived visual position in human observers (male and female). We find a linear relationship between saccade latency and perceived position shift, challenging the classic dissociation between "vision for action" and "vision for perception" for tasks of this kind and showing that oculomotor position representations are either shared with or tightly coupled to perceptual position representations. Altogether, we show that the visual system uses both the spatial and temporal characteristics of an upcoming saccade to localize visual objects for both action and perception.SIGNIFICANCE STATEMENT Accurately localizing moving objects is a computational challenge for the brain due to the inevitable delays that result from neural transmission. To solve this, the brain might implement motion extrapolation, predicting where an object ought to be at the present moment. Here, we use the flash-grab effect to induce perceptual position shifts and show that the latency of imminent saccades predicts the perceived position of the objects they target. This counterintuitive finding is important because it not only shows that motion extrapolation mechanisms indeed work to reduce the behavioral impact of neural transmission delays in the human brain, but also that these mechanisms are closely matched in the perceptual and oculomotor systems.

Presaccadic motion integration drives a predictive postsaccadic following response.

Saccadic eye movements sample the visual world and ensure high acuity across the visual field. To compensate for delays in processing, saccades to moving targets require predictions: The eyes must intercept the target's future position to then pursue its direction of motion. Although prediction is crucial to voluntary pursuit, it is unclear whether it is an obligatory feature of saccade planning. Saccade planning involves an involuntary enhanced processing of the target, called presaccadic attention. Does this presaccadic attention recruit smooth eye movements automatically? To test this, we had human participants perform a saccade to one of four apertures, which were static, but each contained a random dot field with motion tangential to the required saccade. In this task, saccades were deviated along the direction of target motion, and the eyes exhibited a following response upon saccade landing. This postsaccadic following response (PFR) increased with spatial uncertainty of the target position and persisted even when we removed the motion stimulus in midflight of the saccade, confirming that it relied on presaccadic information. Motion from 50-100 ms prior to the saccade had the strongest influence on PFR, consistent with the time course of perceptual enhancements reported in presaccadic attention. Finally, the PFR magnitude related linearly to the logarithm of stimulus velocity and generally had low gain, similar to involuntary ocular following movements commonly observed after sudden motion onsets. These results suggest that presaccadic attention selects motion features of targets predictively, presumably to ensure successful immediate tracking of saccade targets in motion.

Saccadic selection of stabilized items in visuospatial working memory.

Saccadic eye movements prioritize the memory of visual stimuli that had previously been seen at the saccade target. In two experiments, we assessed whether this influence is limited to fragile memory traces or if saccades can also affect consolidated representations in visuospatial working memory (VSWM). To interfere with fragile memory traces, we presented visual masks at different delays following the offset of a memory array and simultaneously prompted participants to generate a saccade to one location. Masking was very effective: Memory performance was lowest right after the disappearance of the memory array and gradually increased for later mask onsets. In spite of that, memory was best for stimuli congruent with the saccade target. This advantage was largest at shortest delays and then decreased over the course of a second. Insofar as only consolidated representations survive interference from masks, we conclude that saccades exert spatially selective biases on stable representations in VSWM.

Saccadic eye movements do not disrupt the deployment of feature-based attention.

The tight link of saccades to covert spatial attention has been firmly established, yet their relation to other forms of visual selection remains poorly understood. Here we studied the temporal dynamics of feature-based attention (FBA) during fixation and across saccades. Participants reported the orientation (on a continuous scale) of one of two sets of spatially interspersed Gabors (black or white). We tested performance at different intervals between the onset of a colored cue (black or white, indicating which stimulus was the most probable target; red: neutral condition) and the stimulus. FBA built up after cue onset: Benefits (errors for valid vs. neutral cues), costs (invalid vs. neutral), and the overall cueing effect (valid vs. invalid) increased with the cue-stimulus interval. Critically, we also tested visual performance at different intervals after a saccade, when FBA had been fully deployed before saccade initiation. Cueing effects were evident immediately after the saccade and were predicted most accurately and most precisely by fully deployed FBA, indicating that FBA was continuous throughout saccades. Finally, a decomposition of orientation reports into target reports and random guesses confirmed continuity of report precision and guess rates across the saccade. We discuss the role of FBA in perceptual continuity across saccades.

Selective enhancement of orientation tuning before saccades.

Saccadic eye movements cause a rapid sweep of the visual image across the retina and bring the saccade's target into high-acuity foveal vision. Even before saccade onset, visual processing is selectively prioritized at the saccade target. To determine how this presaccadic attention shift exerts its influence on visual selection, we compare the dynamics of perceptual tuning curves before movement onset at the saccade target and in the opposite hemifield. Participants monitored a 30-Hz sequence of randomly oriented gratings for a target orientation. Combining a reverse correlation technique previously used to study orientation tuning in neurons and general additive mixed modeling, we found that perceptual reports were tuned to the target orientation. The gain of orientation tuning increased markedly within the last 100 ms before saccade onset. In addition, we observed finer orientation tuning right before saccade onset. This increase in gain and tuning occurred at the saccade target location and was not observed at the incongruent location in the opposite hemifield. The present findings suggest, therefore, that presaccadic attention exerts its influence on vision in a spatially and feature-selective manner, enhancing performance and sharpening feature tuning at the future gaze location before the eyes start moving.

Chances and challenges for an active visual search perspective.

Using fixations as the fundamental unit of visual search is an appealing gear change in a paradigm that has long dominated attention research. To truly inform theories of search, however, additional challenges must be faced, including (1) an empirically motivated definition of fixation in the presence of fixational saccades and (2) the biases and limitations of transsaccadic perception and memory.

Oculomotor inhibition covaries with conscious detection.

Saccadic eye movements occur frequently even during attempted fixation, but they halt momentarily when a new stimulus appears. Here, we demonstrate that this rapid, involuntary "oculomotor freezing" reflex is yoked to fluctuations in explicit visual perception. Human observers reported the presence or absence of a brief visual stimulus while we recorded microsaccades, small spontaneous eye movements. We found that microsaccades were reflexively inhibited if and only if the observer reported seeing the stimulus, even when none was present. By applying a novel Bayesian classification technique to patterns of microsaccades on individual trials, we were able to decode the reported state of perception more accurately than the state of the stimulus (present vs. absent). Moreover, explicit perceptual sensitivity and the oculomotor reflex were both susceptible to orientation-specific adaptation. The adaptation effects suggest that the freezing reflex is mediated by signals processed in the visual cortex before reaching oculomotor control centers rather than relying on a direct subcortical route, as some previous research has suggested. We conclude that the reflexive inhibition of microsaccades immediately and inadvertently reveals when the observer becomes aware of a change in the environment. By providing an objective measure of conscious perceptual detection that does not require explicit reports, this finding opens doors to clinical applications and further investigations of perceptual awareness.

Presaccadic motion integration between current and future retinotopic locations of attended objects.

Object tracking across eye movements is thought to rely on presaccadic updating of attention between the object's current and its "remapped" location (i.e., the postsaccadic retinotopic location). We report evidence for a bifocal, presaccadic sampling between these two positions. While preparing a saccade, participants viewed four spatially separated random dot kinematograms, one of which was cued by a colored flash. They reported the direction of a coherent motion signal at the cued location while a second signal occurred simultaneously either at the cue's remapped location or at one of several control locations. Motion integration between the signals occurred only when the two motion signals were congruent and were shown at the cue and at its remapped location. This shows that the visual system integrates features between both the current and the future retinotopic locations of an attended object and that such presaccadic sampling is feature specific.