Dissociating the Contributions of Frontal Eye Field Activity to Spatial Working Memory and Motor Preparation.

Neurons within dorsolateral prefrontal cortex (PFC) of primates are characterized by robust persistent spiking activity exhibited during the delay period of working memory tasks. This includes the frontal eye field (FEF) where nearly half of the neurons are active when spatial locations are held in working memory. Past evidence has established the FEF's contribution to the planning and triggering of saccadic eye movements as well as to the control of visual spatial attention. However, it remains unclear whether persistent delay activity reflects a similar dual role in movement planning and visuospatial working memory. We trained male monkeys to alternate between different forms of a spatial working memory task which could dissociate remembered stimulus locations from planned eye movements. We tested the effects of inactivation of FEF sites on behavioral performance in the different tasks. Consistent with previous studies, FEF inactivation impaired the execution of memory-guided saccades (MGSs), and impaired performance when remembered locations matched the planned eye movement. In contrast, memory performance was largely unaffected when the remembered location was dissociated from the correct eye movement response. Overall, the inactivation effects demonstrated clear deficits in eye movements, regardless of task type, but little or no evidence of a deficit in spatial working memory. Thus, our results indicate that persistent delay activity in the FEF contributes primarily to the preparation of eye movements and not to spatial working memory.SIGNIFICANCE STATEMENT Many frontal eye field (FEF) neurons exhibit spatially tuned persistent spiking activity during the delay period of working memory tasks. However, the role of the FEF in spatial working memory remains unresolved. We tested the effects of inactivation of FEF sites on behavioral performance in different forms of a spatial working memory task, one of which dissociated the remembered stimulus locations from planned eye movements. We found that FEF inactivation produced clear deficits in eye movements, regardless of task type, but no deficit in spatial working memory when dissociated from those movements.

Action space restructures visual working memory in prefrontal cortex.

Visual working memory enables flexible behavior by decoupling sensory stimuli from behavioral actions. While previous studies have predominantly focused on the storage component of working memory, the role of future actions in shaping working memory remains unknown. To answer this question, we used two working memory tasks that allowed the dissociation of sensory and action components of working memory. We measured behavioral performance and neuronal activity in the macaque prefrontal cortex area, frontal eye fields. We show that the action space reshapes working memory, as evidenced by distinct patterns of memory tuning and attentional orienting between the two tasks. Notably, neuronal activity during the working memory period predicted future behavior and exhibited mixed selectivity in relation to the sensory space but linear selectivity relative to the action space. This linear selectivity was achieved through the rapid transformation from sensory to action space and was subsequently maintained as a stable cross-temporal population activity pattern. Combined, we provide direct physiological evidence of the action-oriented nature of frontal eye field neurons during memory tasks and demonstrate that the anticipation of behavioral outcomes plays a significant role in transforming and maintaining the contents of visual working memory.

Dissociating the Contributions of Frontal Eye Field Activity to Spatial Working Memory and Motor Preparation.

Neurons within dorsolateral prefrontal cortex of primates are characterized by robust persistent spiking activity exhibited during the delay period of working memory tasks. This includes the frontal eye field (FEF) where nearly half of the neurons are active when spatial locations are held in working memory. Past evidence has established the FEF's contribution to the planning and triggering of saccadic eye movements as well as to the control of visual spatial attention. However, it remains unclear if persistent delay activity reflects a similar dual role in movement planning and visuospatial working memory. We trained monkeys to alternate between different forms of a spatial working memory task which could dissociate remembered stimulus locations from planned eye movements. We tested the effects of inactivation of FEF sites on behavioral performance in the different tasks. Consistent with previous studies, FEF inactivation impaired the execution of memory-guided saccades, and impaired performance when remembered locations matched the planned eye movement. In contrast, memory performance was largely unaffected when the remembered location was dissociated from the correct eye movement response. Overall, the inactivation effects demonstrated clear deficits on eye movements, regardless of task type, but little or no evidence of a deficit in spatial working memory. Thus, our results indicate that persistent delay activity in the FEF contributes primarily to the preparation of eye movements and not to spatial working memory.

Eye and hand movements disrupt attentional control.

Voluntary attentional control is the ability to selectively focus on a subset of visual information in the presence of other competing stimuli-a marker of cognitive control enabling flexible, goal-driven behavior. To test its robustness, we contrasted attentional control with the most common source of attentional orienting in daily life: attention shifts prior to goal-directed eye and hand movements. In a multi-tasking paradigm, human participants attended at a location while planning eye or hand movements elsewhere. Voluntary attentional control suffered with every simultaneous action plan, even under reduced task difficulty and memory load-factors known to interfere with attentional control. Furthermore, the performance cost was limited to voluntary attention: We observed simultaneous attention benefits at two movement targets without attentional competition between them. This demonstrates that the visual system allows for the concurrent representation of multiple attentional foci. Since attentional control is extremely fragile and dominated by premotor attention shifts, we propose that action-driven selection plays the superordinate role for visual selection.

Sounds are remapped across saccades.

To achieve visual space constancy, our brain remaps eye-centered projections of visual objects across saccades. Here, we measured saccade trajectory curvature following the presentation of visual, auditory, and audiovisual distractors in a double-step saccade task to investigate if this stability mechanism also accounts for localized sounds. We found that saccade trajectories systematically curved away from the position at which either a light or a sound was presented, suggesting that both modalities are represented in eye-centered oculomotor centers. Importantly, the same effect was observed when the distractor preceded the execution of the first saccade. These results suggest that oculomotor centers keep track of visual, auditory and audiovisual objects by remapping their eye-centered representations across saccades. Furthermore, they argue for the existence of a supra-modal map which keeps track of multi-sensory object locations across our movements to create an impression of space constancy.

The interdependence of attention, working memory and gaze control: behavior and neural circuitry.

Visual attention, visual working memory, and gaze control are basic functions that all select a subset of visual input to guide immediate or subsequent behavior. In this review, we focus on the relationship between these three functions and describe evidence, both at the behavioral and neural circuit levels that they are heavily interdependent. We start with the demonstration that gaze control - or saccade preparation in particular - leads to spatial attention. Next, we show that spatial attention and working memory interact at the behavioral level and rely on a common set of neural mechanisms. Next, we discuss the evidence that gaze control mechanisms are involved in spatial working memory. Lastly, we highlight the links between gaze control and non-spatial memory.

Saccade selection and inhibition: motor and attentional components.

Motor responses are fundamentally spatial in their function and neural organization. However, studies of inhibitory motor control, focused on global stopping of all actions, have ignored whether inhibitory control can be exercised selectively for specific actions. We used a new approach to elicit and measure motor inhibition by asking human participants to either look at (select) or avoid looking at (inhibit) a location in space. We found that instructing a location to be avoided resulted in an inhibitory bias specific to that location. When compared with the facilitatory bias observed in the Look task, it differed significantly in both its spatiotemporal dynamics and its modulation of attentional processing. While action selection was evident in oculomotor system and interacted with attentional processing, action inhibition was evident mainly in the oculomotor system. Our findings suggest that action inhibition is implemented by spatially specific mechanisms that are separate from action selection. NEW & NOTEWORTHY We show that cognitive control of saccadic responses evokes separable action selection and inhibition processes. Both action selection and inhibition are represented in the saccadic system, but only action selection interacts with the attentional system.

Pre-saccadic remapping relies on dynamics of spatial attention.

Each saccade shifts the projections of the visual scene on the retina. It has been proposed that the receptive fields of neurons in oculomotor areas are predictively remapped to account for these shifts. While remapping of the whole visual scene seems prohibitively complex, selection by attention may limit these processes to a subset of attended locations. Because attentional selection consumes time, remapping of attended locations should evolve in time, too. In our study, we cued a spatial location by presenting an attention-capturing cue at different times before a saccade and constructed maps of attentional allocation across the visual field. We observed no remapping of attention when the cue appeared shortly before saccade. In contrast, when the cue appeared sufficiently early before saccade, attentional resources were reallocated precisely to the remapped location. Our results show that pre-saccadic remapping takes time to develop suggesting that it relies on the spatial and temporal dynamics of spatial attention.

Spatial attention during saccade decisions.

Behavioral measures of decision making are usually limited to observations of decision outcomes. In the present study, we made use of the fact that oculomotor and sensory selection are closely linked to track oculomotor decision making before oculomotor responses are made. We asked participants to make a saccadic eye movement to one of two memorized target locations and observed that visual sensitivity increased at both the chosen and the nonchosen saccade target locations, with a clear bias toward the chosen target. The time course of changes in visual sensitivity was related to saccadic latency, with the competition between the chosen and nonchosen targets resolved faster before short-latency saccades. On error trials, we observed an increased competition between the chosen and nonchosen targets. Moreover, oculomotor selection and visual sensitivity were influenced by top-down and bottom-up factors as well as by selection history and predicted the direction of saccades. Our findings demonstrate that saccade decisions have direct visual consequences and show that decision making can be traced in the human oculomotor system well before choices are made. Our results also indicate a strong association between decision making, saccade target selection, and visual sensitivity.NEW & NOTEWORTHY We show that saccadic decisions can be tracked by measuring spatial attention. Spatial attention is allocated in parallel to the two competing saccade targets, and the time course of spatial attention differs for fast-slow and for correct-erroneous decisions. Saccade decisions take the form of a competition between potential saccade goals, which is associated with spatial attention allocation to those locations.

Presaccadic motion integration between current and future retinotopic locations of attended objects.

Object tracking across eye movements is thought to rely on presaccadic updating of attention between the object's current and its "remapped" location (i.e., the postsaccadic retinotopic location). We report evidence for a bifocal, presaccadic sampling between these two positions. While preparing a saccade, participants viewed four spatially separated random dot kinematograms, one of which was cued by a colored flash. They reported the direction of a coherent motion signal at the cued location while a second signal occurred simultaneously either at the cue's remapped location or at one of several control locations. Motion integration between the signals occurred only when the two motion signals were congruent and were shown at the cue and at its remapped location. This shows that the visual system integrates features between both the current and the future retinotopic locations of an attended object and that such presaccadic sampling is feature specific.

Attention allocation before antisaccades.

In the present study, we investigated the distribution of attention before antisaccades. We used a dual task paradigm, in which participants made prosaccades or antisaccades and discriminated the orientation of a visual probe shown at the saccade goal, the visual cue location (antisaccade condition), or a neutral location. Moreover, participants indicated whether they had made a correct antisaccade or an erroneous prosaccade. We observed that, while spatial attention in the prosaccade task was allocated only to the saccade goal, attention in the antisaccade task was allocated both to the cued location and to the antisaccade goal. This suggests parallel attentional selection of the cued and antisaccade locations. We further observed that in error trials--in which participants made an incorrect prosaccade instead of an antisaccade--spatial attention was biased towards the prosaccade goal. These erroneous prosaccades were mostly unnoticed and were often followed by corrective antisaccades with very short latencies (<100 ms). Data from error trials therefore provide further evidence for the parallel programming of the reflexive prosaccade to the cue and the antisaccade to the intended location. Taken together, our results suggest that attention allocation and saccade goal selection in the antisaccade task are mediated by a common competitive process.

Oculomotor selection underlies feature retention in visual working memory.

Oculomotor selection, spatial task relevance, and visual working memory (WM) are described as three processes highly intertwined and sustained by similar cortical structures. However, because task-relevant locations always constitute potential saccade targets, no study so far has been able to distinguish between oculomotor selection and spatial task relevance. We designed an experiment that allowed us to dissociate in humans the contribution of task relevance, oculomotor selection, and oculomotor execution to the retention of feature representations in WM. We report that task relevance and oculomotor selection lead to dissociable effects on feature WM maintenance. In a first task, in which an object's location was encoded as a saccade target, its feature representations were successfully maintained in WM, whereas they declined at nonsaccade target locations. Likewise, we observed a similar WM benefit at the target of saccades that were prepared but never executed. In a second task, when an object's location was marked as task relevant but constituted a nonsaccade target (a location to avoid), feature representations maintained at that location did not benefit. Combined, our results demonstrate that oculomotor selection is consistently associated with WM, whereas task relevance is not. This provides evidence for an overlapping circuitry serving saccade target selection and feature-based WM that can be dissociated from processes encoding task-relevant locations.

Target-distractor competition in the oculomotor system is spatiotopic.

In natural scenes, multiple visual stimuli compete for selection; however, each saccade displaces the stimulus representations in retinotopicaly organized visual and oculomotor maps. In the present study, we used saccade curvature to investigate whether oculomotor competition across eye movements is represented in retinotopic or spatiotopic coordinates. Participants performed a sequence of saccades and we induced oculomotor competition by briefly presenting a task-irrelevant distractor at different times during the saccade sequence. Despite the intervening saccade, the second saccade curved away from a spatial representation of the distractor that was presented before the first saccade. Furthermore, the degree of saccade curvature increased with the salience of the distractor presented before the first saccade. The results suggest that spatiotopic representations of target-distractor competition are crucial for successful interaction with objects of interest despite the intervening eye movements.

Dissociating oculomotor contributions to spatial and feature-based selection.

Saccades not only deliver the high-resolution retinal image requisite for visual perception, but processing stages associated with saccade target selection affect visual perception even before the eye movement starts. These presaccadic effects are thought to arise from two visual selection mechanisms: spatial selection that enhances processing of the saccade target location and feature-based selection that enhances processing of the saccade target features. By measuring oculomotor performance and perceptual discrimination, we determined which selection mechanisms are associated with saccade preparation. We observed both feature-based and space-based selection during saccade preparation but found that feature-based selection was neither related to saccade initiation nor was it affected by simultaneously observed redistribution of spatial selection. We conclude that oculomotor selection biases visual selection only in a spatial, feature-unspecific manner.

Inhibition of saccades elicits attentional suppression.

Visuospatial attention has been shown to have a central role in planning and generation of saccades but what role, if any, it plays in inhibition of saccades remains unclear. In this study, we used an oculomotor delayed match- or nonmatch-to-sample task in which a cued location has to be encoded and memorized for one of two very different goals-to plan a saccade to it or to avoid making a saccade to it. We measured the spatial allocation of attention during the delay and found that while marking a location as a future saccade target resulted in an attentional benefit at that location, marking it as forbidden to saccades led to an attentional cost. Additionally, saccade trajectories were found to deviate away more from the "don't look" location than from a saccade-irrelevant distractor confirming greater inhibition of an actively forbidden location in oculomotor programming. Our finding that attention is suppressed at locations forbidden to saccades confirms and complements the claim of a selective and obligatory coupling between saccades and attention-saccades at the memorized location could neither be planned nor suppressed independent of a corresponding effect on attentional performance.

Allocation of attention across saccades.

Whenever the eyes move, spatial attention must keep track of the locations of targets as they shift on the retina. This study investigated transsaccadic updating of visual attention to cued targets. While observers prepared a saccade, we flashed an irrelevant, but salient, color cue in their visual periphery and measured the allocation of spatial attention before and after the saccade using a tilt discrimination task. We found that just before the saccade, attention was allocated to the cue's future retinal location, its predictively "remapped" location. Attention was sustained at the cue's location in the world across the saccade, despite the change of retinal position whereas it decayed quickly at the retinal location of the cue, after the eye landed. By extinguishing the color cue across the saccade, we further demonstrate that the visual system relies only on predictive allocation of spatial attention, as the presence of the cue after the saccade did not substantially affect attentional allocation. These behavioral results support and extend physiological evidence showing predictive activation of visual neurons when an attended stimulus will fall in their receptive field after a saccade. Our results show that tracking of spatial locations across saccades is a plausible consequence of physiological remapping.

Hearing the speed: visual motion biases the perception of auditory tempo.

The coupling between sensory and motor processes has been established in various scenarios: for example, the perception of auditory rhythm entails an audiomotor representation of the sounds. Similarly, visual action patterns can also be represented via a visuomotor transformation. In this study, we tested the hypothesis that the visual motor information, such as embedded in a coherent motion flow, can interact with the perception of a motor-related aspect in auditory rhythm: the tempo. In the first two experiments, we employed an auditory tempo judgment task where participants listened to a standard auditory sequence while concurrently watching visual stimuli of different motion information, after which they judged the tempo of a comparison sequence related to the standard. In Experiment 1, we found that the same auditory tempo was perceived as faster when it was accompanied by accelerating visual motion than by non-motion luminance change. In Experiment 2, we compared the perceived auditory tempo among three visual motion conditions, increase in speed, decrease in speed, and no speed change, and found the corresponding bias in judgment of auditory tempo: faster than it was, slower than it was, and no bias. In Experiment 3, the perceptual bias induced by the change in motion speed was consistently reflected in the tempo reproduction task. Taken together, these results indicate that between a visual spatiotemporal and an auditory temporal stimulation, the embedded motor representations from each can interact across modalities, leading to a spatial-to-temporal bias. This suggests that the perceptual process in one modality can incorporate concurrent motor information from cross-modal sensory inputs to form a coherent experience.

Independent allocation of attention to eye and hand targets in coordinated eye-hand movements.

When reaching for objects, people frequently look where they reach. This raises the question of whether the targets for the eye and hand in concurrent eye and hand movements are selected by a unitary attentional system or by independent mechanisms. We used the deployment of visual attention as an index of the selection of movement targets and asked observers to reach and look to either the same location or separate locations. Results show that during the preparation of coordinated movements, attention is allocated in parallel to the targets of a saccade and a reaching movement. Attentional allocations for the two movements interact synergistically when both are directed to a common goal. Delaying the eye movement delays the attentional shift to the saccade target while leaving attentional deployment to the reach target unaffected. Our findings demonstrate that attentional resources are allocated independently to the targets of eye and hand movements and suggest that the goals for these effectors are selected by separate attentional mechanisms.

Predictive remapping of attention across eye movements.

Many cells in retinotopic brain areas increase their activity when saccades (rapid eye movements) are about to bring stimuli into their receptive fields. Although previous work has attempted to look at the functional correlates of such predictive remapping, no study has explicitly tested for better attentional performance at the future retinal locations of attended targets. We found that, briefly before the eyes start moving, attention drawn to the targets of upcoming saccades also shifted to those retinal locations that the targets would cover once the eyes had moved, facilitating future movements. This suggests that presaccadic visual attention shifts serve to both improve presaccadic perceptual processing at the target locations and speed subsequent eye movements to their new postsaccadic locations. Predictive remapping of attention provides a sparse, efficient mechanism for keeping track of relevant parts of the scene when frequent rapid eye movements provoke retinal smear and temporal masking.

Preparing coordinated eye and hand movements: dual-task costs are not attentional.

Dual-task costs are observed when people perform two tasks at the same time. It has been suggested that these costs arise from limitations of movement goal selection when multiple goal-directed movements are made simultaneously. To investigate this, we asked participants to reach and look at different locations while we varied the time between the cues to start the eye and the hand movement between 150 ms and 900 ms. In Experiment 1, participants executed the reach first, and the saccade second, in Experiment 2 the order of the movements was reversed. We observed dual-task costs-participants were slower to start the eye or hand movement if they were planning another movement at that time. In Experiment 3, we investigated whether these dual-task costs were due to limited attentional resources needed to select saccade and reach goal locations. We found that the discrimination of a probe improved at both saccade and reach locations, indicating that attention shifted to both movement goals. Importantly, while we again observed the expected dual-task costs as reflected in movement latencies, there was no apparent delay of the associated attention shifts. Our results rule out attentional goal selection as the causal factor leading to the dual-task costs occurring in eye-hand movements.