From Prior Information to Saccade Selection: Evolution of Frontal Eye Field Activity during Natural Scene Search.

Prior knowledge about our environment influences our actions. How does this knowledge evolve into a final action plan and how does the brain represent this? Here, we investigated this question in the monkey oculomotor system during self-guided search of natural scenes. In the frontal eye field (FEF), we found a subset of neurons, "Early neurons," that contain information about the upcoming saccade long before it is executed, often before the previous saccade had even ended. Crucially, much of this early information did not relate to the actual saccade that would eventually be selected. Rather, it related to prior information about the probabilities of possible upcoming saccades based on the presaccade fixation location. Nearer to the time of saccade onset, a greater proportion of these neurons' activities related to the saccade selection, although prior information continued to influence activity throughout. A separate subset of FEF neurons, "Late neurons," only represented the final action plan near saccade onset and not prior information. Our results demonstrate how, across the population of FEF neurons, prior information evolves into definitive saccade plans.

Predicting rhesus monkey eye movements during natural-image search.

There are three prominent factors that can predict human visual-search behavior in natural scenes: the distinctiveness of a location (salience), similarity to the target (relevance), and features of the environment that predict where the object might be (context). We do not currently know how well these factors are able to predict macaque visual search, which matters because it is arguably the most popular model for asking how the brain controls eye movements. Here we trained monkeys to perform the pedestrian search task previously used for human subjects. Salience, relevance, and context models were all predictive of monkey eye fixations and jointly about as precise as for humans. We attempted to disrupt the influence of scene context on search by testing the monkeys with an inverted set of the same images. Surprisingly, the monkeys were able to locate the pedestrian at a rate similar to that for upright images. The best predictions of monkey fixations in searching inverted images were obtained by rotating the results of the model predictions for the original image. The fact that the same models can predict human and monkey search behavior suggests that the monkey can be used as a good model for understanding how the human brain enables natural-scene search.

Visual Experience Is Required for the Development of Eye Movement Maps in the Mouse Superior Colliculus.

Topographic maps are a fundamental feature of the brain's representations of the sensory environment as well as an efficient way to organize motor control networks. Although great progress has been made in our understanding of sensory map development, very little is known about how topographic representations for motor control develop and interface with sensory maps. Here we map the representation for eye movements in the superior colliculus (SC) in awake mice. As stimulation sites were sampled along the anterior-posterior axis, small amplitude, nasally directed (ipsiversive) saccadic eye movements were evoked by microstimulation in anterior SC, followed by a smooth progression to large, temporally directed (contraversive) movements in posterior SC. This progressive change of movement amplitude and direction is consistent with the global polarity of the retinotopic map in the superficial SC, just as in primates and cats. We then investigated the role of visual experience in the development of eye movement map by studying mice reared in complete darkness. Saccades evoked by SC stimulation as well as spontaneous saccadic eye movements were larger in the dark-reared mice, indicating that visual experience is required to fine-tune the gain of saccades and to establish normal eye movement maps in the SC. Our experiments provide a foundation for future studies to investigate the synaptic organization and developmental mechanisms of sensorimotor transformations in mice. SIGNIFICANCE STATEMENT: The superior colliculus (SC) is a midbrain structure important for multisensory integration and sensorimotor transformation. Here we have studied eye movement representations in the SC of mice, a species that has become a popular model in vision research because of available genetic tools. Our studies show mice make saccadic eye movements spontaneously and in response to SC stimulation. The mouse SC contains an eye movement map that has the same global polarity as the overlaying visual map, just like in cats and primates. Furthermore, we show that visual experience is required for establishing the normal eye movement map. Our study provides a necessary basis for future mechanistic studies of how SC motor maps develop and become aligned with sensory maps.

Role of expected reward in frontal eye field during natural scene search.

When a saccade is expected to result in a reward, both neural activity in oculomotor areas and the saccade itself (e.g., its vigor and latency) are altered (compared with when no reward is expected). As such, it is unclear whether the correlations of neural activity with reward indicate a representation of reward beyond a movement representation; the modulated neural activity may simply represent the differences in motor output due to expected reward. Here, to distinguish between these possibilities, we trained monkeys to perform a natural scene search task while we recorded from the frontal eye field (FEF). Indeed, when reward was expected (i.e., saccades to the target), FEF neurons showed enhanced responses. Moreover, when monkeys accidentally made eye movements to the target, firing rates were lower than when they purposively moved to the target. Thus, neurons were modulated by expected reward rather than simply the presence of the target. We then fit a model that simultaneously included components related to expected reward and saccade parameters. While expected reward led to shorter latency and higher velocity saccades, these behavioral changes could not fully explain the increased FEF firing rates. Thus, FEF neurons appear to encode motivational factors such as reward expectation, above and beyond the kinematic and behavioral consequences of imminent reward.

Feature-based attention and spatial selection in frontal eye fields during natural scene search.

When we search for visual objects, the features of those objects bias our attention across the visual landscape (feature-based attention). The brain uses these top-down cues to select eye movement targets (spatial selection). The frontal eye field (FEF) is a prefrontal brain region implicated in selecting eye movements and is thought to reflect feature-based attention and spatial selection. Here, we study how FEF facilitates attention and selection in complex natural scenes. We ask whether FEF neurons facilitate feature-based attention by representing search-relevant visual features or whether they are primarily involved in selecting eye movement targets in space. We show that search-relevant visual features are weakly predictive of gaze in natural scenes and additionally have no significant influence on FEF activity. Instead, FEF activity appears to primarily correlate with the direction of the upcoming eye movement. Our result demonstrates a concrete need for better models of natural scene search and suggests that FEF activity during natural scene search is explained primarily by spatial selection.

Saliency and saccade encoding in the frontal eye field during natural scene search.

The frontal eye field (FEF) plays a central role in saccade selection and execution. Using artificial stimuli, many studies have shown that the activity of neurons in the FEF is affected by both visually salient stimuli in a neuron's receptive field and upcoming saccades in a certain direction. However, the extent to which visual and motor information is represented in the FEF in the context of the cluttered natural scenes we encounter during everyday life has not been explored. Here, we model the activities of neurons in the FEF, recorded while monkeys were searching natural scenes, using both visual and saccade information. We compare the contribution of bottom-up visual saliency (based on low-level features such as brightness, orientation, and color) and saccade direction. We find that, while saliency is correlated with the activities of some neurons, this relationship is ultimately driven by activities related to movement. Although bottom-up visual saliency contributes to the choice of saccade targets, it does not appear that FEF neurons actively encode the kind of saliency posited by popular saliency map theories. Instead, our results emphasize the FEF's role in the stages of saccade planning directly related to movement generation.

Using Natural Scenes to Enhance our Understanding of the Cerebral Cortex's Role in Visual Search.

Using natural scenes is an approach to studying the visual and eye movement systems approximating how these systems function in everyday life. This review examines the results from behavioral and neurophysiological studies using natural scene viewing in humans and monkeys. The use of natural scenes for the study of cerebral cortical activity is relatively new and presents challenges for data analysis. Methods and results from the use of natural scenes for the study of the visual and eye movement cortex are presented, with emphasis on new insights that this method provides enhancing what is known about these cortical regions from the use of conventional methods.

Inhibition in superior colliculus neurons in a brightness discrimination task?

Simultaneous recordings were collected from between two and four buildup neurons from the left and right superior colliculi in rhesus monkeys in a simple two-choice brightness discrimination task. The monkeys were required to move their eyes to one of two response targets to indicate their decision. Neurons were identified whose receptive fields were centered on the response targets. The functional role of inhibition was examined by conditionalizing firing rate on a high versus low rate in target neurons 90 ms to 30 ms before the saccade and examining the firing rate in both contralateral and ipsilateral neurons. Two models with racing diffusion processes were fit to the behavioral data, and the same analysis was performed on simulated paths in the diffusion processes that have been found to represent firing rate. The results produce converging evidence for the lack of a functional role for inhibition between neural populations corresponding to the two decisions.

Predictive activity in macaque frontal eye field neurons during natural scene searching.

Generating sequences of multiple saccadic eye movements allows us to search our environment quickly and efficiently. Although the frontal eye field cortex (FEF) has been linked to target selection and making saccades, little is known about its role in the control and performance of the sequences of saccades made during self-guided visual search. We recorded from FEF cells while monkeys searched for a target embedded in natural scenes and examined the degree to which cells with visual and visuo-movement activity showed evidence of target selection for future saccades. We found that for about half of these cells, activity during the fixation period between saccades predicted the next saccade in a sequence at an early time that precluded selection based on current visual input to a cell's response field. In addition to predicting the next saccade, activity during the fixation prior to two successive saccades also predicted the direction and goal of the second saccade in the sequence. We refer to this as advanced predictive activity. Unlike activity indicating the upcoming saccade, advanced predictive activity occurred later in the fixation period, mirroring the order of the saccade sequence itself. The remaining cells without advanced predictive activity did not predict future saccades but reintroduced the signal for the upcoming saccade at an intermediate time in the fixation period. Together these findings suggest that during natural visual search the timing of FEF cell activity is consistent with a role in specifying targets for one or more future saccades in a search sequence.

Single trial-based prediction of a go/no-go decision in monkey superior colliculus.

While some decision-making processes often result in the generation of an observable action, for example eye or limb movements, others may prevent actions and occur without an overt behavioral response. To understand how these decisions are made, one must look directly at their neuronal substrates. We trained two monkeys on a go/no-go task which requires a saccade to a peripheral cue stimulus (go) or maintenance of fixation (no-go). We performed binary regressions on the activity of single neurons in the superior colliculus (SC), with the go/no-go decision as a predictor variable, and constructed a virtual decision function (VDF) designed to provide a good estimation of decision content and its timing in a single trial decision process. Post hoc analyses by VDF correctly predicted the monkey's choice in more than 80% of trials. These results suggest that monitoring of SC activity has sufficient capacity to predict go/no-go decisions on a trial-by-trial basis.

Neural mind reading of multi-dimensional decisions by monkey mid-brain activity.

Brain-machine interfaces (BMIs) have the potential to improve the quality of life for individuals with disabilities. We engaged in the development of neural mind-reading techniques for cognitive BMIs to provide a readout of decision processes. We trained 2 monkeys on go/no-go tasks, and monitored the activity of groups of neurons in their mid-brain superior colliculus (SC). We designed a virtual decision function (VDF) reflecting the continuous progress of binary decisions on a single-trial basis, and applied it to the ensemble activity of SC neurons. Post hoc analyses using the VDF predicted the cue location as well as the monkey's motor choice (go or no-go) soon after the presentation of the cue. These results suggest that our neural mind-reading techniques have the potential to provide rapid real-time control of communication support devices.

Dual diffusion model for single-cell recording data from the superior colliculus in a brightness-discrimination task.

Monkeys made saccades to one of two peripheral targets based on the brightness of a central stimulus. Task difficulty was manipulated by varying the ratio of stimulus black-and-white pixels. Correct response probability for two monkeys varied directly with difficulty. Deep layer SC neurons exhibited robust presaccadic activity the magnitude of which was unaffected by task difficulty when the stimulus specified a saccade toward a target within the neuron's response field. Activity after stimuli specifying saccades to targets outside the response field was affected by task difficulty, increasing as the task became more difficult. A quantitative model derived from studies of human decision-making was fit to the behavioral data. The model assumes that information from the stimulus drives two independent diffusion processes. Simulated paths from the model were compared with neuron activity, assuming that firing rate is linearly related to position in the accumulation process. The firing rate data show delayed availability of discriminative information for fast, intermediate, and slow decisions when activity is aligned on the stimulus and very small differences in discriminative information when aligned on the saccade. The model produces exactly these patterns of results. The accumulation process is highly variable, allowing the process both to make errors, as is the case for the behavioral performance, and also to account for the firing rate results. Thus the dual diffusion model provides a quantitative account for both the behavior in a simple decision-making task as well as the patterns of activity in competing populations of neurons.

Effects of eye position upon activity of neurons in macaque superior colliculus.

We examined the activity of neurons in the deep layers of the superior colliculus of awake behaving rhesus monkeys during the performance of standard oculomotor tasks as well as during self-guided eye movements made while viewing natural images. The standard tasks were used to characterize the activity of neurons based on established criteria. The natural viewing paradigm enabled the sampling of neuronal activity during saccades and fixations distributed over a wide range of eye positions. Two distinct aspects of eye-movement behavior contributed to the modulation of firing activity in these neurons. The well-established influence of saccade amplitude and direction was strongest and most prevalent surrounding the time of the start of the saccade. However, the activity of these neurons was also affected by the orbital position of the eyes, and this effect was best observed during intervals of fixation. Many neurons were sensitive to both parameters, and the directions of their saccade vector and eye position response fields tended to be aligned. The sample of neurons included visual, build-up, and burst activities, alone or in combination. All of these activity types were included in the subpopulation of neurons with significant eye-position tuning, although position tuning was more common in neurons with build-up or burst activity and less common in neurons with visual activity. The presence of both eye-position as well as saccade-vector signals in the superior colliculus is likely important for its role in the planning and guidance of combined movements of the eyes and head.

Macaque frontal eye field input to saccade-related neurons in the superior colliculus.

Extracellular recordings were made simultaneously in the frontal eye field and superior colliculus in awake, behaving rhesus monkeys. Frontal eye field microstimulation was used to orthodromically activate the superior colliculus both to locate the depth of the strongest frontal eye field input to the superior colliculus and to identify superior colliculus neurons receiving direct frontal eye field input. The activity of orthodromically driven colliculus neurons was characterized during visuomotor tasks. The purpose of this study was to identify the types of superior colliculus neurons that receive excitatory frontal eye field input. We found that microstimulation of the frontal eye field did not activate the superficial layers of the superior colliculus but did activate the deeper layers. This pattern of activation coincided with the prevalence of visual versus saccade-related activity in the superficial and deep layers. A total of 83 orthodromically driven superior colliculus neurons were identified. Of these neurons, 93% (n = 77) exhibited a burst of activity associated with the onset of the saccade, and 25% (n = 21) exhibited prelude/build-up activity prior to the onset of a saccade. In addition, it was common to see some activity synchronized with the onset of a visual target (30%, n = 25). In single neurons, these activity profiles could be observed alone or in combination. Superior colliculus neurons that were exclusively visual, however, were not excited by frontal eye field stimulation. We compared the activity of superior colliculus neurons that received frontal eye field input to descriptions of saccade-related neurons made in earlier reports and found that the distribution of neuron types in the orthodromically driven population was similar to the distribution within the overall population. This suggests that the frontal eye field does not selectively influence a specific class of collicular neurons, but, instead has a direct influence on all preparatory, and saccade-related activity within the deep layers of the superior colliculus.

The primate frontal eye field and the generation of saccadic eye movements: comparison of lesion and acute inactivation/activation studies

The frontal eye field (FEF) of monkeys has been repeatedly implicated in the generation of saccadic eye movements by various experimental approaches. Electrical stimulation of most of the FEF produces saccadic eye movements, many cells have activities related to saccades, and it has anatomical connections with many other oculomotor ares. Surprisingly, complete lesions of the FEF have remarkably little effect on oculomotor behavior. Only when more cognitive aspects are tested is a deficit clearly detected. In contrast, acute inactivation of the FEF on monkeys with the GABA agonist muscimol produced much more severe oculomotor impairment. This difference is probably due to the acute nature of the muscimol effect, which does not allow time for reorganization of the control of eye movements before testing begins. In addition, acute activation of the FEF with the GABA antagonist bicuculline caused the monkey to make irrepressible saccades of the same dimensions as those electrically elicited at the site. These experiments further confirm the strong involvement of the FEF in the control of saccadic eye movements and fixation.