Behavior in a visual search task with moving dot stimuli.

Understanding the neuronal mechanisms underlying the processing of visual attention requires a well-designed behavioral task that allows investigators to clearly describe the behavioral effects of attention. Here, we introduce a behavioral paradigm in which one, two or four moving dot stimuli are used in a visual search paradigm that includes two additional attentional conditions. Two animals were trained to make a saccade to a target (a dot patch with net rightward motion) and hold central fixation if no target was present. To implement covert visual attention, we included trials in which a 100% valid spatial cue appeared and trials in which the color of the fixation point indicated, with 100% validity, which of four colored dot patches the animals should attend to. We analyzed the behavior in terms of reaction times and signal detection theory metrics d-prime (representing sensitivity) and criteria. In both animals, we found that reaction times were greater for larger set-sizes and that the introduction of an attentional cue reduced the reaction times substantially. We also found that both animals showed increases in criteria, but no change in sensitivity, as set-size increased and the attentional cues led to an increase in sensitivity, with only some change in criteria. Our results illustrate how the animals perform this task and imply that both animals chose similar strategies. Importantly, this will allow future neurophysiological studies to probe not only the effects of attention, but will give the possibility of seeing whether different neural mechanisms drive changes in criteria and d-prime.

The roles of the lateral intraparietal area and frontal eye field in guiding eye movements in free viewing search behavior.

The lateral intraparietal area (LIP) and frontal eye field (FEF) have been shown to play significant roles in oculomotor control, yet most studies have found that the two areas behave similarly. To identify the unique roles each area plays in guiding eye movements, we recorded 200 LIP neurons and 231 FEF neurons from four animals performing a free viewing visual foraging task. We analyzed how neuronal responses were modulated by stimulus identity and the animals' choice of where to make a saccade. We additionally analyzed the comodulation of the sensory signals and the choice signal to identify how the sensory signals drove the choice. We found a clearly defined division of labor: LIP provided a stable map integrating task rules and stimulus identity, whereas FEF responses were dynamic, representing more complex information and, just before the saccade, were integrated with task rules and stimulus identity to decide where to move the eye.NEW & NOTEWORTHY The lateral intrapareital area (LIP) and frontal eye field (FEF) are known to contribute to guiding eye movements, but little is known about the unique roles that each area plays. Using a free viewing visual search task, we found that LIP provides a stable map of the visual world, integrating task rules and stimulus identity. FEF activity is consistently modulated by more complex information but, just before the saccade, integrates all the information to make the final decision about where to move.

Suppression of frontal eye field neuronal responses with maintained fixation.

The decision of where to make an eye movement is thought to be driven primarily by responses to stimuli in neurons' receptive fields (RFs) in oculomotor areas, including the frontal eye field (FEF) of prefrontal cortex. It is also thought that a saccade may be generated when the accumulation of this activity in favor of one location or another reaches a threshold. However, in the reading and scene perception fields, it is well known that the properties of the stimulus at the fovea often affect when the eyes leave that stimulus. We propose that if FEF plays a role in generating eye movements, then the identity of the stimulus at fixation should affect the FEF responses so as to reduce the probability of making a saccade when fixating an item of interest. Using a visual foraging task in which animals could make multiple eye movements within a single trial, we found that responses were strongly modulated by the identity of the stimulus at the fovea. Specifically, responses to the stimulus in the RF were suppressed when the animal maintained fixation for longer durations on a stimulus that could be associated with a reward. We suggest that this suppression, which was predicted by models of eye movement behavior, could be a mechanism by which FEF can modulate the temporal flow of saccades based on the importance of the stimulus at the fovea.

Neurons in FEF Keep Track of Items That Have Been Previously Fixated in Free Viewing Visual Search.

When searching a visual scene for a target, we tend not to look at items or locations we have already searched. It is thought that this behavior is driven by an inhibitory tagging mechanism that inhibits responses on priority maps to the relevant items. We hypothesized that this inhibitory tagging signal should be represented as an elevated response in neurons that keep track of stimuli that have been fixated. We recorded from 231 neurons in the frontal eye field (FEF) of 2 male animals performing a visual foraging task, in which they had to find a reward linked to one of five identical targets (Ts) among five distractors. We identified 38 neurons with activity that was significantly greater when the stimulus in the receptive field had been fixated previously in the trial than when it had not been fixated. The response to a fixated object began before the saccade ended, suggesting that this information is remapped. Unlike most FEF neurons, the activity in these cells was not suppressed during active fixation, had minimal motor responses, and did not change through the trial. Yet using traditional classifications from a memory-guided saccade, they were indistinguishable from the rest of the FEF population. We propose that these neurons keep track of any items that have been fixated within the trial and this signal is propagated by remapping. These neurons could be the source of the inhibitory tagging signal to parietal cortex, where a neuronal instantiation of inhibitory tagging is seen.SIGNIFICANCE STATEMENT When we search a scene for an item, we rarely examine the same location twice. It is thought that this is due to a neural mechanism that keeps track of the items at which we have looked. Here we identified a subset of neurons in the frontal eye field that preferentially responded to items that had been fixated earlier in the trial. These responses were remapped, appearing before the saccade even ended, and were not suppressed during maintained fixation. We propose that these neurons keep track of which items have been examined in search and could be the source of feedback that creates the inhibitory tagging seen in parietal cortex.

The functional roles of neural remapping in cortex.

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

Activity in LIP, But not V4, Matches Performance When Attention is Spread.

The enhancement of neuronal responses in many visual areas while animals perform spatial attention tasks has widely been thought to be the neural correlate of visual attention, but it is unclear whether the presence or absence of this modulation contributes to our striking inability to notice changes in change blindness examples. We asked whether neuronal responses in visual area V4 and the lateral intraparietal area (LIP) in posterior parietal cortex could explain the limited ability of subjects to attend multiple items in a display. We trained animals to perform a change detection task in which they had to compare 2 arrays of stimuli separated briefly in time and found that each animal's performance decreased as function of set-size. Neuronal discriminability in V4 was consistent across set-sizes, but decreased for higher set-sizes in LIP. The introduction of a reward bias produced attentional enhancement in V4, but this could not explain the vast improvement in performance, whereas the enhancement in LIP responses could. We suggest that behavioral set-size effects and the marked improvement in performance with focused attention may not be related to response enhancement in V4 but, instead, may occur in or on the way to LIP.

Artificial palpation in robotic surgery using haptic feedback.

BACKGROUND: The loss of tactile feedback in minimally invasive robotic surgery remains a major challenge to the expanding field. With visual cue compensation alone, tissue characterization via palpation proves to be immensely difficult. This work evaluates a bimodal vibrotactile system as a means of conveying applied forces to simulate haptic feedback in two sets of studies simulating an artificial palpation task using the da Vinci surgical robot. METHODS: Subjects in the first study were tasked with localizing an embedded vessel in a soft tissue phantom using a single-sensor unit. In the second study, subjects localized tumor-like structures using a three-sensor array. In both sets of studies, subjects completed the task under three trial conditions: no feedback, normal force tactile feedback, and hybrid vibrotactile feedback. Recordings of correct localization, incorrect localization, and time-to-completion were used to evaluate performance outcomes. RESULTS: With the addition of vibrotactile and pneumatic feedback, significant improvements in the percentage of correct localization attempts were detected (p = 0.0001 and p = 0.0459, respectively) during the first experiment with phantom vessels. Similarly, significant improvements in correct localization were found with the addition of vibrotactile (p = 2.57E-5) and pneumatic significance (p = 8.54E-5) were observed in the second experiment involving tumor phantoms. CONCLUSIONS: This work demonstrates not only the superior benefits of a multi-modal feedback over traditional single-modality feedback, but also the effectiveness of vibration in providing haptic feedback to artificial palpation systems.

Attention, intention, and priority in the parietal lobe.

For many years there has been a debate about the role of the parietal lobe in the generation of behavior. Does it generate movement plans (intention) or choose objects in the environment for further processing? To answer this, we focus on the lateral intraparietal area (LIP), an area that has been shown to play independent roles in target selection for saccades and the generation of visual attention. Based on results from a variety of tasks, we propose that LIP acts as a priority map in which objects are represented by activity proportional to their behavioral priority. We present evidence to show that the priority map combines bottom-up inputs like a rapid visual response with an array of top-down signals like a saccade plan. The spatial location representing the peak of the map is used by the oculomotor system to target saccades and by the visual system to guide visual attention.

Object comparison in the lateral intraparietal area.

We can search for and locate specific objects in our environment by looking for objects with similar features. Object recognition involves stimulus similarity responses in ventral visual areas and task-related responses in prefrontal cortex. We tested whether neurons in the lateral intraparietal area (LIP) of posterior parietal cortex could form an intermediary representation, collating information from object-specific similarity map representations to allow general decisions about whether a stimulus matches the object being looked for. We hypothesized that responses to stimuli would correlate with how similar they are to a sample stimulus. When animals compared two peripheral stimuli to a sample at their fovea, the response to the matching stimulus was similar, independent of the sample identity, but the response to the nonmatch depended on how similar it was to the sample: the more similar, the greater the response to the nonmatch stimulus. These results could not be explained by task difficulty or confidence. We propose that LIP uses its known mechanistic properties to integrate incoming visual information, including that from the ventral stream about object identity, to create a dynamic representation that is concise, low dimensional, and task relevant and that signifies the choice priorities in mental matching behavior.NEW & NOTEWORTHY Studies in object recognition have focused on the ventral stream, in which neurons respond as a function of how similar a stimulus is to their preferred stimulus, and on prefrontal cortex, where neurons indicate which stimulus is being looked for. We found that parietal area LIP uses its known mechanistic properties to form an intermediary representation in this process. This creates a perceptual similarity map that can be used to guide decisions in prefrontal areas.

Remapping, Spatial Stability, and Temporal Continuity: From the Pre-Saccadic to Postsaccadic Representation of Visual Space in LIP.

As our eyes move, we have a strong percept that the world is stable in space and time; however, the signals in cortex coming from the retina change with each eye movement. It is not known how this changing input produces the visual percept we experience, although the predictive remapping of receptive fields has been described as a likely candidate. To explain how remapping accounts for perceptual stability, we examined responses of neurons in the lateral intraparietal area while animals performed a visual foraging task. When a stimulus was brought into the response field of a neuron that exhibited remapping, the onset of the postsaccadic representation occurred shortly after the saccade ends. Whenever a stimulus was taken out of the response field, the presaccadic representation abruptly ended shortly after the eyes stopped moving. In the 38% (20/52) of neurons that exhibited remapping, there was no more than 30 ms between the end of the presaccadic representation and the start of the postsaccadic representation and, in some neurons, and the population as a whole, it was continuous. We conclude by describing how this seamless shift from a presaccadic to postsaccadic representation could contribute to spatial stability and temporal continuity.

Visual-perceptual mismatch in robotic surgery.

BACKGROUND: The principal objective of the experiment was to analyze the effects of the clutch operation of robotic surgical systems on the performance of the operator. The relative coordinate system introduced by the clutch operation can introduce a visual-perceptual mismatch which can potentially have negative impact on a surgeon's performance. We also assess the impact of the introduction of additional tactile sensory information on reducing the impact of visual-perceptual mismatch on the performance of the operator. METHODS: We asked 45 novice subjects to complete peg transfers using the da Vinci IS 1200 system with grasper-mounted, normal force sensors. The task involves picking up a peg with one of the robotic arms, passing it to the other arm, and then placing it on the opposite side of the view. Subjects were divided into three groups: aligned group (no mismatch), the misaligned group (10 cm z axis mismatch), and the haptics-misaligned group (haptic feedback and z axis mismatch). Each subject performed the task five times, during which the grip force, time of completion, and number of faults were recorded. RESULTS: Compared to the subjects that performed the tasks using a properly aligned controller/arm configuration, subjects with a single-axis misalignment showed significantly more peg drops (p = 0.011) and longer time to completion (p < 0.001). Additionally, it was observed that addition of tactile feedback helps reduce the negative effects of visual-perceptual mismatch in some cases. Grip force data recorded from grasper-mounted sensors showed no difference between the different groups. CONCLUSIONS: The visual-perceptual mismatch created by the misalignment of the robotic controls relative to the robotic arms has a negative impact on the operator of a robotic surgical system. Introduction of other sensory information and haptic feedback systems can help in potentially reducing this effect.

Evaluating tactile feedback in robotic surgery for potential clinical application using an animal model.

INTRODUCTION: The aims of this study were to evaluate (1) grasping forces with the application of a tactile feedback system in vivo and (2) the incidence of tissue damage incurred during robotic tissue manipulation. Robotic-assisted minimally invasive surgery has been shown to be beneficial in a variety of surgical specialties, particularly radical prostatectomy. This innovative surgical tool offers advantages over traditional laparoscopic techniques, such as improved wrist-like maneuverability, stereoscopic video displays, and scaling of surgical gestures to increase precision. A widely cited disadvantage associated with robotic systems is the absence of tactile feedback. METHODS AND PROCEDURE: Nineteen subjects were categorized into two groups: 5 experts (six or more robotic cases) and 14 novices (five cases or less). The subjects used the da Vinci with integrated tactile feedback to run porcine bowel in the following conditions: (T1: deactivated tactile feedback; T2: activated tactile feedback; and T3: deactivated tactile feedback). The grasping force, incidence of tissue damage, and the correlation of grasping force and tissue damage were analyzed. Tissue damage was evaluated both grossly and histologically by a pathologist blinded to the sample. RESULTS: Tactile feedback resulted in significantly decreased grasping forces for both experts and novices (P < 0.001 in both conditions). The overall incidence of tissue damage was significantly decreased in all subjects (P < 0.001). A statistically significant correlation was found between grasping forces and incidence of tissue damage (P = 0.008). The decreased forces and tissue damage were retained through the third trial when the system was deactivated (P > 0.05 in all subjects). CONCLUSION: The in vivo application of integrated tactile feedback in the robotic system demonstrates significantly reduced grasping forces, resulting in significantly less tissue damage. This tactile feedback system may improve surgical outcomes and broaden the use of robotic-assisted minimally invasive surgery.

LIP activity in the interstimulus interval of a change detection task biases the behavioral response.

When looking around at the world, we can only attend to a limited number of locations. The lateral intraparietal area (LIP) is thought to play a role in guiding both covert attention and eye movements. In this study, we tested the involvement of LIP in both mechanisms with a change detection task. In the task, animals had to indicate whether an element changed during a blank in the trial by making a saccade to it. If no element changed, they had to maintain fixation. We examine how the animal's behavior is biased based on LIP activity prior to the presentation of the stimulus the animal must respond to. When the activity was high, the animal was more likely to make an eye movement toward the stimulus, even if there was no change; when the activity was low, the animal either had a slower reaction time or maintained fixation, even if a change occurred. We conclude that LIP activity is involved in both covert and overt attention, but when decisions about eye movements are to be made, this role takes precedence over guiding covert attention.

Dissociating activity in the lateral intraparietal area from value using a visual foraging task.

We make decisions about where to look approximately three times per second in normal viewing. It has been suggested that eye movements may be guided by activity in the lateral intraparietal area (LIP), which is thought to represent the relative value of objects in space. However, it is not clear how values for saccade goal selection are prioritized while free-viewing in a cluttered visual environment. To address this question, we compared the neural responses of LIP neurons in two subjects with their saccadic behavior and three estimates of stimulus value. These measures were extracted from the subjects' performance in a visual foraging task, in which we parametrically controlled the number of objects on the screen. We found that the firing rates of LIP neurons did not correlate well with the animals' behavior or any of our estimated measures of value. However, if the LIP activity was further normalized, it became highly correlated with the animals' decisions. These data suggest that LIP activity does not represent value in complex environments, but that the value can easily be extracted with one further step of processing. We propose that activity in LIP represents attentional priority and that the downstream normalization of this activity is an essential process in guiding action.

Extrafoveal preview benefit during free-viewing visual search in the monkey.

Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

Anticipatory remapping of attentional priority across the entire visual field.

It has been suggested that one way we may create a stable percept of the visual world across multiple eye movements is to pass information from one set of neurons to another around the time of each eye movement. Previous studies have shown that some neurons in the lateral intraparietal area (LIP) exhibit anticipatory remapping: these neurons produce a visual response to a stimulus that will enter their receptive field after a saccade but before it actually does so. LIP responses during fixation are thought to represent attentional priority, behavioral relevance, or value. In this study, we test whether the remapped response represents this attentional priority by examining the activity of LIP neurons while animals perform a visual foraging task. We find that the population responds more to a target than to a distractor before the saccade even begins to bring the stimulus into the receptive field. Within 20 ms of the saccade ending, the responses in almost one-third of LIP neurons closely resemble the responses that will emerge during stable fixation. Finally, we show that, in these neurons and in the population as a whole, this remapping occurs for all stimuli in all locations across the visual field and for both long and short saccades. We conclude that this complete remapping of attentional priority across the visual field could underlie spatial stability across saccades.

The role of tactile feedback in grip force during laparoscopic training tasks.

BACKGROUND: Laparoscopic minimally invasive surgery has revolutionized surgical care by reducing trauma to the patient, thereby decreasing the need for medication and shortening recovery times. During open procedures, surgeons can directly feel tissue characteristics. However, in laparoscopic surgery, tactile feedback during grip is attenuated and limited to the resistance felt in the tool handle. Excessive grip force during laparoscopic surgery can lead to tissue damage. Providing additional supplementary tactile feedback may allow subjects to have better control of grip force and identification of tissue characteristics, potentially decreasing the learning curve associated with complex minimally invasive techniques. METHODS: A tactile feedback system has been developed and integrated into a modified laparoscopic grasper that allows forces applied at the grasper tips to be felt by the surgeon's hands. In this study, 15 subjects (11 novices, 4 experts) were asked to perform single-handed peg transfers using these laparoscopic graspers in three trials (feedback OFF, ON, OFF). Peak and average grip forces (newtons) during each grip event were measured and compared using a Wilcoxon ranked test in which each subject served as his or her own control. RESULTS: After activating the tactile feedback system, the novice subject population showed significant decreases in grip force (p < 0.003). When the system was deactivated for the third trial, there were significant increases in grip force (p < 0.003). Expert subjects showed no significant improvements with the addition of tactile feedback (p > 0.05 in all cases). CONCLUSION: Supplementary tactile feedback helped novice subjects reduce grip force during the laparoscopic training task but did not offer improvements for the four expert subjects. This indicates that tactile feedback may be beneficial for laparoscopic training but has limited long-term use in the nonrobotic setting.

A lack of anticipatory remapping of retinotopic receptive fields in the middle temporal area.

The middle temporal (MT) area has traditionally been thought to be a retinotopic area. However, recent functional magnetic resonance imaging and psychophysical evidence have suggested that human MT may have some spatiotopic processing. To gain an understanding of the neural mechanisms underlying this process, we recorded neurons from area MT in awake behaving animals performing a simple saccade task in which a spatially stable moving dot stimulus was presented for 500 ms in one of two locations: the presaccadic receptive field or the postsaccadic receptive field. MT neurons responded as if their receptive fields were purely retinotopic. When the stimulus was placed in the presaccadic receptive field, the response was elevated until the saccade took the stimulus out of the receptive field. When the stimulus was placed in the postsaccadic receptive field, the neuron only began its response after the end of the saccade. No evidence of presaccadic or anticipatory remapping was found. We conclude that gain fields are most likely to be responsible for the spatiotopic signal seen in area MT.

A pure salience response in posterior parietal cortex.

When exploring a visual scene, some objects perceptually popout because of a difference of color, shape, or size. This bottom-up information is an important part of many models describing the allocation of visual attention. It has been hypothesized that the lateral intraparietal area (LIP) acts as a "priority map," integrating bottom-up and top-down information to guide the allocation of attention. Despite a large literature describing top-down influences in LIP, the presence of a pure salience response to a salient stimulus defined by its static features alone has not been reported. We compared LIP responses with colored salient stimuli and distractors in a passive fixation task. Many LIP neurons responded preferentially to 1 of the 2 colored stimuli, yet the mean responses to the salient stimuli were significantly higher than to distractors, independent of the features of the stimuli. These enhanced responses were significant within 75 ms, and the mean responses to salient and distractor stimuli were tightly correlated, suggesting a simple gain control. We propose that a pure salience signal rapidly appears in LIP by collating salience signals from earlier visual areas. This contributes to the creation of a priority map, which is used to guide attention and saccades.

The neural basis of visual attention.

Visual attention is the mechanism the nervous system uses to highlight specific locations, objects or features within the visual field. This can be accomplished by making an eye movement to bring the object onto the fovea (overt attention) or by increased processing of visual information in neurons representing more peripheral regions of the visual field (covert attention). This review will examine two aspects of visual attention: the changes in neural responses within visual cortices due to the allocation of covert attention; and the neural activity in higher cortical areas involved in guiding the allocation of attention. The first section will highlight processes that occur during visual spatial attention and feature-based attention in cortical visual areas and several related models that have recently been proposed to explain this activity. The second section will focus on the parietofrontal network thought to be involved in targeting eye movements and allocating covert attention. It will describe evidence that the lateral intraparietal area, frontal eye field and superior colliculus are involved in the guidance of visual attention, and describe the priority map model, which is thought to operate in at least several of these areas.