A role for visual areas in physics simulations.

To engage with the world, we must regularly make predictions about the outcomes of physical scenes. How do we make these predictions? Recent computational evidence points to simulation-the idea that we can introspectively manipulate rich, mental models of the world-as one explanation for how such predictions are accomplished. However, questions about the potential neural mechanisms of simulation remain. We hypothesized that the process of simulating physical events would evoke imagery-like representations in visual areas of those same events. Using functional magnetic resonance imaging, we find that when participants are asked to predict the likely trajectory of a falling ball, motion-sensitive brain regions are activated. We demonstrate that this activity, which occurs even though no motion is being sensed, resembles activity patterns that arise while participants perceive the ball's motion. This finding thus suggests that mental simulations recreate sensory depictions of how a physical scene is likely to unfold.

Behavioral and oculomotor evidence for visual simulation of object movement.

We regularly interact with moving objects in our environment. Yet, little is known about how we extrapolate the future movements of visually perceived objects. One possibility is that movements are experienced by a mental visual simulation, allowing one to internally picture an object's upcoming motion trajectory, even as the object itself remains stationary. Here we examined this possibility by asking human participants to make judgments about the future position of a falling ball on an obstacle-filled display. We found that properties of the ball's trajectory were highly predictive of subjects' reaction times and accuracy on the task. We also found that the eye movements subjects made while attempting to ascertain where the ball might fall had significant spatiotemporal overlap with those made while actually perceiving the ball fall. These findings suggest that subjects simulated the ball's trajectory to inform their responses. Finally, we trained a convolutional neural network to see whether this problem could be solved by simple image analysis as opposed to the more intricate simulation strategy we propose. We found that while the network was able to solve our task, the model's output did not effectively or consistently predict human behavior. This implies that subjects employed a different strategy for solving our task, and bolsters the conclusion that they were engaging in visual simulation. The current study thus provides support for visual simulation of motion as a means of understanding complex visual scenes and paves the way for future investigations of this phenomenon at a neural level.

The effects of short-term and long-term learning on the responses of lateral intraparietal neurons to visually presented objects.

The lateral intraparietal area (LIP) is thought to play an important role in the guidance of where to look and pay attention. LIP can also respond selectively to differently shaped objects. We sought to understand to what extent short-term and long-term experience with visual orienting determines the responses of LIP to objects of different shapes. We taught monkeys to arbitrarily associate centrally presented objects of various shapes with orienting either toward or away from a preferred spatial location of a neuron. The training could last for less than a single day or for several months. We found that neural responses to objects are affected by such experience, but that the length of the learning period determines how this neural plasticity manifests. Short-term learning affects neural responses to objects, but these effects are only seen relatively late after visual onset; at this time, the responses to newly learned objects resemble those of familiar objects that share their meaning or arbitrary association. Long-term learning affects the earliest bottom-up responses to visual objects. These responses tend to be greater for objects that have been associated with looking toward, rather than away from, LIP neurons' preferred spatial locations. Responses to objects can nonetheless be distinct, although they have been similarly acted on in the past and will lead to the same orienting behavior in the future. Our results therefore indicate that a complete experience-driven override of LIP object responses may be difficult or impossible. We relate these results to behavioral work on visual attention.

Paired neuron recordings in the prefrontal and inferotemporal cortices reveal that spatial selection precedes object identification during visual search.

We addressed the question of how we locate and identify objects in complex natural environments by simultaneously recording single neurons from two brain regions that play different roles in this familiar activity--the frontal eye field (FEF), an area in the prefrontal cortex that is involved in visual spatial selection, and the inferotemporal cortex (IT), which is involved in object recognition--in monkeys performing a covert visual search task. Although the monkeys reported object identity, not location, neural activity specifying target location was evident in FEF before neural activity specifying target identity in IT. These two distinct processes were temporally correlated implying a functional linkage between the end stages of "where" and "what" visual processing and indicating that spatial selection is necessary for the formation of complex object representations associated with visual perception.

The Architecture of Object-Based Attention.

The allocation of attention to objects raises several intriguing questions: What are objects, how does attention access them, what anatomical regions are involved? Here, we review recent progress in the field to determine the mechanisms underlying object-based attention. First, findings from unconscious priming and cueing suggest that the preattentive targets of object-based attention can be fully developed object representations that have reached the level of identity. Next, the control of object-based attention appears to come from ventral visual areas specialized in object analysis that project downward to early visual areas. How feedback from object areas can accurately target the object's specific locations and features is unknown but recent work in autoencoding has made this plausible. Finally, we suggest that the three classic modes of attention may not be as independent as is commonly considered, and instead could all rely on object-based attention. Specifically, studies show that attention can be allocated to the separated members of a group-without affecting the space between them-matching the defining property of feature-based attention. At the same time, object-based attention directed to a single small item has the properties of space-based attention. We outline the architecture of object-based attention, the novel predictions it brings, and discuss how it works in parallel with other attention pathways.

The effects of prefrontal cortex inactivation on object responses of single neurons in the inferotemporal cortex during visual search.

Inferotemporal cortex (IT) is believed to be directly involved in object processing and necessary for accurate and efficient object recognition. The frontal eye field (FEF) is an area in the primate prefrontal cortex that is involved in visual spatial selection and is thought to guide spatial attention and eye movements. We show that object-selective responses of IT neurons and behavioral performance are affected by changes in frontal eye field activity. This was found in monkeys performing a search classification task by temporarily inactivating subregions of FEF while simultaneously recording the activity from single neurons in IT. The effect on object selectivity and performance was specific, occurring in a predictable spatially dependent manner and was strongest when the IT neuron's preferred target was presented in the presence of distractors. FEF inactivation did not affect IT responses on trials in which the nonpreferred target was presented in the search array.

Stimulus selectivity and response latency in putative inhibitory and excitatory neurons of the primate inferior temporal cortex.

The cerebral cortex is composed of many distinct classes of neurons. Numerous studies have demonstrated corresponding differences in neuronal properties across cell types, but these comparisons have largely been limited to conditions outside of awake, behaving animals. Thus the functional role of the various cell types is not well understood. Here, we investigate differences in the functional properties of two widespread and broad classes of cells in inferior temporal cortex of macaque monkeys: inhibitory interneurons and excitatory projection cells. Cells were classified as putative inhibitory or putative excitatory neurons on the basis of their extracellular waveform characteristics (e.g., spike duration). Consistent with previous intracellular recordings in cortical slices, putative inhibitory neurons had higher spontaneous firing rates and higher stimulus-evoked firing rates than putative excitatory neurons. Additionally, putative excitatory neurons were more susceptible to spike waveform adaptation following very short interspike intervals. Finally, we compared two functional properties of each neuron's stimulus-evoked response: stimulus selectivity and response latency. First, putative excitatory neurons showed stronger stimulus selectivity compared with putative inhibitory neurons. Second, putative inhibitory neurons had shorter response latencies compared with putative excitatory neurons. Selectivity differences were maintained and latency differences were enhanced during a visual search task emulating more natural viewing conditions. Our results suggest that short-latency inhibitory responses are likely to sculpt visual processing in excitatory neurons, yielding a sparser visual representation.

Neurons in inferior temporal cortex are sensitive to motion trajectory during degraded object recognition.

Our brains continuously acquire sensory information and make judgments even when visual information is limited. In some circumstances, an ambiguous object can be recognized from how it moves, such as an animal hopping or a plane flying overhead. Yet it remains unclear how movement is processed by brain areas involved in visual object recognition. Here we investigate whether inferior temporal (IT) cortex, an area known for its relevance in visual form processing, has access to motion information during recognition. We developed a matching task that required monkeys to recognize moving shapes with variable levels of shape degradation. Neural recordings in area IT showed that, surprisingly, some IT neurons responded stronger to degraded shapes than clear ones. Furthermore, neurons exhibited motion sensitivity at different times during the presentation of the blurry target. Population decoding analyses showed that motion patterns could be decoded from IT neuron pseudo-populations. Contrary to previous findings, these results suggest that neurons in IT can integrate visual motion and shape information, particularly when shape information is degraded, in a way that has been previously overlooked. Our results highlight the importance of using challenging multifeature recognition tasks to understand the role of area IT in naturalistic visual object recognition.

Evidence for Independent Processing of Shape by Vision and Touch.

Although visual object recognition is well studied and relatively well understood, much less is known about how shapes are recognized by touch and how such haptic stimuli might be compared with visual shapes. One might expect that the processes of visual and haptic object recognition engage similar brain structures given the advantages of avoiding redundant brain circuitry and indeed there is some evidence that this is the case. A potentially fruitful approach to understanding the differences in how shapes might be neurally represented is to find an algorithmic method of comparing shapes, which agrees with human behavior and determines whether that method differs between different modality conditions. If not, it would provide further evidence for a shared representation of shape. We recruited human participants to perform a one-back same-different visual and haptic shape comparison task both within (i.e., comparing two visual shapes or two haptic shapes) and across (i.e., comparing visual with haptic shapes) modalities. We then used various shape metrics to predict performance based on the shape, orientation, and modality of the two stimuli that were being compared on each trial. We found that the metrics that best predict shape comparison behavior heavily depended on the modality of the two shapes, suggesting differences in which features are used for comparing shapes depending on modality and that object recognition is not necessarily performed in a single, modality-agnostic region.

Temporal cortex neurons encode articulated actions as slow sequences of integrated poses.

Form and motion processing pathways of the primate visual system are known to be interconnected, but there has been surprisingly little investigation of how they interact at the cellular level. Here we explore this issue with a series of three electrophysiology experiments designed to reveal the sources of action selectivity in monkey temporal cortex neurons. Monkeys discriminated between actions performed by complex, richly textured, rendered bipedal figures and hands. The firing patterns of neurons contained enough information to discriminate the identity of the character, the action performed, and the particular conjunction of action and character. This suggests convergence of motion and form information within single cells. Form and motion information in isolation were both sufficient to drive action discrimination within these neurons, but removing form information caused a greater disruption to the original response. Finally, we investigated the temporal window across which visual information is integrated into a single pose (or, equivalently, the timing with which poses are differentiated). Temporal cortex neurons under normal conditions represent actions as sequences of poses integrated over approximately 120 ms. They receive both motion and form information, however, and can use either if the other is absent.

Neural dynamics in inferior temporal cortex during a visual working memory task.

Intelligent organisms are capable of tracking objects even when they temporarily disappear from sight, a cognitive capacity commonly referred to as visual working memory (VWM). The neural basis of VWM has been the subject of significant scientific debate, with recent work focusing on the relative roles of posterior visual areas, such as the inferior temporal cortex (ITC), and the prefrontal cortex. Here we reexamined the contribution of ITC to VWM by recording from highly selective individual ITC neurons as monkeys engaged in multiple versions of an occlusion-based memory task. As expected, we found strong evidence for a role of ITC in stimulus encoding. We also found that almost half of these selective cells showed stimulus-selective delay period modulation, with a small but significant fraction exhibiting differential responses even in the presence of simultaneously visible interfering information. When we combined the informational content of multiple neurons, we found that the accuracy with which we could decode memory content increased drastically. The memory epoch analyses suggest that behaviorally relevant visual memories were reinstated in ITC. Furthermore, we observed a population-wide enhancement of neuronal response to a match stimulus compared with the same stimulus presented as a nonmatch. The single-cell enhancement preceded any match effects identified in the local field potential, leading us to speculate that enhancement is the result of neural processing local to ITC. Moreover, match enhancement was only later followed by the more commonly observed match suppression. Altogether, the data support the hypothesis that, when a stimulus is held in memory, ITC neurons are actively biased in favor of task-relevant visual representations and that this bias can immediately impact subsequent recognition events.

Activity of inferior temporal cortical neurons predicts recognition choice behavior and recognition time during visual search.

Although the selectivity for complex stimuli exhibited by neurons in inferior temporal cortex is often taken as evidence of their role in visual perception, few studies have directly tested this hypothesis. Here, we sought to create a relatively natural task with few behavioral constraints to test whether activity in inferior temporal cortex neurons predicts whether or not a monkey will recognize and respond to a complex visual object. Monkeys were trained to freely view an array of images and report the presence of one of many possible target images previously associated with a hand response. On certain trials, the identity of the target was swapped during the monkeys' targeting saccade. Furthermore, the response association of the preswap target and the postswap target differed (e.g., right-to-left target swap). Neural activity in cells selective for the preswap target was significantly higher when the monkeys' response matched the hand association of the preswap target. Furthermore, the monkeys' response time was predicted by the magnitude of the presaccadic firing rate on nonswap trials. Our results provide additional support for the role of inferior temporal cortex in object recognition during natural behavior.

Context familiarity enhances target processing by inferior temporal cortex neurons.

Experience-dependent changes in the response properties of ventral visual stream neurons are thought to underlie our ability to rapidly and efficiently recognize visual objects. How these neural changes are related to efficient visual processing during natural vision remains unclear. Here, we demonstrate a neurophysiological correlate of efficient visual search through highly familiar object arrays. Humans and monkeys are faster at locating the same target when it is surrounded by familiar compared with unfamiliar distractors. We show that this behavioral enhancement is driven by an increased sensitivity of target-selective neurons in inferior temporal cortex. This results from an increased "signal" for target representations and decreased "noise" from neighboring familiar distractors. These data highlight the dynamic properties of the inferior temporal cortex neurons and add to a growing body of evidence demonstrating how experience shapes neural processing in the ventral visual stream.

Effects of temporal context and temporal expectancy on neural activity in inferior temporal cortex.

Timing is critical. The same event can mean different things at different times and some events are more likely to occur at one time than another. We used a cued visual classification task to evaluate how changes in temporal context affect neural responses in inferior temporal cortex, an extrastriate visual area known to be involved in object processing. On each trial a first image cued a temporal delay before a second target image appeared. The animal's task was to classify the second image by pressing one of two buttons previously associated with that target. All images were used as both cues and targets. Whether an image cued a delay time or signaled a button press depended entirely upon whether it was the first or second picture in a trial. This paradigm allowed us to compare inferior temporal cortex neural activity to the same image subdivided by temporal context and expectation. Neuronal spiking was more robust and visually evoked local field potentials (LFP's) larger for target presentations than for cue presentations. On invalidly cued trials, when targets appeared unexpectedly early, the magnitude of the evoked LFP was reduced and delayed and neuronal spiking was attenuated. Spike field coherence increased in the beta-gamma frequency range for expected targets. In conclusion, different neural responses in higher order ventral visual cortex may occur for the same visual image based on manipulations of temporal attention.

The role of category learning in the acquisition and retention of perceptual expertise: a behavioral and neurophysiological study.

This study examined the neural mechanisms underlying perceptual categorization and expertise. Participants were either exposed to or learned to classify three categories of cars (sedans, SUVs, antiques) at either the basic or subordinate level. Event-Related Potentials (ERPs) as well as accuracy and reaction time were recorded before, immediately after, and 1-week after training. Behavioral results showed that only subordinate-level training led to better discrimination of trained cars, and this ability was retained a week after training. ERPs showed an equivalent increase in the N170 across all three training conditions whereas the N250 was only enhanced in response to subordinate-level training. The behavioral and electrophysiological results distinguish category learning at the subordinate level from category learning occurring at the basic level or from simple exposure. Together with data from previous investigations, the current results suggest that subordinate-level training, but not basic-level or exposure training, leads to expert-like improvements in categorization accuracy. These improvements are mirrored by changes in the N250 rather than the N170 component, and these effects persist at least a week after training, so are conceivably related to long-term learning processes supporting perceptual expertise.

A method for the real-time rendering of formless dot field structure-from-motion stimuli.

The perception of visual motion relies on different computations and different neural substrates than the perception of static form. It is therefore useful to have psychophysical stimuli that carry mostly or entirely motion information, conveying little or nothing about form in any single frame. Structure-from-motion stimuli can sometimes achieve this dissociation, with some examples in studies of biological motion using point-light walkers. It is, however, generally not trivial to provide motion information without also providing static form information. The problem becomes more computationally difficult when the structures and the motions in question are complex. Here we present a technique by which an animated three-dimensional scene can be rendered in real-time as a pattern of dots. Each dot follows the trajectory of the underlying object in the animation, but each static frame of the animation appears to be a uniform random field of dots. The resulting stimuli capture motion vectors across arbitrary complex scenes, while providing virtually no instantaneous information about the structure of that scene. We also present the results of a psychophysical experiment demonstrating the efficacy and the limitations of the technique. The ability to create such stimuli on the fly allows for interactive adjustment and control of the stimuli, real-time parametric variations of structure and motion, and the creation of large libraries of actions without the need to pre-render a prohibitive number of movies. This technique provides a powerful tool for the dissociation of complex motion from static form.

Spike count reliability and the Poisson hypothesis.

The variability of cortical activity in response to repeated presentations of a stimulus has been an area of controversy in the ongoing debate regarding the evidence for fine temporal structure in nervous system activity. We present a new statistical technique for assessing the significance of observed variability in the neural spike counts with respect to a minimal Poisson hypothesis, which avoids the conventional but troubling assumption that the spiking process is identically distributed across trials. We apply the method to recordings of inferotemporal cortical neurons of primates presented with complex visual stimuli. On this data, the minimal Poisson hypothesis is rejected: the neuronal responses are too reliable to be fit by a typical firing-rate model, even allowing for sudden, time-varying, and trial-dependent rate changes after stimulus onset. The statistical evidence favors a tightly regulated stimulus response in these neurons, close to stimulus onset, although not further away.

A multielectrode study of the inferotemporal cortex in the monkey: effects of grouping on spike rates and synchrony.

We measured spiking activity from 175 pairs of visually selective neurons while a monkey performed a visual classification task in which the same image pairs appeared in informative and uninformative contexts. Spike counts and synchrony counts were higher during fixations of informative configurations. Spike field coherence showed low frequency (approximately 5-15 Hz) desynchronization when viewing informative stimulus configurations. Jitter statistics, which quantify synchrony and control for spike rates, showed less probable, more surprising patterns of synchronies when the monkey made the correct response than when he erred. Our results suggest that changes in spike counts parallel changes in raw synchrony counts and that both track aspects of stimulus context. Synchrony unexplained by rates changes, however, also occurs and correlates with performance.

XOR style tasks for testing visual object processing in monkeys.

Using visually complex stimuli, three monkeys learned visual exclusive-or (XOR) tasks that required detecting two way visual feature conjunctions. Monkeys with passive exposure to the test images, or prior experience, were quicker to acquire an XOR style task. Training on each pairwise comparison of the stimuli to be used in an XOR task provided nearly complete transfer when stimuli became intermingled in the full XOR task. Task mastery took longer, accuracy was lower, and response times were slower for conjunction stimuli. Rotating features of the XOR stimuli did not adversely effect recognition speed or accuracy.

Initial saccades predict manual recognition choices in the monkey.

In animals with specialized foveae, eye position has a direct influence over the acquisition of detailed visual information. At the same time, eye movements executed during natural behaviors are closely linked with motor actions. In this study, we investigated patterns of eye movements during a simple visual discrimination task. Three rhesus monkeys learned to recognize images of real world objects with no explicit constraints on eye position. Analysis of the monkeys' eye movements showed that although the endpoint of the initial saccade depended on the particular visual stimulus, the trajectory of the first saccades also reliably predicted the manual response associated with that stimulus. We thus observed that initial saccades executed in a recognition task reflect both perceptual and motor aspects of a visual task. This pattern of eye movements emerged spontaneously in all three animals tested despite the fact that saccades were never explicitly rewarded. As the average saccade latency was under 200ms, object specific learned associations must have exerted their influence over the initial saccade even earlier, providing a novel temporal marker for the rapidity of visual recognition processes. Taken together, these results suggest that caution should be exercised when interpreting the meaning of oculomotor patterns observed during perceptual tasks, as these blur the line between perceptual processing and motor preparation.