Neural representations for object perception: structure, category, and adaptive coding.

Object perception is one of the most remarkable capacities of the primate brain. Owing to the large and indeterminate dimensionality of object space, the neural basis of object perception has been difficult to study and remains controversial. Recent work has provided a more precise picture of how 2D and 3D object structure is encoded in intermediate and higher-level visual cortices. Yet, other studies suggest that higher-level visual cortex represents categorical identity rather than structure. Furthermore, object responses are surprisingly adaptive to changes in environmental statistics, implying that learning through evolution, development, and also shorter-term experience during adulthood may optimize the object code. Future progress in reconciling these findings will depend on more effective sampling of the object domain and direct comparison of these competing hypotheses.

Curvature processing dynamics in macaque area V4.

We have previously analyzed shape processing dynamics in macaque monkey posterior inferotemporal cortex (PIT). We described how early PIT responses to individual contour fragments evolve into tuning for multifragment shape configurations. Here, we analyzed curvature processing dynamics in area V4, which provides feedforward inputs to PIT. We contrasted 2 hypotheses: 1) that V4 curvature tuning evolves from tuning for simpler elements, analogous to PIT shape synthesis and 2) that V4 curvature tuning emerges immediately, based on purely feedforward mechanisms. Our results clearly supported the first hypothesis. Early V4 responses carried information about individual contour orientations. Tuning for multiorientation (curved) contours developed gradually over ∼50 ms. Together, the current and previous results suggest a partial sequence for shape synthesis in ventral pathway cortex. We propose that early orientation signals are synthesized into curved contour fragment representations in V4 and that these signals are transmitted to PIT, where they are then synthesized into multifragment shape representations. The observed dynamics might additionally or alternatively reflect influences from earlier (V1, V2) and later (central and anterior IT) processing stages in the ventral pathway. In either case, the dynamics of contour information in V4 and PIT appear to reflect a sequential hierarchical process of shape synthesis.

How Shape Perception Works, in Two Dimensions and Three Dimensions.

The ventral visual pathway transforms retinal images into neural representations that support object understanding, including exquisite appreciation of precise 2D pattern shape and 3D volumetric shape. We articulate a framework for understanding the goals of this transformation and how they are achieved by neural coding at successive ventral pathway stages. The critical goals are (a) radical compression to make shape information communicable across axonal bundles and storable in memory, (b) explicit coding to make shape information easily readable by the rest of the brain and thus accessible for cognition and behavioral control, and (c) representational stability to maintain consistent perception across highly variable viewing conditions. We describe how each transformational step in ventral pathway vision serves one or more of these goals. This three-goal framework unifies discoveries about ventral shape processing into a neural explanation for our remarkable experience of shape as a vivid, richly detailed aspect of the natural world.

An inferotemporal coding strategy robust to partial object occlusion.

Object coding in primate ventral pathway cortex progresses in sparseness/compression/efficiency, from many orientation signals in V1, to fewer 2D/3D part signals in V4, to still fewer multi-part configuration signals in AIT (anterior inferotemporal cortex). 1-11 This progression could lead to individual neurons exclusively selective for unique objects, the sparsest code for identity, especially for highly familiar, important objects. 12-18 To test this, we trained macaque monkeys to discriminate 8 simple letter-like shapes in a match-to-sample task, a design in which one-to-one coding of letters by neurons could streamline behavior. Performance increased from chance to >80% correct over a period of weeks, after which AIT neurons showed clear learning effects, with increased selectivity for multi-part configurations within the trained alphabet shapes. But these neurons were not exclusively tuned for unique letters based on training, since their responsiveness generalized to different, non-trained shapes containing the same configurations. This multi-part configuration coding limit in AIT is not maximally sparse, but it could explain the robustness of primate vision to partial object occlusion, which is common in the natural world and problematic for computer vision. Multi-part configurations are highly diagnostic of identity, and neural signals for various partial object structures can provide different but equally sufficient evidence for whole object identity across most occlusion conditions.

Representation of tactile curvature in macaque somatosensory area 2.

Tactile shape information is elaborated in a cortical hierarchy spanning primary (SI) and secondary somatosensory cortex (SII). Indeed, SI neurons in areas 3b and 1 encode simple contour features such as small oriented bars and edges, whereas higher order SII neurons represent large curved contour features such as angles and arcs. However, neural coding of these contour features has not been systematically characterized in area 2, the most caudal SI subdivision in the postcentral gyrus. In the present study, we analyzed area 2 neural responses to embossed oriented bars and curved contour fragments to establish whether curvature representations are generated in the postcentral gyrus. We found that many area 2 neurons (26 of 112) exhibit clear curvature tuning, preferring contours pointing in a particular direction. Fewer area 2 neurons (15 of 112) show preferences for oriented bars. Because area 2 response patterns closely resembled SII patterns, we also compared area 2 and SII response time courses to characterize the temporal dynamics of curvature synthesis in the somatosensory system. We found that curvature representations develop and peak concurrently in area 2 and SII. These results reveal that transitions from orientation tuning to curvature selectivity in the somatosensory cortical hierarchy occur within SI rather than between SI and SII.

Multi-finger receptive field properties in primary somatosensory cortex: A revised account of the spatiotemporal integration functions of area 3b.

The leading view in the somatosensory system indicates that area 3b serves as a cortical relay site that primarily encodes (cutaneous) tactile features limited to individual digits. Our recent work argues against this model by showing that area 3b cells can integrate both cutaneous and proprioceptive information from the hand. Here, we further test the validity of this model by studying multi-digit (MD) integration properties in area 3b. In contrast to the prevailing view, we show that most cells in area 3b have a receptive field (RF) that extends to multiple digits, with the size of the RF (i.e., the number of responsive digits) increasing across time. Further, we show that MD cells' orientation angle preference is highly correlated across digits. Taken together, these data show that area 3b plays a larger role in generating neural representations of tactile objects, as opposed to just being a "feature detector" relay site.

Object representation in a gravitational reference frame.

When your head tilts laterally, as in sports, reaching, and resting, your eyes counterrotate less than 20%, and thus eye images rotate, over a total range of about 180°. Yet, the world appears stable and vision remains normal. We discovered a neural strategy for rotational stability in anterior inferotemporal cortex (IT), the final stage of object vision in primates. We measured object orientation tuning of IT neurons in macaque monkeys tilted +25 and -25° laterally, producing ~40° difference in retinal image orientation. Among IT neurons with consistent object orientation tuning, 63% remained stable with respect to gravity across tilts. Gravitational tuning depended on vestibular/somatosensory but also visual cues, consistent with previous evidence that IT processes scene cues for gravity's orientation. In addition to stability across image rotations, an internal gravitational reference frame is important for physical understanding of a world where object position, posture, structure, shape, movement, and behavior interact critically with gravity.

Analogous intermediate shape coding in vision and touch.

We recognize, understand, and interact with objects through both vision and touch. Conceivably, these two sensory systems encode object shape in similar ways, which could facilitate cross-modal communication. To test this idea, we studied single neurons in macaque monkey intermediate visual (area V4) and somatosensory (area SII) cortex, using matched shape stimuli. We found similar patterns of shape sensitivity characterized by tuning for curvature direction. These parallel tuning patterns imply analogous shape coding mechanisms in intermediate visual and somatosensory cortex.

Early Emergence of Solid Shape Coding in Natural and Deep Network Vision.

Area V4 is the first object-specific processing stage in the ventral visual pathway, just as area MT is the first motion-specific processing stage in the dorsal pathway. For almost 50 years, coding of object shape in V4 has been studied and conceived in terms of flat pattern processing, given its early position in the transformation of 2D visual images. Here, however, in awake monkey recording experiments, we found that roughly half of V4 neurons are more tuned and responsive to solid, 3D shape-in-depth, as conveyed by shading, specularity, reflection, refraction, or disparity cues in images. Using 2-photon functional microscopy, we found that flat- and solid-preferring neurons were segregated into separate modules across the surface of area V4. These findings should impact early shape-processing theories and models, which have focused on 2D pattern processing. In fact, our analyses of early object processing in AlexNet, a standard visual deep network, revealed a similar distribution of sensitivities to flat and solid shape in layer 3. Early processing of solid shape, in parallel with flat shape, could represent a computational advantage discovered by both primate brain evolution and deep-network training.

Dynamic shape synthesis in posterior inferotemporal cortex.

How does the brain synthesize low-level neural signals for simple shape parts into coherent representations of complete objects? Here, we present evidence for a dynamic process of object part integration in macaque posterior inferotemporal cortex (IT). Immediately after stimulus onset, neural responses carried information about individual object parts (simple contour fragments) only. Subsequently, information about specific multipart configurations emerged, building gradually over the course of approximately 60 ms, producing a sparser and more explicit representation of object shape. We show that this gradual transformation can be explained by a recurrent network process that effectively compares parts signals across neurons to generate inferences about multipart shape configurations.

Transformation of shape information in the ventral pathway.

Object perception seems effortless to us, but it depends on intensive neural processing across multiple stages in ventral pathway visual cortex. Shape information at the retinal level is hopelessly complex, variable and implicit. The ventral pathway must somehow transform retinal signals into much more compact, stable and explicit representations of object shape. Recent findings highlight key aspects of this transformation: higher-order contour derivatives, structural representation in object-based coordinates, composite shape tuning dimensions, and long-term storage of object knowledge. These coding principles could help to explain our remarkable ability to perceive, distinguish, remember and understand a virtual infinity of objects.

Vision: Dialogues between Deep Networks and the Brain.

Two new papers show how deep neural networks interacting with the brain can generate visual images that drive surprisingly strong neural responses. These images are tantalizing reflections of visual information in the brain.

Active vision and visual activation in area V4.

During normal vision, the focus of gaze continually jumps from one important image feature to the next. In this issue of Neuron, Mazer and Gallant analyze neural activity in higher-level visual cortex during this kind of active visual exploration, and they demonstrate a localized enhancement of visual responses that predicts the target of the upcoming eye movement.

Population coding of shape in area V4.

Shape is represented in the visual system by patterns of activity across populations of neurons. We studied the population code for shape in area V4 of macaque monkeys, which is part of the ventral (object-related) pathway in primate visual cortex. We have previously found that many macaque V4 neurons are tuned for the curvature and object-centered position of boundary fragments (such as 'concavity on the right'). Here we tested the hypothesis that populations of such cells represent complete shapes as aggregates of boundary fragments. To estimate the population representation of a given shape, we scaled each cell's tuning peak by its response to that shape, summed across cells and smoothed. The resulting population response surface contained 3-8 peaks that represented major boundary features and could be used to reconstruct (approximately) the original shape. This exemplifies how a multi-peaked neural population response can represent a complex stimulus in terms of its constituent elements.

Underlying principles of visual shape selectivity in posterior inferotemporal cortex.

Object perception depends on shape processing in the ventral visual pathway, which in monkeys culminates in inferotemporal cortex (IT). Here we provide a description of fundamental quantitative principles governing neural selectivity for complex shape in IT. By measuring responses to large, parametric sets of two-dimensional (2D) silhouette shapes, we found that neurons in posterior IT (Brodmann's areas TEO and posterior TE) integrate information about multiple contour elements (straight and curved edge fragments of the type represented in lower-level areas) using both linear and nonlinear mechanisms. This results in complex, distributed response patterns that cannot be characterized solely in terms of example stimuli. We explained these response patterns with tuning functions in multidimensional shape space and accurately predicted neural responses to the widely varying shapes in our stimulus set. Integration of contour element information in earlier stages of IT represents an important step in the transformation from low-level shape signals to complex object representation.

Three-dimensional orientation tuning in macaque area V4.

Tuning for the orientation of elongated, linear image elements (edges, bars, gratings), first discovered by Hubel and Wiesel, is considered a key feature of visual processing in the brain. It has been studied extensively in two dimensions (2D) using frontoparallel stimuli, but in real life most lines, edges and contours are slanted with respect to the viewer. Here we report that neurons in macaque area V4, an intermediate stage in the ventral (object-related) pathway of visual cortex, were tuned for 3D orientation--that is,for specific slants as well as for 2D orientation. The tuning for 3D orientation was consistent across depth position (binocular disparity) and position within the 2D classical receptive field. The existence of 3D orientation signals in the ventral pathway suggests that the brain may use such information to interpret 3D shape.

First-Pass Processing of Value Cues in the Ventral Visual Pathway.

Real-world value often depends on subtle, continuously variable visual cues specific to particular object categories, like the tailoring of a suit, the condition of an automobile, or the construction of a house. Here, we used microelectrode recording in behaving monkeys to test two possible mechanisms for category-specific value-cue processing: (1) previous findings suggest that prefrontal cortex (PFC) identifies object categories, and based on category identity, PFC could use top-down attentional modulation to enhance visual processing of category-specific value cues, providing signals to PFC for calculating value, and (2) a faster mechanism would be first-pass visual processing of category-specific value cues, immediately providing the necessary visual information to PFC. This, however, would require learned mechanisms for processing the appropriate cues in a given object category. To test these hypotheses, we trained monkeys to discriminate value in four letter-like stimulus categories. Each category had a different, continuously variable shape cue that signified value (liquid reward amount) as well as other cues that were irrelevant. Monkeys chose between stimuli of different reward values. Consistent with the first-pass hypothesis, we found early signals for category-specific value cues in area TE (the final stage in monkey ventral visual pathway) beginning 81 ms after stimulus onset-essentially at the start of TE responses. Task-related activity emerged in lateral PFC approximately 40 ms later and consisted mainly of category-invariant value tuning. Our results show that, for familiar, behaviorally relevant object categories, high-level ventral pathway cortex can implement rapid, first-pass processing of category-specific value cues.

Visual attention: bottom-up versus top-down.

Visual attention is attracted by salient stimuli that 'pop out' from their surroundings. Attention can also be voluntarily directed to objects of current importance to the observer. What happens in the brain when these two processes interact?

Integration of objects and space in perception and memory.

Distinct processing of objects and space has been an organizing principle for studying higher-level vision and medial temporal lobe memory. Here, however, we discuss how object and spatial information are in fact closely integrated in vision and memory. The ventral, object-processing visual pathway carries precise spatial information, transformed from retinotopic coordinates into relative dimensions. At the final stages of the ventral pathway, including the dorsal anterior temporal lobe (TEd), object-sensitive neurons are intermixed with neurons that process large-scale environmental space. TEd projects primarily to perirhinal cortex (PRC), which in turn projects to lateral entorhinal cortex (LEC). PRC and LEC also combine object and spatial information. For example, PRC and LEC neurons exhibit place fields that are evoked by landmark objects or the remembered locations of objects. Thus, spatial information, on both local and global scales, is deeply integrated into the ventral (temporal) object-processing pathway in vision and memory.