Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences.

What specific features should visual neurons encode, given the infinity of real-world images and the limited number of neurons available to represent them? We investigated neuronal selectivity in monkey inferotemporal cortex via the vast hypothesis space of a generative deep neural network, avoiding assumptions about features or semantic categories. A genetic algorithm searched this space for stimuli that maximized neuronal firing. This led to the evolution of rich synthetic images of objects with complex combinations of shapes, colors, and textures, sometimes resembling animals or familiar people, other times revealing novel patterns that did not map to any clear semantic category. These results expand our conception of the dictionary of features encoded in the cortex, and the approach can potentially reveal the internal representations of any system whose input can be captured by a generative model.

Macaques recognize features in synthetic images derived from ventral stream neurons.

Primates can recognize features in virtually all types of images, an ability that still requires a comprehensive computational explanation. One hypothesis is that visual cortex neurons learn patterns from scenes, objects, and textures, and use these patterns to interpolate incoming visual information. We have used machine learning algorithms to instantiate visual patterns stored by neurons-we call these highly activating images prototypes. Prototypes from inferotemporal (IT) neurons often resemble parts of real-world objects, such as monkey faces and body parts, a similarity established via pretrained neural networks [C. R. Ponce et al., Cell 177, 999-1009.e10 (2019)] and naïve human participants [A. Bardon, W. Xiao, C. R. Ponce, M. S. Livingstone, G. Kreiman, Proc. Natl. Acad. Sci. U.S.A. 119, e2118705119 (2022)]. However, it is not known whether monkeys themselves perceive similarities between neuronal prototypes and real-world objects. Here, we investigated whether monkeys reported similarities between prototypes and real-world objects using a two-alternative forced choice task. We trained the animals to saccade to synthetic images of monkeys, and subsequently tested how they classified prototypes synthesized from IT and primary visual cortex (V1). We found monkeys classified IT prototypes as conspecifics more often than they did random generator images and V1 prototypes, and their choices were partially predicted by convolutional neural networks. Further, we confirmed that monkeys could abstract general shape information from images of real-world objects. Finally, we verified these results with human participants. Our results provide further evidence that prototypes from cortical neurons represent interpretable abstractions from the visual world.

Face neurons encode nonsemantic features.

The primate inferior temporal cortex contains neurons that respond more strongly to faces than to other objects. Termed "face neurons," these neurons are thought to be selective for faces as a semantic category. However, face neurons also partly respond to clocks, fruits, and single eyes, raising the question of whether face neurons are better described as selective for visual features related to faces but dissociable from them. We used a recently described algorithm, XDream, to evolve stimuli that strongly activated face neurons. XDream leverages a generative neural network that is not limited to realistic objects. Human participants assessed images evolved for face neurons and for nonface neurons and natural images depicting faces, cars, fruits, etc. Evolved images were consistently judged to be distinct from real faces. Images evolved for face neurons were rated as slightly more similar to faces than images evolved for nonface neurons. There was a correlation among natural images between face neuron activity and subjective "faceness" ratings, but this relationship did not hold for face neuron­evolved images, which triggered high activity but were rated low in faceness. Our results suggest that so-called face neurons are better described as tuned to visual features rather than semantic categories.

End-Stopping Predicts Curvature Tuning along the Ventral Stream.

Neurons in primate inferotemporal cortex (IT) are clustered into patches of shared image preferences. Functional imaging has shown that these patches are activated by natural categories (e.g., faces, body parts, and places), artificial categories (numerals, words) and geometric features (curvature and real-world size). These domains develop in the same cortical locations across monkeys and humans, which raises the possibility of common innate mechanisms. Although these commonalities could be high-level template-based categories, it is alternatively possible that the domain locations are constrained by low-level properties such as end-stopping, eccentricity, and the shape of the preferred images. To explore this, we looked for correlations among curvature preference, receptive field (RF) end-stopping, and RF eccentricity in the ventral stream. We recorded from sites in V1, V4, and posterior IT (PIT) from six monkeys using microelectrode arrays. Across all visual areas, we found a tendency for end-stopped sites to prefer curved over straight contours. Further, we found a progression in population curvature preferences along the visual hierarchy, where, on average, V1 sites preferred straight Gabors, V4 sites preferred curved stimuli, and many PIT sites showed a preference for curvature that was concave relative to fixation. Our results provide evidence that high-level functional domains may be mapped according to early rudimentary properties of the visual system. SIGNIFICANCE STATEMENT: The macaque occipitotemporal cortex contains clusters of neurons with preferences for categories such as faces, body parts, and places. One common question is how these clusters (or "domains") acquire their cortical position along the ventral stream. We and other investigators previously established an fMRI-level correlation among these category domains, retinotopy, and curvature preferences: for example, in inferotemporal cortex, face- and curvature-preferring domains show a central visual field bias whereas place- and rectilinear-preferring domains show a more peripheral visual field bias. Here, we have found an electrophysiological-level explanation for the correlation among domain preference, curvature, and retinotopy based on neuronal preference for short over long contours, also called end-stopping.

Posterior Inferotemporal Cortex Cells Use Multiple Input Pathways for Shape Encoding.

In the macaque monkey brain, posterior inferior temporal (PIT) cortex cells contribute to visual object recognition. They receive concurrent inputs from visual areas V4, V3, and V2. We asked how these different anatomical pathways shape PIT response properties by deactivating them while monitoring PIT activity in two male macaques. We found that cooling of V4 or V2|3 did not lead to consistent changes in population excitatory drive; however, population pattern analyses showed that V4-based pathways were more important than V2|3-based pathways. We did not find any image features that predicted decoding accuracy differences between both interventions. Using the HMAX hierarchical model of visual recognition, we found that different groups of simulated "PIT" units with different input histories (lacking "V2|3" or "V4" input) allowed for comparable levels of object-decoding performance and that removing a large fraction of "PIT" activity resulted in similar drops in performance as in the cooling experiments. We conclude that distinct input pathways to PIT relay similar types of shape information, with V1-dependent V4 cells providing more quantitatively useful information for overall encoding than cells in V2 projecting directly to PIT.SIGNIFICANCE STATEMENT Convolutional neural networks are the best models of the visual system, but most emphasize input transformations across a serial hierarchy akin to the primary "ventral stream" (V1 → V2 → V4 → IT). However, the ventral stream also comprises parallel "bypass" pathways: V1 also connects to V4, and V2 to IT. To explore the advantages of mixing long and short pathways in the macaque brain, we used cortical cooling to silence inputs to posterior IT and compared the findings with an HMAX model with parallel pathways.

Contributions of indirect pathways to visual response properties in macaque middle temporal area MT.

The primate visual cortex exhibits a remarkable degree of interconnectivity. Each visual area receives an average of 10 to 15 inputs, many of them from cortical areas with overlapping, but not identical, functional properties. In this study, we assessed the functional significance of this anatomical parallelism to the middle temporal area (MT) of the macaque visual cortex. MT receives major feedforward inputs from areas V1, V2, and V3, but little is known about the properties of each of these pathways. We previously demonstrated that reversible inactivation of V2 and V3 causes a disproportionate degradation of tuning for binocular disparity of MT neurons, relative to direction tuning (Ponce et al., 2008). Here we show that MT neurons continued to encode speed and size information during V2/3 inactivation; however, many became significantly less responsive to fast speeds and others showed a modest decrease in surround suppression. These changes resemble previously reported effects of reducing stimulus contrast (Pack et al., 2005; Krekelberg et al., 2006), but we show here that they differ in their temporal dynamics. We find no evidence that the indirect pathways selectively target different functional regions within MT. Overall, our findings suggest that the indirect pathways to MT primarily convey modality-specific information on binocular disparity, but that they also contribute to the processing of stimuli moving at fast speeds.

Integrating motion and depth via parallel pathways.

Processing of visual information is both parallel and hierarchical, with each visual area richly interconnected with other visual areas. An example of the parallel architecture of the primate visual system is the existence of two principal pathways providing input to the middle temporal visual area (MT): namely, a direct projection from striate cortex (V1), and a set of indirect projections that also originate in V1 but then relay through V2 and V3. Here we have reversibly inactivated the indirect pathways while recording from MT neurons and measuring eye movements in alert monkeys, a procedure that has enabled us to assess whether the two different input pathways are redundant or whether they carry different kinds of information. We find that this inactivation causes a disproportionate degradation of binocular disparity tuning relative to direction tuning in MT neurons, suggesting that the indirect pathways are important in the recovery of depth in three-dimensional scenes.

Tuning landscapes of the ventral stream.

A goal in visual neuroscience is to explain how neurons respond to natural scenes. However, neurons are generally tested using simpler stimuli, often because they can be transformed smoothly, allowing the measurement of tuning functions (i.e., response peaks and slopes). Here, we test the idea that all classic tuning curves can be viewed as slices of a higher-dimensional tuning landscape. We use activation-maximizing stimuli ("prototypes") as landmarks in a generative image space and map tuning functions around these peaks. We find that neurons show smooth bell-shaped tuning consistent with radial basis functions, spanning a vast image transformation range, with systematic differences in landscape geometry from V1 to inferotemporal cortex. By modeling these trends, we infer that neurons in the higher visual cortex have higher intrinsic feature dimensionality. Overall, these results suggest that visual neurons are better viewed as signaling distances to prototypes on an image manifold.

Visual prototypes in the ventral stream are attuned to complexity and gaze behavior.

Early theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours. This view was validated by the discovery that neurons in posterior visual cortex respond to edges and curvature. Still, it remains unclear what other information-rich features are encoded by neurons in more anterior cortical regions (e.g., inferotemporal cortex). Here, we use a generative deep neural network to synthesize images guided by neuronal responses from across the visuocortical hierarchy, using floating microelectrode arrays in areas V1, V4 and inferotemporal cortex of two macaque monkeys. We hypothesize these images ("prototypes") represent such predicted information-rich features. Prototypes vary across areas, show moderate complexity, and resemble salient visual attributes and semantic content of natural images, as indicated by the animals' gaze behavior. This suggests the code for object recognition represents compressed features of behavioral relevance, an underexplored aspect of efficient coding.

Seeing faces is necessary for face-domain formation.

Here we report that monkeys raised without exposure to faces did not develop face domains, but did develop domains for other categories and did show normal retinotopic organization, indicating that early face deprivation leads to a highly selective cortical processing deficit. Therefore, experience must be necessary for the formation (or maintenance) of face domains. Gaze tracking revealed that control monkeys looked preferentially at faces, even at ages prior to the emergence of face domains, but face-deprived monkeys did not, indicating that face looking is not innate. A retinotopic organization is present throughout the visual system at birth, so selective early viewing behavior could bias category-specific visual responses toward particular retinotopic representations, thereby leading to domain formation in stereotyped locations in inferotemporal cortex, without requiring category-specific templates or biases. Thus, we propose that environmental importance influences viewing behavior, viewing behavior drives neuronal activity, and neuronal activity sculpts domain formation.

Temporal evolution of 2-dimensional direction signals used to guide eye movements.

The smooth pursuit system must integrate many local motion measurements into a coherent estimate of target velocity. Several laboratories have studied this integration process using eye movements elicited by targets, such as tilted bars, containing conflicts between local motion signals measured along contours [one dimensional (1D)] and those measured at the bar's endpoints, or terminators [two dimensional (2D)]. The general finding is that 1D signals dominate early responses, whereas later components of the behavior are determined by 2D signals. We studied the dynamics of the integration process in macaque monkeys by systematically varying the relative proportions of 1D and 2D signals and the retinal eccentricities at which they appeared. Predictably, longer bars produced greater and longer-lasting contour-induced deviations. The evolution of the 2D response occurred over a period of 50-400 ms, depending on the relative proportions of 1D and 2D signals. As contours were displaced from the fovea the deviation decreased but much less so for early (1st 40 ms) than for late (subsequent 40 ms) pursuit initiation. These bottom-up effects could be overcome to a limited extent by the top-down influence of predictability. Finally, we observed that when animals were free to track any part of the bar, they spontaneously made short-latency saccades to the terminators on most trials, especially when the bars were tilted. This suggests an increased saliency of moving terminators, particularly when discrepancies exist among local motion signals.