Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences.

What specific features should visual neurons encode, given the infinity of real-world images and the limited number of neurons available to represent them? We investigated neuronal selectivity in monkey inferotemporal cortex via the vast hypothesis space of a generative deep neural network, avoiding assumptions about features or semantic categories. A genetic algorithm searched this space for stimuli that maximized neuronal firing. This led to the evolution of rich synthetic images of objects with complex combinations of shapes, colors, and textures, sometimes resembling animals or familiar people, other times revealing novel patterns that did not map to any clear semantic category. These results expand our conception of the dictionary of features encoded in the cortex, and the approach can potentially reveal the internal representations of any system whose input can be captured by a generative model.

On the relationship between maps and domains in inferotemporal cortex.

How does the brain encode information about the environment? Decades of research have led to the pervasive notion that the object-processing pathway in primate cortex consists of multiple areas that are each specialized to process different object categories (such as faces, bodies, hands, non-face objects and scenes). The anatomical consistency and modularity of these regions have been interpreted as evidence that these regions are innately specialized. Here, we propose that ventral-stream modules do not represent clusters of circuits that each evolved to process some specific object category particularly important for survival, but instead reflect the effects of experience on a domain-general architecture that evolved to be able to adapt, within a lifetime, to its particular environment. Furthermore, we propose that the mechanisms underlying the development of domains are both evolutionarily old and universal across cortex. Topographic maps are fundamental, governing the development of specializations across systems, providing a framework for brain organization.

Triggers for mother love.

Previous studies showed that baby monkeys separated from their mothers develop strong and lasting attachments to inanimate surrogate mothers, but only if the surrogate has a soft texture; soft texture is more important for the infant's attachment than is the provision of milk. Here I report that postpartum female monkeys also form strong and persistent attachments to inanimate surrogate infants, that the template for triggering maternal attachment is also tactile, and that even a brief period of attachment formation can dominate visual and auditory cues indicating a more appropriate target.

Face neurons encode nonsemantic features.

The primate inferior temporal cortex contains neurons that respond more strongly to faces than to other objects. Termed "face neurons," these neurons are thought to be selective for faces as a semantic category. However, face neurons also partly respond to clocks, fruits, and single eyes, raising the question of whether face neurons are better described as selective for visual features related to faces but dissociable from them. We used a recently described algorithm, XDream, to evolve stimuli that strongly activated face neurons. XDream leverages a generative neural network that is not limited to realistic objects. Human participants assessed images evolved for face neurons and for nonface neurons and natural images depicting faces, cars, fruits, etc. Evolved images were consistently judged to be distinct from real faces. Images evolved for face neurons were rated as slightly more similar to faces than images evolved for nonface neurons. There was a correlation among natural images between face neuron activity and subjective "faceness" ratings, but this relationship did not hold for face neuron­evolved images, which triggered high activity but were rated low in faceness. Our results suggest that so-called face neurons are better described as tuned to visual features rather than semantic categories.

Anatomical correlates of face patches in macaque inferotemporal cortex.

Primate brains typically have regions within the ventral visual stream that are selectively responsive to faces. In macaques, these face patches are located in similar parts of inferotemporal cortex across individuals although correspondence with particular anatomical features has not been reported previously. Here, using high-resolution functional and anatomical imaging, we show that small "bumps," or buried gyri, along the lower bank of the superior temporal sulcus are predictive of the location of face-selective regions. Recordings from implanted multielectrode arrays verified that these bumps contain face-selective neurons. These bumps were present in monkeys raised without seeing faces and that lack face patches, indicating that these anatomical landmarks are predictive of, but not sufficient for, the presence of face selectivity. These bumps are found across primate species that span taxonomy lines, indicating common evolutionary developmental mechanisms. The bumps emerge during fetal development in macaques, indicating that they arise from general developmental mechanisms that result in the regularity of cortical folding of the entire brain.

Evolution of Osteocrin as an activity-regulated factor in the primate brain.

Sensory stimuli drive the maturation and function of the mammalian nervous system in part through the activation of gene expression networks that regulate synapse development and plasticity. These networks have primarily been studied in mice, and it is not known whether there are species- or clade-specific activity-regulated genes that control features of brain development and function. Here we use transcriptional profiling of human fetal brain cultures to identify an activity-dependent secreted factor, Osteocrin (OSTN), that is induced by membrane depolarization of human but not mouse neurons. We find that OSTN has been repurposed in primates through the evolutionary acquisition of DNA regulatory elements that bind the activity-regulated transcription factor MEF2. In addition, we demonstrate that OSTN is expressed in primate neocortex and restricts activity-dependent dendritic growth in human neurons. These findings suggest that, in response to sensory input, OSTN regulates features of neuronal structure and function that are unique to primates.

Body map proto-organization in newborn macaques.

Topographic sensory maps are a prominent feature of the adult primate brain. Here, we asked whether topographic representations of the body are present at birth. Using functional MRI (fMRI), we find that the newborn somatomotor system, spanning frontoparietal cortex and subcortex, comprises multiple topographic representations of the body. The organization of these large-scale body maps was indistinguishable from those in older monkeys. Finer-scale differentiation of individual fingers increased over the first 2 y, suggesting that topographic representations are refined during early development. Last, we found that somatomotor representations were unchanged in 2 visually impaired monkeys who relied on touch for interacting with their environment, demonstrating that massive shifts in early sensory experience in an otherwise anatomically intact brain are insufficient for driving cross-modal plasticity. We propose that a topographic scaffolding is present at birth that both directs and constrains experience-driven modifications throughout somatosensory and motor systems.

Retinotopic Organization of Scene Areas in Macaque Inferior Temporal Cortex.

Primates have specialized domains in inferior temporal (IT) cortex that are responsive to particular image categories. Though IT traditionally has been regarded as lacking retinotopy, several recent studies in monkeys have shown that retinotopic maps extend to face patches along the lower bank of the superior temporal sulcus (STS) and neighboring regions of IT cortex. Here, we used fMRI to map the retinotopic organization of medial ventral temporal cortex in four monkeys (2 male and 2 female). We confirm the presence of visual field maps within and around the lower bank of the STS and extend these prior findings to scene-selective cortex in the ventral-most regions of IT. Within the occipitotemporal sulcus (OTS), we identified two retinotopic areas, OTS1 and OTS2. The polar angle representation of OTS2 was a mirror reversal of the OTS1 representation. These regions contained representations of the contralateral periphery and were selectively active for scene versus face, body, or object images. The extent of this retinotopy parallels that in humans and shows that the organization of the scene network is preserved across primate species. In addition retinotopic maps were identified in dorsal extrastriate, posterior parietal, and frontal cortex as well as the thalamus, including both the lateral geniculate nucleus and pulvinar. Together, it appears that most, if not all, of the macaque visual system contains organized representations of visual space.SIGNIFICANCE STATEMENT Primates have specialized domains in inferior temporal (IT) cortex that are responsive to particular image categories. Though retinotopic maps are considered a fundamental organizing principle of posterior visual cortex, IT traditionally has been regarded as lacking retinotopy. Recent imaging studies have demonstrated the presence of several visual field maps within the lateral IT. Using neuroimaging, we found multiple representations of visual space within ventral IT cortex of macaques that included scene-selective cortex. Scene domains were biased toward the peripheral visual field. These data demonstrate the prevalence of visual field maps throughout the primate visual system, including late stages in the ventral visual hierarchy, and support the idea that domains representing different categories are biased toward different parts of the visual field.

End-Stopping Predicts Curvature Tuning along the Ventral Stream.

Neurons in primate inferotemporal cortex (IT) are clustered into patches of shared image preferences. Functional imaging has shown that these patches are activated by natural categories (e.g., faces, body parts, and places), artificial categories (numerals, words) and geometric features (curvature and real-world size). These domains develop in the same cortical locations across monkeys and humans, which raises the possibility of common innate mechanisms. Although these commonalities could be high-level template-based categories, it is alternatively possible that the domain locations are constrained by low-level properties such as end-stopping, eccentricity, and the shape of the preferred images. To explore this, we looked for correlations among curvature preference, receptive field (RF) end-stopping, and RF eccentricity in the ventral stream. We recorded from sites in V1, V4, and posterior IT (PIT) from six monkeys using microelectrode arrays. Across all visual areas, we found a tendency for end-stopped sites to prefer curved over straight contours. Further, we found a progression in population curvature preferences along the visual hierarchy, where, on average, V1 sites preferred straight Gabors, V4 sites preferred curved stimuli, and many PIT sites showed a preference for curvature that was concave relative to fixation. Our results provide evidence that high-level functional domains may be mapped according to early rudimentary properties of the visual system. SIGNIFICANCE STATEMENT: The macaque occipitotemporal cortex contains clusters of neurons with preferences for categories such as faces, body parts, and places. One common question is how these clusters (or "domains") acquire their cortical position along the ventral stream. We and other investigators previously established an fMRI-level correlation among these category domains, retinotopy, and curvature preferences: for example, in inferotemporal cortex, face- and curvature-preferring domains show a central visual field bias whereas place- and rectilinear-preferring domains show a more peripheral visual field bias. Here, we have found an electrophysiological-level explanation for the correlation among domain preference, curvature, and retinotopy based on neuronal preference for short over long contours, also called end-stopping.

Posterior Inferotemporal Cortex Cells Use Multiple Input Pathways for Shape Encoding.

In the macaque monkey brain, posterior inferior temporal (PIT) cortex cells contribute to visual object recognition. They receive concurrent inputs from visual areas V4, V3, and V2. We asked how these different anatomical pathways shape PIT response properties by deactivating them while monitoring PIT activity in two male macaques. We found that cooling of V4 or V2|3 did not lead to consistent changes in population excitatory drive; however, population pattern analyses showed that V4-based pathways were more important than V2|3-based pathways. We did not find any image features that predicted decoding accuracy differences between both interventions. Using the HMAX hierarchical model of visual recognition, we found that different groups of simulated "PIT" units with different input histories (lacking "V2|3" or "V4" input) allowed for comparable levels of object-decoding performance and that removing a large fraction of "PIT" activity resulted in similar drops in performance as in the cooling experiments. We conclude that distinct input pathways to PIT relay similar types of shape information, with V1-dependent V4 cells providing more quantitatively useful information for overall encoding than cells in V2 projecting directly to PIT.SIGNIFICANCE STATEMENT Convolutional neural networks are the best models of the visual system, but most emphasize input transformations across a serial hierarchy akin to the primary "ventral stream" (V1 → V2 → V4 → IT). However, the ventral stream also comprises parallel "bypass" pathways: V1 also connects to V4, and V2 to IT. To explore the advantages of mixing long and short pathways in the macaque brain, we used cortical cooling to silence inputs to posterior IT and compared the findings with an HMAX model with parallel pathways.

Symbol addition by monkeys provides evidence for normalized quantity coding.

Weber's law can be explained either by a compressive scaling of sensory response with stimulus magnitude or by a proportional scaling of response variability. These two mechanisms can be distinguished by asking how quantities are added or subtracted. We trained Rhesus monkeys to associate 26 distinct symbols with 0-25 drops of reward, and then tested how they combine, or add, symbolically represented reward magnitude. We found that they could combine symbolically represented magnitudes, and they transferred this ability to a novel symbol set, indicating that they were performing a calculation, not just memorizing the value of each combination. The way they combined pairs of symbols indicated neither a linear nor a compressed scale, but rather a dynamically shifting, relative scaling.

Feature-selective responses in macaque visual cortex follow eye movements during natural vision.

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

When the whole is only the parts: non-holistic object parts predominate face-cell responses to illusory faces.

Humans are inclined to perceive faces in everyday objects with a face-like configuration. This illusion, known as face pareidolia, is often attributed to a specialized network of 'face cells' in primates. We found that face cells in macaque inferotemporal cortex responded selectively to pareidolia images, but this selectivity did not require a holistic, face-like configuration, nor did it encode human faceness ratings. Instead, it was driven mostly by isolated object parts that are perceived as eyes only within a face-like context. These object parts lack usual characteristics of primate eyes, pointing to the role of lower-level features. Our results suggest that face-cell responses are dominated by local, generic features, unlike primate visual perception, which requires holistic information. These findings caution against interpreting neural activity through the lens of human perception. Doing so could impose human perceptual biases, like seeing faces where none exist, onto our understanding of neural activity.

The neural code for "face cells" is not face-specific.

Face cells are neurons that respond more to faces than to non-face objects. They are found in clusters in the inferotemporal cortex, thought to process faces specifically, and, hence, studied using faces almost exclusively. Analyzing neural responses in and around macaque face patches to hundreds of objects, we found graded response profiles for non-face objects that predicted the degree of face selectivity and provided information on face-cell tuning beyond that from actual faces. This relationship between non-face and face responses was not predicted by color and simple shape properties but by information encoded in deep neural networks trained on general objects rather than face classification. These findings contradict the long-standing assumption that face versus non-face selectivity emerges from face-specific features and challenge the practice of focusing on only the most effective stimulus. They provide evidence instead that category-selective neurons are best understood by their tuning directions in a domain-general object space.

Colored halos around faces and emotion-evoked colors: a new form of synesthesia.

The claim that some individuals see colored halos or auras around faces has long been part of popular folklore. Here we report on a 23-year-old man (subject TK) diagnosed with Asperger's disorder, who began to consistently experience colors around individuals at the age of 10. TK's colors are based on the individual's identity and emotional connotation. We interpret these experiences as a form of synesthesia, and confirm their authenticity through a target detection paradigm. Additionally, we investigate TK's claim that emotions evoke highly specific colors, allowing him, despite his Asperger's, to introspect on emotions and recognize them in others.

The retrocalcarine sulcus maps different retinotopic representations in macaques and humans.

Primate cerebral cortex is highly convoluted with much of the cortical surface buried in sulcal folds. The origins of cortical folding and its functional relevance have been a major focus of systems and cognitive neuroscience, especially when considering stereotyped patterns of cortical folding that are shared across individuals within a primate species and across multiple species. However, foundational questions regarding organizing principles shared across species remain unanswered. Taking a cross-species comparative approach with a careful consideration of historical observations, we investigate cortical folding relative to primary visual cortex (area V1). We identify two macroanatomical structures-the retrocalcarine and external calcarine sulci-in 24 humans and 6 macaque monkeys. We show that within species, these sulci are identifiable in all individuals, fall on a similar part of the V1 retinotopic map, and thus, serve as anatomical landmarks predictive of functional organization. Yet, across species, the underlying eccentricity representations corresponding to these macroanatomical structures differ strikingly across humans and macaques. Thus, the correspondence between retinotopic representation and cortical folding for an evolutionarily old structure like V1 is species-specific and suggests potential differences in developmental and experiential constraints across primates.

Multivariate patterns in object-selective cortex dissociate perceptual and physical shape similarity.

Prior research has identified the lateral occipital complex (LOC) as a critical cortical region for the representation of object shape in humans. However, little is known about the nature of the representations contained in the LOC and their relationship to the perceptual experience of shape. We used human functional MRI to measure the physical, behavioral, and neural similarity between pairs of novel shapes to ask whether the representations of shape contained in subregions of the LOC more closely reflect the physical stimuli themselves, or the perceptual experience of those stimuli. Perceptual similarity measures for each pair of shapes were obtained from a psychophysical same-different task; physical similarity measures were based on stimulus parameters; and neural similarity measures were obtained from multivoxel pattern analysis methods applied to anterior LOC (pFs) and posterior LOC (LO). We found that the pattern of pairwise shape similarities in LO most closely matched physical shape similarities, whereas shape similarities in pFs most closely matched perceptual shape similarities. Further, shape representations were similar across participants in LO but highly variable across participants in pFs. Together, these findings indicate that activation patterns in subregions of object-selective cortex encode objects according to a hierarchy, with stimulus-based representations in posterior regions and subjective and observer-specific representations in anterior regions.

Noninvasive functional MRI in alert monkeys.

Functional magnetic resonance imaging (fMRI) is now widely used to study human brain function. Alert monkey fMRI experiments have been used to localize functions and to compare the workings of the human and monkey brains. Monkey fMRI poses considerable challenges because of the monkey's small brain and naturally uncooperative disposition. While training can encourage monkeys to be more obliging during scanning, the usual procedure is to hold the monkey's head motionless by means of a surgically implanted head post. Such implants are invasive and require regular maintenance. In order to overcome these problems we developed a technique for holding monkeys' heads motionless during scanning using a custom-fitted plastic helmet, a chin strap, and a mild suction supplied by a vacuum blower. This vacuum helmet method is totally noninvasive and has shown no adverse effects after repeated use for several months. The motion of a trained monkey's head in the helmet during scanning was comparable to that of a trained monkey implanted with a conventional head post.

The benefit of symbols: monkeys show linear, human-like, accuracy when using symbols to represent scalar value.

When humans and animals estimate numbers of items, their error rate is proportional to the number. To date, however, only humans show the capacity to represent large numbers symbolically, which endows them with increased precision, especially for large numbers, and with tools for manipulating numbers. This ability depends critically on our capacity to acquire and represent explicit symbols. Here we show that when rhesus monkeys are trained to use an explicit symbol system, they too show more precise, and linear, scaling than they do using a one-to-one corresponding numerosity representation. We also found that when taught two different types of representations for reward amount, the monkeys systematically undervalued the less precise representation. The results indicate that monkeys, like humans, can learn alternative mechanisms for representing a single value scale and that performance variability and relative value depend on the distinguishability of each representation.

End stopping in V1 is sensitive to contrast.

Common situations that result in different perceptions of grouping and border ownership, such as shadows and occlusion, have distinct sign-of-contrast relationships at their edge-crossing junctions. Here we report a property of end stopping in V1 that distinguishes among different sign-of-contrast situations, thereby obviating the need for explicit junction detectors. We show that the inhibitory effect of the end zones in end-stopped cells is highly selective for the relative sign of contrast between the central activating stimulus and stimuli presented at the end zones. Conversely, the facilitatory effect of end zones in length-summing cells is not selective for the relative sign of contrast between the central activating stimulus and stimuli presented at the end zones. This finding indicates that end stopping belongs in the category of cortical computations that are selective for sign of contrast, such as direction selectivity and disparity selectivity, but length summation does not.