The Code for Facial Identity in the Primate Brain.

Primates recognize complex objects such as faces with remarkable speed and reliability. Here, we reveal the brain's code for facial identity. Experiments in macaques demonstrate an extraordinarily simple transformation between faces and responses of cells in face patches. By formatting faces as points in a high-dimensional linear space, we discovered that each face cell's firing rate is proportional to the projection of an incoming face stimulus onto a single axis in this space, allowing a face cell ensemble to encode the location of any face in the space. Using this code, we could precisely decode faces from neural population responses and predict neural firing rates to faces. Furthermore, this code disavows the long-standing assumption that face cells encode specific facial identities, confirmed by engineering faces with drastically different appearance that elicited identical responses in single face cells. Our work suggests that other objects could be encoded by analogous metric coordinate systems. PAPERCLIP.

Temporal multiplexing of perception and memory codes in IT cortex.

A central assumption of neuroscience is that long-term memories are represented by the same brain areas that encode sensory stimuli1. Neurons in inferotemporal (IT) cortex represent the sensory percept of visual objects using a distributed axis code2-4. Whether and how the same IT neural population represents the long-term memory of visual objects remains unclear. Here we examined how familiar faces are encoded in the IT anterior medial face patch (AM), perirhinal face patch (PR) and temporal pole face patch (TP). In AM and PR we observed that the encoding axis for familiar faces is rotated relative to that for unfamiliar faces at long latency; in TP this memory-related rotation was much weaker. Contrary to previous claims, the relative response magnitude to familiar versus unfamiliar faces was not a stable indicator of familiarity in any patch5-11. The mechanism underlying the memory-related axis change is likely intrinsic to IT cortex, because inactivation of PR did not affect axis change dynamics in AM. Overall, our results suggest that memories of familiar faces are represented in AM and perirhinal cortex by a distinct long-latency code, explaining how the same cell population can encode both the percept and memory of faces.

A map of object space in primate inferotemporal cortex.

The inferotemporal (IT) cortex is responsible for object recognition, but it is unclear how the representation of visual objects is organized in this part of the brain. Areas that are selective for categories such as faces, bodies, and scenes have been found1-5, but large parts of IT cortex lack any known specialization, raising the question of what general principle governs IT organization. Here we used functional MRI, microstimulation, electrophysiology, and deep networks to investigate the organization of macaque IT cortex. We built a low-dimensional object space to describe general objects using a feedforward deep neural network trained on object classification6. Responses of IT cells to a large set of objects revealed that single IT cells project incoming objects onto specific axes of this space. Anatomically, cells were clustered into four networks according to the first two components of their preferred axes, forming a map of object space. This map was repeated across three hierarchical stages of increasing view invariance, and cells that comprised these maps collectively harboured sufficient coding capacity to approximately reconstruct objects. These results provide a unified picture of IT organization in which category-selective regions are part of a coarse map of object space whose dimensions can be extracted from a deep network.

Functional modules for visual scene segmentation in macaque visual cortex.

Segmentation, the computation of object boundaries, is one of the most important steps in intermediate visual processing. Previous studies have reported cells across visual cortex that are modulated by segmentation features, but the functional role of these cells remains unclear. First, it is unclear whether these cells encode segmentation consistently since most studies used only a limited variety of stimulus types. Second, it is unclear whether these cells are organized into specialized modules or instead randomly scattered across the visual cortex: the former would lend credence to a functional role for putative segmentation cells. Here, we used fMRI-guided electrophysiology to systematically characterize the consistency and spatial organization of segmentation-encoding cells across the visual cortex. Using fMRI, we identified a set of patches in V2, V3, V3A, V4, and V4A that were more active for stimuli containing figures compared to ground, regardless of whether figures were defined by texture, motion, luminance, or disparity. We targeted these patches for single-unit recordings and found that cells inside segmentation patches were tuned to both figure-ground and borders more consistently across types of stimuli than cells in the visual cortex outside the patches. Remarkably, we found clusters of cells inside segmentation patches that showed the same border-ownership preference across all stimulus types. Finally, using a population decoding approach, we found that segmentation could be decoded with higher accuracy from segmentation patches than from either color-selective or control regions. Overall, our results suggest that segmentation signals are preferentially encoded in spatially discrete patches.

The macaque face patch system: a turtle's underbelly for the brain.

Objects constitute the fundamental currency of our consciousness: they are the things that we perceive, remember and think about. One of the most important objects for a primate is a face. Research on the macaque face patch system in recent years has given us a remarkable window into the detailed processes underlying object recognition. Here, we review the macaque face patch system, including its anatomical organization, coding principles, role in behaviour and interactions with other brain regions. We highlight not only how it constitutes an archetypal object recognition system but also how it may provide a key to understanding mechanisms for higher cognitive function.

A topological solution to object segmentation and tracking.

The world is composed of objects, the ground, and the sky. Visual perception of objects requires solving two fundamental challenges: 1) segmenting visual input into discrete units and 2) tracking identities of these units despite appearance changes due to object deformation, changing perspective, and dynamic occlusion. Current computer vision approaches to segmentation and tracking that approach human performance all require learning, raising the question, Can objects be segmented and tracked without learning? Here, we show that the mathematical structure of light rays reflected from environment surfaces yields a natural representation of persistent surfaces, and this surface representation provides a solution to both the segmentation and tracking problems. We describe how to generate this surface representation from continuous visual input and demonstrate that our approach can segment and invariantly track objects in cluttered synthetic video despite severe appearance changes, without requiring learning.

Combining brain perturbation and neuroimaging in non-human primates.

Brain perturbation studies allow detailed causal inferences of behavioral and neural processes. Because the combination of brain perturbation methods and neural measurement techniques is inherently challenging, research in humans has predominantly focused on non-invasive, indirect brain perturbations, or neurological lesion studies. Non-human primates have been indispensable as a neurobiological system that is highly similar to humans while simultaneously being more experimentally tractable, allowing visualization of the functional and structural impact of systematic brain perturbation. This review considers the state of the art in non-human primate brain perturbation with a focus on approaches that can be combined with neuroimaging. We consider both non-reversible (lesions) and reversible or temporary perturbations such as electrical, pharmacological, optical, optogenetic, chemogenetic, pathway-selective, and ultrasound based interference methods. Method-specific considerations from the research and development community are offered to facilitate research in this field and support further innovations. We conclude by identifying novel avenues for further research and innovation and by highlighting the clinical translational potential of the methods.

Consistency of Border-Ownership Cells across Artificial Stimuli, Natural Stimuli, and Stimuli with Ambiguous Contours.

Segmentation and recognition of objects in a visual scene are two problems that are hard to solve separately from each other. When segmenting an ambiguous scene, it is helpful to already know the present objects and their shapes. However, for recognizing an object in clutter, one would like to consider its isolated segment alone to avoid confounds from features of other objects. Border-ownership cells (Zhou et al., 2000) appear to play an important role in segmentation, as they signal the side-of-figure of artificial stimuli. The present work explores the role of border-ownership cells in dorsal macaque visual areas V2 and V3 in the segmentation of natural object stimuli and locally ambiguous stimuli. We report two major results. First, compared with previous estimates, we found a smaller percentage of cells that were consistent across artificial stimuli used previously. Second, we found that the average response of those neurons that did respond consistently to the side-of-figure of artificial stimuli also consistently signaled, as a population, the side-of-figure for borders of single faces, occluding faces and, with higher latencies, even stimuli with illusory contours, such as Mooney faces and natural faces completely missing local edge information. In contrast, the local edge or the outlines of the face alone could not always evoke a significant border-ownership signal. Our results underscore that border ownership is coded by a population of cells, and indicate that these cells integrate a variety of cues, including low-level features and global object context, to compute the segmentation of the scene. SIGNIFICANCE STATEMENT: To distinguish different objects in a natural scene, the brain must segment the image into regions corresponding to objects. The so-called "border-ownership" cells appear to be dedicated to this task, as they signal for a given edge on which side the object is that owns it. Here, we report that individual border-ownership cells are unreliable when tested across a battery of artificial stimuli used previously but can signal border-ownership consistently as a population. We show that these border-ownership population signals are also suited for signaling border-ownership for natural objects and at longer latency, even for stimuli without local edge information. Our results suggest that border-ownership cells integrate both local, low-level and global, high-level cues to segment the scene.

Saccade modulation by optical and electrical stimulation in the macaque frontal eye field.

Recent studies have demonstrated that strong neural modulations can be evoked with optogenetic stimulation in macaque motor cortex without observing any evoked movements (Han et al., 2009, 2011; Diester et al., 2011). It remains unclear why such perturbations do not generate movements and if conditions exist under which they may evoke movements. In this study, we examine the effects of five optogenetic constructs in the macaque frontal eye field and use electrical microstimulation to assess whether optical perturbation of the local network leads to observable motor changes during optical, electrical, and combined stimulation. We report a significant increase in the probability of evoking saccadic eye movements when low current electrical stimulation is coupled to optical stimulation compared with when electrical stimulation is used alone. Experiments combining channelrhodopsin 2 (ChR2) and electrical stimulation with simultaneous fMRI revealed no discernible fMRI activity at the electrode tip with optical stimulation but strong activity with electrical stimulation. Our findings suggest that stimulation with current ChR2 optogenetic constructs generates subthreshold activity that contributes to the initiation of movements but, in most cases, is not sufficient to evoke a motor response.

Faces in motion: selectivity of macaque and human face processing areas for dynamic stimuli.

Face recognition mechanisms need to extract information from static and dynamic faces. It has been hypothesized that the analysis of dynamic face attributes is performed by different face areas than the analysis of static facial attributes. To date, there is no evidence for such a division of labor in macaque monkeys. We used fMRI to determine specializations of macaque face areas for motion. Face areas in the fundus of the superior temporal sulcus responded to general object motion; face areas outside of the superior temporal sulcus fundus responded more to facial motion than general object motion. Thus, the macaque face-processing system exhibits regional specialization for facial motion. Human face areas, processing the same stimuli, exhibited specializations for facial motion as well. Yet the spatial patterns of facial motion selectivity differed across species, suggesting that facial dynamics are analyzed differently in humans and macaques.

Effects of focused ultrasound in a "clean" mouse model of ultrasonic neuromodulation.

Recent studies on ultrasonic neuromodulation (UNM) in rodents have shown that focused ultrasound (FUS) can activate peripheral auditory pathways, leading to off-target and brain-wide excitation, which obscures the direct activation of the target area by FUS. To address this issue, we developed a new mouse model, the double transgenic Pou4f3+/DTR × Thy1-GCaMP6s, which allows for inducible deafening using diphtheria toxin and minimizes off-target effects of UNM while allowing effects on neural activity to be visualized with fluorescent calcium imaging. Using this model, we found that the auditory confounds caused by FUS can be significantly reduced or eliminated within a certain pressure range. At higher pressures, FUS can result in focal fluorescence dips at the target, elicit non-auditory sensory confounds, and damage tissue, leading to spreading depolarization. Under the acoustic conditions we tested, we did not observe direct calcium responses in the mouse cortex. Our findings provide a cleaner animal model for UNM and sonogenetics research, establish a parameter range within which off-target effects are confidently avoided, and reveal the non-auditory side effects of higher-pressure stimulation.

Effects of focused ultrasound in a "clean" mouse model of ultrasonic neuromodulation.

Recent studies on ultrasonic neuromodulation (UNM) in rodents have shown that focused ultrasound (FUS) can activate peripheral auditory pathways, leading to off-target and brain-wide excitation, which obscures the direct activation of the target area by FUS. To address this issue, we developed a new mouse model, the double transgenic Pou4f3+/DTR × Thy1-GCaMP6s, which allows for inducible deafening using diphtheria toxin and minimizes off-target effects of UNM while allowing effects on neural activity to be visualized with fluorescent calcium imaging. Using this model, we found that the auditory confounds caused by FUS can be significantly reduced or eliminated within a certain pressure range. At higher pressures, FUS can result in focal fluorescence dips at the target, elicit non-auditory sensory confounds, and damage tissue, leading to spreading depolarization. Under the acoustic conditions we tested, we did not observe direct calcium responses in the mouse cortex. Our findings provide a cleaner animal model for UNM and sonogenetics research, establish a parameter range within which off-target effects are confidently avoided, and reveal the non-auditory side effects of higher-pressure stimulation.

Rapid, concerted switching of the neural code in inferotemporal cortex.

A fundamental paradigm in neuroscience is the concept of neural coding through tuning functions 1 . According to this idea, neurons encode stimuli through fixed mappings of stimulus features to firing rates. Here, we report that the tuning of visual neurons can rapidly and coherently change across a population to attend to a whole and its parts. We set out to investigate a longstanding debate concerning whether inferotemporal (IT) cortex uses a specialized code for representing specific types of objects or whether it uses a general code that applies to any object. We found that face cells in macaque IT cortex initially adopted a general code optimized for face detection. But following a rapid, concerted population event lasting < 20 ms, the neural code transformed into a face-specific one with two striking properties: (i) response gradients to principal detection-related dimensions reversed direction, and (ii) new tuning developed to multiple higher feature space dimensions supporting fine face discrimination. These dynamics were face specific and did not occur in response to objects. Overall, these results show that, for faces, face cells shift from detection to discrimination by switching from an object-general code to a face-specific code. More broadly, our results suggest a novel mechanism for neural representation: concerted, stimulus-dependent switching of the neural code used by a cortical area.

Mice and primates use distinct strategies for visual segmentation.

The rodent visual system has attracted great interest in recent years due to its experimental tractability, but the fundamental mechanisms used by the mouse to represent the visual world remain unclear. In the primate, researchers have argued from both behavioral and neural evidence that a key step in visual representation is 'figure-ground segmentation', the delineation of figures as distinct from backgrounds. To determine if mice also show behavioral and neural signatures of figure-ground segmentation, we trained mice on a figure-ground segmentation task where figures were defined by gratings and naturalistic textures moving counterphase to the background. Unlike primates, mice were severely limited in their ability to segment figure from ground using the opponent motion cue, with segmentation behavior strongly dependent on the specific carrier pattern. Remarkably, when mice were forced to localize naturalistic patterns defined by opponent motion, they adopted a strategy of brute force memorization of texture patterns. In contrast, primates, including humans, macaques, and mouse lemurs, could readily segment figures independent of carrier pattern using the opponent motion cue. Consistent with mouse behavior, neural responses to the same stimuli recorded in mouse visual areas V1, RL, and LM also did not support texture-invariant segmentation of figures using opponent motion. Modeling revealed that the texture dependence of both the mouse's behavior and neural responses could be explained by a feedforward neural network lacking explicit segmentation capabilities. These findings reveal a fundamental limitation in the ability of mice to segment visual objects compared to primates.

Color-tuned neurons are spatially clustered according to color preference within alert macaque posterior inferior temporal cortex.

Large islands of extrastriate cortex that are enriched for color-tuned neurons have recently been described in alert macaque using a combination of functional magnetic resonance imaging (fMRI) and single-unit recording. These millimeter-sized islands, dubbed "globs," are scattered throughout the posterior inferior temporal cortex (PIT), a swath of brain anterior to area V3, including areas V4, PITd, and posterior TEO. We investigated the micro-organization of neurons within the globs. We used fMRI to identify the globs and then used MRI-guided microelectrodes to test the color properties of single glob cells. We used color stimuli that sample the CIELUV perceptual color space at regular intervals to test the color tuning of single units, and make two observations. First, color-tuned neurons of various color preferences were found within single globs. Second, adjacent glob cells tended to have the same color tuning, demonstrating that glob cells are clustered by color preference and suggesting that they are arranged in color columns. Neurons separated by 50 microm, measured parallel to the cortical sheet, had more similar color tuning than neurons separated by 100 microm, suggesting that the scale of the color columns is <100 microm. These results show that color-tuned neurons in PIT are organized by color preference on a finer scale than the scale of single globs. Moreover, the color preferences of neurons recorded sequentially along a given electrode penetration shifted gradually in many penetrations, suggesting that the color columns are arranged according to a chromotopic map reflecting perceptual color space.

Specialized color modules in macaque extrastriate cortex.

Imaging studies are consistent with the existence of brain regions specialized for color, but electrophysiological studies have produced conflicting results. Here we address the neural basis for color, using targeted single-unit recording in alert macaque monkeys, guided by functional magnetic resonance imaging (fMRI) of the same subjects. Distributed within posterior inferior temporal cortex, a large region encompassing V4, PITd, and posterior TEO that some have proposed functions as a single visual complex, we found color-biased fMRI hotspots that we call "globs," each several millimeters wide. Almost all cells located in globs showed strong luminance-invariant color tuning and some shape selectivity. Cells in different globs represented distinct visual field locations, consistent with the coarse retinotopy of this brain region. Cells in "interglob" regions were not color tuned, but were more strongly shape selective. Neither population was direction selective. These results suggest that color perception is mediated by specialized neurons that are clustered within the extrastriate brain.

Comparing face patch systems in macaques and humans.

Face recognition is of central importance for primate social behavior. In both humans and macaques, the visual analysis of faces is supported by a set of specialized face areas. The precise organization of these areas and the correspondence between individual macaque and human face-selective areas are debated. Here, we examined the organization of face-selective regions across the temporal lobe in a large number of macaque and human subjects. Macaques showed 6 regions of face-selective cortex arranged in a stereotypical pattern along the temporal lobe. Human subjects showed, in addition to 3 reported face areas (the occipital, fusiform, and superior temporal sulcus face areas), a face-selective area located anterior to the fusiform face area, in the anterior collateral sulcus. These results suggest a closer anatomical correspondence between macaque and human face-processing systems than previously realized.

Explaining face representation in the primate brain using different computational models.

Understanding how the brain represents the identity of complex objects is a central challenge of visual neuroscience. The principles governing object processing have been extensively studied in the macaque face patch system, a sub-network of inferotemporal (IT) cortex specialized for face processing. A previous study reported that single face patch neurons encode axes of a generative model called the "active appearance" model, which transforms 50D feature vectors separately representing facial shape and facial texture into facial images. However, a systematic investigation comparing this model to other computational models, especially convolutional neural network models that have shown success in explaining neural responses in the ventral visual stream, has been lacking. Here, we recorded responses of cells in the most anterior face patch anterior medial (AM) to a large set of real face images and compared a large number of models for explaining neural responses. We found that the active appearance model better explained responses than any other model except CORnet-Z, a feedforward deep neural network trained on general object classification to classify non-face images, whose performance it tied on some face image sets and exceeded on others. Surprisingly, deep neural networks trained specifically on facial identification did not explain neural responses well. A major reason is that units in the network, unlike neurons, are less modulated by face-related factors unrelated to facial identification, such as illumination.

Functional connectivity of the macaque brain across stimulus and arousal states.

Cortical networks generate temporally correlated brain activity. To clarify the functional significance of this correlated activity, we asked whether and how its structure depends on stimulus and arousal state. Using independent components analysis of macaque functional magnetic resonance imaging data, we identified a large number of brain networks that were strikingly reproducible across different visual stimulus contexts. Fewer networks were reproducible across alert and anesthetized brain states. Network complexity ranged from bilateral single-node networks to networks comprising multiple discrete nodes distributed over 3 cm of cortex; one network identified in our survey included parts of the temporal parietal occipital junction, dorsal premotor cortex, insula, and posterior cingulate cortex bilaterally. Our results reveal the wealth of spatially structured correlated networks throughout the brain in both alert and anesthetized monkeys, and show that anesthesia significantly alters the spatial structure of these networks.