Specialized Networks for Social Cognition in the Primate Brain.

Primates have evolved diverse cognitive capabilities to navigate their complex social world. To understand how the brain implements critical social cognitive abilities, we describe functional specialization in the domains of face processing, social interaction understanding, and mental state attribution. Systems for face processing are specialized from the level of single cells to populations of neurons within brain regions to hierarchically organized networks that extract and represent abstract social information. Such functional specialization is not confined to the sensorimotor periphery but appears to be a pervasive theme of primate brain organization all the way to the apex regions of cortical hierarchies. Circuits processing social information are juxtaposed with parallel systems involved in processing nonsocial information, suggesting common computations applied to different domains. The emerging picture of the neural basis of social cognition is a set of distinct but interacting subnetworks involved in component processes such as face perception and social reasoning, traversing large parts of the primate brain.

Evolutionary and biomedical insights from a marmoset diploid genome assembly.

The accurate and complete assembly of both haplotype sequences of a diploid organism is essential to understanding the role of variation in genome functions, phenotypes and diseases1. Here, using a trio-binning approach, we present a high-quality, diploid reference genome, with both haplotypes assembled independently at the chromosome level, for the common marmoset (Callithrix jacchus), an primate model system that is widely used in biomedical research2,3. The full spectrum of heterozygosity between the two haplotypes involves 1.36% of the genome-much higher than the 0.13% indicated by the standard estimation based on single-nucleotide heterozygosity alone. The de novo mutation rate is 0.43 × 10-8 per site per generation, and the paternal inherited genome acquired twice as many mutations as the maternal. Our diploid assembly enabled us to discover a recent expansion of the sex-differentiation region and unique evolutionary changes in the marmoset Y chromosome. In addition, we identified many genes with signatures of positive selection that might have contributed to the evolution of Callithrix biological features. Brain-related genes were highly conserved between marmosets and humans, although several genes experienced lineage-specific copy number variations or diversifying selection, with implications for the use of marmosets as a model system.

Encoding of dynamic facial information in the middle dorsal face area.

Faces in motion reveal a plethora of information through visual dynamics. Faces can move in complex patterns while transforming facial shape, e.g., during the generation of different emotional expressions. While motion and shape processing have been studied extensively in separate research enterprises, much less is known about their conjunction during biological motion. Here, we took advantage of the discovery in brain-imaging studies of an area in the dorsal portion of the macaque monkey superior temporal sulcus (STS), the middle dorsal face area (MD), with selectivity for naturalistic face motion. To gain mechanistic insights into the coding of facial motion, we recorded single-unit activity from MD, testing whether and how MD cells encode face motion. The MD population was highly sensitive to naturalistic facial motion and facial shape. Some MD cells responded only to the conjunction of facial shape and motion, others were selective for facial shape even without movement, and yet others were suppressed by facial motion. We found that this heterogeneous MD population transforms face motion into a higher dimensional activity space, a representation that would allow for high sensitivity to relevant small-scale movements. Indeed, we show that many MD cells carry such sensitivity for eye movements. We further found that MD cells encode motion of head, mouth, and eyes in a separable manner, requiring the use of multiple reference frames. Thus, MD is a bona fide face-motion area that uses highly heterogeneous cell populations to create codes capturing even complex facial motion trajectories.

Geometric deep learning enables 3D kinematic profiling across species and environments.

Comprehensive descriptions of animal behavior require precise three-dimensional (3D) measurements of whole-body movements. Although two-dimensional approaches can track visible landmarks in restrictive environments, performance drops in freely moving animals, due to occlusions and appearance changes. Therefore, we designed DANNCE to robustly track anatomical landmarks in 3D across species and behaviors. DANNCE uses projective geometry to construct inputs to a convolutional neural network that leverages learned 3D geometric reasoning. We trained and benchmarked DANNCE using a dataset of nearly seven million frames that relates color videos and rodent 3D poses. In rats and mice, DANNCE robustly tracked dozens of landmarks on the head, trunk, and limbs of freely moving animals in naturalistic settings. We extended DANNCE to datasets from rat pups, marmosets, and chickadees, and demonstrate quantitative profiling of behavioral lineage during development.

Joint encoding of facial identity, orientation, gaze, and expression in the middle dorsal face area.

The last two decades have established that a network of face-selective areas in the temporal lobe of macaque monkeys supports the visual processing of faces. Each area within the network contains a large fraction of face-selective cells. And each area encodes facial identity and head orientation differently. A recent brain-imaging study discovered an area outside of this network selective for naturalistic facial motion, the middle dorsal (MD) face area. This finding offers the opportunity to determine whether coding principles revealed inside the core network would generalize to face areas outside the core network. We investigated the encoding of static faces and objects, facial identity, and head orientation, dimensions which had been studied in multiple areas of the core face-processing network before, as well as facial expressions and gaze. We found that MD populations form a face-selective cluster with a degree of selectivity comparable to that of areas in the core face-processing network. MD encodes facial identity robustly across changes in head orientation and expression, it encodes head orientation robustly against changes in identity and expression, and it encodes expression robustly across changes in identity and head orientation. These three dimensions are encoded in a separable manner. Furthermore, MD also encodes the direction of gaze in addition to head orientation. Thus, MD encodes both structural properties (identity) and changeable ones (expression and gaze) and thus provides information about another animal's direction of attention (head orientation and gaze). MD contains a heterogeneous population of cells that establish a multidimensional code for faces.

Evidence for an attentional priority map in inferotemporal cortex.

From incoming sensory information, our brains make selections according to current behavioral goals. This process, selective attention, is controlled by parietal and frontal areas. Here, we show that another brain area, posterior inferotemporal cortex (PITd), also exhibits the defining properties of attentional control. We discovered this area with functional magnetic resonance imaging (fMRI) during an attentive motion discrimination task. Single-cell recordings from PITd revealed strong attentional modulation across 3 attention tasks yet no tuning to task-relevant stimulus features, like motion direction or color. Instead, PITd neurons closely tracked the subject's attention state and predicted upcoming errors of attentional selection. Furthermore, artificial electrical PITd stimulation controlled the location of attentional selection without altering feature discrimination. These are the defining properties of a feature-blind priority map encoding the locus of attention. Together, these results suggest area PITd, located strategically to gather information about object properties, as an attentional priority map.

Face Patch Resting State Networks Link Face Processing to Social Cognition.

Faces transmit a wealth of social information. How this information is exchanged between face-processing centers and brain areas supporting social cognition remains largely unclear. Here we identify these routes using resting state functional magnetic resonance imaging in macaque monkeys. We find that face areas functionally connect to specific regions within frontal, temporal, and parietal cortices, as well as subcortical structures supporting emotive, mnemonic, and cognitive functions. This establishes the existence of an extended face-recognition system in the macaque. Furthermore, the face patch resting state networks and the default mode network in monkeys show a pattern of overlap akin to that between the social brain and the default mode network in humans: this overlap specifically includes the posterior superior temporal sulcus, medial parietal, and dorsomedial prefrontal cortex, areas supporting high-level social cognition in humans. Together, these results reveal the embedding of face areas into larger brain networks and suggest that the resting state networks of the face patch system offer a new, easily accessible venue into the functional organization of the social brain and into the evolution of possibly uniquely human social skills.

Occipital White Matter Tracts in Human and Macaque.

We compare several major white-matter tracts in human and macaque occipital lobe using diffusion magnetic resonance imaging. The comparison suggests similarities but also significant differences in the tracts. There are several apparently homologous tracts in the 2 species, including the vertical occipital fasciculus (VOF), optic radiation, forceps major, and inferior longitudinal fasciculus (ILF). There is one large human tract, the inferior fronto-occipital fasciculus, with no corresponding fasciculus in macaque. We could identify the macaque VOF (mVOF), which has been little studied. Its position is consistent with classical invasive anatomical studies by Wernicke. VOF homology is supported by similarity of the endpoints in V3A and ventral V4 across species. The mVOF fibers intertwine with the dorsal segment of the ILF, but the human VOF appears to be lateral to the ILF. These similarities and differences between the occipital lobe tracts will be useful in establishing which circuitry in the macaque can serve as an accurate model for human visual cortex.

Whole-agent selectivity within the macaque face-processing system.

The primate brain contains a set of face-selective areas, which are thought to extract the rich social information that faces provide, such as emotional state and personal identity. The nature of this information raises a fundamental question about these face-selective areas: Do they respond to a face purely because of its visual attributes, or because the face embodies a larger social agent? Here, we used functional magnetic resonance imaging to determine whether the macaque face patch system exhibits a whole-agent response above and beyond its responses to individually presented faces and bodies. We found a systematic development of whole-agent preference through the face patches, from subadditive integration of face and body responses in posterior face patches to superadditive integration in anterior face patches. Superadditivity was not observed for faces atop nonbody objects, implying categorical specificity of face-body interaction. Furthermore, superadditivity was robust to visual degradation of facial detail, suggesting whole-agent selectivity does not require prior face recognition. In contrast, even the body patches immediately adjacent to anterior face areas did not exhibit superadditivity. This asymmetry between face- and body-processing systems may explain why observers attribute bodies' social signals to faces, and not vice versa. The development of whole-agent selectivity from posterior to anterior face patches, in concert with the recently described development of natural motion selectivity from ventral to dorsal face patches, identifies a single face patch, AF (anterior fundus), as a likely link between the analysis of facial shape and semantic inferences about other agents.

Attentive Motion Discrimination Recruits an Area in Inferotemporal Cortex.

Attentional selection requires the interplay of multiple brain areas. Theoretical accounts of selective attention predict different areas with different functional properties to support endogenous covert attention. To test these predictions, we devised a demanding attention task requiring motion discrimination and spatial selection and performed whole-brain imaging in macaque monkeys. Attention modulated the early visual cortex, motion-selective dorsal stream areas, the lateral intraparietal area, and the frontal eye fields. This pattern of activation supports early selection, feature-based, and biased-competition attention accounts, as well as the frontoparietal theory of attentional control. While high-level motion-selective dorsal stream areas did not exhibit strong attentional modulation, ventral stream areas V4d and the dorsal posterior inferotemporal cortex (PITd) did. The PITd in fact was, consistently across task variations, the most significantly and most strongly attention-modulated area, even though it did not exhibit signs of motion selectivity. Thus the recruitment of the PITd in attention tasks involving different kinds of motion analysis is not predicted by any theoretical account of attention. These functional data, together with known anatomical connections, suggest a general and possibly critical role of the PITd in attentional selection. SIGNIFICANCE STATEMENT: Attention is the key cognitive function that selects sensory information relevant to the current goals, relegating other information to the shadows of consciousness. To better understand the neural mechanisms of this interplay between sensory processing and internal cognitive state, we must learn more about the brain areas supporting attentional selection. Here, to test theoretical accounts of attentional selection, we used a novel task requiring sustained attention to motion. We found that, surprisingly, among the most strongly attention-modulated areas is one that is neither selective for the sensory feature relevant for current goals nor one hitherto thought to be involved in attentional control. This discovery suggests a need for an extension of current theoretical accounts of the brain circuits for attentional selection.

Intelligent information loss: the coding of facial identity, head pose, and non-face information in the macaque face patch system.

Faces are a behaviorally important class of visual stimuli for primates. Recent work in macaque monkeys has identified six discrete face areas where most neurons have higher firing rates to images of faces compared with other objects (Tsao et al., 2006). While neurons in these areas appear to have different tuning (Freiwald and Tsao, 2010; Issa and DiCarlo, 2012), exactly what types of information and, consequently, which visual behaviors neural populations within each face area can support, is unknown. Here we use population decoding to better characterize three of these face patches (ML/MF, AL, and AM). We show that neural activity in all patches contains information that discriminates between the broad categories of face and nonface objects, individual faces, and nonface stimuli. Information is present in both high and lower firing rate regimes. However, there were significant differences between the patches, with the most anterior patch showing relatively weaker representation of nonface stimuli. Additionally, we find that pose-invariant face identity information increases as one moves to more anterior patches, while information about the orientation of the head decreases. Finally, we show that all the information we can extract from the population is present in patterns of activity across neurons, and there is relatively little information in the total activity of the population. These findings give new insight into the representations constructed by the face patch system and how they are successively transformed.

Faces in motion: selectivity of macaque and human face processing areas for dynamic stimuli.

Face recognition mechanisms need to extract information from static and dynamic faces. It has been hypothesized that the analysis of dynamic face attributes is performed by different face areas than the analysis of static facial attributes. To date, there is no evidence for such a division of labor in macaque monkeys. We used fMRI to determine specializations of macaque face areas for motion. Face areas in the fundus of the superior temporal sulcus responded to general object motion; face areas outside of the superior temporal sulcus fundus responded more to facial motion than general object motion. Thus, the macaque face-processing system exhibits regional specialization for facial motion. Human face areas, processing the same stimuli, exhibited specializations for facial motion as well. Yet the spatial patterns of facial motion selectivity differed across species, suggesting that facial dynamics are analyzed differently in humans and macaques.

Neural correlates of individual facial recognition in a social wasp.

Individual recognition is critical for social behavior across species. Whether recognition is mediated by circuits specialized for social information processing has been a matter of debate. Here we examine the neurobiological underpinning of individual visual facial recognition in Polistes fuscatus paper wasps. Front-facing images of conspecific wasps broadly increase activity across many brain regions relative to other stimuli. Notably, we identify a localized subpopulation of neurons in the protocerebrum which show specialized selectivity for front-facing wasp images, which we term wasp cells. These wasp cells encode information regarding the facial patterns, with ensemble activity correlating with facial identity. Wasp cells are strikingly analogous to face cells in primates, indicating that specialized circuits are likely an adaptive feature of neural architecture to support visual recognition.

Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks.

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric views (e.g. left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

Multiple objects evoke fluctuating responses in several regions of the visual pathway.

How neural representations preserve information about multiple stimuli is mysterious. Because tuning of individual neurons is coarse (e.g., visual receptive field diameters can exceed perceptual resolution), the populations of neurons potentially responsive to each individual stimulus can overlap, raising the question of how information about each item might be segregated and preserved in the population. We recently reported evidence for a potential solution to this problem: when two stimuli were present, some neurons in the macaque visual cortical areas V1 and V4 exhibited fluctuating firing patterns, as if they responded to only one individual stimulus at a time (Jun et al., 2022). However, whether such an information encoding strategy is ubiquitous in the visual pathway and thus could constitute a general phenomenon remains unknown. Here, we provide new evidence that such fluctuating activity is also evoked by multiple stimuli in visual areas responsible for processing visual motion (middle temporal visual area, MT), and faces (middle fundus and anterolateral face patches in inferotemporal cortex - areas MF and AL), thus extending the scope of circumstances in which fluctuating activity is observed. Furthermore, consistent with our previous results in the early visual area V1, MT exhibits fluctuations between the representations of two stimuli when these form distinguishable objects but not when they fuse into one perceived object, suggesting that fluctuating activity patterns may underlie visual object formation. Taken together, these findings point toward an updated model of how the brain preserves sensory information about multiple stimuli for subsequent processing and behavioral action.

Attention to surfaces modulates motion processing in extrastriate area MT.

In the visual system, early atomized representations are grouped into higher-level entities through processes of perceptual organization. Here we present neurophysiological evidence that a representation of a simple object, a surface defined by color and motion, can be the unit of attentional selection at an early stage of visual processing. Monkeys were cued by the color of a fixation spot to attend to one of two transparent random-dot surfaces, one red and one green, which occupied the same region of space. Motion of the attended surface drove neurons in the middle temporal (MT) visual area more strongly than physically identical motion of the non-attended surface, even though both occurred within the spotlight of attention. Surface-based effects of attention persisted even without differential surface coloring, but attentional modulation was stronger with color. These results show that attention can select surface representations to modulate visual processing as early as cortical area MT.

Comparing face patch systems in macaques and humans.

Face recognition is of central importance for primate social behavior. In both humans and macaques, the visual analysis of faces is supported by a set of specialized face areas. The precise organization of these areas and the correspondence between individual macaque and human face-selective areas are debated. Here, we examined the organization of face-selective regions across the temporal lobe in a large number of macaque and human subjects. Macaques showed 6 regions of face-selective cortex arranged in a stereotypical pattern along the temporal lobe. Human subjects showed, in addition to 3 reported face areas (the occipital, fusiform, and superior temporal sulcus face areas), a face-selective area located anterior to the fusiform face area, in the anterior collateral sulcus. These results suggest a closer anatomical correspondence between macaque and human face-processing systems than previously realized.

Neuroscience: A Face's Journey through Space and Time.

Faces are complex objects of great variety, which the visual brain somehow manages to organize by similarity. Two such orderings in fact exist and one, a new study finds, is transformed into the other over time, enhancing a face's distinctiveness.

A fast link between face perception and memory in the temporal pole.

The question of how the brain recognizes the faces of familiar individuals has been important throughout the history of neuroscience. Cells linking visual processing to person memory have been proposed but not found. Here, we report the discovery of such cells through recordings from an area in the macaque temporal pole identified with functional magnetic resonance imaging. These cells responded to faces that were personally familiar. They responded nonlinearly to stepwise changes in face visibility and detail and holistically to face parts, reflecting key signatures of familiar face recognition. They discriminated between familiar identities, as fast as a general face identity area. The discovery of these cells establishes a new pathway for the fast recognition of familiar individuals.