The Code for Facial Identity in the Primate Brain.

Primates recognize complex objects such as faces with remarkable speed and reliability. Here, we reveal the brain's code for facial identity. Experiments in macaques demonstrate an extraordinarily simple transformation between faces and responses of cells in face patches. By formatting faces as points in a high-dimensional linear space, we discovered that each face cell's firing rate is proportional to the projection of an incoming face stimulus onto a single axis in this space, allowing a face cell ensemble to encode the location of any face in the space. Using this code, we could precisely decode faces from neural population responses and predict neural firing rates to faces. Furthermore, this code disavows the long-standing assumption that face cells encode specific facial identities, confirmed by engineering faces with drastically different appearance that elicited identical responses in single face cells. Our work suggests that other objects could be encoded by analogous metric coordinate systems. PAPERCLIP.

Specialized Networks for Social Cognition in the Primate Brain.

Primates have evolved diverse cognitive capabilities to navigate their complex social world. To understand how the brain implements critical social cognitive abilities, we describe functional specialization in the domains of face processing, social interaction understanding, and mental state attribution. Systems for face processing are specialized from the level of single cells to populations of neurons within brain regions to hierarchically organized networks that extract and represent abstract social information. Such functional specialization is not confined to the sensorimotor periphery but appears to be a pervasive theme of primate brain organization all the way to the apex regions of cortical hierarchies. Circuits processing social information are juxtaposed with parallel systems involved in processing nonsocial information, suggesting common computations applied to different domains. The emerging picture of the neural basis of social cognition is a set of distinct but interacting subnetworks involved in component processes such as face perception and social reasoning, traversing large parts of the primate brain.

Inactivation of face-selective neurons alters eye movements when free viewing faces.

During free viewing, faces attract gaze and induce specific fixation patterns corresponding to the facial features. This suggests that neurons encoding the facial features are in the causal chain that steers the eyes. However, there is no physiological evidence to support a mechanistic link between face-encoding neurons in high-level visual areas and the oculomotor system. In this study, we targeted the middle face patches of the inferior temporal (IT) cortex in two macaque monkeys using an functional magnetic resonance imaging (fMRI) localizer. We then utilized muscimol microinjection to unilaterally suppress IT neural activity inside and outside the face patches and recorded eye movements while the animals free viewing natural scenes. Inactivation of the face-selective neurons altered the pattern of eye movements on faces: The monkeys found faces in the scene but neglected the eye contralateral to the inactivation hemisphere. These findings reveal the causal contribution of the high-level visual cortex in eye movements.

Encoding of 3D physical dimensions by face-selective cortical neurons.

Neurons throughout the primate inferior temporal (IT) cortex respond selectively to visual images of faces and other complex objects. The response magnitude of neurons to a given image often depends on the size at which the image is presented, usually on a flat display at a fixed distance. While such size sensitivity might simply reflect the angular subtense of retinal image stimulation in degrees, one unexplored possibility is that it tracks the real-world geometry of physical objects, such as their size and distance to the observer in centimeters. This distinction bears fundamentally on the nature of object representation in IT and on the scope of visual operations supported by the ventral visual pathway. To address this question, we assessed the response dependency of neurons in the macaque anterior fundus (AF) face patch to the angular versus physical size of faces. We employed a macaque avatar to stereoscopically render three-dimensional (3D) photorealistic faces at multiple sizes and distances, including a subset of size/distance combinations designed to cast the same size retinal image projection. We found that most AF neurons were modulated principally by the 3D physical size of the face rather than its two-dimensional (2D) angular size on the retina. Further, most neurons responded strongest to extremely large and small faces, rather than to those of normal size. Together, these findings reveal a graded encoding of physical size among face patch neurons, providing evidence that category-selective regions of the primate ventral visual pathway participate in a geometric analysis of real-world objects.

Encoding of dynamic facial information in the middle dorsal face area.

Faces in motion reveal a plethora of information through visual dynamics. Faces can move in complex patterns while transforming facial shape, e.g., during the generation of different emotional expressions. While motion and shape processing have been studied extensively in separate research enterprises, much less is known about their conjunction during biological motion. Here, we took advantage of the discovery in brain-imaging studies of an area in the dorsal portion of the macaque monkey superior temporal sulcus (STS), the middle dorsal face area (MD), with selectivity for naturalistic face motion. To gain mechanistic insights into the coding of facial motion, we recorded single-unit activity from MD, testing whether and how MD cells encode face motion. The MD population was highly sensitive to naturalistic facial motion and facial shape. Some MD cells responded only to the conjunction of facial shape and motion, others were selective for facial shape even without movement, and yet others were suppressed by facial motion. We found that this heterogeneous MD population transforms face motion into a higher dimensional activity space, a representation that would allow for high sensitivity to relevant small-scale movements. Indeed, we show that many MD cells carry such sensitivity for eye movements. We further found that MD cells encode motion of head, mouth, and eyes in a separable manner, requiring the use of multiple reference frames. Thus, MD is a bona fide face-motion area that uses highly heterogeneous cell populations to create codes capturing even complex facial motion trajectories.

Perception and Memory Reinstatement Engage Overlapping Face-Selective Regions within Human Ventral Temporal Cortex.

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions specialized for processing specific sensory stimuli are reactivated to support content-specific retrieval. Recently, several studies have emphasized transformations in the spatial organization of these reinstated activity patterns. Specifically, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with the activity observed during perception. However, it is not clear that such transformations occur universally, with inconsistent evidence for other important stimulus categories, particularly faces. One challenge in addressing this question is the careful delineation of face-selective cortices, which are interdigitated with other selective regions, in configurations that spatially differ across individuals. Therefore, we conducted a multisession neuroimaging study to first carefully map individual participants' (nine males and seven females) face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the activity patterns within these regions during face memory encoding and retrieval. While face-selective regions were expectedly engaged during face perception at encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions, but did not show any consistent direction of spatial transformation (e.g., anteriorization). We also report on unique human intracranial recordings from VTC under the same experimental conditions. These findings highlight the importance of considering the complex configuration of category-selective cortex in elucidating principles shaping the neural transformations that occur from perception to memory.

A Multidimensional Neural Representation of Face Impressions.

From a glimpse of a face, people form trait impressions that operate as facial stereotypes, which are largely inaccurate yet nevertheless drive social behavior. Behavioral studies have long pointed to dimensions of trustworthiness and dominance that are thought to underlie face impressions due to their evolutionarily adaptive nature. Using human neuroimaging (N = 26, 19 female, 7 male), we identify a two-dimensional representation of faces' inferred traits in the middle temporal gyrus (MTG), a region involved in domain-general conceptual processing including the activation of social concepts. The similarity of neural-response patterns for any given pair of faces in the bilateral MTG was predicted by their proximity in trustworthiness-dominance space, an effect that could not be explained by mere visual similarity. This MTG trait-space representation occurred automatically, was relatively invariant across participants, and did not depend on the explicit endorsement of face impressions (i.e., beliefs that face impressions are valid and accurate). In contrast, regions involved in high-level social reasoning (the bilateral temporoparietal junction and posterior superior temporal sulcus; TPJ-pSTS) and entity-specific social knowledge (the left anterior temporal lobe; ATL) also exhibited this trait-space representation but only among participants who explicitly endorsed forming these impressions. Together, the findings identify a two-dimensional neural representation of face impressions and suggest that multiple implicit and explicit mechanisms give rise to biases based on facial appearance. While the MTG implicitly represents a multidimensional trait space for faces, the TPJ-pSTS and ATL are involved in the explicit application of this trait space for social evaluation and behavior.

Information content and temporal structure of face selective local field potentials frequency bands in IT cortex.

Sensory stimulation triggers synchronized bioelectrical activity in the brain across various frequencies. This study delves into network-level activities, specifically focusing on local field potentials as a neural signature of visual category representation. Specifically, we studied the role of different local field potential frequency oscillation bands in visual stimulus category representation by presenting images of faces and objects to three monkeys while recording local field potential from inferior temporal cortex. We found category selective local field potential responses mainly for animate, but not inanimate, objects. Notably, face-selective local field potential responses were evident across all tested frequency bands, manifesting in both enhanced (above mean baseline activity) and suppressed (below mean baseline activity) local field potential powers. We observed four different local field potential response profiles based on frequency bands and face selective excitatory and suppressive responses. Low-frequency local field potential bands (1-30 Hz) were more prodominstaly suppressed by face stimulation than the high-frequency (30-170 Hz) local field potential bands. Furthermore, the low-frequency local field potentials conveyed less face category informtion than the high-frequency local field potential in both enhansive and suppressive conditions. Furthermore, we observed a negative correlation between face/object d-prime values in all the tested local field potential frequency bands and the anterior-posterior position of the recording sites. In addition, the power of low-frequency local field potential systematically declined across inferior temporal anterior-posterior positions, whereas high-frequency local field potential did not exhibit such a pattern. In general, for most of the above-mentioned findings somewhat similar results were observed for body, but not, other stimulus categories. The observed findings suggest that a balance of face selective excitation and inhibition across time and cortical space shape face category selectivity in inferior temporal cortex.

Intracranial Electroencephalography and Deep Neural Networks Reveal Shared Substrates for Representations of Face Identity and Expressions.

According to a classical view of face perception (Bruce and Young, 1986; Haxby et al., 2000), face identity and facial expression recognition are performed by separate neural substrates (ventral and lateral temporal face-selective regions, respectively). However, recent studies challenge this view, showing that expression valence can also be decoded from ventral regions (Skerry and Saxe, 2014; Li et al., 2019), and identity from lateral regions (Anzellotti and Caramazza, 2017). These findings could be reconciled with the classical view if regions specialized for one task (either identity or expression) contain a small amount of information for the other task (that enables above-chance decoding). In this case, we would expect representations in lateral regions to be more similar to representations in deep convolutional neural networks (DCNNs) trained to recognize facial expression than to representations in DCNNs trained to recognize face identity (the converse should hold for ventral regions). We tested this hypothesis by analyzing neural responses to faces varying in identity and expression. Representational dissimilarity matrices (RDMs) computed from human intracranial recordings (n = 11 adults; 7 females) were compared with RDMs from DCNNs trained to label either identity or expression. We found that RDMs from DCNNs trained to recognize identity correlated with intracranial recordings more strongly in all regions tested-even in regions classically hypothesized to be specialized for expression. These results deviate from the classical view, suggesting that face-selective ventral and lateral regions contribute to the representation of both identity and expression.SIGNIFICANCE STATEMENT Previous work proposed that separate brain regions are specialized for the recognition of face identity and facial expression. However, identity and expression recognition mechanisms might share common brain regions instead. We tested these alternatives using deep neural networks and intracranial recordings from face-selective brain regions. Deep neural networks trained to recognize identity and networks trained to recognize expression learned representations that correlate with neural recordings. Identity-trained representations correlated with intracranial recordings more strongly in all regions tested, including regions hypothesized to be expression specialized in the classical hypothesis. These findings support the view that identity and expression recognition rely on common brain regions. This discovery may require reevaluation of the roles that the ventral and lateral neural pathways play in processing socially relevant stimuli.

Neurophysiological indices of face processing in people with psychosis and their siblings: An event-related potential study.

People with schizophrenia experience difficulties with social interactions. One contributor to these social deficits is dysfunction in processing facial features and facial emotional expressions. However, it is not known whether face processing deficits are evident in those with other psychotic disorders or in those genetically at-risk for psychosis (i.e., first-degree relatives of those with psychosis). We assessed event-related potentials (ERPs) during a facial and emotion processing task in 100 people with a diagnosis of schizophrenia or another psychotic condition (PSY), 32 of their siblings (SIB) and 45 healthy comparison participants (CTL). In separate blocks, participants identified the sex (male or female) or emotion (happy, angry, neutral) of faces. In a comparison condition, participants indicated whether buildings had one or two floors. ERPs were examined in two stages. First, we compared ERPs across the emotion, sex and building identification conditions. Second, we compared ERPs among the three different facial emotions. PSY exhibited significantly lower amplitudes over parietal-occipital regions between 111 and 151 ms when viewing faces but not buildings than CTL, consistent with a face-selective N170 ERP component deficit. The SIB group was intermediate for faces, but not significantly different than PSY or CTL. During emotion identification, all three groups showed increased N170 amplitudes to angry and happy versus neutral expressions, with no group differences. In follow up analyses, we examined differences between PSY with or without affective psychosis, and differences between those with schizophrenia versus other psychotic disorders; there were no significant differences in these analyses. Face processing deficits assessed with ERPs were observed in a group of diverse psychotic disorders, though deficits were not seen to be modulated by facial emotion expression. Additionally, N170 deficits are not evident in siblings of those with PSY.

Amygdala and cortical gamma-band responses to emotional faces are modulated by attention to valence.

The amygdala might support an attentional bias for emotional faces. However, whether and how selective attention toward a specific valence modulates this bias is not fully understood. Likewise, it is unclear whether amygdala and cortical signals respond to emotion and attention in a similar way. We recorded gamma-band activity (GBA, > 30 Hz) intracranially in the amygdalae of 11 patients with epilepsy and collected scalp recordings from 19 healthy participants. We presented angry, neutral, and happy faces randomly, and we denoted one valence as the target. Participants detected happy targets most quickly and accurately. In the amygdala, during attention to negative faces, low gamma-band activity (LGBA, < 90 Hz) increased for angry compared with happy faces from 160 ms. From 220 ms onward, amygdala high gamma-band activity (HGBA, > 90 Hz) was higher for angry and neutral faces than for happy ones. Monitoring neutral faces increased amygdala HGBA for emotions compared with neutral faces from 40 ms. Expressions were not differentiated in GBA while monitoring positive faces. On the scalp, only threat monitoring resulted in expression differentiation. Here, posterior LGBA was increased selectively for angry targets from 60 ms. The data show that GBA differentiation of emotional expressions is modulated by attention to valence: Top-down-controlled threat vigilance coordinates widespread GBA in favor of angry faces. Stimulus-driven emotion differentiation in amygdala GBA occurs during a neutral attentional focus. These findings align with a multi-pathway model of emotion processing and specify the role of GBA in this process.

Social impairment in survivors of pediatric brain tumors via reduced social attention and emotion-specific facial expression recognition.

BACKGROUND/OBJECTIVES: Survivors of pediatric brain tumors (SPBT) experience significant social challenges, including fewer friends and greater isolation than peers. Difficulties in face processing and visual social attention have been implicated in these outcomes. This study evaluated facial expression recognition (FER), social attention, and their associations with social impairments in SPBT. METHODS: SPBT (N = 54; ages 7-16) at least 2 years post treatment completed a measure of FER, while parents completed measures of social impairment. A subset (N = 30) completed a social attention assessment that recorded eye gaze patterns while watching videos depicting pairs of children engaged in joint play. Social Prioritization scores were calculated, with higher scores indicating more face looking. Correlations and regression analyses evaluated associations between variables, while a path analysis modeling tool (PROCESS) evaluated the indirect effects of Social Prioritization on social impairments through emotion-specific FER. RESULTS: Poorer recognition of angry and sad facial expressions was significantly correlated with greater social impairment. Social Prioritization was positively correlated with angry FER but no other emotions. Social Prioritization had significant indirect effects on social impairments through angry FER. CONCLUSION: Findings suggest interventions aimed at improving recognition of specific emotions may mitigate social impairments in SPBT. Further, reduced social attention (i.e., diminished face looking) could be a factor in reduced face processing ability, which may result in social impairments. Longitudinal research is needed to elucidate temporal associations between social attention, face processing, and social impairments.

A paleo-neurologic investigation of the social brain hypothesis in frontotemporal dementia.

The social brain hypothesis posits that a disproportionate encephalization in primates enabled to adapt behavior to a social context. Also, it has been proposed that phylogenetically recent brain areas are disproportionally affected by neurodegeneration. Using structural and functional magnetic resonance imaging, the present study investigates brain-behavior associations and neural integrity of hyperspecialized and domain-general cortical social brain areas in behavioral variant frontotemporal dementia (bvFTD). The results revealed that both structure and function of hyperspecialized social areas in the middle portion of the superior temporal sulcus (STS) are compromised in bvFTD, while no deterioration was observed in domain general social areas in the posterior STS. While the structural findings adhered to an anterior-posterior gradient, the functional group differences only occurred in the hyperspecialized locations. Activity in specialized regions was associated with structural integrity of the amygdala and with social deficits in bvFTD. In conclusion, the results are in line with the paleo-neurology hypothesis positing that neurodegeneration primarily hits cortical areas showing increased specialization, but also with the compatible alternative explanation that anterior STS regions degenerate earlier, based on stronger connections to and trans-neuronal spreading from regions affected early in bvFTD.

Neither Parents' Sex Nor the Type of Family Modulates Attentional Bias Toward Infant Faces: A Preliminary Study in Different-Sex and Same-Sex Parents.

An attentional bias toward infant versus adult faces has been detected in parents and positively associated with sensitive caregiving behaviors. In previous research, the attentional bias has been measured as the difference in attention, in terms of reaction times, captured by infant versus adult faces; the larger the difference, the greater the cognitive engagement that adults deployed to infant faces. However, research so far has been mostly confined to samples of mothers, who have been more represented than fathers. Moreover, new family forms, especially same-sex families of men, have been left out of research. To clarify potential sex differences and extend previous findings to diverse family forms, we implemented a modified Go/no-Go attentional task measuring attentional bias to infant faces in parents with children aged from 2 to 36 months. The sample (N = 86) was matched and included 22 fathers and 22 mothers from different-sex families and 20 fathers and 22 mothers from same-sex families. Overall, the results confirmed that infant faces induced a greater attentional bias compared to adult faces. Moreover, we found that neither the type of family nor parents' sex modulated the attentional bias toward infant faces. The findings are discussed in relation to the importance of understanding the correlates of parental response to infant cues going beyond a heteronormative perspective on parenting.

Perceptual narrowing continues throughout childhood: Evidence from specialization of face processing.

Face recognition shows a long trajectory of development and is known to be closely associated with the development of social skills. However, it is still debated whether this long trajectory is perceptually based and what the role is of experience-based refinements of face representations throughout development. We examined the effects of short and long-term experienced stimulus history on face processing, using regression biases of face representations towards the experienced mean. Children and adults performed same-different judgments in a serial discrimination task where two consecutive faces were drawn from a distribution of morphed faces. The results show that face recognition continues to improve after 9 years of age, with more pronounced improvements for own-race faces. This increased narrowing with age is also indicated by similar use of stimulus statistics for own-race and other-race faces in children, contrary to the different use of the overall stimulus history for these two face types in adults. Increased face proficiency in adulthood renders the perceptual system less tuned to other-race face statistics. Altogether, the results demonstrate associations between levels of specialization and the extent to which perceptual representations become narrowly tuned with age.

Resistance of a short-term memory concealed information test with famous faces to countermeasures.

The concealed information test (CIT) aims at identifying knowledge that a person wants to hide, by measuring physiological indices during the presentation of known versus unknown items. Recently, Lancry-Dayan et al. (Journal of Applied Research in Memory and Cognition, 7 (2), 291-302, 2018) proposed a new version of this test that included a short-term memory task to maximize differences between responses to items. Participants were asked to memorize four pictures of faces that included one face of an acquaintance. The authors observed that participants looked at the familiar face during the first second and then tended to avoid it. This specific orientation-avoidance pattern occurred even in participants instructed to conceal their familiarity with the known faces (in a spontaneous or a guided manner). In a first experiment, we reproduced Lancry-Dayan et al.'s (2018) study using photos of famous faces. The pattern found by Lancry-Dayan et al. was observed in participants asked to perform the memory task only, participants asked to conceal their familiarity with the famous faces, and participants of a countermeasure group. In a second experiment, we tested the robustness of Lancry-Dayan et al.'s countermeasure. We modified the instructions by emphasizing the oculomotor task or giving feedback. While between-group differences in gaze-pattern appeared after feedback was provided, classification analyses were still able to distinguish between familiar and unfamiliar faces accurately, which revealed the good resistance of this new CIT protocol to countermeasures.

Capuchin monkeys' (Sapajus [Cebus] apella) categorization of photos of unknown male conspecifics suggests attention to fWHR and a dominance bias.

The ability to quickly perceive others' rank minimizes costs by helping individuals behave appropriately when interacting with strangers. Indeed, humans and at least some other species can quickly determine strangers' rank or dominance based only on physical features without observing others' interactions or behavior. Nonhuman primates can determine strangers' ranks by observing their interactions, and some evidence suggests that at least some cues to dominance, such as facial width-to-height ratio (fWHR), are also present in other primates. However, it is unknown whether they can determine strangers' rank simply by looking at their faces, rather than observing their interactions. If so, this would suggest selective pressure across the primates on both cues to dominance and the ability to detect those cues accurately. To address this, we examined the ability of male and female tufted capuchin monkeys (Sapajus [Cebus] apella) to categorize images of the faces of unknown conspecifics (Sapajus from different colonies) and humans (computer-generated and real) as dominant or nondominant based only on still images. Capuchins' categorization of unknown conspecific faces was consistent with fWHR, a cue to dominance, although there was a strong tendency to categorize strangers as dominant, particularly for males. This was true despite the continued correct categorization of known individuals. In addition, capuchins did not categorize human strangers in accordance with external pre-ratings of dominance by independent human raters, despite the availability of the same cue, fWHR. We consider these results in the context of capuchin socio-ecology and what they mean for the evolution of rapid decision-making in social contexts.

Face masks degrade our ability to remember face-name associations more than predicted by judgments of learning.

In response to the COVID-19 pandemic, face masks became required attire. Face masks obstruct the bottom portion of faces, restricting face processing. The present study examined the influence face masks have on memory predictions and memory performance for new face-name associations. Participants studied face-name pairs presented for 8 s (Experiment 1) or 10 s (Experiment 2). Half of the face-name pairs included a face mask obstructing the nose and mouth of the pictured face, counterbalanced across participants. Participants provided item-by-item judgements of learning (JOLs) and completed subsequent cued recall and associative recognition memory tests. Both experiments demonstrated that face masks impaired memory for newly-learned names, however, the magnitude of the mask impact was under-predicted by JOLs. The presence of a face mask negatively influenced memory performance to a greater degree than participants' JOLs predicted. Results have implications for name learning during pandemics, as well as in settings where face masks are common (e.g., medical field).

Joint encoding of facial identity, orientation, gaze, and expression in the middle dorsal face area.

The last two decades have established that a network of face-selective areas in the temporal lobe of macaque monkeys supports the visual processing of faces. Each area within the network contains a large fraction of face-selective cells. And each area encodes facial identity and head orientation differently. A recent brain-imaging study discovered an area outside of this network selective for naturalistic facial motion, the middle dorsal (MD) face area. This finding offers the opportunity to determine whether coding principles revealed inside the core network would generalize to face areas outside the core network. We investigated the encoding of static faces and objects, facial identity, and head orientation, dimensions which had been studied in multiple areas of the core face-processing network before, as well as facial expressions and gaze. We found that MD populations form a face-selective cluster with a degree of selectivity comparable to that of areas in the core face-processing network. MD encodes facial identity robustly across changes in head orientation and expression, it encodes head orientation robustly against changes in identity and expression, and it encodes expression robustly across changes in identity and head orientation. These three dimensions are encoded in a separable manner. Furthermore, MD also encodes the direction of gaze in addition to head orientation. Thus, MD encodes both structural properties (identity) and changeable ones (expression and gaze) and thus provides information about another animal's direction of attention (head orientation and gaze). MD contains a heterogeneous population of cells that establish a multidimensional code for faces.

Shared neural codes for visual and semantic information about familiar faces in a common representational space.

Processes evoked by seeing a personally familiar face encompass recognition of visual appearance and activation of social and person knowledge. Whereas visual appearance is the same for all viewers, social and person knowledge may be more idiosyncratic. Using between-subject multivariate decoding of hyperaligned functional magnetic resonance imaging data, we investigated whether representations of personally familiar faces in different parts of the distributed neural system for face perception are shared across individuals who know the same people. We found that the identities of both personally familiar and merely visually familiar faces were decoded accurately across brains in the core system for visual processing, but only the identities of personally familiar faces could be decoded across brains in the extended system for processing nonvisual information associated with faces. Our results show that personal interactions with the same individuals lead to shared neural representations of both the seen and unseen features that distinguish their identities.