A uniform human multimodal dataset for emotion perception and judgment.

Face perception is a fundamental aspect of human social interaction, yet most research on this topic has focused on single modalities and specific aspects of face perception. Here, we present a comprehensive multimodal dataset for examining facial emotion perception and judgment. This dataset includes EEG data from 97 unique neurotypical participants across 8 experiments, fMRI data from 19 neurotypical participants, single-neuron data from 16 neurosurgical patients (22 sessions), eye tracking data from 24 neurotypical participants, behavioral and eye tracking data from 18 participants with ASD and 15 matched controls, and behavioral data from 3 rare patients with focal bilateral amygdala lesions. Notably, participants from all modalities performed the same task. Overall, this multimodal dataset provides a comprehensive exploration of facial emotion perception, emphasizing the importance of integrating multiple modalities to gain a holistic understanding of this complex cognitive process. This dataset serves as a key missing link between human neuroimaging and neurophysiology literature, and facilitates the study of neuropsychiatric populations.

The role of visual imagery in face recognition and the construction of facial composites. Evidence from Aphantasia.

People with aphantasia have a markedly impaired ability to form visual images in the mind's eye. Here, by testing people with and without aphantasia, we examine the relationship between visual imagery and face processing. We show that aphantasics have weaker face recognition than people with visual imagery, using both self-report (Prosopagnosia Index) and behavioural measures (Cambridge Face Memory Test). However, aphantasics nonetheless have a fully intact ability to construct facial composites from memory (i.e., composites produced using EFIT6 by aphantasics and imagers were rated as equally accurate in terms of their resemblance to a target face). Additionally, we show that aphantasics were less able than imagers to see the resemblance between composites and a target face, suggestive of potential issues with face matching (perception). Finally, we show that holistic and featural methods of composite construction using EFIT6 produce equally accurate composites. Our results suggest that face recognition, but not face composite construction, is facilitated by the ability to represent visual properties as 'pictures in the mind'. Our findings have implications for the study of aphantasia, and also for forensic settings, where face composite systems are commonly used to aid criminal investigations.

Robust holistic face processing in early childhood during the COVID-19 pandemic.

The timing of the developmental emergence of holistic face processing and its sensitivity to experience in early childhood are somewhat controversial topics. To investigate holistic face perception in early childhood, we used an online testing platform and administered a two-alternative forced-choice task to 4-, 5-, and 6-year-old children. The children saw pairs of composite faces and needed to decide whether the faces were the same or different. To determine whether experience with masked faces may have negatively affected holistic processing, we also administered a parental questionnaire to assess the children's exposure to masked faces during the COVID-19 pandemic. We found that all three age groups performed holistic face processing when the faces were upright (Experiment 1) but not when the faces were inverted (Experiment 2), that response accuracy increased with age, and that response accuracy was not related to degree of exposure to masked faces. These results indicate that holistic face processing is relatively robust in early childhood and that short-term exposure to partially visible faces does not negatively affect young children's holistic face perception.

Developmental emergence of holistic processing in word recognition.

Holistic processing (HP) of faces refers to the obligatory, simultaneous processing of the parts and their relations, and it emerges over the course of development. HP is manifest in a decrement in the perception of inverted versus upright faces and a reduction in face processing ability when the relations between parts are perturbed. Here, adopting the HP framework for faces, we examined the developmental emergence of HP in another domain for which human adults have expertise, namely, visual word processing. Children, adolescents, and adults performed a lexical decision task and we used two established signatures of HP for faces: the advantage in perception of upright over inverted words and nonwords and the reduced sensitivity to increasing parts (word length). Relative to the other groups, children showed less of an advantage for upright versus inverted trials and lexical decision was more affected by increasing word length. Performance on these HP indices was strongly associated with age and with reading proficiency. Also, the emergence of HP for word perception was not simply a result of improved visual perception over the course of development as no group differences were observed on an object decision task. These results reveal the developmental emergence of HP for orthographic input, and reflect a further instance of experience-dependent tuning of visual perception. These results also add to existing findings on the commonalities of mechanisms of word and face recognition. RESEARCH HIGHLIGHTS: Children showed less of an advantage for upright versus inverted trials compared to adolescents and adults. Relative to the other groups, lexical decision in children was more affected by increasing word length. Performance on holistic processing (HP) indices was strongly associated with age and with reading proficiency. HP emergence for word perception was not due to improved visual perception over development as there were no group differences on an object decision task.

Investigating the Influence of Autism Spectrum Traits on Face Processing Mechanisms in Developmental Prosopagnosia.

Autism traits are common exclusionary criteria in developmental prosopagnosia (DP) studies. We investigated whether autism traits produce qualitatively different face processing in 43 DPs with high vs. low autism quotient (AQ) scores. Compared to controls (n = 27), face memory and perception were similarly deficient in the high- and low-AQ DPs, with the high-AQ DP group additionally showing deficient face emotion recognition. Task-based fMRI revealed reduced occipito-temporal face selectivity in both groups, with high-AQ DPs additionally demonstrating decreased posterior superior temporal sulcus selectivity. Resting-state fMRI showed similar reduced face-selective network connectivity in both DP groups compared with controls. Together, this demonstrates that high- and low-AQ DP groups have very similar face processing deficits, with additional facial emotion deficits in high-AQ DPs.

Attentional Features of Mindfulness are Better Predictors of Face Recognition than Empathy and Compassion-Based Constructs.

Recent research has employed measures of either empathy, compassion or mindfulness and linked better face recognition memory to higher scores of identification with all humanity and mindfulness but not empathy or compassion. Additionally, empathy, compassion and mindfulness have been suggested as concepts that intertwine, but research has not yet examined how their respective personality questionnaires map onto latent concepts. We employed these measures together to explore their factor structure and, using structural equation modelling, we investigated if the suggested latent variables predict recognition memory performance for face and non-face stimuli. Attentional notions of mindfulness described a latent factor that predicted face recognition. All self-compassion facets and the non-react mindfulness facet described a latent factor, which predicted false alarms in face recognition. Finally, empathy and compassion-based notions described one latent factor, which did not predict recognition performance. None of the latent variables predicted performance in either object or voice recognition. Collectively, findings indicate attention-based mindfulness to benefit face recognition, prompting further research into the potential of mindfulness to support the face recognition process.

Individual differences in classification images of Mooney faces.

Human face recognition is robust even under conditions of extreme lighting and in situations where there is high noise and uncertainty. Mooney faces are a canonical example of this: Mooney faces are two-tone shadow-defined images that are readily and holistically recognized despite lacking easily segmented face features. Face perception in such impoverished situations-and Mooney face perception in particular-is often thought to be supported by comparing encountered faces to stored templates. Here, we used a classification image approach to measure the templates that observers use to recognize Mooney faces. Visualizing these templates reveals the regions and structures of the image that best predict individual observer recognition, and they reflect the underlying internal representation of faces. Using this approach, we tested whether there are classification images that are consistent from session to session, whether the classification images are observer-specific, and whether they allow for pattern completion of holistic representations even in the absence of an underlying signal. We found that classification images of Mooney faces were indeed non-random (i.e., consistent session from session) within each observer, but they were different between observers. This result is in line with previously proposed existence of face templates that support face recognition, and further suggests that these templates may be unique to each observer and could drive idiosyncratic individual differences in holistic face recognition. Moreover, we found classification images that reflected information within the blank regions of the original Mooney faces, suggesting that observers may fill in missing information using idiosyncratic internal information about faces.

Transcutaneous auricular vagus nerve stimulation increases eye-gaze on salient facial features and oxytocin release.

Non-invasive, transcutaneous electrical stimulation of the auricular branch of the vagus nerve (taVNS) via the ear is used therapeutically in epilepsy, pain, and depression, and may also have beneficial effects on social cognition. However, the underlying mechanisms of taVNS are unclear and evidence regarding its role in social cognition improvement is limited. To investigate the impact of taVNS on social cognition we have studied its effects on gaze toward emotional faces in combination with eye-tracking and on the release of the neuropeptide oxytocin which plays a key role in influencing social cognition and motivation. A total of 54 subjects were enrolled (49 were included in the final analysis) in a sham-controlled, participant-blind, crossover experiment, consisting of two treatment sessions 1 week apart. In one session participants received 30-min taVNS (tragus), and in the other, they received 30-min sham (earlobe) stimulation with the treatment order counterbalanced. The proportion of time spent viewing the faces and facial features (eyes, nose, and mouth) was measured together with resting pupil size. Additionally, saliva samples were taken for the measurement of oxytocin concentrations by enzyme-linked immunoassay. Saliva oxytocin concentrations increased significantly after taVNS compared to sham stimulation, while resting pupil size did not. In addition, taVNS increased time spent viewing the nose region irrespective of face emotion, and this was positively correlated with increased saliva oxytocin concentrations. Our findings suggest that taVNS biases visual attention toward socially salient facial features across different emotions and this is associated with its effects on increasing endogenous oxytocin release.

Relative tuning of holistic face processing towards the fovea.

Humans quickly detect and gaze at faces in the world, which reflects their importance in cognition and may lead to tuning of face recognition toward the central visual field. Although sometimes reported, foveal selectivity in face processing is debated: brain imaging studies have found evidence for a central field bias specific to faces, but behavioral studies have found little foveal selectivity in face recognition. These conflicting results are difficult to reconcile, but they could arise from stimulus-specific differences. Recent studies, for example, suggest that individual faces vary in the degree to which they require holistic processing. Holistic processing is the perception of faces as a whole rather than as a set of separate features. We hypothesized that the dissociation between behavioral and neuroimaging studies arises because of this stimulus-specific dependence on holistic processing. Specifically, the central bias found in neuroimaging studies may be specific to holistic processing. Here, we tested whether the eccentricity-dependence of face perception is determined by the degree to which faces require holistic processing. We first measured the holistic-ness of individual Mooney faces (two-tone shadow images readily perceived as faces). In a group of independent observers, we then used a gender discrimination task to measured recognition of these Mooney faces as a function of their eccentricity. Face gender was recognized across the visual field, even at substantial eccentricities, replicating prior work. Importantly, however, holistic face gender recognition was relatively tuned-slightly, but reliably stronger in the central visual field. Our results may reconcile the debate on the eccentricity-dependance of face perception and reveal a spatial inhomogeneity specifically in the holistic representations of faces.

Face perception: computational insights from phylogeny.

Studies of face perception in primates elucidate the psychological and neural mechanisms that support this critical and complex ability. Recent progress in characterizing face perception across species, for example in insects and reptiles, has highlighted the ubiquity over phylogeny of this key ability for social interactions and survival. Here, we review the competence in face perception across species and the types of computation that support this behavior. We conclude that the computational complexity of face perception evinced by a species is not related to phylogenetic status and is, instead, largely a product of environmental context and social and adaptive pressures. Integrating findings across evolutionary data permits the derivation of computational principles that shed further light on primate face perception.

Does Cross-Race Contact Improve Cross-Race Face Perception? A Meta-Analysis of the Cross-Race Deficit and Contact.

Contact with racial outgroups is thought to reduce the cross-race recognition deficit (CRD), the tendency for people to recognize same-race (i.e., ingroup) faces more accurately than cross-race (i.e., outgroup) faces. In 2001, Meissner and Brigham conducted a meta-analysis in which they examined this question and found a meta-analytic effect of r = -.13. We conduct a new meta-analysis based on 20 years of additional data to update the estimate of this relationship and examine theoretical and methodological moderators of the effect. We find a meta-analytic effect of r = -.15. In line with theoretical predictions, we find some evidence that the magnitude of this relationship is stronger when contact occurs during childhood rather than adulthood. We find no evidence that the relationship differs for measures of holistic/configural processing compared with normal processing. Finally, we find that the magnitude of the relationship depends on the operationalization of contact and that it is strongest when contact is manipulated. We consider recommendations for further research on this topic.

Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning.

Facial expression recognition (FER) has received significant attention in the past decade with witnessed progress, but data inconsistencies among different FER datasets greatly hinder the generalization ability of the models learned on one dataset to another. Recently, a series of cross-domain FER algorithms (CD-FERs) have been extensively developed to address this issue. Although each declares to achieve superior performance, comprehensive and fair comparisons are lacking due to inconsistent choices of the source/target datasets and feature extractors. In this work, we first propose to construct a unified CD-FER evaluation benchmark, in which we re-implement the well-performing CD-FER and recently published general domain adaptation algorithms and ensure that all these algorithms adopt the same source/target datasets and feature extractors for fair CD-FER evaluations. Based on the analysis, we find that most of the current state-of-the-art algorithms use adversarial learning mechanisms that aim to learn holistic domain-invariant features to mitigate domain shifts. However, these algorithms ignore local features, which are more transferable across different datasets and carry more detailed content for fine-grained adaptation. Therefore, we develop a novel adversarial graph representation adaptation (AGRA) framework that integrates graph representation propagation with adversarial learning to realize effective cross-domain holistic-local feature co-adaptation. Specifically, our framework first builds two graphs to correlate holistic and local regions within each domain and across different domains, respectively. Then, it extracts holistic-local features from the input image and uses learnable per-class statistical distributions to initialize the corresponding graph nodes. Finally, two stacked graph convolution networks (GCNs) are adopted to propagate holistic-local features within each domain to explore their interaction and across different domains for holistic-local feature co-adaptation. In this way, the AGRA framework can adaptively learn fine-grained domain-invariant features and thus facilitate cross-domain expression recognition. We conduct extensive and fair comparisons on the unified evaluation benchmark and show that the proposed AGRA framework outperforms previous state-of-the-art methods.

Distinct neural-behavioral correspondence within face processing and attention networks for the composite face effect.

The composite face effect (CFE) is recognized as a hallmark for holistic face processing, but our knowledge remains sparse about its cognitive and neural loci. Using functional magnetic resonance imaging with independent localizer and complete composite face task, we here investigated its neural-behavioral correspondence within face processing and attention networks. Complementing classical comparisons, we adopted a dimensional reduction approach to explore the core cognitive constructs of the behavioral CFE measurement. Our univariate analyses found an alignment effect in regions associated with both the extended face processing network and attention networks. Further representational similarity analyses based on Euclidian distances among all experimental conditions were used to identify cortical regions with reliable neural-behavioral correspondences. Multidimensional scaling and hierarchical clustering analyses for neural-behavioral correspondence data revealed two principal components underlying the behavioral CFE effect, which fit best to the neural responses in the bilateral insula and medial frontal gyrus. These findings highlight the distinct neurocognitive contributions of both face processing and attentional networks to the behavioral CFE outcome, which bridge the gaps between face recognition and attentional control models.

Holistic face recognition is an emergent phenomenon of spatial processing in face-selective regions.

Spatial processing by receptive fields is a core property of the visual system. However, it is unknown how spatial processing in high-level regions contributes to recognition behavior. As face inversion is thought to disrupt typical holistic processing of information in faces, we mapped population receptive fields (pRFs) with upright and inverted faces in the human visual system. Here we show that in face-selective regions, but not primary visual cortex, pRFs and overall visual field coverage are smaller and shifted downward in response to face inversion. From these measurements, we successfully predict the relative behavioral detriment of face inversion at different positions in the visual field. This correspondence between neural measurements and behavior demonstrates how spatial processing in face-selective regions may enable holistic perception. These results not only show that spatial processing in high-level visual regions is dynamically used towards recognition, but also suggest a powerful approach for bridging neural computations by receptive fields to behavior.

A fast link between face perception and memory in the temporal pole.

The question of how the brain recognizes the faces of familiar individuals has been important throughout the history of neuroscience. Cells linking visual processing to person memory have been proposed but not found. Here, we report the discovery of such cells through recordings from an area in the macaque temporal pole identified with functional magnetic resonance imaging. These cells responded to faces that were personally familiar. They responded nonlinearly to stepwise changes in face visibility and detail and holistically to face parts, reflecting key signatures of familiar face recognition. They discriminated between familiar identities, as fast as a general face identity area. The discovery of these cells establishes a new pathway for the fast recognition of familiar individuals.

Beyond imagination: Hypnotic visual hallucination induces greater lateralised brain activity than visual mental imagery.

Hypnotic suggestions can produce a broad range of perceptual experiences, including hallucinations. Visual hypnotic hallucinations differ in many ways from regular mental images. For example, they are usually experienced as automatic, vivid, and real images, typically compromising the sense of reality. While both hypnotic hallucination and mental imagery are believed to mainly rely on the activation of the visual cortex via top-down mechanisms, it is unknown how they differ in the neural processes they engage. Here we used an adaptation paradigm to test and compare top-down processing between hypnotic hallucination, mental imagery, and visual perception in very highly hypnotisable individuals whose ability to hallucinate was assessed. By measuring the N170/VPP event-related complex and using multivariate decoding analysis, we found that hypnotic hallucination of faces involves greater top-down activation of sensory processing through lateralised neural mechanisms in the right hemisphere compared to mental imagery. Our findings suggest that the neural signatures that distinguish hypnotically hallucinated faces from imagined faces lie in the right brain hemisphere.

The N400 and late occipital positivity in processing dynamic facial expressions with natural emotional voice.

People require multimodal emotional interactions to live in a social environment. Several studies using dynamic facial expressions and emotional voices have reported that multimodal emotional incongruency evokes an early sensory component of event-related potentials (ERPs), while others have found a late cognitive component. The integration mechanism of two different results remains unclear. We speculate that it is semantic analysis in a multimodal integration framework that evokes the late ERP component. An electrophysiological experiment was conducted using emotionally congruent or incongruent dynamic faces and natural voices to promote semantic analysis. To investigate the top-down modulation of the ERP component, attention was manipulated via two tasks that directed participants to attend to facial versus vocal expressions. Our results revealed interactions between facial and vocal emotional expressions, manifested as modulations of the auditory N400 ERP amplitudes but not N1 and P2 amplitudes, for incongruent emotional face-voice combinations only in the face-attentive task. A late occipital positive potential amplitude emerged only during the voice-attentive task. Overall, these findings support the idea that semantic analysis is a key factor in evoking the late cognitive component. The task effect for these ERPs suggests that top-down attention alters not only the amplitude of ERP but also the ERP component per se. Our results implicate a principle of emotional face-voice processing in the brain that may underlie complex audiovisual interactions in everyday communication.

Effects of Ayahuasca on the Recognition of Facial Expressions of Emotions in Naive Healthy Volunteers: A Pilot, Proof-of-Concept, Randomized Controlled Trial.

BACKGROUND: The recognition of emotions in facial expressions (REFE) is a core aspect of social cognition. Previous studies with the serotonergic hallucinogens lysergic acid diethylamide and psilocybin showed that these drugs reduced the recognition of negative (fear) faces in healthy volunteers. This trial assessed the acute and prolonged effects of a single dose of ayahuasca on the REFE. METHODS: Twenty-two healthy volunteers participated in a pilot, proof-of-concept, randomized trial. Study variables included a REFE task performed before and 4 hours after drug intake, subjective effects (self-reports/observer impressions), tolerability measures (cardiovascular measures, self-reports), and brain-derived neurotrophic factor plasma levels. The REFE task was applied again 1, 7, 14, and 21 days and 3 months after drug intake. Stability of ayahuasca alkaloids during the study was also assessed (room temperature, 18 months). FINDINGS: Compared with placebo, ayahuasca did not modify the REFE. No significant effects were observed on cardiovascular measures and brain-derived neurotrophic factor levels. Volunteers reported visual effects, tranquility/relaxation, and well-being, with few reports of transient anxiety/confusion. Ayahuasca was well tolerated, producing mainly nausea, gastrointestinal discomfort, and vomiting. A significant time-dependent deterioration of alkaloids was observed, especially for dimethyltryptamine. CONCLUSIONS: Absence of significant effects on the REFE task could be due to lack of effects of ayahuasca (at the doses used), alkaloid degradation, learning effects, and the high educational level of the sample. Further trials with different samples are needed to better understand the effects of ayahuasca and other serotonergic hallucinogens on the REFE. Future trials should improve methods to guarantee the stability of ayahuasca alkaloids.

Audiovisual integration in macaque face patch neurons.

Primate social communication depends on the perceptual integration of visual and auditory cues, reflected in the multimodal mixing of sensory signals in certain cortical areas. The macaque cortical face patch network, identified through visual, face-selective responses measured with fMRI, is assumed to contribute to visual social interactions. However, whether face patch neurons are also influenced by acoustic information, such as the auditory component of a natural vocalization, remains unknown. Here, we recorded single-unit activity in the anterior fundus (AF) face patch, in the superior temporal sulcus, and anterior medial (AM) face patch, on the undersurface of the temporal lobe, in macaques presented with audiovisual, visual-only, and auditory-only renditions of natural movies of macaques vocalizing. The results revealed that 76% of neurons in face patch AF were significantly influenced by the auditory component of the movie, most often through enhancement of visual responses but sometimes in response to the auditory stimulus alone. By contrast, few neurons in face patch AM exhibited significant auditory responses or modulation. Control experiments in AF used an animated macaque avatar to demonstrate, first, that the structural elements of the face were often essential for audiovisual modulation and, second, that the temporal modulation of the acoustic stimulus was more important than its frequency spectrum. Together, these results identify a striking contrast between two face patches and specifically identify AF as playing a potential role in the integration of audiovisual cues during natural modes of social communication.

Face-Selective Units in Human Ventral Temporal Cortex Reactivate during Free Recall.

Research in functional neuroimaging has suggested that category-selective regions of visual cortex, including the ventral temporal cortex (VTC), can be reactivated endogenously through imagery and recall. Face representation in the monkey face-patch system has been well studied and is an attractive domain in which to explore these processes in humans. The VTCs of 8 human subjects (4 female) undergoing invasive monitoring for epilepsy surgery were implanted with microelectrodes. Most (26 of 33) category-selective units showed specificity for face stimuli. Different face exemplars evoked consistent and discriminable responses in the population of units sampled. During free recall, face-selective units preferentially reactivated in the absence of visual stimulation during a 2 s window preceding face recall events. Furthermore, we show that in at least 1 subject, the identity of the recalled face could be predicted by comparing activity preceding recall events to activity evoked by visual stimulation. We show that face-selective units in the human VTC are reactivated endogenously, and present initial evidence that consistent representations of individual face exemplars are specifically reactivated in this manner.SIGNIFICANCE STATEMENT The role of "top-down" endogenous reactivation of native representations in higher sensory areas is poorly understood in humans. We conducted the first detailed single-unit survey of ventral temporal cortex (VTC) in human subjects, showing that, similarly to nonhuman primates, humans encode different faces using different rate codes. Then, we demonstrated that, when subjects recalled and imagined a given face, VTC neurons reactivated with the same rate codes as when subjects initially viewed that face. This suggests that the VTC units not only carry durable representations of faces, but that those representations can be endogenously reactivated via "top-down" mechanisms.