A uniform human multimodal dataset for emotion perception and judgment.

Face perception is a fundamental aspect of human social interaction, yet most research on this topic has focused on single modalities and specific aspects of face perception. Here, we present a comprehensive multimodal dataset for examining facial emotion perception and judgment. This dataset includes EEG data from 97 unique neurotypical participants across 8 experiments, fMRI data from 19 neurotypical participants, single-neuron data from 16 neurosurgical patients (22 sessions), eye tracking data from 24 neurotypical participants, behavioral and eye tracking data from 18 participants with ASD and 15 matched controls, and behavioral data from 3 rare patients with focal bilateral amygdala lesions. Notably, participants from all modalities performed the same task. Overall, this multimodal dataset provides a comprehensive exploration of facial emotion perception, emphasizing the importance of integrating multiple modalities to gain a holistic understanding of this complex cognitive process. This dataset serves as a key missing link between human neuroimaging and neurophysiology literature, and facilitates the study of neuropsychiatric populations.

Robust holistic face processing in early childhood during the COVID-19 pandemic.

The timing of the developmental emergence of holistic face processing and its sensitivity to experience in early childhood are somewhat controversial topics. To investigate holistic face perception in early childhood, we used an online testing platform and administered a two-alternative forced-choice task to 4-, 5-, and 6-year-old children. The children saw pairs of composite faces and needed to decide whether the faces were the same or different. To determine whether experience with masked faces may have negatively affected holistic processing, we also administered a parental questionnaire to assess the children's exposure to masked faces during the COVID-19 pandemic. We found that all three age groups performed holistic face processing when the faces were upright (Experiment 1) but not when the faces were inverted (Experiment 2), that response accuracy increased with age, and that response accuracy was not related to degree of exposure to masked faces. These results indicate that holistic face processing is relatively robust in early childhood and that short-term exposure to partially visible faces does not negatively affect young children's holistic face perception.

Distinct neural-behavioral correspondence within face processing and attention networks for the composite face effect.

The composite face effect (CFE) is recognized as a hallmark for holistic face processing, but our knowledge remains sparse about its cognitive and neural loci. Using functional magnetic resonance imaging with independent localizer and complete composite face task, we here investigated its neural-behavioral correspondence within face processing and attention networks. Complementing classical comparisons, we adopted a dimensional reduction approach to explore the core cognitive constructs of the behavioral CFE measurement. Our univariate analyses found an alignment effect in regions associated with both the extended face processing network and attention networks. Further representational similarity analyses based on Euclidian distances among all experimental conditions were used to identify cortical regions with reliable neural-behavioral correspondences. Multidimensional scaling and hierarchical clustering analyses for neural-behavioral correspondence data revealed two principal components underlying the behavioral CFE effect, which fit best to the neural responses in the bilateral insula and medial frontal gyrus. These findings highlight the distinct neurocognitive contributions of both face processing and attentional networks to the behavioral CFE outcome, which bridge the gaps between face recognition and attentional control models.

Holistic face recognition is an emergent phenomenon of spatial processing in face-selective regions.

Spatial processing by receptive fields is a core property of the visual system. However, it is unknown how spatial processing in high-level regions contributes to recognition behavior. As face inversion is thought to disrupt typical holistic processing of information in faces, we mapped population receptive fields (pRFs) with upright and inverted faces in the human visual system. Here we show that in face-selective regions, but not primary visual cortex, pRFs and overall visual field coverage are smaller and shifted downward in response to face inversion. From these measurements, we successfully predict the relative behavioral detriment of face inversion at different positions in the visual field. This correspondence between neural measurements and behavior demonstrates how spatial processing in face-selective regions may enable holistic perception. These results not only show that spatial processing in high-level visual regions is dynamically used towards recognition, but also suggest a powerful approach for bridging neural computations by receptive fields to behavior.

Beyond imagination: Hypnotic visual hallucination induces greater lateralised brain activity than visual mental imagery.

Hypnotic suggestions can produce a broad range of perceptual experiences, including hallucinations. Visual hypnotic hallucinations differ in many ways from regular mental images. For example, they are usually experienced as automatic, vivid, and real images, typically compromising the sense of reality. While both hypnotic hallucination and mental imagery are believed to mainly rely on the activation of the visual cortex via top-down mechanisms, it is unknown how they differ in the neural processes they engage. Here we used an adaptation paradigm to test and compare top-down processing between hypnotic hallucination, mental imagery, and visual perception in very highly hypnotisable individuals whose ability to hallucinate was assessed. By measuring the N170/VPP event-related complex and using multivariate decoding analysis, we found that hypnotic hallucination of faces involves greater top-down activation of sensory processing through lateralised neural mechanisms in the right hemisphere compared to mental imagery. Our findings suggest that the neural signatures that distinguish hypnotically hallucinated faces from imagined faces lie in the right brain hemisphere.

The N400 and late occipital positivity in processing dynamic facial expressions with natural emotional voice.

People require multimodal emotional interactions to live in a social environment. Several studies using dynamic facial expressions and emotional voices have reported that multimodal emotional incongruency evokes an early sensory component of event-related potentials (ERPs), while others have found a late cognitive component. The integration mechanism of two different results remains unclear. We speculate that it is semantic analysis in a multimodal integration framework that evokes the late ERP component. An electrophysiological experiment was conducted using emotionally congruent or incongruent dynamic faces and natural voices to promote semantic analysis. To investigate the top-down modulation of the ERP component, attention was manipulated via two tasks that directed participants to attend to facial versus vocal expressions. Our results revealed interactions between facial and vocal emotional expressions, manifested as modulations of the auditory N400 ERP amplitudes but not N1 and P2 amplitudes, for incongruent emotional face-voice combinations only in the face-attentive task. A late occipital positive potential amplitude emerged only during the voice-attentive task. Overall, these findings support the idea that semantic analysis is a key factor in evoking the late cognitive component. The task effect for these ERPs suggests that top-down attention alters not only the amplitude of ERP but also the ERP component per se. Our results implicate a principle of emotional face-voice processing in the brain that may underlie complex audiovisual interactions in everyday communication.

Audiovisual integration in macaque face patch neurons.

Primate social communication depends on the perceptual integration of visual and auditory cues, reflected in the multimodal mixing of sensory signals in certain cortical areas. The macaque cortical face patch network, identified through visual, face-selective responses measured with fMRI, is assumed to contribute to visual social interactions. However, whether face patch neurons are also influenced by acoustic information, such as the auditory component of a natural vocalization, remains unknown. Here, we recorded single-unit activity in the anterior fundus (AF) face patch, in the superior temporal sulcus, and anterior medial (AM) face patch, on the undersurface of the temporal lobe, in macaques presented with audiovisual, visual-only, and auditory-only renditions of natural movies of macaques vocalizing. The results revealed that 76% of neurons in face patch AF were significantly influenced by the auditory component of the movie, most often through enhancement of visual responses but sometimes in response to the auditory stimulus alone. By contrast, few neurons in face patch AM exhibited significant auditory responses or modulation. Control experiments in AF used an animated macaque avatar to demonstrate, first, that the structural elements of the face were often essential for audiovisual modulation and, second, that the temporal modulation of the acoustic stimulus was more important than its frequency spectrum. Together, these results identify a striking contrast between two face patches and specifically identify AF as playing a potential role in the integration of audiovisual cues during natural modes of social communication.

Investigating holistic face processing within and outside of face-responsive brain regions.

It has been shown that human faces are processed holistically (i.e. as indecomposable wholes, rather than by their component parts) and this holistic face processing is linked to brain activity in face-responsive brain regions. Although several brain regions outside of the face-responsive network are also sensitive to relational processing and perceptual grouping, whether these non-face-responsive regions contribute to holistic processing remains unclear. Here, we investigated holistic face processing in the composite face paradigm both within and outside of face-responsive brain regions. We recorded participants' brain activity using fMRI while they performed a composite face task. Behavioural results indicate that participants tend to judge the same top face halves as different when they are aligned with different bottom face halves but not when they are misaligned, demonstrating a composite face effect. Neuroimaging results revealed significant differences in responses to aligned and misaligned faces in the lateral occipital complex (LOC), and trends in the anterior part of the fusiform face area (FFA2) and transverse occipital sulcus (TOS), suggesting that these regions are sensitive to holistic versus part-based face processing. Furthermore, the retrosplenial cortex (RSC) and the parahippocampal place area (PPA) showed a pattern of neural activity consistent with a holistic representation of face identity, which also correlated with the strength of the behavioural composite face effect. These results suggest that neural activity in brain regions both within and outside of the face-responsive network contributes to the composite-face effect.

Prosodic influence in face emotion perception: evidence from functional near-infrared spectroscopy.

Emotion is communicated via the integration of concurrently presented information from multiple information channels, such as voice, face, gesture and touch. This study investigated the neural and perceptual correlates of emotion perception as influenced by facial and vocal information by measuring changes in oxygenated hemoglobin (HbO) using functional near-infrared spectroscopy (fNIRS) and acquiring psychometrics. HbO activity was recorded from 103 channels while participants ([Formula: see text], [Formula: see text]) were presented with vocalizations produced in either a happy, angry or neutral prosody. Voices were presented alone or paired with an emotional face and compared with a face-only condition. Behavioral results indicated that when voices were paired with faces, a bias in the direction of the emotion of the voice was present. Subjects' responses also showed greater variance and longer reaction times when responding to the bimodal conditions when compared to the face-only condition. While both the happy and angry prosody conditions exhibited right lateralized increases in HbO compared to the neutral condition, these activations were segregated into posterior-anterior subdivisions by emotion. Specific emotional prosodies may therefore differentially influence emotion perception, with happy voices exhibiting posterior activity in receptive emotion areas and angry voices displaying activity in anterior expressive emotion areas.

Acute effects of oxytocin in music performance anxiety: a crossover, randomized, placebo-controlled trial.

RATIONALE: Individuals with music performance anxiety (MPA) present physical, behavioral, and cognitive manifestations of anxiety, in addition to information processing deficits, especially in facial emotion recognition (FER). OBJECTIVES: To assess the effects of a single dose of intranasal oxytocin (24 IU) on FER in a sample of musicians with high and low MPA (primary outcome), as well as indicators of mood/anxiety and self-assessed performance (secondary outcomes). METHODS: Crossover, randomized, double-blind, placebo-controlled trial involving 43 male musicians with different levels of MPA. Participants completed a static facial emotion recognition task and self-rated mood and performance scales. Data were analyzed using ANOVA 2 × 0 for crossover trials and the Omnibus test (measure of separability between intervention and carryover effects). RESULTS: Only musicians with high MPA treated with oxytocin had a higher accuracy in the recognition of happiness (p < 0.03; d > 0.72). No effects of oxytocin were found on mood indicators or on self-perceived performance, regardless of MPA level. CONCLUSIONS: The results indicate possible benefits of the acute treatment with oxytocin in MPA, which may improve the management of this common and disabling condition that affects professional musicians. The appropriate perception of positive feedback may increase confidence and feelings of social acceptance, reducing symptoms associated with the condition. The lack of effects on mood/anxiety and cognition may be explained by the context-dependent characteristic of the effects of oxytocin, since the experiment did not represent an actual situation of social threat. TRIAL REGISTRATION: Brazilian Clinical Trials Registry (Registro Brasileiro de Ensaios Clínicos): No. RBR-9cph2q.

Altered Effective Connectivity of Central Autonomic Network in Response to Negative Facial Expression in Adults With Cannabis Use Disorder.

BACKGROUND: Cannabis use is associated with an increased risk of stress-related adverse cardiovascular events. Because brain regions of the central autonomic network largely overlap with brain regions related to the neural response to emotion and stress, the central autonomic network may mediate the autonomic response to negative emotional stimuli. We aimed to obtain evidence to determine whether neural connectivity of the central autonomic network is altered in individuals with cannabis use disorder (CUD) when they are exposed to negative emotional stimuli. METHODS: Effective (directional) connectivity (EC) analysis using dynamic causal modeling was applied to functional magnetic resonance imaging data acquired from 23 subjects with CUD and 23 control subjects of the Human Connectome Project while they performed an emotional face-matching task with interleaving periods of negative-face (fearful/angry) and neutral-shape stimuli. The EC difference (modulatory change) was measured during the negative-face trials relative to the neutral-shape trials. RESULTS: The CUD group was similar to the control group in nonimaging measures and brain activations but showed greater modulatory changes in left amygdala to hypothalamus EC (positively associated with Perceived Stress Scale score), right amygdala to bilateral fusiform gyri ECs (positively associated with Perceived Stress Scale score), and left ventrolateral prefrontal cortex to bilateral fusiform gyri ECs (negatively associated with Perceived Stress Scale score). CONCLUSIONS: Left amygdala to hypothalamus EC and right amygdala to bilateral fusiform gyri ECs are possibly part of circuits underlying the risk of individuals with CUD to stress-related disorders. Correspondingly, left ventrolateral prefrontal cortex to bilateral fusiform gyri ECs are possibly part of circuits reflecting a protective mechanism.

Differential responses to infant faces in relation to maternal substance use: An exploratory study.

BACKGROUND: Maternal substance use and addiction has been associated with negative consequences for parenting and may increase addiction vulnerability in the developing child. Neuroimaging research suggests that substance use may decrease the reward of caring for infants and heighten stress reactivity to affective infant cues. METHODS: Thirty-two substance-using mothers and twenty-two non-substance-using mothers were presented with emotional face and cry stimuli generated from their own and a demographically matched unknown infant during fMRI scanning. Between-group differences in neural activity during task performance were assessed using whole-brain, mixed-effects models corrected for multiple comparisons (voxel-level p < 0.001, pFWE<0.05). RESULTS: Relative to non-substance-using mothers, substance-using mothers exhibited greater activation when viewing their own infant's face as compared to an unknown infant's face across multiple brain regions, including superior medial frontal, inferior parietal, and middle temporal regions. Substance-using mothers also had a decreased response to sad infant faces in the ventral striatum relative to the non-substance-using mothers. Neural responses to own vs. unknown infant cries did not significantly differ between substance-using and non-substance-using mothers. CONCLUSIONS: Findings suggest overlapping cortical and subcortical brain regions implicated in responding to infant faces, with activation differences related to infant familiarity, emotional expression, and maternal substance use. While prior work has focused on attenuated neural responses to infant cues, greater attention is needed toward understanding the increased reactivity to affective infant cues observed in substance-using mothers.

Hippocampal GABA levels correlate with retrieval performance in an associative learning paradigm.

Neural plasticity is a complex process dependent on neurochemical underpinnings. Next to the glutamatergic system which contributes to memory formation via long-term potentiation (LTP) and long-term depression (LTD), the main inhibitory neurotransmitter, GABA is crucially involved in neuroplastic processes. Hence, we investigated changes in glutamate and GABA levels in the brain in healthy participants performing an associative learning paradigm. Twenty healthy participants (10 female, 25 ±â€¯5 years) underwent paired multi-voxel magnetic resonance spectroscopy imaging before and after completing 21 days of a facial associative learning paradigm in a longitudinal study design. Changes of GABA and glutamate were compared to retrieval success in the hippocampus, insula and thalamus. No changes in GABA and glutamate concentration were found after 21 days of associative learning. However, baseline hippocampal GABA levels were significantly correlated with initial retrieval success (pcor = 0.013, r = 0.690). In contrast to the thalamus and insula (pcor>0.1), higher baseline GABA levels in the hippocampus were associated with better retrieval performance in an associative learning paradigm. Therefore, our findings support the importance of hippocampal GABA levels in memory formation in the human brain in vivo.

Face recognition and memory in congenital amusia.

Congenital amusia, commonly known as tone deafness, is a lifelong impairment of music perception and production. It remains a question of debate whether the impairments in musical domain observed in congenital amusia are paralleled in other non-musical perceptual abilities. Using behavioral measures in two experiments, the current study explored face perception and memory in congenital amusics. Both congenital amusics and matched controls performed a face perception task (Experiment 1) and an old/novel object memory task (for both faces and houses, Experiment 2). The results showed that the congenital amusic group had significantly slower reaction times than that in matched control group when identifying whether two faces presented together were the same or different. For different face-pairs, the deficit was greater for upright faces compared with inverted faces. For object memory task, the congenital amusic group also showed worse memory performance than the control group. The results of the present study suggest that the impairment attributed to congenital amusia is not only limited to music, but also extends to visual perception and visual memory domain.

Parametric face drawings: A demographically diverse and customizable face space model.

We introduce a novel face space model-parametric face drawings (or PFDs)-to generate schematic, though realistic, parameterized line drawings of faces based on the statistical distribution of human facial features. A review of existing face space models (including FaceGen Modeller, Synthetic Faces, MPI, and active appearance model) indicates that current models are constrained by their reliance on ethnically homogeneous face databases. This constraint has led to negative consequences for underrepresented populations, such as impairments in automatized identity recognition of certain demographic groups. Our model is based on a demographically diverse sample of 400 faces (200 female, 200 male; 100 East Asian/Pacific Islander, 100 Latinx/Hispanic, 100 black/African-American, and 100 white/Caucasian) compiled from several face databases (including FERET face recognition technology and the Chicago Face Database). Each front-view face image is manually coded with 85 landmark points that are then normalized and rendered with MATLAB (MathWorks, Natick, MA) tools to produce a smooth, parameterized face line drawing. We present data from two behavioral experiments to validate our model and demonstrate its applicability. In Experiment 1 we show that PFDs produce a reliable "inversion effect" in short-term recognition, a hallmark of holistic processing. In Experiment 2, we conduct a celebrity recognition task, comparing performance on PFDs to performance on untextured renderings from FaceGen Modeller. Participants successfully recognized approximately 50% of celebrity faces based on the PFD models, comparable to performance based on FaceGen Modeler (also 50% correct). We highlight a range of potential applications of our model, list some limitations, and provide MATLAB resources for researchers to utilize our face space, including the ability to customize the demographic makeup of the face space, add new faces, and produce morphs and caricatures.

Mindfulness training reduces stress and amygdala reactivity to fearful faces in middle-school children.

The impact of mindfulness training on stress and associated brain plasticity has been shown in adults, whereas the impact of such training in the developing brain remains unknown. To address this open question, 40 middle-school children were randomized to either mindfulness or coding training (active control) interventions during the school day for eight weeks. Outcome measures were ratings of self-perceived stress and right amygdala activation while viewing fearful, happy, and neutral facial expressions during functional MRI. Prior to intervention, greater stress correlated with greater right amygdala activation in response to fearful versus neutral facial expressions across all children. After intervention, children who received mindfulness training reported lower stress associated with reduced right amygdala activation to fearful faces relative to children in the control condition. Amygdala responses to happy faces were unrelated to either initial stress or mindfulness reduction of stress. Moreover, mindfulness training led to relatively stronger functional connectivity between the right amygdala and ventromedial prefrontal cortex during the viewing of fearful facial expressions. Changes in perceived stress and neuroplasticity occurred in nonmeditative states, indicating that the benefits of mindfulness training generalized beyond the active meditative state. This study provides initial evidence that mindfulness training in children reduces stress and promotes functional brain changes and that such training can be integrated into the school curriculum for entire classes. This study also reveals first evidence that a neurocognitive mechanism for both stress and its reduction by mindfulness training is related specifically to reduced amygdala responses to negative stimuli. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Differential Patterns of Visual Sensory Alteration Underlying Face Emotion Recognition Impairment and Motion Perception Deficits in Schizophrenia and Autism Spectrum Disorder.

BACKGROUND: Impaired face emotion recognition (FER) and abnormal motion processing are core features in schizophrenia (SZ) and autism spectrum disorder (ASD) that have been linked to atypical activity within the visual cortex. Despite overlaps, only a few studies have directly explored convergent versus divergent neural mechanisms of altered visual processing in ASD and SZ. We employed a multimodal imaging approach to evaluate FER and motion perception in relation to functioning of subcortical and cortical visual regions. METHODS: Subjects were 20 high-functioning adults with ASD, 19 patients with SZ, and 17 control participants. Behavioral measures of coherent motion sensitivity and FER along with electrophysiological and functional magnetic resonance imaging measures of visual pattern and motion processing were obtained. Resting-state functional magnetic resonance imaging was used to assess the relationship between corticocortical and thalamocortical connectivity and atypical visual processing. RESULTS: SZ and ASD participants had intercorrelated deficits in FER and motion sensitivity. In both groups, reduced motion sensitivity was associated with reduced functional magnetic resonance imaging activation in the occipitotemporal cortex and lower delta-band electroencephalogram power. In ASD, FER deficits correlated with hyperactivation of dorsal stream regions and increased evoked theta power. Activation of the pulvinar correlated with abnormal alpha-band modulation in SZ and ASD with under- and overmodulation, respectively, predicting increased clinical symptoms in both groups. CONCLUSIONS: SZ and ASD participants showed equivalent deficits in FER and motion sensitivity but markedly different profiles of physiological dysfunction. The specific pattern of deficits observed in each group may help guide development of treatments designed to downregulate versus upregulate visual processing within the respective clinical groups.

Behind the face of holistic perception: Holistic processing of Gestalt stimuli and faces recruit overlapping perceptual mechanisms.

Holistic processing, demonstrated by a failure of selective attention to individual parts within stimuli, is often considered a relatively unique feature of the processing of faces and objects of expertise. However, face-like holistic processing has been recently demonstrated for novel line stimuli with salient Gestalt perceptual grouping cues. Further, disrupting such cues within face stimuli disrupts holistic face perception. There is evidence that holistic processing of these gestalt stimuli and faces does not overlap mechanistically in the same way as does the processing of faces and objects of expertise. However, the relationship between these different manifestations of holistic processing is unclear. We developed a task to probe whether a holistic processing-specific overlap occurs at an earlier, perceptual level between the mechanisms supporting processing of faces and strong gestalt stimuli. Faces and gestalt line stimuli were overlaid, and participants made part judgments about either the faces (Experiment 1) or line stimuli (Experiment 2) in a composite task indexing holistic perception. The data revealed evidence of reciprocal interference between holistic processing of line and face stimuli, with indices of holistic processing of face and line stimuli reduced when the overlaid stimuli were also processed holistically (e.g., intact line/face stimuli) compared with when the overlaid stimuli did not commandeer holistic processing resources (e.g., misaligned line/face stimuli). This pattern is consistent with a mechanistic overlap between the holistic perception of faces and gestalt stimuli. Our results support a dual-stimulus-based and experienced-based-pathway model of holistic processing, with face stimuli using both.

Happy and Angry Faces Elicit Atypical Neural Activation in Children With Autism Spectrum Disorder.

BACKGROUND: Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by significant impairments in social interactions and communication. The ability to accurately perceive and interpret emotional faces is critical to successful social interactions. However, few studies have investigated the spatiotemporal profile of the neural mechanisms underlying emotional face processing in ASD, particularly in children. The current study fills this important gap. METHODS: Participants were 55 children: 28 children with ASD (mean age = 9.5 ± 1.3 years) and 27 control children (mean age = 8.5 ± 1.3 years). All children completed an implicit emotional face task while magnetoencephalography was recorded. We examined spatiotemporal differences between the groups in neural activation during implicit processing of emotional faces. RESULTS: Within-group analyses demonstrated greater right middle temporal (300-375 ms) and superior temporal (300-400 ms) activation to angry faces than to happy faces in control children, while children with ASD showed greater activation from 250 to 500 ms to happy faces than to angry faces across frontal and temporal regions. Between-group analyses demonstrated that children with ASD showed similar patterns of late (425-500 ms) posterior cingulate and thalamic underactivity to both angry and happy faces relative to control children, suggesting general atypical processing of emotional information. CONCLUSIONS: Atypical posterior cingulate cortex and thalamus recruitment in children with ASD to emotional faces suggests poor modulation of toggling between the default mode network and task-based processing. Increased neural activity to happy faces compared with angry faces in children with ASD suggests reduced salience or immature response to anger, which in turn could contribute to deficits in social cognition in ASD.

The role of the IPL in person identification.

We investigate the brain activations when identifying a newly encountered individual as being the same as a person previously perceived, a fundamental but seldom acknowledged process. In an identity condition, two faces had to be identified as the same person in contrast to a control condition, in which two faces had to be recognised as belonging to similar looking twins. Our results demonstrate an increase of neural activation in frontal as well as in parietal areas including the left inferior parietal lobe and the precuneus during identification. We introduce mental files theory to model this process as a linking of co-referential files and identify important connections to other domains in neurological and cognitive science (e.g., delusional misidentification syndromes, theory of mind).