Are adaptation aftereffects for facial emotional expressions affected by prior knowledge about the emotion?

Accurate perception of the emotional signals conveyed by others is crucial for successful social interaction. Such perception is influenced not only by sensory input, but also by knowledge we have about the others' emotions. This study addresses the issue of whether knowing that the other's emotional state is congruent or incongruent with their displayed emotional expression ("genuine" and "fake", respectively) affects the neural mechanisms underpinning the perception of their facial emotional expressions. We used a visual adaptation paradigm to investigate this question in three experiments employing increasing adaptation durations. The adapting stimuli consisted of photographs of emotional facial expressions of joy and anger, purported to reflect (in-)congruency between felt and expressed emotion, displayed by professional actors. A Validity checking procedure ensured participants had the correct knowledge about the (in-)congruency. Significantly smaller adaptation aftereffects were obtained when participants knew that the displayed expression was incongruent with the felt emotion, following all tested adaptation periods. This study shows that knowledge relating to the congruency between felt and expressed emotion modulates face expression aftereffects. We argue that this reflects that the neural substrate responsible for the perception of facial expressions of emotion incorporates the presumed felt emotion underpinning the expression.

Audience facial expressions detected by automated face analysis software reflect emotions in music.

An abundance of studies on emotional experiences in response to music have been published over the past decades, however, most have been carried out in controlled laboratory settings and rely on subjective reports. Facial expressions have been occasionally assessed but measured using intrusive methods such as facial electromyography (fEMG). The present study investigated emotional experiences of fifty participants in a live concert. Our aims were to explore whether automated face analysis could detect facial expressions of emotion in a group of people in an ecologically valid listening context, to determine whether emotions expressed by the music predicted specific facial expressions and examine whether facial expressions of emotion could be used to predict subjective ratings of pleasantness and activation. During the concert, participants were filmed and facial expressions were subsequently analyzed with automated face analysis software. Self-report on participants' subjective experience of pleasantness and activation were collected after the concert for all pieces (two happy, two sad). Our results show that the pieces that expressed sadness resulted in more facial expressions of sadness (compared to happiness), whereas the pieces that expressed happiness resulted in more facial expressions of happiness (compared to sadness). Differences for other facial expression categories (anger, fear, surprise, disgust, and neutral) were not found. Independent of the musical piece or emotion expressed in the music facial expressions of happiness predicted ratings of subjectively felt pleasantness, whilst facial expressions of sadness and disgust predicted low and high ratings of subjectively felt activation, respectively. Together, our results show that non-invasive measurements of audience facial expressions in a naturalistic concert setting are indicative of emotions expressed by the music, and the subjective experiences of the audience members themselves.

Interactions between Conscious and Subconscious Signals: Selective Attention under Feature-Based Competition Increases Neural Selectivity during Brain Adaptation.

Efficient perception in natural environments depends on neural interactions between voluntary processes within cognitive control, such as attention, and those that are automatic and subconscious, such as brain adaptation to predictable input (also called repetition suppression). Although both attention and adaptation have been studied separately and there is considerable knowledge of the neurobiology involved in each of these processes, how attention interacts with adaptation remains equivocal. We examined how attention interacts with visual and auditory adaptation by measuring neuroimaging effects consistent with changes in either neural gain or selectivity. Male and female human participants were scanned with functional magnetic resonance imaging (fMRI) first while they discriminated repetition of morphed faces or voices and either directed their attention to stimulus identity or spatial location. Attention to face or voice identity, while ignoring stimulus location, solely increased the gain of respectively face- or voice-sensitive cortex. The results were strikingly different in an experiment when participants attended to voice identity versus stimulus loudness. In this case, attention to voice while ignoring sound loudness increased neural selectivity. The combined results show that how attention affects adaptation depends on the level of feature-based competition, reconciling prior conflicting observations. The findings are theoretically important and are discussed in relation to neurobiological interactions between attention and different types of predictive signals.SIGNIFICANCE STATEMENT Adaptation to repeated environmental events is ubiquitous in the animal brain, an automatic typically subconscious, predictive signal. Cognitive influences, such as by attention, powerfully affect sensory processing and can overcome brain adaptation. However, how neural interactions occur between adaptation and attention remains controversial. We conducted fMRI experiments regulating the focus of attention during adaptation to repeated stimuli with perceptually balanced stimulus expectancy. We observed an interaction between attention and adaptation consistent with increased neural selectivity, but only under conditions of feature-based competition, challenging the notion that attention interacts with brain adaptation by only affecting response gain. This demonstrates that attention retains its full complement of mechanistic influences on sensory cortex even as it interacts with more automatic or subconscious predictive processes.

Action adaptation during natural unfolding social scenes influences action recognition and inferences made about actor beliefs.

When observing another individual's actions, we can both recognize their actions and infer their beliefs concerning the physical and social environment. The extent to which visual adaptation influences action recognition and conceptually later stages of processing involved in deriving the belief state of the actor remains unknown. To explore this we used virtual reality (life-size photorealistic actors presented in stereoscopic three dimensions) to see how visual adaptation influences the perception of individuals in naturally unfolding social scenes at increasingly higher levels of action understanding. We presented scenes in which one actor picked up boxes (of varying number and weight), after which a second actor picked up a single box. Adaptation to the first actor's behavior systematically changed perception of the second actor. Aftereffects increased with the duration of the first actor's behavior, declined exponentially over time, and were independent of view direction. Inferences about the second actor's expectation of box weight were also distorted by adaptation to the first actor. Distortions in action recognition and actor expectations did not, however, extend across different actions, indicating that adaptation is not acting at an action-independent abstract level but rather at an action-dependent level. We conclude that although adaptation influences more complex inferences about belief states of individuals, this is likely to be a result of adaptation at an earlier action recognition stage rather than adaptation operating at a higher, more abstract level in mentalizing or simulation systems.

A database of whole-body action videos for the study of action, emotion, and untrustworthiness.

We present a database of high-definition (HD) videos for the study of traits inferred from whole-body actions. Twenty-nine actors (19 female) were filmed performing different actions-walking, picking up a box, putting down a box, jumping, sitting down, and standing and acting-while conveying different traits, including four emotions (anger, fear, happiness, sadness), untrustworthiness, and neutral, where no specific trait was conveyed. For the actions conveying the four emotions and untrustworthiness, the actions were filmed multiple times, with the actor conveying the traits with different levels of intensity. In total, we made 2,783 action videos (in both two-dimensional and three-dimensional format), each lasting 7 s with a frame rate of 50 fps. All videos were filmed in a green-screen studio in order to isolate the action information from all contextual detail and to provide a flexible stimulus set for future use. In order to validate the traits conveyed by each action, we asked participants to rate each of the actions corresponding to the trait that the actor portrayed in the two-dimensional videos. To provide a useful database of stimuli of multiple actions conveying multiple traits, each video name contains information on the gender of the actor, the action executed, the trait conveyed, and the rating of its perceived intensity. All videos can be downloaded free at the following address: http://www-users.york.ac.uk/~neb506/databases.html. We discuss potential uses for the database in the analysis of the perception of whole-body actions.

Inter-subject correlation of audience facial expressions predicts audience engagement during theatrical performances.

During performances, audiences experience various emotional states, and these are reflected in their ongoing facial expressions. We investigated if audience engagement could be determined by measuring the inter-subject correlation (ISC) of non-invasively recorded audience facial expressions. We filmed the faces of multiple audience members at theatrical performances and determined the intensity of their different facial expressions throughout the performances. Neutral, happy, anger, and disgust expression ISCs accounted for up to 24% of the performance dramaturge's predictions of audience engagement. Expression synchrony was greater between individuals in close proximity, suggesting effects of emotional contagion or cognitive similarities between neighboring individuals, whereas expression synchrony was greatest between individuals who were younger, female, and with greater levels of empathy, showing that individual characteristics impact shared audience experiences. Together, our results show that facial expression synchronization could be used as a real-time non-invasive indicator of engagement in audiences larger than achieved using previous approaches.

Perceptual discrimination of action formidableness and friendliness and the impact of autistic traits.

The ability to determine whether the actions of other individuals are friendly or formidable are key decisions we need to make to successfully navigate our complex social environment. In this study we measured perceptual performance when discriminating actions that vary in their friendliness or formidableness, and whether performance was related to the autistic traits of individuals. To do this, we developed an action morphing method to generate novel actions that lied along the action quality dimensions of formidableness and friendliness. In Experiment 1 we show that actions that vary along the formidableness or friendliness continua were rated as varying monotonically along the respective quality. In Experiment 2 we measured the ability of individuals with different levels of autistic traits to discriminate action formidableness and friendliness using adaptive 2-AFC procedures. We found considerable variation in perceptual thresholds when discriminating action formidableness (~ 540% interindividual variation) or friendliness (~ 1100% interindividual variation). Importantly, we found no evidence that autistic traits influenced perceptual discrimination of these action qualities. These results confirm that sensory enhancements with autistic traits are limited to lower level stimuli, and suggest that the perceptual processing of these complex social signals are not affected by autistic traits.

Four fundamental dimensions underlie the perception of human actions.

We evaluate the actions of other individuals based upon a variety of movements that reveal critical information to guide decision making and behavioural responses. These signals convey a range of information about the actor, including their goals, intentions and internal mental states. Although progress has been made to identify cortical regions involved in action processing, the organising principles underlying our representation of actions still remains unclear. In this paper we investigated the conceptual space that underlies action perception by assessing which qualities are fundamental to the perception of human actions. We recorded 240 different actions using motion-capture and used these data to animate a volumetric avatar that performed the different actions. 230 participants then viewed these actions and rated the extent to which each action demonstrated 23 different action characteristics (e.g., avoiding-approaching, pulling-pushing, weak-powerful). We analysed these data using Exploratory Factor Analysis to examine the latent factors underlying visual action perception. The best fitting model was a four-dimensional model with oblique rotation. We named the factors: friendly-unfriendly, formidable-feeble, planned-unplanned, and abduction-adduction. The first two factors of friendliness and formidableness explained approximately 22% of the variance each, compared to planned and abduction, which explained approximately 7-8% of the variance each; as such we interpret this representation of action space as having 2 + 2 dimensions. A closer examination of the first two factors suggests a similarity to the principal factors underlying our evaluation of facial traits and emotions, whilst the last two factors of planning and abduction appear unique to actions.

Implied motion activation in cortical area MT can be explained by visual low-level features.

To investigate form-related activity in motion-sensitive cortical areas, we recorded cell responses to animate implied motion in macaque middle temporal (MT) and medial superior temporal (MST) cortex and investigated these areas using fMRI in humans. In the single-cell studies, we compared responses with static images of human or monkey figures walking or running left or right with responses to the same human and monkey figures standing or sitting still. We also investigated whether the view of the animate figure (facing left or right) that elicited the highest response was correlated with the preferred direction for moving random dot patterns. First, figures were presented inside the cell's receptive field. Subsequently, figures were presented at the fovea while a dynamic noise pattern was presented at the cell's receptive field location. The results show that MT neurons did not discriminate between figures on the basis of the implied motion content. Instead, response preferences for implied motion correlated with preferences for low-level visual features such as orientation and size. No correlation was found between the preferred view of figures implying motion and the preferred direction for moving random dot patterns. Similar findings were obtained in a smaller population of MST cortical neurons. Testing human MT+ responses with fMRI further corroborated the notion that low-level stimulus features might explain implied motion activation in human MT+. Together, these results suggest that prior human imaging studies demonstrating animate implied motion processing in area MT+ can be best explained by sensitivity for low-level features rather than sensitivity for the motion implied by animate figures.

Reduced connectivity between mentalizing and mirror systems in autism spectrum condition.

The mentalizing system and mirror system are thought to play important roles in inferring the internal mental states of others - a process known as mentalizing. Autism spectrum condition (ASC) is associated with difficulties in mentalizing. The aim of this study was to determine whether the behavioural difficulties in mentalizing associated with ASC can be explained by changes in functional connectivity between the mentalizing and mirror system. We recruited 40 adult participants (20 with ASC and 20 typically-developing). Brain activity was monitored using functional magnetic resonance imaging while participants watched videos in which actors performed hand actions. The videos were shown in separate mentalizing and non-mentalizing blocks. During mentalizing blocks, participants were asked to indicate whether hand actions were clumsy or spiteful (i.e. to judge the intent of the action). During non-mentalizing blocks, participants indicated whether the actions were successful or unsuccessful (i.e. to judge the outcome of the action). Higher activity during the mentalizing blocks compared to non-mentalizing blocks was found in regions associated with the mentalizing system: the dorsal medial prefrontal cortex (dmPFC) and the temporo-parietal junction (TPJ), as well as in regions typically associated with the mirror system: the inferior frontal gyrus (IFG) and the inferior parietal lobe (IPL). Next, functional connectivity between regions was evaluated as a function of task. During mentalizing blocks, there was increased functional connectivity between the dmPFC and the mirror system in typically developing participants. In contrast, there was no increase in functional connectivity between these regions in ASC participants. Connectivity between the dmPFC and IFG was negatively correlated with autistic traits. The reduced connectivity in ASC participants was consistent with behavioural performance on the mentalizing task, which was also negatively correlated with the level of autistic traits. Together, these data emphasise the importance of functional connectivity between the mentalizing and mirror systems when inferring social intentions and show that reduced connectivity between these systems may explain some of the behavioural difficulties experienced by adults with ASC.

Timing of mirror system activation when inferring the intentions of others.

Neuroimaging studies have shown mirror system (MS) activation when participants infer internal states e.g. emotions, intentions or beliefs (known as 'mentalizing') from others' actions. However, the exact role of the MS in mentalizing tasks is unknown. Dysfunctional MS activation may underlie mentalizing deficits experienced by adults with autism spectrum disorder (ASD). This study investigated the timing of MS activity when inferring intentions in order to delineate between existing models of MS involvement. Single-pulse transcranial magnetic stimulation (TMS) was applied to the primary motor cortex at different time points during the observation of hand actions whilst participants inferred intentions (mentalizing task) and performed a non-mentalizing task. Electromyographic activity in the contralateral hand was used as an indirect measure of MS activity. Greater corticospinal activity was found during the mentalizing task than the non-mentalizing task, but only at the end of observed actions, suggesting late MS involvement in processing intentions. Enhanced corticospinal activity was not related to autistic traits or behavioural performance suggesting the MS has a more automatic role in processing others' intentions, irrespective of mentalizing ability. Our results extend current knowledge of MS activation when mentalizing, allowing initial delineation between different models of MS involvement in mentalizing.

Investigating Mirror System (MS) Activity in Adults with ASD When Inferring Others' Intentions Using Both TMS and EEG.

ASD is associated with mentalizing deficits that may correspond with atypical mirror system (MS) activation. We investigated MS activity in adults with and without ASD when inferring others' intentions using TMS-induced motor evoked potentials (MEPs) and mu suppression measured by EEG. Autistic traits were measured for all participants. Our EEG data show, high levels of autistic traits predicted reduced right mu (8-10 Hz) suppression when mentalizing. Higher left mu (8-10 Hz) suppression was associated with superior mentalizing performances. Eye-tracking and TMS data showed no differences associated with autistic traits. Our data suggest ASD is associated with reduced right MS activity when mentalizing, TMS-induced MEPs and mu suppression measure different aspects of MS functioning and the MS is directly involved in inferring intentions.

Abilities to Explicitly and Implicitly Infer Intentions from Actions in Adults with Autism Spectrum Disorder.

Previous research suggests that Autism Spectrum Disorder (ASD) might be associated with impairments on implicit but not explicit mentalizing tasks. However, such comparisons are made difficult by the heterogeneity of stimuli and the techniques used to measure mentalizing capabilities. We tested the abilities of 34 individuals (17 with ASD) to derive intentions from others' actions during both explicit and implicit tasks and tracked their eye-movements. Adults with ASD displayed explicit but not implicit mentalizing deficits. Adults with ASD displayed typical fixation patterns during both implicit and explicit tasks. These results illustrate an explicit mentalizing deficit in adults with ASD, which cannot be attributed to differences in fixation patterns.

Visual adaptation enhances action sound discrimination.

Prolonged exposure, or adaptation, to a stimulus in 1 modality can bias, but also enhance, perception of a subsequent stimulus presented within the same modality. However, recent research has also found that adaptation in 1 modality can bias perception in another modality. Here, we show a novel crossmodal adaptation effect, where adaptation to a visual stimulus enhances subsequent auditory perception. We found that when compared to no adaptation, prior adaptation to visual, auditory, or audiovisual hand actions enhanced discrimination between 2 subsequently presented hand action sounds. Discrimination was most enhanced when the visual action "matched" the auditory action. In addition, prior adaptation to a visual, auditory, or audiovisual action caused subsequent ambiguous action sounds to be perceived as less like the adaptor. In contrast, these crossmodal action aftereffects were not generated by adaptation to the names of actions. Enhanced crossmodal discrimination and crossmodal perceptual aftereffects may result from separate mechanisms operating in audiovisual action sensitive neurons within perceptual systems. Adaptation-induced crossmodal enhancements cannot be explained by postperceptual responses or decisions. More generally, these results together indicate that adaptation is a ubiquitous mechanism for optimizing perceptual processing of multisensory stimuli.

The sensitivity of primate STS neurons to walking sequences and to the degree of articulation in static images.

We readily use the form of human figures to determine if they are moving. Human figures that have arms and legs outstretched (articulated) appear to be moving more than figures where the arms and legs are near the body (standing). We tested whether neurons in the macaque monkey superior temporal sulcus (STS), a region known to be involved in processing social stimuli, were sensitive to the degree of articulation of a static human figure. Additionally, we tested sensitivity to the same stimuli within forward and backward walking sequences. We found that 57% of cells that responded to the static image of a human figure was also sensitive to the degree of articulation of the figure. Some cells displayed selective responses for articulated postures, while others (in equal numbers) displayed selective responses for standing postures. Cells selective for static images of articulated figures were more likely to respond to movies of walking forwards than walking backwards. Cells selective for static images of standing figures were more likely to respond to movies of walking backwards than forwards. An association between form sensitivity and walking sensitivity could be consistent with an interpretation that cell responses to articulated figures act as an implied motion signal.

Emotional Actions Are Coded via Two Mechanisms: With and without Identity Representation.

Accurate perception of an individual's identity and emotion derived from their actions and behavior is essential for successful social functioning. Here we determined the role of identity in the representation of emotional whole-body actions using visual adaptation paradigms. Participants adapted to actors performing different whole-body actions in a happy and sad fashion. Following adaptation subsequent neutral actions appeared to convey the opposite emotion. We demonstrate two different emotional action aftereffects showing distinctive adaptation characteristics. For one short-lived aftereffect, adaptation to the emotion expressed by an individual resulted in biases in the perception of the expression of emotion by other individuals, indicating an identity-independent representation of emotional actions. A second, longer lasting, aftereffect was observed where adaptation to the emotion expressed by an individual resulted in longer-term biases in the perception of the expressions of emotion only by the same individual; this indicated an additional identity-dependent representation of emotional actions. Together, the presence of these two aftereffects indicates the existence of two mechanisms for coding emotional actions, only one of which takes into account the actor's identity. The results that we observe might parallel processing of emotion from face and voice.

Adaptation to facial trustworthiness is different in female and male observers.

Face adaptation paradigms have been used extensively to investigate the mechanisms underlying the processing of several different facial characteristics including face shape, identity, view and emotional expression. Judgements of facial trustworthiness can also be influenced by visual adaptation; to date these (un)trustworthy face aftereffects have only been shown following adaptation to emotional expression and facial masculinity/femininity. In this study we assessed how exposure to trustworthy and untrustworthy faces influenced the perception of the trustworthiness of subsequent test faces. In a mixed factorial design experiment, we tested the influence of adaptation to female and male faces on the perception of subsequent female and male faces in both female and male observers. In female observers, we found that following adaptation to trustworthy and untrustworthy faces subsequent test faces appeared less like the adapting stimuli. Sex of the adapting and test faces did not have significant influence on these (un)trustworthy face aftereffects. In male observers, however, we found no significant influence of the effect of adaptation on the subsequent perception of face trustworthiness. The clear difference in the visual aftereffects induced in female and male observers indicates the operation of different mechanisms underlying the perception of facial trustworthiness, and future studies should investigate these mechanisms separately in female and male observers.

Dynamics of walking adaptation aftereffects induced in static images of walking actors.

Visual adaptation to walking actions results in subsequent aftereffects that bias perception of static images of walkers in different postures so that they are interpreted as walking in the opposite direction to the adapting actor. It is not clear, however, if the walking aftereffect is comparable to other well studied low- and high-level visual aftereffects. We therefore measured the dynamics of the walking aftereffect in order to assess the characteristics of the adapting mechanism. We found that walking aftereffects showed similar characteristic dynamics as for face aftereffects and some motion aftereffects. Walking aftereffects could be induced in a broad range of different static images of walking actors and were not restricted to images of actors in any particular posture. Walking aftereffects increased with adapting stimulus repetition and declined over time. The duration of the aftereffect was dependent upon time spent observing the adapting stimulus and could be well modelled by a power-law function that characterises this relationship in both face and motion aftereffects. Increasing the speed of the adapting stimulus by increasing actor walk speed increased aftereffect magnitude, as seen for some motion aftereffects. The nature of the aftereffects induced by observing walking actors indicates that they behave like traditional high-level visual aftereffects.

From single cells to social perception.

Research describing the cellular coding of faces in non-human primates often provides the underlying physiological framework for our understanding of face processing in humans. Models of face perception, explanations of perceptual after-effects from viewing particular types of faces, and interpretation of human neuroimaging data rely on monkey neurophysiological data and the assumption that neurophysiological responses of humans are comparable to those recorded in the non-human primate. Here, we review studies that describe cells that preferentially respond to faces, and assess the link between the physiological characteristics of single cells and social perception. Principally, we describe cells recorded from the non-human primate, although a limited number of cells have been recorded in humans, and are included in order to appraise the validity of non-human physiological data for our understanding of human face and social perception.

Seeing the future: Natural image sequences produce "anticipatory" neuronal activity and bias perceptual report.

This paper relates human perception to the functioning of cells in the temporal cortex that are engaged in high-level pattern processing. We review historical developments concerning (a) the functional organization of cells processing faces and (b) the selectivity for faces in cell responses. We then focus on (c) the comparison of perception and cell responses to images of faces presented in sequences of unrelated images. Specifically the paper concerns the cell function and perception in circumstances where meaningful patterns occur momentarily in the context of a naturally or unnaturally changing visual environment. Experience of visual sequences allows anticipation, yet one sensory stimulus also "masks" perception and neural processing of subsequent stimuli. To understand this paradox we compared cell responses in monkey temporal cortex to body images presented individually, in pairs and in action sequences. Responses to one image suppressed responses to similar images for approximately 500 ms. This suppression led to responses peaking 100 ms earlier to image sequences than to isolated images (e.g., during head rotation, face-selective activity peaks before the face confronts the observer). Thus forward masking has unrecognized benefits for perception because it can transform neuronal activity to make it predictive during natural change.