Time course of selective attention to face regions in social anxiety: eye-tracking and computational modelling.

We investigated the time course of selective attention to face regions during judgment of dis/approval by low (LSA) and high (HSA) social anxiety undergraduates (with clinical levels on questionnaire measures). The viewers' gaze direction was assessed and the stimulus visual saliency of face regions was computed, for video-clips displaying dynamic facial expressions. Social anxiety was related to perception of disapproval from faces with an ambiguous smile (i.e. with non-happy eyes), but not those with congruent happy eyes and a smile. HSA observers selectively looked earlier at the eye region, whereas LSA ones preferentially looked at the smiling mouth. Consistently, gaze allocation was less related to visual saliency of the smile for HSA than for LSA viewers. The attentional bias towards the less salient eye region - thus opposing the automatic capture by the smile - suggests that it is strategically driven in HSA individuals, possibly aimed at detecting negative evaluators.

Facial attractiveness impressions precede trustworthiness inferences: lower detection thresholds and faster decision latencies.

Prior research has found a relationship between perceived facial attractiveness and perceived personal trustworthiness. We examined the time course of attractiveness relative to trustworthiness evaluation of emotional and neutral faces. This served to explore whether attractiveness might be used as an easily accessible cue and a quick shortcut for judging trustworthiness. Detection thresholds and judgment latencies as a function of expressive intensity were measured. Significant correlations between attractiveness and trustworthiness consistently held for six emotional expressions at four intensities, and neutral faces. Importantly, perceived attractiveness preceded perceived trustworthiness, with lower detection thresholds and shorter decision latencies. This reveals a time course advantage for attractiveness, and suggests that earlier attractiveness impressions could bias trustworthiness inferences. A heuristic cognitive mechanism is hypothesised to ease processing demands by relying on simple and observable clues (attractiveness) as a substitute for more complex and not easily accessible information (trustworthiness).

Neural time course and brain sources of facial attractiveness vs. trustworthiness judgment.

Prior research has shown that the more (or less) attractive a face is judged, the more (or less) trustworthy the person is deemed and that some common neural networks are recruited during facial attractiveness and trustworthiness evaluation. To interpret the relationship between attractiveness and trustworthiness (e.g., whether perception of personal trustworthiness may depend on perception of facial attractiveness), we investigated their relative neural processing time course. An event-related potential (ERP) paradigm was used, with localization of brain sources of the scalp neural activity. Face stimuli with a neutral, angry, happy, or surprised expression were presented in an attractiveness judgment, a trustworthiness judgment, or a control (no explicit social judgment) task. Emotional facial expression processing occurred earlier (N170 and EPN, 150-290 ms post-stimulus onset) than attractiveness and trustworthiness processing (P3b, 400-700 ms). Importantly, right-central ERP (C2, C4, C6) differences reflecting discrimination between "yes" (attractive or trustworthy) and "no" (unattractive or untrustworthy) decisions occurred at least 400 ms earlier for attractiveness than for trustworthiness, in the absence of LRP motor preparation differences. Neural source analysis indicated that facial processing brain networks (e.g., LG, FG, and IPL-extending to pSTS), also right-lateralized, were involved in the discrimination time course differences. This suggests that attractiveness impressions precede and might prime trustworthiness inferences and that the neural time course differences reflect truly facial encoding processes.

What makes a smiling face look happy? Visual saliency, distinctiveness, and affect.

We investigated the relative contribution of (a) perceptual (eyes and mouth visual saliency), (b) conceptual or categorical (eye expression distinctiveness), and (c) affective (rated valence and arousal) factors, and (d) specific morphological facial features (Action Units; AUs), to the recognition of facial happiness. The face stimuli conveyed truly happy expressions with a smiling mouth and happy eyes, or blended expressions with a smile but non-happy eyes (neutral, sad, fearful, disgusted, surprised, or angry). Saliency, distinctiveness, affect, and AUs served as predictors; the probability of judging a face as happy was the criterion. Both for truly happy and for blended expressions, the probability of perceiving happiness increased mainly as a function of positive valence of the facial configuration. In addition, for blended expressions, the probability of being (wrongly) perceived as happy increased as a function of (a) delayed saliency and (b) reduced distinctiveness of the non-happy eyes, and (c) enhanced AU 6 (cheek raiser) or (d) reduced AUs 4, 5, and 9 (brow lowerer, upper lid raiser, and nose wrinkler, respectively). Importantly, the later the eyes become visually salient relative to the smiling mouth, the more likely it is that faces will look happy.

Adaptive attunement of selective covert attention to evolutionary-relevant emotional visual scenes.

We investigated selective attention to emotional scenes in peripheral vision, as a function of adaptive relevance of scene affective content for male and female observers. Pairs of emotional-neutral images appeared peripherally-with perceptual stimulus differences controlled-while viewers were fixating on a different stimulus in central vision. Early selective orienting was assessed by the probability of directing the first fixation towards either scene, and the time until first fixation. Emotional scenes selectively captured covert attention even when they were task-irrelevant, thus revealing involuntary, automatic processing. Sex of observers and specific emotional scene content (e.g., male-to-female-aggression, families and babies, etc.) interactively modulated covert attention, depending on adaptive priorities and goals for each sex, both for pleasant and unpleasant content. The attentional system exhibits domain-specific and sex-specific biases and attunements, probably rooted in evolutionary pressures to enhance reproductive and protective success. Emotional cues selectively capture covert attention based on their bio-social significance.

Perceptual and affective mechanisms in facial expression recognition: An integrative review.

Facial expressions of emotion involve a physical component of morphological changes in a face and an affective component conveying information about the expresser's internal feelings. It remains unresolved how much recognition and discrimination of expressions rely on the perception of morphological patterns or the processing of affective content. This review of research on the role of visual and emotional factors in expression recognition reached three major conclusions. First, behavioral, neurophysiological, and computational measures indicate that basic expressions are reliably recognized and discriminated from one another, albeit the effect may be inflated by the use of prototypical expression stimuli and forced-choice responses. Second, affective content along the dimensions of valence and arousal is extracted early from facial expressions, although this coarse affective representation contributes minimally to categorical recognition of specific expressions. Third, the physical configuration and visual saliency of facial features contribute significantly to expression recognition, with "emotionless" computational models being able to reproduce some of the basic phenomena demonstrated in human observers. We conclude that facial expression recognition, as it has been investigated in conventional laboratory tasks, depends to a greater extent on perceptual than affective information and mechanisms.

Brain signatures of perceiving a smile: Time course and source localization.

Facial happiness is consistently recognized faster than other expressions of emotion. In this study, to determine when and where in the brain such a recognition advantage develops, EEG activity during an expression categorization task was subjected to temporospatial PCA analysis and LAURA source localizations. Happy, angry, and neutral faces were presented either in whole or bottom-half format (with the mouth region visible). The comparison of part- versus whole-face conditions served to examine the role of the smile. Two neural signatures underlying the happy face advantage emerged. One peaked around 140 ms (left N140) and was source-located at the left IT cortex (MTG), with greater activity for happy versus non-happy faces in both whole and bottom-half face format. This suggests an enhanced perceptual encoding mechanism for salient smiles. The other peaked around 370 ms (P3b and N3) and was located at the right IT (FG) and dorsal cingulate (CC) cortices, with greater activity specifically for bottom-half happy versus non-happy faces. This suggests an enhanced recruitment of face-specific information to categorize (or reconstruct) facial happiness from diagnostic smiling mouths. Additional differential brain responses revealed a specific "anger effect," with greater activity for angry versus non-angry expressions (right N170 and P230; right pSTS and IPL); and a coarse "emotion effect," with greater activity for happy and angry versus neutral expressions (anterior P2 and posterior N170; vmPFC and right IFG).

Lateralized discrimination of emotional scenes in peripheral vision.

This study investigates whether there is lateralized processing of emotional scenes in the visual periphery, in the absence of eye fixations; and whether this varies with emotional valence (pleasant vs. unpleasant), specific emotional scene content (babies, erotica, human attack, mutilation, etc.), and sex of the viewer. Pairs of emotional (positive or negative) and neutral photographs were presented for 150 ms peripherally (≥6.5° away from fixation). Observers judged on which side the emotional picture was located. Low-level image properties, scene visual saliency, and eye movements were controlled. Results showed that (a) correct identification of the emotional scene exceeded the chance level; (b) performance was more accurate and faster when the emotional scene appeared in the left than in the right visual field; (c) lateralization was equivalent for females and males for pleasant scenes, but was greater for females and unpleasant scenes; and (d) lateralization occurred similarly for different emotional scene categories. These findings reveal discrimination between emotional and neutral scenes, and right brain hemisphere dominance for emotional processing, which is modulated by sex of the viewer and scene valence, and suggest that coarse affective significance can be extracted in peripheral vision.

Brain lateralization of holistic versus analytic processing of emotional facial expressions.

This study investigated the neurocognitive mechanisms underlying the role of the eye and the mouth regions in the recognition of facial happiness, anger, and surprise. To this end, face stimuli were shown in three formats (whole face, upper half visible, and lower half visible) and behavioral categorization, computational modeling, and ERP (event-related potentials) measures were combined. N170 (150-180 ms post-stimulus; right hemisphere) and EPN (early posterior negativity; 200-300 ms; mainly, right hemisphere) were modulated by expression of whole faces, but not by separate halves. This suggests that expression encoding (N170) and emotional assessment (EPN) require holistic processing, mainly in the right hemisphere. In contrast, the mouth region of happy faces enhanced left temporo-occipital activity (150-180 ms), and also the LPC (late positive complex; centro-parietal) activity (350-450 ms) earlier than the angry eyes (450-600 ms) or other face regions. Relatedly, computational modeling revealed that the mouth region of happy faces was also visually salient by 150 ms following stimulus onset. This suggests that analytical or part-based processing of the salient smile occurs early (150-180 ms) and lateralized (left), and is subsequently used as a shortcut to identify the expression of happiness (350-450 ms). This would account for the happy face advantage in behavioral recognition tasks when the smile is visible.

Facial expression recognition in peripheral versus central vision: role of the eyes and the mouth.

This study investigated facial expression recognition in peripheral relative to central vision, and the factors accounting for the recognition advantage of some expressions in the visual periphery. Whole faces or only the eyes or the mouth regions were presented for 150 ms, either at fixation or extrafoveally (2.5° or 6°), followed by a backward mask and a probe word. Results indicated that (a) all the basic expressions were recognized above chance level, although performance in peripheral vision was less impaired for happy than for non-happy expressions, (b) the happy face advantage remained when only the mouth region was presented, and (c) the smiling mouth was the most visually salient and most distinctive facial feature of all expressions. This suggests that the saliency and the diagnostic value of the smile account for the advantage in happy face recognition in peripheral vision. Because of saliency, the smiling mouth accrues sensory gain and becomes resistant to visual degradation due to stimulus eccentricity, thus remaining accessible extrafoveally. Because of diagnostic value, the smile provides a distinctive single cue of facial happiness, thus bypassing integration of face parts and reducing susceptibility to breakdown of configural processing in peripheral vision.

When does the brain distinguish between genuine and ambiguous smiles? An ERP study.

Event-related brain potentials (ERPs) were recorded to assess the processing time course of ambiguous facial expressions with a smiling mouth but neutral, fearful, or angry eyes, in comparison with genuinely happy faces (a smile and happy eyes) and non-happy faces (neutral, fearful, or angry mouth and eyes). Participants judged whether the faces looked truly happy or not. Electroencephalographic recordings were made from 64 scalp electrodes to generate ERPs. The neural activation patterns showed early P200 sensitivity (differences between negative and positive or neutral expressions) and EPN sensitivity (differences between positive and neutral expressions) to emotional valence. In contrast, sensitivity to ambiguity (differences between genuine and ambiguous expressions) emerged only in later LPP components. Discrimination of emotional vs. neutral affect occurs between 180 and 430ms from stimulus onset, whereas the detection and resolution of ambiguity takes place between 470 and 720ms. In addition, while blended expressions involving a smile with angry eyes can be identified as not happy in the P200 (175-240ms) component, smiles with fearful or neutral eyes produce the same ERP pattern as genuinely happy faces, thus revealing poor discrimination.

Primacy of emotional vs. semantic scene recognition in peripheral vision.

Emotional scenes were presented peripherally (5.2° away from fixation) or foveally (at fixation) for 150 ms. In affective evaluation tasks viewers judged whether a scene was unpleasant or not, or whether it was pleasant or not. In semantic categorisation tasks viewers judged whether a scene involved animals or humans (superordinate-level task), or whether it portrayed females or males (subordinate-level task). The same stimuli were used for the affective and the semantic task. Results indicated that in peripheral vision affective evaluation was less accurate and slower than animal/human discrimination, and did not show any advantage over gender discrimination. In addition, performance impairment in the peripheral relative to the foveal condition was greater or equivalent for affective than for semantic categorisation. These findings cast doubts on the specialness and the primacy of affective over semantic recognition. The findings are also relevant when considering the role of the subcortical "low route" in emotional processing.

Emotion recognition from posed and spontaneous dynamic expressions: Human observers versus machine analysis.

The majority of research on the judgment of emotion from facial expressions has focused on deliberately posed displays, often sampled from single stimulus sets. Herein, we investigate emotion recognition from posed and spontaneous expressions, comparing classification performance between humans and machine in a cross-corpora investigation. For this, dynamic facial stimuli portraying the six basic emotions were sampled from a broad range of different databases, and then presented to human observers and a machine classifier. Recognition performance by the machine was found to be superior for posed expressions containing prototypical facial patterns, and comparable to humans when classifying emotions from spontaneous displays. In both humans and machine, accuracy rates were generally higher for posed compared to spontaneous stimuli. The findings suggest that automated systems rely on expression prototypicality for emotion classification and may perform just as well as humans when tested in a cross-corpora context. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

How spatial attention and attentional resources influence the processing of emotional visual scenes.

Pleasant, neutral, or unpleasant pictures appeared for 150 ms in either peripheral or foveal vision, with or without a concurrent foveal load task. Participants indicated whether the visual scene in the picture was or was not pleasant, or was or was not unpleasant. The manipulation of picture location (foveal vs. peripheral) aimed to tap spatial attention, while the perceptual load task was used to manipulate the availability of attentional resources. Results showed that emotional valence was discriminated above the chance level even in the attentionally-constrained conditions (peripheral presentation combined with perceptual load). Nevertheless, valence encoding depended on both attentional mechanisms, as indicated by reductions in accuracy and by slowed reaction times in valence identification when attention was allocated elsewhere, relative to when the scene appeared at fixation and when there was no concurrent task. This indicates that emotional processing requires attention.

Eye-movement assessment of the time course in facial expression recognition: Neurophysiological implications.

Happy, surprised, disgusted, angry, sad, fearful, and neutral faces were presented extrafoveally, with fixations on faces allowed or not. The faces were preceded by a cue word that designated the face to be saccaded in a two-alternative forced-choice discrimination task (2AFC; Experiments 1 and 2), or were followed by a probe word for recognition (Experiment 3). Eye tracking was used to decompose the recognition process into stages. Relative to the other expressions, happy faces (1) were identified faster (as early as 160 msec from stimulus onset) in extrafoveal vision, as revealed by shorter saccade latencies in the 2AFC task; (2) required less encoding effort, as indexed by shorter first fixations and dwell times; and (3) required less decision-making effort, as indicated by fewer refixations on the face after the recognition probe was presented. This reveals a happy-face identification advantage both prior to and during overt attentional processing. The results are discussed in relation to prior neurophysiological findings on latencies in facial expression recognition.

Emotional scene content drives the saccade generation system reflexively.

The authors assessed whether parafoveal perception of emotional content influences saccade programming. In Experiment 1, paired emotional and neutral scenes were presented to parafoveal vision. Participants performed voluntary saccades toward either of the scenes according to an imperative signal (color cue). Saccadic reaction times were faster when the cue pointed toward the emotional picture rather than toward the neutral picture. Experiment 2 replicated these findings with a reflexive saccade task, in which abrupt luminosity changes were used as exogenous saccade cues. In Experiment 3, participants performed vertical reflexive saccades that were orthogonal to the emotional-neutral picture locations. Saccade endpoints and trajectories deviated away from the visual field in which the emotional scenes were presented. Experiment 4 showed that computationally modeled visual saliency does not vary as a function of scene content and that inversion abolishes the rapid orienting toward the emotional scenes. Visual confounds cannot thus explain the results. The authors conclude that early saccade target selection and execution processes are automatically influenced by emotional picture content. This reveals processing of meaningful scene content prior to overt attention to the stimulus.

Semantic word priming in the absence of eye fixations: relative contributions of overt and covert attention.

In the present study, we investigated the role of covert and overt attention in word identification. In repetition and semantic priming paradigms, prime words were followed by a probe for lexical decision. To make the primes available only to covert attention, we presented them for 150 msec, parafoveally (2.2 degrees away from fixation), and under gaze-contingent foveal masking. To make the primes available to overt attention, we presented them for 150 msec, at fixation, with no masking. Results showed both repetition and semantic priming in the absence of eye fixations on the primes: There was facilitation for identical and semantically related probe words, relative to an unrelated prime-probe condition. This revealed that both word form and meaning can be processed by covert attention alone. The pattern of relative contributions of covert (approximately 25%) and overt (approximately 75%) attention was similar for repetition and semantic priming.

Lateralised covert attention in word identification.

The right visual field superiority in word recognition has been attributed to an attentional advantage by the left brain hemisphere. We investigated whether such advantage involves lateralised covert attention, in the absence of overt fixations on prime words. In a lexical decision task target words were preceded by an identical or an unrelated prime word. Eye movements were monitored. In Experiment 1 lateralised (to the left or right of fixation) prime words were parafoveally visible but foveally masked, thus allowing for covert attention but preventing overt attention. In Experiment 2 prime words were presented at fixation, thus allowing for both overt and covert attention. Results revealed positive priming in the absence of fixations on the primes when these were presented in the right visual field. The effects of covertly attended primes were nevertheless significantly reduced in comparison with those of overtly attended primes. It is concluded that word identification can be accomplished to a significant extent by lateralised covert attention alone, with right visual field advantage.

Enhanced processing of emotional gist in peripheral vision.

Emotional (pleasant or unpleasant) and neutral scenes were presented foveally (at fixation) or peripherally (5.2 degrees away from fixation) as primes for 150 ms. The prime was followed by a mask and a centrally presented probe scene for recognition. The probe was either identical in specific content (i.e., same people and objects) to the prime, or it was related to the prime in general content and affective valence. The probe was always different from the prime in color, size, and spatial orientation. Results showed an interaction between prime location and emotional valence for the recognition hit rate, but also for the false alarm rate and correct rejection times. There were no differences as a function of emotional valence in the foveal display condition. In contrast, in the peripheral display condition both hit and false alarm rates were higher and correct rejection times were longer for emotional than for neutral scenes. It is concluded that emotional gist, or a coarse affective impression, is extracted from emotional scenes in peripheral vision, which then leads to confuse them with others of related affective valence. The underlying neurophysiological mechanisms are discussed. An alternative explanation based on the physical characteristics of the scene images was ruled out.

Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions.

A happy facial expression makes a person look (more) trustworthy. Do perceptions of happiness and trustworthiness rely on the same face regions and visual attention processes? In an eye-tracking study, eye movements and fixations were recorded while participants judged the un/happiness or the un/trustworthiness of dynamic facial expressions in which the eyes and/or the mouth unfolded from neutral to happy or vice versa. A smiling mouth and happy eyes enhanced perceived happiness and trustworthiness similarly, with a greater contribution of the smile relative to the eyes. This comparable judgement output for happiness and trustworthiness was reached through shared as well as distinct attentional mechanisms: (a) entry times and (b) initial fixation thresholds for each face region were equivalent for both judgements, thereby revealing the same attentional orienting in happiness and trustworthiness processing. However, (c) greater and (d) longer fixation density for the mouth region in the happiness task, and for the eye region in the trustworthiness task, demonstrated different selective attentional engagement. Relatedly, (e) mean fixation duration across face regions was longer in the trustworthiness task, thus showing increased attentional intensity or processing effort.