Deep models of superficial face judgments.

The diversity of human faces and the contexts in which they appear gives rise to an expansive stimulus space over which people infer psychological traits (e.g., trustworthiness or alertness) and other attributes (e.g., age or adiposity). Machine learning methods, in particular deep neural networks, provide expressive feature representations of face stimuli, but the correspondence between these representations and various human attribute inferences is difficult to determine because the former are high-dimensional vectors produced via black-box optimization algorithms. Here we combine deep generative image models with over 1 million judgments to model inferences of more than 30 attributes over a comprehensive latent face space. The predictive accuracy of our model approaches human interrater reliability, which simulations suggest would not have been possible with fewer faces, fewer judgments, or lower-dimensional feature representations. Our model can be used to predict and manipulate inferences with respect to arbitrary face photographs or to generate synthetic photorealistic face stimuli that evoke impressions tuned along the modeled attributes.

A neuronal social trait space for first impressions in the human amygdala and hippocampus.

People instantaneously evaluate faces with significant agreement on evaluations of social traits. However, the neural basis for such rapid spontaneous face evaluation remains largely unknown. Here, we recorded from 490 neurons in the human amygdala and hippocampus and found that the neuronal activity was associated with the geometry of a social trait space. We further investigated the temporal evolution and modulation on the social trait representation, and we employed encoding and decoding models to reveal the critical social traits for the trait space. We also recorded from another 938 neurons and replicated our findings using different social traits. Together, our results suggest that there exists a neuronal population code for a comprehensive social trait space in the human amygdala and hippocampus that underlies spontaneous first impressions. Changes in such neuronal social trait space may have implications for the abnormal processing of social information observed in some neurological and psychiatric disorders.

Trustworthiness judgments without the halo effect: A data-driven computational modeling approach.

Trustworthy-looking faces are also perceived as more attractive, but are there other meaningful cues that contribute to perceived trustworthiness? Using data-driven models, we identify these cues after removing attractiveness cues. In Experiment 1, we show that both judgments of trustworthiness and attractiveness of faces manipulated by a model of perceived trustworthiness change in the same direction. To control for the effect of attractiveness, we build two new models of perceived trustworthiness: a subtraction model, which forces the perceived attractiveness and trustworthiness to be negatively correlated (Experiment 2), and an orthogonal model, which reduces their correlation (Experiment 3). In both experiments, faces manipulated to appear more trustworthy were indeed perceived to be more trustworthy, but not more attractive. Importantly, in both experiments, these faces were also perceived as more approachable and with more positive expressions, as indicated by both judgments and machine learning algorithms. The current studies show that the visual cues used for trustworthiness and attractiveness judgments can be separated, and that apparent approachability and facial emotion are driving trustworthiness judgments and possibly general valence evaluation.

Pain and satisfaction: healthcare providers' facial appearance matters.

Trait inferences based solely on facial appearance affect many social decisions. Here we tested whether the effects of such inferences extend to the perception of physical sensations. In an actual clinical setting, we show that healthcare providers' facial appearance is a strong predictor of pain experienced by patients during a medical procedure. The effect was specific to familiarity: facial features of healthcare providers that convey feelings of familiarity were associated with a decrease in patients' perception of pain. In addition, caring appearance of the healthcare providers was significantly related to patients' satisfaction with the care they received. Besides indicating that rapid, unreflective trait inferences from facial appearance may affect important healthcare outcomes, these findings contribute to the understanding of the mechanisms underlying social modulation of pain perception.

Stimulus generalization as a mechanism for learning to trust.

How do humans learn to trust unfamiliar others? Decisions in the absence of direct knowledge rely on our ability to generalize from past experiences and are often shaped by the degree of similarity between prior experience and novel situations. Here, we leverage a stimulus generalization framework to examine how perceptual similarity between known individuals and unfamiliar strangers shapes social learning. In a behavioral study, subjects play an iterative trust game with three partners who exhibit highly trustworthy, somewhat trustworthy, or highly untrustworthy behavior. After learning who can be trusted, subjects select new partners for a second game. Unbeknownst to subjects, each potential new partner was parametrically morphed with one of the three original players. Results reveal that subjects prefer to play with strangers who implicitly resemble the original player they previously learned was trustworthy and avoid playing with strangers resembling the untrustworthy player. These decisions to trust or distrust strangers formed a generalization gradient that converged toward baseline as perceptual similarity to the original player diminished. In a second imaging experiment we replicate these behavioral gradients and leverage multivariate pattern similarity analyses to reveal that a tuning profile of activation patterns in the amygdala selectively captures increasing perceptions of untrustworthiness. We additionally observe that within the caudate adaptive choices to trust rely on neural activation patterns similar to those elicited when learning about unrelated, but perceptually familiar, individuals. Together, these findings suggest an associative learning mechanism efficiently deploys moral information encoded from past experiences to guide future choice.

Quantifying idiosyncratic and shared contributions to judgment.

Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement-faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)-we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.

Revealing Hidden Gender Biases in Competence Impressions of Faces.

Competence impressions from faces affect important decisions, such as hiring and voting. Here, using data-driven computational models, we identified the components of the competence stereotype. Faces manipulated by a competence model varied in attractiveness (Experiment 1a). However, faces could be manipulated on perceived competence controlling for attractiveness (Experiment 1b); moreover, faces perceived as more competent but not attractive were also perceived as more confident and masculine, suggesting a bias to perceive male faces as more competent than female faces (Experiment 2). Correspondingly, faces manipulated to appear competent but not attractive were more likely to be classified as male (Experiment 3). When masculinity cues that induced competence impressions were applied to real-life images, these cues were more effective on male faces (Experiment 4). These findings suggest that the main components of competence impressions are attractiveness, confidence, and masculinity, and they reveal gender biases in how we form important impressions of other people.

Person information facilitates memory for face identity.

We tested whether episodic information about people facilitates memory for their faces (Experiment 1) and whether this effect is specific for face identity (Experiment 2). Participants were presented with faces paired with behavioral descriptions (positive, neutral, or negative) and faces displayed alone. In both experiments, participants were more likely to recognize faces paired with behavioral descriptions, and after 1-week delay, their memory was better for faces paired with descriptions of salient behavior (i.e., with positive and negative valence) than faces paired with neutral behaviors or faces presented without information. To examine whether these effects are about memory for face identity rather than face image memory, in Experiment 2, we presented different facial images (varying in facial angle) of the same people at the encoding and at the recognition test. Although this manipulation decreased the overall recognition, the findings of Experiment 1 were fully replicated. The findings suggest that minimal affective information is sufficient to facilitate memory for face identity.

Behavioral and Neural Adaptation in Approach Behavior.

People often make approachability decisions based on perceived facial trustworthiness. However, it remains unclear how people learn trustworthiness from a population of faces and whether this learning influences their approachability decisions. Here we investigated the neural underpinning of approach behavior and tested two important hypotheses: whether the amygdala adapts to different trustworthiness ranges and whether the amygdala is modulated by task instructions and evaluative goals. We showed that participants adapted to the stimulus range of perceived trustworthiness when making approach decisions and that these decisions were further modulated by the social context. The right amygdala showed both linear response and quadratic response to trustworthiness level, as observed in prior studies. Notably, the amygdala's response to trustworthiness was not modulated by stimulus range or social context, a possible neural dynamic adaptation. Together, our data have revealed a robust behavioral adaptation to different trustworthiness ranges as well as a neural substrate underlying approach behavior based on perceived facial trustworthiness.

The Dorsomedial Prefrontal Cortex Plays a Causal Role in Integrating Social Impressions from Faces and Verbal Descriptions.

Several neuroimaging studies point to a key role of the dorsomedial prefrontal cortex (dmPFC) in the formation of socially relevant impressions. In 3 different experiments, participants were required to form socially relevant impressions about other individuals on the basis of text descriptions of their social behaviors, and to decide whether a face alone, a trait adjective (e.g., "selfish"), or a face presented with a trait adjective was consistent or inconsistent with the impression they had formed. Before deciding whether the target stimulus matched the impression they had previously formed, participants received transcranial magnetic stimulation (TMS) over the dmPFC, the inferior frontal gyrus (IFG, also implicated in social impression formation), or over a control site (vertex). Results from the 3 experiments converged in showing that interfering with dmPFC activity significantly delayed participants in responding whether a face-adjective pair was consistent with the impression they had formed. No effects of TMS were observed following stimulation of the IFG or when evaluations had to be made on faces or trait adjectives presented alone. Our findings critically extend previous neuroimaging evidence by indicating a causal role of the dmPFC in creating coherent impressions based on the integration of face and verbal description of social behaviors.

For Your Local Eyes Only: Culture-Specific Face Typicality Influences Perceptions of Trustworthiness.

Recent findings show that typical faces are judged as more trustworthy than atypical faces. However, it is not clear whether employment of typicality cues in trustworthiness judgment happens across cultures and if these cues are culture specific. In two studies, conducted in Japan and Israel, participants judged trustworthiness and attractiveness of faces. In Study 1, faces varied along a cross-cultural dimension ranging from a Japanese to an Israeli typical face. Own-culture typical faces were perceived as more trustworthy than other-culture typical faces, suggesting that people in both cultures employ typicality cues when judging trustworthiness, but that the cues, indicative of typicality, are culture dependent. Because perceivers may be less familiar with other-culture typicality cues, Study 2 tested the extent to which they rely on available facial information other than typicality, when judging other-culture faces. In Study 2, Japanese and Israeli faces varied from either Japanese or Israeli attractive to unattractive with the respective typical face at the midpoint. For own-culture faces, trustworthiness judgments peaked around own-culture typical face. However, when judging other-culture faces, both cultures also employed attractiveness cues, but this effect was more apparent for Japanese participants. Our findings highlight the importance of culture when considering the effect of typicality on trustworthiness judgments.

The shape of novel objects contributes to shared impressions.

How do people share impressions of novel objects, and is this even possible? We tested whether the shape of novel 3-D objects can lead to similar impressions across people. To do this, we introduced a technique for manipulating highly complex shapes and measured four types of evaluative impressions (approachable, dangerous, beautiful, likable). Because relatively little is understood regarding how people form impressions of novel objects, we first sought to confirm the reliability of this behavior by examining how similar impressions are for an individual asked to re-evaluate the stimuli (i.e., impression consistency). To situate the magnitude of reliability, we compared novel objects to faces-familiar and extensively studied stimuli. Impression consistency was always present for both types of stimuli and comparable across all evaluations. Second, and more importantly, we tested how similar impressions are across people (i.e., impression consensus). Impression consensus was always present for faces, but not always for novel objects. In Study 2 we examined a greater diversity of shapes and replicated the findings of Study 1 for novel objects. The findings suggest that impression consensus for novel objects only emerges when certain types of shapes and evaluations map together. When such mapping is possible, impressions are isomorphic with the parametrized shapes.

Modelling perceptions of criminality and remorse from faces using a data-driven computational approach.

Perceptions of criminality and remorse are critical for legal decision-making. While faces perceived as criminal are more likely to be selected in police lineups and to receive guilty verdicts, faces perceived as remorseful are more likely to receive less severe punishment recommendations. To identify the information that makes a face appear criminal and/or remorseful, we successfully used two different data-driven computational approaches that led to convergent findings: one relying on the use of computer-generated faces, and the other on photographs of people. In addition to visualising and validating the perceived looks of criminality and remorse, we report correlations with earlier face models of dominance, threat, trustworthiness, masculinity/femininity, and sadness. The new face models of criminal and remorseful appearance contribute to our understanding of perceived criminality and remorse. They can be used to study the effects of perceived criminality and remorse on decision-making; research that can ultimately inform legal policies.

The biasing effects of appearances go beyond physical attractiveness and mating motives.

The influence of appearances goes well beyond physical attractiveness and includes the surprisingly powerful impact of "face-ism" - the tendency to stereotype individuals based on their facial features. A growing body of research has revealed that these face-based social attributions bias the outcomes of labor markets and experimental economic games in ways that are hard to explain via evolutionary mating motives.

Interfering with activity in the dorsomedial prefrontal cortex via TMS affects social impressions updating.

In our everyday social interactions we often need to deal with others' unpredictable behaviors. Integrating unexpected information in a consistent representation of another agent is a cognitively demanding process. Several neuroimaging studies point to the medial prefrontal cortex (mPFC) as a critical structure in mediating social evaluations. Our aim here was to shed light on the possible causal role of the mPFC in the dynamic process of forming and updating social impressions about others. We addressed this issue by suppressing activity in the mPFC by means of 1 Hz offline transcranial magnetic stimulation (TMS) prior to a task requiring participants to evaluate other agents' trustworthiness after reading about their social behavior. In two different experiments, we found that inhibiting activity in the mPFC increased perceived trustworthiness when inconsistent information about one agent's behavior was provided. In turn, when only negative or positive behaviors of a person were described, TMS over the mPFC did not affect judgments. Our results indicate that the mPFC is causally involved in mediating social impressions updating-at least in cases in which judgment is uncertain due to conflicting information to be processed.

Social attributions from faces: determinants, consequences, accuracy, and functional significance.

Since the early twentieth century, psychologists have known that there is consensus in attributing social and personality characteristics from facial appearance. Recent studies have shown that surprisingly little time and effort are needed to arrive at this consensus. Here we review recent research on social attributions from faces. Section I outlines data-driven methods capable of identifying the perceptual basis of consensus in social attributions from faces (e.g., What makes a face look threatening?). Section II describes nonperceptual determinants of social attributions (e.g., person knowledge and incidental associations). Section III discusses evidence that attributions from faces predict important social outcomes in diverse domains (e.g., investment decisions and leader selection). In Section IV, we argue that the diagnostic validity of these attributions has been greatly overstated in the literature. In the final section, we offer an account of the functional significance of these attributions.

Contributions of facial expressions and body language to the rapid perception of dynamic emotions.

Correctly perceiving emotions in others is a crucial part of social interactions. We constructed a set of dynamic stimuli to determine the relative contributions of the face and body to the accurate perception of basic emotions. We also manipulated the length of these dynamic stimuli in order to explore how much information is needed to identify emotions. The findings suggest that even a short exposure time of 250 milliseconds provided enough information to correctly identify an emotion above the chance level. Furthermore, we found that recognition patterns from the face alone and the body alone differed as a function of emotion. These findings highlight the role of the body in emotion perception and suggest an advantage for angry bodies, which, in contrast to all other emotions, were comparable to the recognition rates from the face and may be advantageous for perceiving imminent threat from a distance.

Contextual alignment of cognitive and neural dynamics.

Effective real-world communication requires the alignment of multiple individuals to a common perspective or mental framework. To study how this alignment occurs at the level of the brain, we measured BOLD response during fMRI while participants (n = 24) listened to a series of vignettes either in the presence or absence of a valid contextual cue. The valid contextual cue was necessary to understand the information in each vignette. We then examined where and to what extent the shared valid context led to greater intersubject similarity of neural processing. Regions of the default mode network including posterior cingulate cortex and medial pFC became more aligned when participants shared a valid contextual framework, whereas other regions, including primary sensory cortices, responded to the stimuli reliably regardless of contextual factors. Taken in conjunction with previous research, the present results suggest that default mode regions help the brain to organize incoming verbal information in the context of previous knowledge.

What is typical is good: the influence of face typicality on perceived trustworthiness.

The role of face typicality in face recognition is well established, but it is unclear whether face typicality is important for face evaluation. Prior studies have focused mainly on typicality's influence on attractiveness, although recent studies have cast doubt on its importance for attractiveness judgments. Here, we argue that face typicality is an important factor for social perception because it affects trustworthiness judgments, which approximate the basic evaluation of faces. This effect has been overlooked because trustworthiness and attractiveness judgments have a high level of shared variance for most face samples. We show that for a continuum of faces that vary on a typicality-attractiveness dimension, trustworthiness judgments peak around the typical face. In contrast, perceived attractiveness increases monotonically past the typical face, as faces become more like the most attractive face. These findings suggest that face typicality is an important determinant of face evaluation.

The development of race-based perceptual categorization: skin color dominates early category judgments.

Prior research on the development of race-based categorization has concluded that children understand the perceptual basis of race categories from as early as age 4 (e.g. Aboud, 1988). However, such work has rarely separated the influence of skin color from other physiognomic features considered by adults to be diagnostic of race categories. In two studies focusing on Black-White race categorization judgments in children between the ages of 4 and 9, as well as in adults, we find that categorization decisions in early childhood are determined almost entirely by attention to skin color, with attention to other physiognomic features exerting only a small influence on judgments as late as middle childhood. We further find that when skin color cues are largely eliminated from the stimuli, adults readily shift almost entirely to focus on other physiognomic features. However, 6- and 8-year-old children show only a limited ability to shift attention to facial physiognomy and so perform poorly on the task. These results demonstrate that attention to 'race' in younger children is better conceptualized as attention to skin color, inviting a reinterpretation of past work focusing on children's race-related cognition.