Identity recognition from faces and bodies in schizophrenia spectrum disorders.

Deficits in facial identity recognition and its association with poor social functioning are well documented in schizophrenia, but none of these studies have assessed the role of the body in these processes. Recent research in healthy populations shows that the body is also an important source of information in identity recognition, and the current study aimed to thoroughly examine identity recognition from both faces and bodies in schizophrenia. Sixty-five individuals with schizophrenia and forty-nine healthy controls completed three conditions of an identity matching task in which they attempted to match unidentified persons in unedited photos of faces and bodies, edited photos showing faces only, or edited photos showing bodies only. Results revealed global deficits in identity recognition in individuals with schizophrenia (ηp2 = 0.068), but both groups showed better recognition from bodies alone as compared to faces alone (ηp2 = 0.573), suggesting that the ability to extract useful information from bodies when identifying persons may remain partially preserved in schizophrenia. Further research is necessary to understand the relationship between face/body processing, identity recognition, and functional outcomes in individuals with schizophrenia-spectrum disorders.

Face identification proficiency test designed using item response theory.

Measures of face-identification proficiency are essential to ensure accurate and consistent performance by professional forensic face examiners and others who perform face-identification tasks in applied scenarios. Current proficiency tests rely on static sets of stimulus items and so cannot be administered validly to the same individual multiple times. To create a proficiency test, a large number of items of "known" difficulty must be assembled. Multiple tests of equal difficulty can be constructed then using subsets of items. We introduce the Triad Identity Matching (TIM) test and evaluate it using item response theory (IRT). Participants view face-image "triads" (N = 225) (two images of one identity, one image of a different identity) and select the different identity. In Experiment 3, university students (N = 197) showed wide-ranging accuracy on the TIM test, and IRT modeling demonstrated that the TIM items span various difficulty levels. In Experiment 3, we used IRT-based item metrics to partition the test into subsets of specific difficulties. Simulations showed that subsets of the TIM items yielded reliable estimates of subject ability. In Experiments 3a and b, we found that the student-derived IRT model reliably evaluated the ability of non-student participants and that ability generalized across different test sessions. In Experiment 3c, we show that TIM test performance correlates with other common face-recognition tests. In summary, the TIM test provides a starting point for developing a framework that is flexible and calibrated to measure proficiency across various ability levels (e.g., professionals or populations with face-processing deficits).

Collaboration to improve cross-race face identification: Wisdom of the multi-racial crowd?

Face identification is particularly prone to error when individuals identify people of a race other than their own - a phenomenon known as the other-race effect (ORE). Here, we show that collaborative "wisdom-of-crowds" decision-making substantially improves face identification accuracy for own- and other-race faces over individuals working alone. In two online experiments, East Asian and White individuals recognized own- and other-race faces as individuals and as part of a collaborative dyad. Collaboration never proved more beneficial in a social setting than when individual identification decisions were combined computationally. The reliable benefit of non-social collaboration may stem from its ability to avoid the potential negative outcomes of group diversity such as conflict. Consistent with this benefit, the racial diversity of collaborators did not influence either general or race-specific face identification accuracy. Our findings suggest that collaboration between two individuals is a promising strategy for improving cross-race face identification that may translate effectively into forensic and eyewitness settings.

Confidence judgments are associated with face identification accuracy: Findings from a confidence forced-choice task.

Confidence is assumed to be an indicator of identification accuracy in legal practices (e.g., forensic face examination). However, it is not clear whether people can evaluate the correctness of their face-identification decisions reliably using confidence reports. In the current experiment, confidence in the correctness of the perceptual decision was measured with a confidence forced-choice methodology: Upon completion of two perceptual face-identity matching trials, the participants were asked to compare the two decisions and to select the trial on which they felt more confident. On each face-identity matching trial, participants viewed three face images (two same-identity images, one different-identity image) and were instructed to select the image of the different identity. In order to measure the extent to which difficulty level informs confidence decisions, we selected face-image triads using item-difficulty estimates extracted from psychometric modeling applied in a prior study. The difference in difficulty between the paired face-image triads predicted the proportion of high-confidence judgments allocated to the easier trial of the pair. Consistent with the impact of difficulty monitoring on confidence judgments, performance was significantly more accurate on trials associated with higher confidence. Overall, the results suggested that people reliably evaluate the correctness of their perceptual face-identity matching decisions and use trial difficulty to evaluate confidence.

First impressions from faces in the real world: Commentary on Sutherland and Young (2022).

The study of first impressions from faces now emphasizes the need to understand trait inferences made to naturalistic face images (British Journal of Psychology, 113, 2022, 1056). Face recognition algorithms based on deep convolutional neural networks simultaneously represent invariant, changeable and environmental variables in face images. Therefore, we suggest them as a comprehensive 'face space' model of first impressions of naturalistic faces. We also suggest that to understand trait inferences in the real world, a logical next step is to consider trait inferences made to whole people (faces and bodies). On the role of cultural contributions to trait perception, we think it is important for the field to begin to consider the way in which trait inferences motivate (or not) behaviour in independent and interdependent cultures.

First impressions: Integrating faces and bodies in personality trait perception.

Faces and bodies spontaneously elicit personality trait judgments (e.g., trustworthy, dominant, lazy). We examined how trait information from the face and body combine to form first impressions of the whole person and whether trait judgments from the face and body are affected by seeing the whole person. Consistent with the trait-dependence hypothesis, Experiment 1 showed that the relative contribution of the face and body to whole-person perception varied with the trait judged. Agreeableness traits (e.g., warm, aggressive, sympathetic, trustworthy) were inferred primarily from the face, conscientiousness traits (e.g., dependable, careless) from the body, and extraversion traits (e.g., dominant, quiet, confident) from the whole person. A control experiment showed that both clothing and body shape contributed to whole-person judgments. In Experiment 2, we found that a face (body) rated in the whole person elicited a different rating than when it was rated in isolation. Specifically, when trait ratings differed for an isolated face and body of the same identity, the whole-person context biased in-context ratings of the faces and bodies towards the ratings of the context. These results showed that face and body trait perception interact more than previously assumed. We combine current and established findings to propose a novel framework to account for face-body integration in trait perception. This framework incorporates basic elements such as perceptual determinants, nonperceptual determinants, trait formation, and integration, as well as predictive factors such as the rater, the person rated, and the situation.

Cortical processing of dynamic bodies in the superior occipito-temporal regions of the infants' brain: Difference from dynamic faces and inversion effect.

Previous functional neuroimaging studies imply a crucial role of the superior temporal regions (e.g., superior temporal sulcus: STS) for processing of dynamic faces and bodies. However, little is known about the cortical processing of moving faces and bodies in infancy. The current study used functional near-infrared spectroscopy (fNIRS) to directly compare cortical hemodynamic responses to dynamic faces (videos of approaching people with blurred bodies) and dynamic bodies (videos of approaching people with blurred faces) in infants' brain. We also examined the body-inversion effect in 5- to 8-month-old infants using hemodynamic responses as a measure. We found significant brain activity for the dynamic faces and bodies in the superior area of bilateral temporal cortices in both 5- to 6-month-old and 7- to 8-month-old infants. The hemodynamic responses to dynamic faces occurred across a broader area of cortex in 7- to 8-month-olds than in 5- to 6-month-olds, but we did not find a developmental change for dynamic bodies. There was no significant activation when the stimuli were presented upside down, indicating that these activation patterns did not result from the low-level visual properties of dynamic faces and bodies. Additionally, we found that the superior temporal regions showed a body inversion effect in infants aged over 5 months: the upright dynamic body stimuli induced stronger activation compared to the inverted stimuli. The most important contribution of the present study is that we identified cortical areas responsive to dynamic bodies and faces in two groups of infants (5-6-months and 7-8-months of age) and we found different developmental trends for the processing of bodies and faces.

Face Recognition by Humans and Machines: Three Fundamental Advances from Deep Learning.

Deep learning models currently achieve human levels of performance on real-world face recognition tasks. We review scientific progress in understanding human face processing using computational approaches based on deep learning. This review is organized around three fundamental advances. First, deep networks trained for face identification generate a representation that retains structured information about the face (e.g., identity, demographics, appearance, social traits, expression) and the input image (e.g., viewpoint, illumination). This forces us to rethink the universe of possible solutions to the problem of inverse optics in vision. Second, deep learning models indicate that high-level visual representations of faces cannot be understood in terms of interpretable features. This has implications for understanding neural tuning and population coding in the high-level visual cortex. Third, learning in deep networks is a multistep process that forces theoretical consideration of diverse categories of learning that can overlap, accumulate over time, and interact. Diverse learning types are needed to model the development of human face processing skills, cross-race effects, and familiarity with individual faces.

Closing the gap between single-unit and neural population codes: Insights from deep learning in face recognition.

Single-unit responses and population codes differ in the "read-out" information they provide about high-level visual representations. Diverging local and global read-outs can be difficult to reconcile with in vivo methods. To bridge this gap, we studied the relationship between single-unit and ensemble codes for identity, gender, and viewpoint, using a deep convolutional neural network (DCNN) trained for face recognition. Analogous to the primate visual system, DCNNs develop representations that generalize over image variation, while retaining subject (e.g., gender) and image (e.g., viewpoint) information. At the unit level, we measured the number of single units needed to predict attributes (identity, gender, viewpoint) and the predictive value of individual units for each attribute. Identification was remarkably accurate using random samples of only 3% of the network's output units, and all units had substantial identity-predicting power. Cross-unit responses were minimally correlated, indicating that single units code non-redundant identity cues. Gender and viewpoint classification required large-scale pooling of units-individual units had weak predictive power. At the ensemble level, principal component analysis of face representations showed that identity, gender, and viewpoint separated into high-dimensional subspaces, ordered by explained variance. Unit-based directions in the representational space were compared with the directions associated with the attributes. Identity, gender, and viewpoint contributed to all individual unit responses, undercutting a neural tuning analogy. Instead, single-unit responses carry superimposed, distributed codes for face identity, gender, and viewpoint. This undermines confidence in the interpretation of neural representations from unit response profiles for both DCNNs and, by analogy, high-level vision.

Facial expression is retained in deep networks trained for face identification.

Facial expressions distort visual cues for identification in two-dimensional images. Face processing systems in the brain must decouple image-based information from multiple sources to operate in the social world. Deep convolutional neural networks (DCNN) trained for face identification retain identity-irrelevant, image-based information (e.g., viewpoint). We asked whether a DCNN trained for identity also retains expression information that generalizes over viewpoint change. DCNN representations were generated for a controlled dataset containing images of 70 actors posing 7 facial expressions (happy, sad, angry, surprised, fearful, disgusted, neutral), from 5 viewpoints (frontal, 90° and 45° left and right profiles). Two-dimensional visualizations of the DCNN representations revealed hierarchical groupings by identity, followed by viewpoint, and then by facial expression. Linear discriminant analysis of full-dimensional representations predicted expressions accurately, mean 76.8% correct for happiness, followed by surprise, disgust, anger, neutral, sad, and fearful at 42.0%; chance \(\approx\)14.3%. Expression classification was stable across viewpoints. Representational similarity heatmaps indicated that image similarities within identities varied more by viewpoint than by expression. We conclude that an identity-trained, deep network retains shape-deformable information about expression and viewpoint, along with identity, in a unified form-consistent with a recent hypothesis for ventral visual stream processing.

Accuracy comparison across face recognition algorithms: Where are we on measuring race bias?

Previous generations of face recognition algorithms differ in accuracy for images of different races (race bias). Here, we present the possible underlying factors (data-driven and scenario modeling) and methodological considerations for assessing race bias in algorithms. We discuss data-driven factors (e.g., image quality, image population statistics, and algorithm architecture), and scenario modeling factors that consider the role of the "user" of the algorithm (e.g., threshold decisions and demographic constraints). To illustrate how these issues apply, we present data from four face recognition algorithms (a previous-generation algorithm and three deep convolutional neural networks, DCNNs) for East Asian and Caucasian faces. First, dataset difficulty affected both overall recognition accuracy and race bias, such that race bias increased with item difficulty. Second, for all four algorithms, the degree of bias varied depending on the identification decision threshold. To achieve equal false accept rates (FARs), East Asian faces required higher identification thresholds than Caucasian faces, for all algorithms. Third, demographic constraints on the formulation of the distributions used in the test, impacted estimates of algorithm accuracy. We conclude that race bias needs to be measured for individual applications and we provide a checklist for measuring this bias in face recognition algorithms.

Seeing through disguise: Getting to know you with a deep convolutional neural network.

People use disguise to look unlike themselves (evasion) or to look like someone else (impersonation). Evasion disguise challenges human ability to see an identity across variable images; Impersonation challenges human ability to tell people apart. Personal familiarity with an individual face helps humans to see through disguise. Here we propose a model of familiarity based on high-level visual learning mechanisms that we tested using a deep convolutional neural network (DCNN) trained for face identification. DCNNs generate a face space in which identities and images co-exist in a unified computational framework, that is categorically structured around identity, rather than retinotopy. This allows for simultaneous manipulation of mechanisms that contrast identities and cluster images. In Experiment 1, we measured the DCNN's baseline accuracy (unfamiliar condition) for identification of faces in no disguise and disguise conditions. Disguise affected DCNN performance in much the same way it affects human performance for unfamiliar faces in disguise (cf. Noyes & Jenkins, 2019). In Experiment 2, we simulated familiarity for individual identities by averaging the DCNN-generated representations from multiple images of each identity. Averaging improved DCNN recognition of faces in evasion disguise, but reduced the ability of the DCNN to differentiate identities of similar appearance. In Experiment 3, we implemented a contrast learning technique to simultaneously teach the DCNN appearance variation and identity contrasts between different individuals. This facilitated identification with both evasion and impersonation disguise. Familiar face recognition requires an ability to group images of the same identity together and separate different identities. The deep network provides a high-level visual representation for face recognition that supports both of these mechanisms of face learning simultaneously.

Integrating faces and bodies: Psychological and neural perspectives on whole person perception.

The human "person" is a common percept we encounter. Research on person perception has been focused either on face or body perception-with less attention paid to whole person perception. We review psychological and neuroscience studies aimed at understanding how face and body processing operate in concert to support intact person perception. We address this question considering: a.) the task to be accomplished (identification, emotion processing, detection), b.) the neural stage of processing (early/late visual mechanisms), and c.) the relevant brain regions for face/body/person processing. From the psychological perspective, we conclude that the integration of faces and bodies is mediated by the goal of the processing (e.g., emotion analysis, identification, etc.). From the neural perspective, we propose a hierarchical functional neural architecture of face-body integration that retains a degree of separation between the dorsal and ventral visual streams. We argue for two centers of integration: a ventral semantic integration hub that is the result of progressive, posterior-to-anterior, face-body integration; and a social agent integration hub in the dorsal stream STS.

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification.

Faces provide information about a person's identity, as well as their sex, age, and ethnicity. People also infer social and personality traits from the face - judgments that can have important societal and personal consequences. In recent years, deep convolutional neural networks (DCNNs) have proven adept at representing the identity of a face from images that vary widely in viewpoint, illumination, expression, and appearance. These algorithms are modeled on the primate visual cortex and consist of multiple processing layers of simulated neurons. Here, we examined whether a DCNN trained for face identification also retains a representation of the information in faces that supports social-trait inferences. Participants rated male and female faces on a diverse set of 18 personality traits. Linear classifiers were trained with cross validation to predict human-assigned trait ratings from the 512 dimensional representations of faces that emerged at the top-layer of a DCNN trained for face identification. The network was trained with 494,414 images of 10,575 identities and consisted of seven layers and 19.8 million parameters. The top-level DCNN features produced by the network predicted the human-assigned social trait profiles with good accuracy. Human-assigned ratings for the individual traits were also predicted accurately. We conclude that the face representations that emerge from DCNNs retain facial information that goes beyond the strict limits of their training.

Learning context and the other-race effect: Strategies for improving face recognition.

People recognize faces of their own race more accurately than faces of other races-a phenomenon known as the "Other-Race Effect" (ORE). Previous studies show that training with multiple variable images improves face recognition. Building on multi-image training, we take a novel approach to improving own- and other-race face recognition by testing the role of learning context on accuracy. Learning context was either contiguous, with multiple images of each identity seen in sequence, or distributed, with multiple images of an identity randomly interspersed among different identities. In two experiments, East Asian and Caucasian participants learned own- and other-races faces either in a contiguous or distributed order. In Experiment 1, people learned each identity from four highly variable face images. In Experiment 2, identities were learned from one image, repeated four times. In both experiments we found a robust other-race effect. The effect of learning context, however, differed depending on the variability of the learned images. The distributed presentation yielded better recognition when people learned from single repeated images (Exp. 1), but not when they learned from multiple variable images (Exp. 2). Overall, performance was better with multiple-image training than repeated single image training. We conclude that multiple-image training and distributed learning can both improve recognition accuracy, but via distinct processes. The former broadens perceptual tolerance for image variation from a face, when there are diverse images available to learn. The latter effectively strengthens the representation of differences among similar faces, when there is only a single learning image.

First Impressions of Personality Traits From Body Shapes.

People infer the personalities of others from their facial appearance. Whether they do so from body shapes is less studied. We explored personality inferences made from body shapes. Participants rated personality traits for male and female bodies generated with a three-dimensional body model. Multivariate spaces created from these ratings indicated that people evaluate bodies on valence and agency in ways that directly contrast positive and negative traits from the Big Five domains. Body-trait stereotypes based on the trait ratings revealed a myriad of diverse body shapes that typify individual traits. Personality-trait profiles were predicted reliably from a subset of the body-shape features used to specify the three-dimensional bodies. Body features related to extraversion and conscientiousness were predicted with the highest consensus, followed by openness traits. This study provides the first comprehensive look at the range, diversity, and reliability of personality inferences that people make from body shapes.

Face Space Representations in Deep Convolutional Neural Networks.

Inspired by the primate visual system, deep convolutional neural networks (DCNNs) have made impressive progress on the complex problem of recognizing faces across variations of viewpoint, illumination, expression, and appearance. This generalized face recognition is a hallmark of human recognition for familiar faces. Despite the computational advances, the visual nature of the face code that emerges in DCNNs is poorly understood. We review what is known about these codes, using the long-standing metaphor of a 'face space' to ground them in the broader context of previous-generation face recognition algorithms. We show that DCNN face representations are a fundamentally new class of visual representation that allows for, but does not assure, generalized face recognition.

Face recognition ability does not predict person identification performance: using individual data in the interpretation of group results.

There are large individual differences in people's face recognition ability. These individual differences provide an opportunity to recruit the best face-recognisers into jobs that require accurate person identification, through the implementation of ability-screening tasks. To date, screening has focused exclusively on face recognition ability; however real-world identifications can involve the use of other person-recognition cues. Here we incorporate body and biological motion recognition as relevant skills for person identification. We test whether performance on a standardised face-matching task (the Glasgow Face Matching Test) predicts performance on three other identity-matching tasks, based on faces, bodies, and biological motion. We examine the results from group versus individual analyses. We found stark differences between the conclusions one would make from group analyses versus analyses that retain information about individual differences. Specifically, tests of correlation and analysis of variance suggested that face recognition ability was related to performance for all person identification tasks. These analyses were strikingly inconsistent with the individual differences data, which suggested that the screening task was related only to performance on the face task. This study highlights the importance of individual data in the interpretation of results of person identification ability.

Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms.

Achieving the upper limits of face identification accuracy in forensic applications can minimize errors that have profound social and personal consequences. Although forensic examiners identify faces in these applications, systematic tests of their accuracy are rare. How can we achieve the most accurate face identification: using people and/or machines working alone or in collaboration? In a comprehensive comparison of face identification by humans and computers, we found that forensic facial examiners, facial reviewers, and superrecognizers were more accurate than fingerprint examiners and students on a challenging face identification test. Individual performance on the test varied widely. On the same test, four deep convolutional neural networks (DCNNs), developed between 2015 and 2017, identified faces within the range of human accuracy. Accuracy of the algorithms increased steadily over time, with the most recent DCNN scoring above the median of the forensic facial examiners. Using crowd-sourcing methods, we fused the judgments of multiple forensic facial examiners by averaging their rating-based identity judgments. Accuracy was substantially better for fused judgments than for individuals working alone. Fusion also served to stabilize performance, boosting the scores of lower-performing individuals and decreasing variability. Single forensic facial examiners fused with the best algorithm were more accurate than the combination of two examiners. Therefore, collaboration among humans and between humans and machines offers tangible benefits to face identification accuracy in important applications. These results offer an evidence-based roadmap for achieving the most accurate face identification possible.

Wisdom of the social versus non-social crowd in face identification.

Face identification is more accurate when people collaborate in social dyads than when they work alone (Dowsett & Burton, 2015, Br. J. Psychol., 106, 433). Identification accuracy is also increased when the responses of two people are averaged for each item to create a 'non-social' dyad (White, Burton, Kemp, & Jenkins, 2013, Appl. Cogn. Psychol., 27, 769; White et al., 2015, Proc. R. Soc. B Biol. Sci., 282, 20151292). Does social collaboration add to the benefits of response averaging for face identification? We compared individuals, social dyads, and non-social dyads on an unfamiliar face identity-matching test. We also simulated non-social collaborations for larger groups of people. Individuals and social dyads judged whether face image pairs depicted the same- or different identities, responding on a 5-point certainty scale. Non-social dyads were constructed by averaging the responses of paired individuals. Both social and non-social dyads were more accurate than individuals. There was no advantage for social over non-social dyads. For larger non-social groups, performance peaked at near perfection with a crowd size of eight participants. We tested three computational models of social collaboration and found that social dyad performance was predicted by the decision of the more accurate partner. We conclude that social interaction does not bolster accuracy for unfamiliar face identity matching in dyads beyond what can be achieved by averaging judgements.