Trust in automation and the accuracy of human-algorithm teams performing one-to-one face matching tasks.

The human face is commonly used for identity verification. While this task was once exclusively performed by humans, technological advancements have seen automated facial recognition systems (AFRS) integrated into many identification scenarios. Although many state-of-the-art AFRS are exceptionally accurate, they often require human oversight or involvement, such that a human operator actions the final decision. Previously, we have shown that on average, humans assisted by a simulated AFRS (sAFRS) failed to reach the level of accuracy achieved by the same sAFRS alone, due to overturning the system's correct decisions and/or failing to correct sAFRS errors. The aim of the current study was to investigate whether participants' trust in automation was related to their performance on a one-to-one face matching task when assisted by a sAFRS. Participants (n = 160) completed a standard face matching task in two phases: an unassisted baseline phase, and an assisted phase where they were shown the identification decision (95% accurate) made by a sAFRS prior to submitting their own decision. While most participants improved with sAFRS assistance, those with greater relative trust in automation achieved larger gains in performance. However, the average aided performance of participants still failed to reach that of the sAFRS alone, regardless of trust status. Nonetheless, further analysis revealed a small sample of participants who achieved 100% accuracy when aided by the sAFRS. Our results speak to the importance of considering individual differences when selecting employees for roles requiring human-algorithm interaction, including identity verification tasks that incorporate facial recognition technologies.

Face masks and fake masks: the effect of real and superimposed masks on face matching with super-recognisers, typical observers, and algorithms.

Mask wearing has been required in various settings since the outbreak of COVID-19, and research has shown that identity judgements are difficult for faces wearing masks. To date, however, the majority of experiments on face identification with masked faces tested humans and computer algorithms using images with superimposed masks rather than images of people wearing real face coverings. In three experiments we test humans (control participants and super-recognisers) and algorithms with images showing different types of face coverings. In all experiments we tested matching concealed or unconcealed faces to an unconcealed reference image, and we found a consistent decrease in face matching accuracy with masked compared to unconcealed faces. In Experiment 1, typical human observers were most accurate at face matching with unconcealed images, and poorer for three different types of superimposed mask conditions. In Experiment 2, we tested both typical observers and super-recognisers with superimposed and real face masks, and found that performance was poorer for real compared to superimposed masks. The same pattern was observed in Experiment 3 with algorithms. Our results highlight the importance of testing both humans and algorithms with real face masks, as using only superimposed masks may underestimate their detrimental effect on face identification.

A new way of classifying developmental prosopagnosia: Balanced Integration Score.

Despite severe everyday problems recognising faces, some individuals with developmental prosopagnosia (DP) can achieve typical accuracy scores on laboratory face recognition tests. To address this, studies sometimes also examine response times (RTs), which tend to be longer in DPs relative to control participants. In the present study, 24 potential (according to self-report) DPs and 110 age-matched controls completed the Cambridge Face and Bicycle Memory Tests, old new faces task, and a famous faces test. We used accuracy and the Balanced Integration Score (BIS), a measure that adjusts accuracy for RTs, to classify our sample at the group and individual levels. Subjective face recognition ability was assessed using the PI20 questionnaire and semi structured interviews. Fifteen DPs showed a major impairment using BIS compared with only five using accuracy alone. Logistic regression showed that a model incorporating the BIS measures was the most sensitive for classifying DP and showed highest area under the curve (AUC). Furthermore, larger between-group effect sizes were observed for a derived global (averaged) memory measure calculated using BIS versus accuracy alone. BIS is thus an extremely sensitive novel measure for attenuating speed-accuracy trade-offs that can otherwise mask impairment measured only by accuracy in DP.

Data-driven studies in face identity processing rely on the quality of the tests and data sets.

There is growing interest in how data-driven approaches can help understand individual differences in face identity processing (FIP). However, researchers employ various FIP tests interchangeably, and it is unclear whether these tests 1) measure the same underlying ability/ies and processes (e.g., confirmation of identity match or elimination of identity match) 2) are reliable, 3) provide consistent performance for individuals across tests online and in laboratory. Together these factors would influence the outcomes of data-driven analyses. Here, we asked 211 participants to perform eight tests frequently reported in the literature. We used Principal Component Analysis and Agglomerative Clustering to determine factors underpinning performance. Importantly, we examined the reliability of these tests, relationships between them, and quantified participant consistency across tests. Our findings show that participants' performance can be split into two factors (called here confirmation and elimination of an identity match) and that participants cluster according to whether they are strong on one of the factors or equally on both. We found that the reliability of these tests is at best moderate, the correlations between them are weak, and that the consistency in participant performance across tests and is low. Developing reliable and valid measures of FIP and consistently scrutinising existing ones will be key for drawing meaningful conclusions from data-driven studies.

Simulated automated facial recognition systems as decision-aids in forensic face matching tasks.

Automated Facial Recognition Systems (AFRS) are used by governments, law enforcement agencies, and private businesses to verify the identity of individuals. Although previous research has compared the performance of AFRS and humans on tasks of one-to-one face matching, little is known about how effectively human operators can use these AFRS as decision-aids. Our aim was to investigate how the prior decision from an AFRS affects human performance on a face matching task, and to establish whether human oversight of AFRS decisions can lead to collaborative performance gains for the human-algorithm team. The identification decisions from our simulated AFRS were informed by the performance of a real, state-of-the-art, Deep Convolutional Neural Network (DCNN) AFRS on the same task. Across five pre-registered experiments, human operators used the decisions from highly accurate AFRS (> 90%) to improve their own face matching performance compared with baseline (sensitivity gain: Cohen's d = 0.71-1.28; overall accuracy gain: d = 0.73-1.46). Yet, despite this improvement, AFRS-aided human performance consistently failed to reach the level that the AFRS achieved alone. Even when the AFRS erred only on the face pairs with the highest human accuracy (> 89%), participants often failed to correct the system's errors, while also overruling many correct decisions, raising questions about the conditions under which human oversight might enhance AFRS operation. Overall, these data demonstrate that the human operator is a limiting factor in this simple model of human-AFRS teaming. These findings have implications for the "human-in-the-loop" approach to AFRS oversight in forensic face matching scenarios. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

How do looking patterns, anti-fat bias, and causal weight attributions relate to adults' judgements of child weight?

Prevailing weight-normative approaches to health pressure adults to visually categorise children's weight, despite little understanding of how such judgements are made. There is no evidence this strategy improves child health, and it may harm children with higher weights. To understand decision-making processes and identify potential mechanisms of harm we examined perceptual and attitudinal factors involved in adults' child weight category judgements. Eye movements of 42 adults were tracked while categorizing the weight of 40 computer-generated images of children (aged 4-5 & 10-11 years) varying in size. Questionnaires assessed child-focused weight bias and causal attributions for child weight. Participants' eye movement patterns resembled those previously reported for adult bodies. Categorisation data showed a perceptual bias towards the 'mid-range' category. For higher weight stimuli, participants whose category judgements most closely matched the stimulus's objective weight had higher child-focused anti-fat bias and weaker genetic attributions for child weight - i.e,. adults who 'label' higher weight in children in line with BMI categories report more stigmatising beliefs about such children, suggesting a possible mechanism of harm. Overall, adults' judgements reflect both unalterable perceptual biases and potentially harmful attitudinal factors, calling into question the feasibility and appropriateness of public health efforts to promote visual child weight categorisation.

Masked face identification is improved by diagnostic feature training.

To slow the spread of COVID-19, many people now wear face masks in public. Face masks impair our ability to identify faces, which can cause problems for professional staff who identify offenders or members of the public. Here, we investigate whether performance on a masked face matching task can be improved by training participants to compare diagnostic facial features (the ears and facial marks)-a validated training method that improves matching performance for unmasked faces. We show this brief diagnostic feature training, which takes less than two minutes to complete, improves matching performance for masked faces by approximately 5%. A control training course, which was unrelated to face identification, had no effect on matching performance. Our findings demonstrate that comparing the ears and facial marks is an effective means of improving face matching performance for masked faces. These findings have implications for professions that regularly perform face identification.

Visual search performance in 'CCTV' and mobile phone-like video footage.

Finding an unfamiliar person in a crowd of others is an integral task for police officers, CCTV-operators, and security staff who may be looking for a suspect or missing person; however, research suggests that it is difficult and accuracy in such tasks is low. In two real-world visual-search experiments, we examined whether being provided with four images versus one image of an unfamiliar target person would help improve accuracy when searching for that person through video footage. In Experiment 1, videos were taken from above and at a distance to simulate CCTV, and images of the target showed their face and torso. In Experiment 2, videos were taken from approximately shoulder height, such as one would expect from body-camera or mobile phone recordings, and target images included only the face. Our findings suggest that having four images as exemplars leads to higher accuracy in the visual search tasks, but this only reached significance in Experiment 2. There also appears to be a conservative bias whereby participants are more likely to respond that the target is not in the video when presented with only one image as opposed to 4. These results point to there being an advantage for providing multiple images of targets for use in video visual-search.

Familiar faces as islands of expertise.

Most people recognise and match pictures of familiar faces effortlessly, while struggling to match unfamiliar face images. This has led to the suggestion that true human expertise for faces applies only to familiar faces. This paper develops that idea to propose that we have isolated 'islands of expertise' surrounding each familiar face that allow us to perform better with faces that resemble those we already know. This idea is tested in three experiments. The first shows that familiarity with a person facilitates identification of their relatives. The second shows that people are better able to remember faces that resemble someone they already know. The third shows that while prompting participants to think about resemblance at study produces a large positive effect on subsequent recognition, there is still a significant effect if there is no such prompt. Face-space-R (Lewis, 2004) is used to illustrate a possible computational explanation of the processes involved.

Bilinguals' inhibitory control and attentional processes in a visual perceptual task.

The aim was to examine theories of bilingual inhibitory control superiority in the visual domain. In an ambiguous figure task, the ability to reverse (switch) interpretations (e.g., duck-rabbit) was examined in 3-5-year-olds bilinguals and monolinguals (N = 67). Bilingualism was no performance predictor in conceptual tasks (Droodle task, false belief task, ambiguous figures production task) that did not pose inhibitory demands. Bilinguals outperformed monolinguals in the ability to reverse, suggesting superior inhibitory capacity per se. Once reversal was experienced there was no difference in the time it took to reverse or reversal frequency between bilinguals and monolinguals. Bayesian analyses confirmed statistical result patterns. Findings support the established view of bilinguals' superior domain-general inhibitory control. This might be brought to bear by attending the environment differently.

Surgical face masks impair human face matching performance for familiar and unfamiliar faces.

In response to the COVID-19 pandemic, many governments around the world now recommend, or require, that their citizens cover the lower half of their face in public. Consequently, many people now wear surgical face masks in public. We investigated whether surgical face masks affected the performance of human observers, and a state-of-the-art face recognition system, on tasks of perceptual face matching. Participants judged whether two simultaneously presented face photographs showed the same person or two different people. We superimposed images of surgical masks over the faces, creating three different mask conditions: control (no masks), mixed (one face wearing a mask), and masked (both faces wearing masks). We found that surgical face masks have a large detrimental effect on human face matching performance, and that the degree of impairment is the same regardless of whether one or both faces in each pair are masked. Surprisingly, this impairment is similar in size for both familiar and unfamiliar faces. When matching masked faces, human observers are biased to reject unfamiliar faces as "mismatches" and to accept familiar faces as "matches". Finally, the face recognition system showed very high classification accuracy for control and masked stimuli, even though it had not been trained to recognise masked faces. However, accuracy fell markedly when one face was masked and the other was not. Our findings demonstrate that surgical face masks impair the ability of humans, and naïve face recognition systems, to perform perceptual face matching tasks. Identification decisions for masked faces should be treated with caution.

Convolutional neural net face recognition works in non-human-like ways.

Convolutional neural networks (CNNs) give the state-of-the-art performance in many pattern recognition problems but can be fooled by carefully crafted patterns of noise. We report that CNN face recognition systems also make surprising 'errors'. We tested six commercial face recognition CNNs and found that they outperform typical human participants on standard face-matching tasks. However, they also declare matches that humans would not, where one image from the pair has been transformed to appear a different sex or race. This is not due to poor performance; the best CNNs perform almost perfectly on the human face-matching tasks, but also declare the most matches for faces of a different apparent race or sex. Although differing on the salience of sex and race, humans and computer systems are not working in completely different ways. They tend to find the same pairs of images difficult, suggesting some agreement about the underlying similarity space.

Constructing identifiable composite faces: The importance of cognitive alignment of interview and construction procedure.

We investigated the impact of congruency between the witness interview and method used to construct a composite face. Experiment 1, using a typical feature-by-feature composite method, revealed that aligning cognitive processes during interview and face construction enhanced the effectiveness of composites compared with composites produced following unaligned (incongruent) procedures. Experiment 2 revealed that incorporating character judgments in the witness interview substantially enhanced identification of feature-based composites when constructing the central (internal) features first, suggesting that such judgments focus attention on this region of the face. Experiment 3 explored alignment of processes using an approach based on an evolutionary algorithm, a method requiring witnesses to create a composite by selecting from arrays based on the eye-region. A combination of character judgments, first for the whole face and then for the eye region, led to best-identified composites. Overall, results indicate that more effective composites are produced when both interview and construction procedures are aligned cognitively. Results are discussed with relevance to the theory of transfer-appropriate processing (Morris, Bransford, & Franks, 1977). (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Eye see through you! Eye tracking unmasks concealed face recognition despite countermeasures.

BACKGROUND: Criminal associates such as terrorist members are likely to deny knowing members of their network when questioned by police. Eye tracking research suggests that lies about familiar faces can be detected by distinct markers of recognition (e.g. fewer fixations and longer fixation durations) across multiple eye fixation parameters. However, the effect of explicit eye movement strategies to concealed recognition on such markers has not been examined. Our aim was to assess the impact of fixed-sequence eye movement strategies (across the forehead, ears, eyes, nose, mouth and chin) on markers of familiar face recognition. Participants were assigned to one of two groups: a standard guilty group who were simply instructed to conceal knowledge but with no specific instructions on how to do so; and a countermeasures group who were instructed to look at every familiar and unfamiliar face in the same way by executing a consistent sequence of fixations. RESULTS: In the standard guilty group, lies about recognition of familiar faces showed longer average fixation durations, a lower proportion of fixations to the inner face regions, and proportionately more viewing of the eyes than honest responses to genuinely unknown faces. In the countermeasures condition, familiar face recognition was detected by longer fixations durations, fewer fixations to the inner regions of the face, and fewer interest areas of the face viewed. Longer fixation durations were a consistent marker of recognition across both conditions for most participants; differences were detectable from the first fixation. CONCLUSION: The results suggest that individuals can exert a degree of executive control over fixation patterns but that: the eyes are particularly attention-grabbing for familiar faces; the more viewers look around the face, the more they give themselves away; and attempts to deploy the same fixation patterns to familiar and unfamiliar faces were unsuccessful. The results suggest that the best strategy for concealing recognition might be to keep the eyes fixated in the centre of the screen but, even then, recognition is apparent in longer fixation durations. We discuss potential optimal conditions for detecting concealed knowledge of faces.

A grey area: how does image hue affect unfamiliar face matching?

The role of image colour in face identification has received little attention in research despite the importance of identifying people from photographs in identity documents (IDs). Here, in two experiments, we investigated whether colour congruency of two photographs, shown side by side, affects face-matching accuracy. Participants were presented with two images from the Models Face Matching Test (experiment 1) and a newly devised matching task incorporating female faces (experiment 2) and asked to decide whether they show the same person or two different people. The photographs were either both in colour, both in grayscale, or mixed (one in grayscale and one in colour). Participants were more likely to accept a pair of images as a "match", i.e. same person, in the mixed condition, regardless of whether the identity of the pair was the same or not. This demonstrates a clear shift in bias between "congruent" colour conditions and the mixed trials. In addition, there was a small decline in accuracy in the mixed condition, relative to when the images were presented in colour. Our study provides the first evidence that the hue of document photographs matters for face-matching performance. This finding has important implications for the design and regulation of photographic ID worldwide.

The impact of external facial features on the construction of facial composites.

Witnesses may construct a composite face of a perpetrator using a computerised interface. Police practitioners guide witnesses through this unusual process, the goal being to produce an identifiable image. However, any changes a perpetrator makes to their external facial-features may interfere with this process. In Experiment 1, participants constructed a composite using a holistic interface one day after target encoding. Target faces were unaltered, or had altered external-features: (i) changed hair, (ii) external-features removed or (iii) naturally-concealed external-features (hair, ears, face-shape occluded by a hooded top). These manipulations produced composites with more error-prone internal-features: participants' familiar with a target's unaltered appearance less often provided a correct name. Experiment 2 applied external-feature alterations to composites of unaltered targets; although whole-face composites contained less error-prone internal-features, identification was impaired. Experiment 3 replicated negative effects of changing target hair on construction and tested a practical solution: selectively concealing hair and eyes improved identification. Practitioner Summary: The research indicates that when a target identity disguises or changes hair, this can lead to a witness (or victim) constructing a composite that is less readily identified. We assess a practical method to overcome this forensic issue. Abbreviation: GEE: Generalized Estimating Equations.

An item's status in semantic memory determines how it is recognized: Dissociable patterns of brain activity observed for famous and unfamiliar faces.

Are all faces recognized in the same way, or does previous experience with a face change how it is retrieved? Previous research using human scalp-recorded Event-Related Potentials (ERPs) demonstrates that recognition memory can produce dissociable brain signals under a variety of circumstances. While many studies have reported dissociations between the putative 'dual processes' of familiarity and recollection, a growing number of reports demonstrate that recollection itself may be fractionated into component processes. Many recognition memory studies using lexical materials as stimuli have reported a left parietal ERP old/new effect for recollection; however, when unfamiliar faces are recollected, an anterior effect can be observed. This paper addresses two separate hypotheses concerning the functional significance of the anterior old/new effect: perceptual retrieval and semantic status. The perceptual retrieval view is that the anterior effect reflects reinstatement of perceptual information bound up in an episodic representation, while the semantic status view is that information not represented in semantic memory pre-experimentally elicits the anterior effect instead of the left parietal effect. We tested these two competing accounts by investigating recognition memory for unfamiliar faces and famous faces in two separate experiments, in which same or different pictures of studied faces were presented as test items to permit brain activity associated with retrieving face and perceptual information to be examined independently. The difference in neural activity between same and different picture hits was operationalized as a pattern of activation associated with perceptual retrieval; while the contrast between different picture hits and correct rejection of new faces was assumed to reflect face retrieval. In Experiment 1, using unfamiliar faces, the anterior old/new effect (500-700 ms) was observed for face retrieval but not for perceptual retrieval, challenging the perceptual retrieval hypothesis. In Experiment 2, using famous faces, face retrieval was associated with a left parietal effect (500-700 ms), supporting the semantic representation hypothesis. A between-subjects analysis comparing scalp topography across the two experiments found that the anterior effect observed for unfamiliar faces is dissociable from the left parietal effect found for famous faces. This pattern of results supports the hypothesis that an item's status in semantic memory determines how it is recognized.

Ego depletion in visual perception: Ego-depleted viewers experience less ambiguous figure reversal.

This study examined the effects of ego depletion on ambiguous figure perception. Adults (N = 315) received an ego depletion task and were subsequently tested on their inhibitory control abilities that were indexed by the Stroop task (Experiment 1) and their ability to perceive both interpretations of ambiguous figures that was indexed by reversal (Experiment 2). Ego depletion had a very small effect on reducing inhibitory control (Cohen's d = .15) (Experiment 1). Ego-depleted participants had a tendency to take longer to respond in Stroop trials. In Experiment 2, ego depletion had small to medium effects on the experience of reversal. Ego-depleted viewers tended to take longer to reverse ambiguous figures (duration to first reversal) when naïve of the ambiguity and experienced less reversal both when naïve and informed of the ambiguity. Together, findings suggest that ego depletion has small effects on inhibitory control and small to medium effects on bottom-up and top-down perceptual processes. The depletion of cognitive resources can reduce our visual perceptual experience.

Fixation patterns, not clinical diagnosis, predict body size over-estimation in eating disordered women and healthy controls.

OBJECTIVE: A core feature of anorexia nervosa (AN) is an over-estimation of body size. Women with AN have a different pattern of eye-movements when judging bodies, but it is unclear whether this is specific to their diagnosis or whether it is found in anyone over-estimating body size. METHOD: To address this question, we compared the eye movement patterns from three participant groups while they carried out a body size estimation task: (i) 20 women with recovering/recovered anorexia (rAN) who had concerns about body shape and weight and who over-estimated body size, (ii) 20 healthy controls who had normative levels of concern about body shape and who estimated body size accurately (iii) 20 healthy controls who had normative levels of concern about body shape but who did over-estimate body size. RESULTS: Comparisons between the three groups showed that: (i) accurate body size estimators tended to look more in the waist region, and this was independent of clinical diagnosis; (ii) there is a pattern of looking at images of bodies, particularly viewing the upper parts of the torso and face, which is specific to participants with rAN but which is independent of accuracy in body size estimation. DISCUSSION: Since the over-estimating controls did not share the same body image concerns that women with rAN report, their over-estimation cannot be explained by attitudinal concerns about body shape and weight. These results suggest that a distributed fixation pattern is associated with over-estimation of body size and should be addressed in treatment programs. © 2016 Wiley Periodicals, Inc. (Int J Eat Disord 2016; 49:507-518).

Holistic face processing can inhibit recognition of forensic facial composites.

Facial composite systems help eyewitnesses to show the appearance of criminals. However, likenesses created by unfamiliar witnesses will not be completely accurate, and people familiar with the target can find them difficult to identify. Faces are processed holistically; we explore whether this impairs identification of inaccurate composite images and whether recognition can be improved. In Experiment 1 (n = 64) an imaging technique was used to make composites of celebrity faces more accurate and identification was contrasted with the original composite images. Corrected composites were better recognized, confirming that errors in production of the likenesses impair identification. The influence of holistic face processing was explored by misaligning the top and bottom parts of the composites (cf. Young, Hellawell, & Hay, 1987). Misalignment impaired recognition of corrected composites but identification of the original, inaccurate composites significantly improved. This effect was replicated with facial composites of noncelebrities in Experiment 2 (n = 57). We conclude that, like real faces, facial composites are processed holistically: recognition is impaired because unlike real faces, composites contain inaccuracies and holistic face processing makes it difficult to perceive identifiable features. This effect was consistent across composites of celebrities and composites of people who are personally familiar. Our findings suggest that identification of forensic facial composites can be enhanced by presenting composites in a misaligned format.