Using deep neural networks to disentangle visual and semantic information in human perception and memory.

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Perceptual similarity modulates effects of learning from variability on face recognition.

Face recognition is a challenging classification task that humans perform effortlessly for familiar faces. Recent studies have emphasized the importance of exposure to high variability appearances of the same identity to perform this task. However, these studies did not explicitly measure the perceptual similarity between the learned images and the images presented at test, which may account for the advantage of learning from high variability. Particularly, randomly selected test images are more likely to be perceptually similar to learned high variability images, and dissimilar to learned low variability images. Here we dissociated effects of learning from variability and study-test perceptual similarity, by collecting human similarity ratings for the study and test images. Using these measures, we independently manipulated the variability between the learning images and their perceptual similarity to the test images. Different groups of participants learned face identities from a low or high variability set of images. The learning phase was followed by a face matching test (Experiment 1) or a face recognition task (Experiment 2) that presented novel images of the learned identities that were perceptually dissimilar or similar to the learned images. Results of both experiments show that perceptual similarity between study and test, rather than image variability at learning per se, predicts face recognition. We conclude that learning from high variability improves face recognition for perceptually similar but not for perceptually dissimilar images. These findings may not be specific to faces and should be similarly evaluated for other domains.

Learning Faces as Concepts Improves Face Recognition by Engaging the Social Brain Network.

Face recognition benefits from associating social information to faces during learning. This has been demonstrated by better recognition for faces that underwent social than perceptual evaluations. Two hypotheses were proposed to account for this effect. According to the feature-elaboration hypothesis, social-evaluations encourage elaborated processing of perceptual information from faces (Winograd, 1981). According to a social-representation hypothesis, social-evaluations convert faces from a perceptual representation to a socially meaningful representation of a person. To decide between these two hypotheses, we ran a functional MRI study in which we functionally localized the posterior face-selective brain areas and social processing brain areas. Participants watched video-clips of young adults and were asked to study them for a recognition test, while making either perceptual evaluations or social evaluations about them. During the fMRI scan, participants performed an old/new recognition test. Behavioural findings replicated better recognition for faces that underwent social then perceptual evaluations. fMRI results showed higher response during the recognition phase for the faces that were learned socially than perceptually, in the social-brain network but not in posterior face-selective network. These results support the social-representation hypothesis and highlight the important role that social processing mechanisms, rather than purely perceptual processes, play in face recognition.