Distinct Yet Proximal Face- and Body-Selective Brain Regions Enable Clutter-Tolerant Representations of the Face, Body, and Whole Person.

Faces and bodies are processed in separate but adjacent regions in the primate visual cortex. Yet, the functional significance of dividing the whole person into areas dedicated to its face and body components and their neighboring locations remains unknown. Here we hypothesized that this separation and proximity together with a normalization mechanism generate clutter-tolerant representations of the face, body, and whole person when presented in complex multi-category scenes. To test this hypothesis, we conducted a fMRI study, presenting images of a person within a multi-category scene to human male and female participants and assessed the contribution of each component to the response to the scene. Our results revealed a clutter-tolerant representation of the whole person in areas selective for both faces and bodies, typically located at the border between the two category-selective regions. Regions exclusively selective for faces or bodies demonstrated clutter-tolerant representations of their preferred category, corroborating earlier findings. Thus, the adjacent locations of face- and body-selective areas enable a hardwired machinery for decluttering of the whole person, without the need for a dedicated population of person-selective neurons. This distinct yet proximal functional organization of category-selective brain regions enhances the representation of the socially significant whole person, along with its face and body components, within multi-category scenes.

Face Processing Systems: From Neurons to Real-World Social Perception.

Primate face processing depends on a distributed network of interlinked face-selective areas composed of face-selective neurons. In both humans and macaques, the network is divided into a ventral stream and a dorsal stream, and the functional similarities of the areas in humans and macaques indicate they are homologous. Neural correlates for face detection, holistic processing, face space, and other key properties of human face processing have been identified at the single neuron level, and studies providing causal evidence have established firmly that face-selective brain areas are central to face processing. These mechanisms give rise to our highly accurate familiar face recognition but also to our error-prone performance with unfamiliar faces. This limitation of the face system has important implications for consequential situations such as eyewitness identification and policing.

Deep learning models challenge the prevailing assumption that face-like effects for objects of expertise support domain-general mechanisms.

The question of whether task performance is best achieved by domain-specific, or domain-general processing mechanisms is fundemental for both artificial and biological systems. This question has generated a fierce debate in the study of expert object recognition. Because humans are experts in face recognition, face-like neural and cognitive effects for objects of expertise were considered support for domain-general mechanisms. However, effects of domain, experience and level of categorization, are confounded in human studies, which may lead to erroneous inferences. To overcome these limitations, we trained deep learning algorithms on different domains (objects, faces, birds) and levels of categorization (basic, sub-ordinate, individual), matched for amount of experience. Like humans, the models generated a larger inversion effect for faces than for objects. Importantly, a face-like inversion effect was found for individual-based categorization of non-faces (birds) but only in a network specialized for that domain. Thus, contrary to prevalent assumptions, face-like effects for objects of expertise do not support domain-general mechanisms but may originate from domain-specific mechanisms. More generally, we show how deep learning algorithms can be used to dissociate factors that are inherently confounded in the natural environment of biological organisms to test hypotheses about their isolated contributions to cognition and behaviour.

Why psychologists should embrace rather than abandon DNNs.

Deep neural networks (DNNs) are powerful computational models, which generate complex, high-level representations that were missing in previous models of human cognition. By studying these high-level representations, psychologists can now gain new insights into the nature and origin of human high-level vision, which was not possible with traditional handcrafted models. Abandoning DNNs would be a huge oversight for psychological sciences.

The Functional Organization of High-Level Visual Cortex Determines the Representation of Complex Visual Stimuli.

A hallmark of high-level visual cortex is its functional organization of neighboring areas that are selective for single categories, such as faces, bodies, and objects. However, visual scenes are typically composed of multiple categories. How does a category-selective cortex represent such complex stimuli? Previous studies have shown that the representation of multiple stimuli can be explained by a normalization mechanism. Here we propose that a normalization mechanism that operates in a cortical region composed of neighboring category-selective areas would generate a representation of multi-category stimuli that varies continuously across a category-selective cortex as a function of the magnitude of category selectivity for its components. By using fMRI, we can examine this correspondence between category selectivity and the representation of multi-category stimuli along a large, continuous region of cortex. To test these predictions, we used a linear model to fit the fMRI response of human participants (both sexes) to a multi-category stimulus (e.g., a whole person) based on the response to its component stimuli presented in isolation (e.g., a face or a body). Consistent with our predictions, the response of cortical areas in high-level visual cortex to multi-category stimuli varies in a continuous manner along a weighted mean line, as a function of the magnitude of its category selectivity. This was the case for both related (face + body) and unrelated (face+wardrobe) multi-category pairs. We conclude that the functional organization of neighboring category-selective areas may enable a dynamic and flexible representation of complex visual scenes that can be modulated by higher-level cognitive systems according to task demands.SIGNIFICANCE STATEMENT It is well established that the high-level visual cortex is composed of category-selective areas that reside in nearby locations. Here we predicted that this functional organization together with a normalization mechanism would generate a representation for multi-category stimuli that varies as a function of the category selectivity for its components. Consistent with this prediction, in an fMRI study we found that the representation of multi-category stimuli varies along high-level visual cortex, in a continuous manner, along a weighted mean line, in accordance with the category selectivity for a given area. These findings suggest that the functional organization of high-level visual cortex enables a flexible representation of complex scenes that can be modulated by high-level cognitive systems according to task demands.

What Is a Face? Critical Features for Face Detection.

Faces convey very rich information that is critical for intact social interaction. To extract this information efficiently, faces should be easily detected from a complex visual scene. Here, we asked which features are critical for face detection. To answer this question, we presented non-face objects that generate a strong percept of a face (i.e., Pareidolia). One group of participants rated the faceness of this set of inanimate images. A second group rated the presence of a set of 12 local and global facial features. Regression analysis revealed that only the eyes or mouth significantly contributed to faceness scores. We further showed that removing eyes or mouth, but not teeth or ears, significantly reduced faceness scores. These findings show that face detection depends on specific facial features, the eyes and the mouth. This minimal information leads to over-generalization that generates false face percepts but assures that real faces are not missed.

Recognizing Facial Slivers.

We report here an unexpectedly robust ability of healthy human individuals ( n = 40) to recognize extremely distorted needle-like facial images, challenging the well-entrenched notion that veridical spatial configuration is necessary for extracting facial identity. In face identification tasks of parametrically compressed internal and external features, we found that the sum of performances on each cue falls significantly short of performance on full faces, despite the equal visual information available from both measures (with full faces essentially being a superposition of internal and external features). We hypothesize that this large deficit stems from the use of positional information about how the internal features are positioned relative to the external features. To test this, we systematically changed the relations between internal and external features and found preferential encoding of vertical but not horizontal spatial relationships in facial representations ( n = 20). Finally, we employ magnetoencephalography imaging ( n = 20) to demonstrate a close mapping between the behavioral psychometric curve and the amplitude of the M250 face familiarity, but not M170 face-sensitive evoked response field component, providing evidence that the M250 can be modulated by faces that are perceptually identifiable, irrespective of extreme distortions to the face's veridical configuration. We theorize that the tolerance to compressive distortions has evolved from the need to recognize faces across varying viewpoints. Our findings help clarify the important, but poorly defined, concept of facial configuration and also enable an association between behavioral performance and previously reported neural correlates of face perception.

The Electrophysiological Signature of Remember-Know Is Confounded with Memory Strength and Cannot Be Interpreted as Evidence for Dual-process Theory of Recognition.

The quantity and nature of the processes underlying recognition memory remains an open question. A majority of behavioral, neuropsychological, and brain studies have suggested that recognition memory is supported by two dissociable processes: recollection and familiarity. It has been conversely argued, however, that recollection and familiarity map onto a single continuum of mnemonic strength and hence that recognition memory is mediated by a single process. Previous electrophysiological studies found marked dissociations between recollection and familiarity, which have been widely held as corroborating the dual-process account. However, it remains unknown whether a strength interpretation can likewise apply for these findings. Here we describe an ERP study, using a modified remember-know (RK) procedure, which allowed us to control for mnemonic strength. We find that ERPs of high and low mnemonic strength mimicked the electrophysiological distinction between R and K responses, in a lateral positive component (LPC), 500-1000 msec poststimulus onset. Critically, when contrasting strength with RK experience, by comparing weak R to strong K responses, the electrophysiological signal mapped onto strength, not onto subjective RK experience. Invoking the LPC as support for dual-process accounts may, therefore, be amiss.

Bodies are Represented as Wholes Rather Than Their Sum of Parts in the Occipital-Temporal Cortex.

Behavioral studies suggested that bodies are represented as wholes rather than in a part-based manner. However, neural selectivity for body stimuli is found for both whole bodies and body parts. It is therefore undetermined whether the neural representation of bodies is configural or part-based. We used functional MRI to test the role of first-order configuration on body representation in the human occipital-temporal cortex by comparing the response to a whole body versus the sum of its parts. Results show that body-selective areas, whether defined by selectivity to headless bodies or body parts, preferred whole bodies over their sum of parts and successfully decoded body configuration. This configural representation was specific to body stimuli and not found for faces. In contrast, general object areas showed no preference for wholes over parts and decoded the configuration of both bodies and faces. Finally, whereas effects of inversion on configural face representation were specific to face-selective mechanisms, effects of body inversion were not unique to body-selective mechanisms. We conclude that the neural representation of body parts is strengthened by their arrangement into an intact body, thereby demonstrating a central role of first-order configuration in the neural representation of bodies in their category-selective areas.

Neural Correlates of Subliminal Language Processing.

Language is a high-level cognitive function, so exploring the neural correlates of unconscious language processing is essential for understanding the limits of unconscious processing in general. The results of several functional magnetic resonance imaging studies have suggested that unconscious lexical and semantic processing is confined to the posterior temporal lobe, without involvement of the frontal lobe-the regions that are indispensable for conscious language processing. However, previous studies employed a similarly designed masked priming paradigm with briefly presented single and contextually unrelated words. It is thus possible, that the stimulation level was insufficiently strong to be detected in the high-level frontal regions. Here, in a high-resolution fMRI and multivariate pattern analysis study we explored the neural correlates of subliminal language processing using a novel paradigm, where written meaningful sentences were suppressed from awareness for extended duration using continuous flash suppression. We found that subjectively and objectively invisible meaningful sentences and unpronounceable nonwords could be discriminated not only in the left posterior superior temporal sulcus (STS), but critically, also in the left middle frontal gyrus. We conclude that frontal lobes play a role in unconscious language processing and that activation of the frontal lobes per se might not be sufficient for achieving conscious awareness.

Reverse engineering the face space: Discovering the critical features for face identification.

How do we identify people? What are the critical facial features that define an identity and determine whether two faces belong to the same person or different people? To answer these questions, we applied the face space framework, according to which faces are represented as points in a multidimensional feature space, such that face space distances are correlated with perceptual similarities between faces. In particular, we developed a novel method that allowed us to reveal the critical dimensions (i.e., critical features) of the face space. To that end, we constructed a concrete face space, which included 20 facial features of natural face images, and asked human observers to evaluate feature values (e.g., how thick are the lips). Next, we systematically and quantitatively changed facial features, and measured the perceptual effects of these manipulations. We found that critical features were those for which participants have high perceptual sensitivity (PS) for detecting differences across identities (e.g., which of two faces has thicker lips). Furthermore, these high PS features vary minimally across different views of the same identity, suggesting high PS features support face recognition across different images of the same face. The methods described here set an infrastructure for discovering the critical features of other face categories not studied here (e.g., Asians, familiar) as well as other aspects of face processing, such as attractiveness or trait inferences.

Cross-modal reorganization in cochlear implant users: Auditory cortex contributes to visual face processing.

There is converging evidence that the auditory cortex takes over visual functions during a period of auditory deprivation. A residual pattern of cross-modal take-over may prevent the auditory cortex to adapt to restored sensory input as delivered by a cochlear implant (CI) and limit speech intelligibility with a CI. The aim of the present study was to investigate whether visual face processing in CI users activates auditory cortex and whether this has adaptive or maladaptive consequences. High-density electroencephalogram data were recorded from CI users (n=21) and age-matched normal hearing controls (n=21) performing a face versus house discrimination task. Lip reading and face recognition abilities were measured as well as speech intelligibility. Evaluation of event-related potential (ERP) topographies revealed significant group differences over occipito-temporal scalp regions. Distributed source analysis identified significantly higher activation in the right auditory cortex for CI users compared to NH controls, confirming visual take-over. Lip reading skills were significantly enhanced in the CI group and appeared to be particularly better after a longer duration of deafness, while face recognition was not significantly different between groups. However, auditory cortex activation in CI users was positively related to face recognition abilities. Our results confirm a cross-modal reorganization for ecologically valid visual stimuli in CI users. Furthermore, they suggest that residual takeover, which can persist even after adaptation to a CI is not necessarily maladaptive.

Clutter modulates the representation of target objects in the human occipitotemporal cortex.

Target objects required for goal-directed behavior are typically embedded within multiple irrelevant objects that may interfere with their encoding. Most neuroimaging studies of high-level visual cortex have examined the representation of isolated objects, and therefore, little is known about how surrounding objects influence the neural representation of target objects. To investigate the effect of different types of clutter on the distributed responses to target objects in high-level visual areas, we used fMRI and manipulated the type of clutter. Specifically, target objects (i.e., a face and a house) were presented either in isolation, in the presence of a homogeneous (identical objects from another category) clutter ("pop-out" display), or in the presence of a heterogeneous (different objects) clutter, while participants performed a target identification task. Using multivoxel pattern analysis (MVPA) we found that in the posterior fusiform object area a heterogeneous but not homogeneous clutter interfered with decoding of the target objects. Furthermore, multivoxel patterns evoked by isolated objects were more similar to multivoxel patterns evoked by homogenous compared with heterogeneous clutter in the lateral occipital and posterior fusiform object areas. Interestingly, there was no effect of clutter on the neural representation of the target objects in their category-selective areas, such as the fusiform face area and the parahippocampal place area. Our findings show that the variation among irrelevant surrounding objects influences the neural representation of target objects in the object general area, but not in object category-selective cortex, where the representation of target objects is invariant to their surroundings.

An integrated face-body representation in the fusiform gyrus but not the lateral occipital cortex.

Faces and bodies are processed by distinct category-selective brain areas. Neuroimaging studies have so far presented isolated faces and headless bodies, and therefore little is known on whether and where faces and headless bodies are grouped together to one object, as they appear in the real world. The current study examined whether a face presented above a body are represented as two separate images or as an integrated face-body representation in face and body-selective brain areas by employing a fMRI competition paradigm. This paradigm has been shown to reveal higher fMRI response to sequential than simultaneous presentation of multiple stimuli (i.e., the competition effect), indicating competitive interactions among simultaneously presented multiple stimuli. We therefore hypothesized that if a face above a body is integrated to an image of a person whereas a body above a face is represented as two separate objects, the competition effect will be larger for the latter than the former. Consistent with our hypothesis, our findings reveal a competition effect when a body is presented above a face, but not when a face is presented above a body, suggesting that a body above a face is represented as two separate objects whereas a face above a body is represented as an integrated image of a person. Interestingly, this integration of a face and a body to an image of a person was found in the fusiform, but not the lateral-occipital face and body areas. We conclude that faces and bodies are processed separately at early stages and are integrated to a unified image of a person at mid-level stages of object processing.

Separate parts of occipito-temporal white matter fibers are associated with recognition of faces and places.

A central finding of functional MRI studies is the highly selective response of distinct brain areas in the occipital temporal cortex to faces and places. However, little is known about the association of white matter fibers with the processing of these object categories. In the current study we used DTI-based tractography to reconstruct two main fibers that connect the occipital lobe with the anterior temporal lobe (inferior longitudinal fasciculus-ILF) and with the frontal lobe (inferior fronto-occipital fasciculus-IFOF) in normal individuals. In addition to MRI scans subjects performed face, scene and body recognition tasks outside the scanner. Results show that recognition of faces and scenes were selectively associated with separate parts of the ILF. In particular, face recognition was highly associated with the fractional anisotropy (FA) of the anterior part of the ILF in the right hemisphere. In contrast, scene recognition was strongly correlated with the FA of the posterior and middle but not the anterior part of the ILF bilaterally. Our findings provide the first demonstration that faces and places are not only associated with distinct brain areas but also with separate parts of white matter fibers.

Faces in the eye of the beholder: unique and stable eye scanning patterns of individual observers.

Eye scanning patterns while viewing pictures have provided valuable information in many domains of visual cognition. Such patterns are determined by the type of image viewed (e.g., faces, scenes) as well as the task individuals are asked to perform (e.g., visual search, memory). Here we show that another key factor that significantly influences eye scanning patterns but has been mostly overlooked is the individual observer. During face viewing, we found that individuals showed diverse scanning patterns that, in many cases, were inconsistent with the typical triangular shape pattern that is commonly observed when eye scanning patterns are averaged across individuals. These idiosyncratic eye scanning patterns were not random but highly stable even when examined 18 months later. Interestingly, these eye-tracking patterns were not predictive of behavioral performance. Such stable and unique scanning patterns may represent a specific behavioral trait/signature and be formed early in development, reflecting idiosyncratic strategies for performing visual recognition tasks.

Using deep neural networks to disentangle visual and semantic information in human perception and memory.

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Hierarchical processing of face viewpoint in human visual cortex.

The ability to recognize objects across different viewpoints (view invariance) is a remarkable property of the primate visual system. According to a prominent theory, view information is represented by view-selective mechanisms at early stages of visual processing and gradually becomes view invariant in high-level visual areas. Single-cell recording studies have also reported an intermediate step of partial view invariance for mirror-symmetric face views. Nevertheless, similar evidence for this type of hierarchical processing for face view has not been reported yet in the human visual cortex. The present functional magnetic resonance imaging study used state-of-the-art multivariate pattern analysis to explore face-view tuning in the human visual cortex. Our results revealed that consistent with a view-selective representation, face view can be successfully decoded in face and object-selective regions as well as in early visual cortex. Critically, similar neural representations for mirror-symmetric views were found in high-level but not in low-level visual areas. Our results support the notion of gradual emergence of view-invariant representation with invariance for mirror-symmetric images as an intermediate step and propose putative neural correlates of mirror-image confusion in the human brain.

The challenge of localizing the anterior temporal face area: a possible solution.

Humans recognize faces exceptionally well. However, the neural correlates of face recognition are still elusive. Accumulated evidence in recent years suggests that the anterior temporal lobe (ATL), in particular face-selective region in the ATL, is a probable locus of face recognition. Unfortunately, functional MRI (fMRI) studies encounter severe signal drop-out in the ventral ATL, where that ATL face area resides. Consequently, all previous studies localized this region in no more than half of the subjects and its volume was relatively small. Thus, a systematic exploration of the properties of the ATL face area is scarce. In the current high-resolution fMRI study we used coronal slice orientation, which permitted us to localize the ATL face area in all the subjects. Furthermore, the volume of the area was much larger than was reported in previous studies. Direct within subjects comparison with data collected with the commonly used axial slice orientation confirmed that the advantage of the coronal slice orientation in revealing a reliable and larger face-selective area in the ATL. Finally, by displaying the face-selective activations resultant from coronal and axial scanning together, we demonstrate an organization principle of a chain of face-selective regions along the posterior-anterior axis in the ventral temporal lobe that is highly reproducible across all subjects. By using the procedure proposed here, a significant progress can be made in studying the neural correlates of face recognition.

The social-encoding benefit in face recognition is generalized to other-race faces.

Faces are visual stimuli that convey rich social information. Previous experiments found better recognition for faces that were evaluated based on their social traits than on their perceptual features during encoding. Here, we ask whether this social-encoding benefit in face recognition is also found for categories of faces that we have no previous social experience with, such as other-race faces. To answer this question, we first explored whether social and perceptual evaluations for other-race faces are consistent and valid. We then asked whether social evaluations during encoding improve recognition for other-race faces. Results show that social and perceptual evaluations of own- and other-race faces were valid. We also found high agreement in social and perceptual evaluations across individuals from different races. This indicates that evaluations of other-race faces are not random but meaningful. Furthermore, we found that social evaluations facilitated face recognition regardless of race, demonstrating a social-encoding benefit for both own- and other-race faces. Our findings highlight the role of social information in face recognition and show how it can be used to improve recognition of categories of faces that are hard to recognize due to lack of experience with them.