The Neural Dynamics of Face Ensemble and Central Face Processing.

Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.

Computation on demand: Action-specific representations of visual task features arise during distinct movement phases.

The intention to act influences the computations of various task-relevant features. However, little is known about the time course of these computations. Furthermore, it is commonly held that these computations are governed by conjunctive neural representations of the features. But, support for this view comes from paradigms arbitrarily combining task features and affordances, thus, requiring representations in working memory. Therefore, the present study used electroencephalography and a well-rehearsed task with features that afford minimal working memory representations to investigate the temporal evolution of feature representations and their potential integration in the brain. Female and male human participants viewed and grasped objects or touched them with a knuckle. Objects had different shapes and were made of heavy or light materials with shape and weight being relevant for grasping, not for "knuckling." Using multivariate analysis showed that representations of object shape were similar for grasping and knuckling. However, only for grasping did early shape representations reactivate at later phases of grasp planning, suggesting that sensorimotor control signals feed back to early visual cortex. Grasp-specific representations of material/weight only arose during grasp execution after object contact during the load phase. A trend for integrated representations of shape and material also became grasp-specific but only briefly during Movement onset. These results suggest that the brain generates action-specific representations of relevant features as required for the different subcomponents of its action computations. Our results argue against the view that goal-directed actions inevitably join all features of a task into a sustained and unified neural representation.Significance statement The idea that all the features of a task are integrated into a joint representation or event file is widely supported but importantly based on paradigms with arbitrary stimulus-response combinations. Our study is the first to investigate grasping using electroencephalography to search for the neural basis of feature integration in such a daily-life task with overlearned stimulus-response mappings. Contrary to the notion of event files we find limited evidence for integrated representations. Instead, we find that task-relevant features form representations at specific phases of the action, suggesting that action intentions reactivate representations of relevant features. Our results show that integrated representations do not occur universally for any kind of goal-directed behaviour but in a manner of computation on demand.

Objective Evaluation of Laparoscopic Experience Based on Muscle Electromyography and Accelerometry Performing Circular Pattern Cutting Tasks: A Pilot Study.

Purpose.The aim of this work is to present a new physical laparoscopy simulator with an electromyography (EMG)/accelerometry-based muscle activity recording system, EvalLap EMG-ACC, and perform objective evaluation of laparoscopic skills based on the quantification of muscle activity of participants with different levels of laparoscopic experience. Methods. EMG and ACC signals were obtained from 14 participants (6 experts, 8 medical students) performing circular pattern cutting tasks using a laparoscopic box trainer with the Trigno (Delsys Inc, Natick, MA) portable wireless system of 16 wireless sensors. Sensors were placed on the proximal and distal muscles of the upper extremities. Seven evaluation metrics were proposed and compared between skilled and novice surgeons. Results. The proximal and distal arm muscles (trapezius, deltoids, biceps, and forearms) were most active while executing laparoscopic tasks. Laparoscopic experience was associated with differences in EMG amplitude (Aavg), muscle activity (iEMG), hand acceleration (iACH), user movement (iAC), and muscle fatigue. For the cutting task, the deltoid, bicep, forearm EMG amplitude, and user movement significantly differed between experience groups. Conclusion. This pilot study demonstrates that different muscle groups are preferentially activated during laparoscopic tasks depending on the level of surgical experience. Expert surgeons showed less muscle activity compared with novices. EvalLap EMG-ACC represents a promising means to distinguish surgeons with basic cutting skills from those who have not yet developed these skills.

Evidence for the incorporation of temporal duration information in human hippocampal long-term memory sequence representations.

There has been much interest in how the hippocampus codes time in support of episodic memory. Notably, while rodent hippocampal neurons, including populations in subfield CA1, have been shown to represent the passage of time in the order of seconds between events, there is limited support for a similar mechanism in humans. Specifically, there is no clear evidence that human hippocampal activity during long-term memory processing is sensitive to temporal duration information that spans seconds. To address this gap, we asked participants to first learn short event sequences that varied in image content and interval durations. During fMRI, participants then completed a recognition memory task, as well as a recall phase in which they were required to mentally replay each sequence in as much detail as possible. We found that individual sequences could be classified using activity patterns in the anterior hippocampus during recognition memory. Critically, successful classification was dependent on the conjunction of event content and temporal structure information (with unsuccessful classification of image content or interval duration alone), and further analyses suggested that the most informative voxels resided in the anterior CA1. Additionally, a classifier trained on anterior CA1 recognition data could successfully identify individual sequences from the mental replay data, suggesting that similar activity patterns supported participants' recognition and recall memory. Our findings complement recent rodent hippocampal research, and provide evidence that long-term sequence memory representations in the human hippocampus can reflect duration information in the order of seconds.

Elucidating the Neural Representation and the Processing Dynamics of Face Ensembles.

Extensive behavioral work has documented the ability of the human visual system to extract summary representations from face ensembles (e.g., the average identity of a crowd of faces). Yet, the nature of such representations, their underlying neural mechanisms, and their temporal dynamics await elucidation. Here, we examine summary representations of facial identity in human adults (of both sexes) with the aid of pattern analyses, as applied to EEG data, along with behavioral testing. Our findings confirm the ability of the visual system to form such representations both explicitly and implicitly (i.e., with or without the use of specific instructions). We show that summary representations, rather than individual ensemble constituents, can be decoded from neural signals elicited by ensemble perception, we describe the properties of such representations by appeal to multidimensional face space constructs, and we visualize their content through neural-based image reconstruction. Further, we show that the temporal profile of ensemble processing diverges systematically from that of single faces consistent with a slower, more gradual accumulation of perceptual information. Thus, our findings reveal the representational basis of ensemble processing, its fine-grained visual content, and its neural dynamics.SIGNIFICANCE STATEMENT Humans encounter groups of faces, or ensembles, in a variety of environments. Previous behavioral research has investigated how humans process face ensembles as well as the types of summary representations that can be derived from them, such as average emotion, gender, and identity. However, the neural mechanisms mediating these processes are unclear. Here, we demonstrate that ensemble representations, with different facial identity summaries, can be decoded and even visualized from neural data through multivariate analyses. These results provide, to our knowledge, the first detailed investigation into the status and the visual content of neural ensemble representations of faces. Further, the current findings shed light on the temporal dynamics of face ensembles and its relationship with single-face processing.

Multivariate Analysis of Electrophysiological Signals Reveals the Temporal Properties of Visuomotor Computations for Precision Grips.

The frontoparietal networks underlying grasping movements have been extensively studied, especially using fMRI. Accordingly, whereas much is known about their cortical locus much less is known about the temporal dynamics of visuomotor transformations. Here, we show that multivariate EEG analysis allows for detailed insights into the time course of visual and visuomotor computations of precision grasps. Male and female human participants first previewed one of several objects and, upon its reappearance, reached to grasp it with the thumb and index finger along one of its two symmetry axes. Object shape classifiers reached transient accuracies of 70% at ∼105 ms, especially based on scalp sites over visual cortex, dropping to lower levels thereafter. Grasp orientation classifiers relied on a system of occipital-to-frontal electrodes. Their accuracy rose concurrently with shape classification but ramped up more gradually, and the slope of the classification curve predicted individual reaction times. Further, cross-temporal generalization revealed that dynamic shape representation involved early and late neural generators that reactivated one another. In contrast, grasp computations involved a chain of generators attaining a sustained state about 100 ms before movement onset. Our results reveal the progression of visual and visuomotor representations over the course of planning and executing grasp movements.SIGNIFICANCE STATEMENT Grasping an object requires the brain to perform visual-to-motor transformations of the object's properties. Although much of the neuroanatomic basis of visuomotor transformations has been uncovered, little is known about its time course. Here, we orthogonally manipulated object visual characteristics and grasp orientation, and used multivariate EEG analysis to reveal that visual and visuomotor computations follow similar time courses but display different properties and dynamics.

Spatiotemporal dynamics of similarity-based neural representations of facial identity.

Humans' remarkable ability to quickly and accurately discriminate among thousands of highly similar complex objects demands rapid and precise neural computations. To elucidate the process by which this is achieved, we used magnetoencephalography to measure spatiotemporal patterns of neural activity with high temporal resolution during visual discrimination among a large and carefully controlled set of faces. We also compared these neural data to lower level "image-based" and higher level "identity-based" model-based representations of our stimuli and to behavioral similarity judgments of our stimuli. Between ∼50 and 400 ms after stimulus onset, face-selective sources in right lateral occipital cortex and right fusiform gyrus and sources in a control region (left V1) yielded successful classification of facial identity. In all regions, early responses were more similar to the image-based representation than to the identity-based representation. In the face-selective regions only, responses were more similar to the identity-based representation at several time points after 200 ms. Behavioral responses were more similar to the identity-based representation than to the image-based representation, and their structure was predicted by responses in the face-selective regions. These results provide a temporally precise description of the transformation from low- to high-level representations of facial identity in human face-selective cortex and demonstrate that face-selective cortical regions represent multiple distinct types of information about face identity at different times over the first 500 ms after stimulus onset. These results have important implications for understanding the rapid emergence of fine-grained, high-level representations of object identity, a computation essential to human visual expertise.

Multimodal evidence on shape and surface information in individual face processing.

The significance of shape and surface information for face perception is well established, yet their relative contribution to recognition and their neural underpinnings await clarification. Here, we employ image reconstruction to retrieve, assess and visualize such information using behavioral, electroencephalography and functional magnetic resonance imaging data. Our results indicate that both shape and surface information can be successfully recovered from each modality but that the latter is better recovered than the former, consistent with its key role for face representations. Further, shape and surface information exhibit similar spatiotemporal profiles, rely on the extraction of specific visual features, such as eye shape or skin tone, and reveal a systematic representational structure, albeit with more cross-modal consistency for shape than surface. More generally, the present work illustrates a novel approach to relating and comparing different modalities in terms of perceptual information content. Thus, our results help elucidate the representational basis of individual face recognition while, methodologically, they showcase the utility of image reconstruction and clarify its reliance on diagnostic visual information.

How are visual words represented? Insights from EEG-based visual word decoding, feature derivation and image reconstruction.

Investigations into the neural basis of reading have shed light on the cortical locus and the functional role of visual-orthographic processing. Yet, the fine-grained structure of neural representations subserving reading remains to be clarified. Here, we capitalize on the spatiotemporal structure of electroencephalography (EEG) data to examine if and how EEG patterns can serve to decode and reconstruct the internal representation of visually presented words in healthy adults. Our results show that word classification and image reconstruction were accurate well above chance, that their temporal profile exhibited an early onset, soon after 100 ms, and peaked around 170 ms. Further, reconstruction results were well explained by a combination of visual-orthographic word properties. Last, systematic individual differences were detected in orthographic representations across participants. Collectively, our results establish the feasibility of EEG-based word decoding and image reconstruction. More generally, they help to elucidate the specific features, dynamics, and neurocomputational principles underlying word recognition.

Feature-based face representations and image reconstruction from behavioral and neural data.

The reconstruction of images from neural data can provide a unique window into the content of human perceptual representations. Although recent efforts have established the viability of this enterprise using functional magnetic resonance imaging (MRI) patterns, these efforts have relied on a variety of prespecified image features. Here, we take on the twofold task of deriving features directly from empirical data and of using these features for facial image reconstruction. First, we use a method akin to reverse correlation to derive visual features from functional MRI patterns elicited by a large set of homogeneous face exemplars. Then, we combine these features to reconstruct novel face images from the corresponding neural patterns. This approach allows us to estimate collections of features associated with different cortical areas as well as to successfully match image reconstructions to corresponding face exemplars. Furthermore, we establish the robustness and the utility of this approach by reconstructing images from patterns of behavioral data. From a theoretical perspective, the current results provide key insights into the nature of high-level visual representations, and from a practical perspective, these findings make possible a broad range of image-reconstruction applications via a straightforward methodological approach.

Independence of viewpoint and identity in face ensemble processing.

Ensemble encoding refers to the visual system's ability to extract a summary representation from multiple items in a set-such as the mean identity of faces in a crowd-circumventing capacity limitations in visual working memory. In the present study we investigated face ensemble representations of higher level identity and lower level viewpoint with the aim of elucidating the extent of their overlap or independence. To this end, we used ensemble displays consisting of six face stimuli which could vary in identity, viewpoint, or both. Across three experiments, participants were asked to report an average identity, a single identity, an average viewpoint, or a single viewpoint, as cued by a central probe face following a stimulus display. In Experiment 1, we observed a dissociation between the processing of identity and viewpoint, as well as between average- and single-viewpoint extraction. Experiment 2 showed viewpoint-invariant identity processing across a wide range of mean viewpoints, spanning 120°. In Experiment 3, accuracy in reporting a response-relevant attribute was unaffected by changes in an irrelevant attribute. Participants were also capable of extracting both attributes simultaneously with little change in accuracy. Taken together, these results argue for the independence of identity and viewpoint in face ensemble processing.

The time course of individual face recognition: A pattern analysis of ERP signals.

An extensive body of work documents the time course of neural face processing in the human visual cortex. However, the majority of this work has focused on specific temporal landmarks, such as N170 and N250 components, derived through univariate analyses of EEG data. Here, we take on a broader evaluation of ERP signals related to individual face recognition as we attempt to move beyond the leading theoretical and methodological framework through the application of pattern analysis to ERP data. Specifically, we investigate the spatiotemporal profile of identity recognition across variation in emotional expression. To this end, we apply pattern classification to ERP signals both in time, for any single electrode, and in space, across multiple electrodes. Our results confirm the significance of traditional ERP components in face processing. At the same time though, they support the idea that the temporal profile of face recognition is incompletely described by such components. First, we show that signals associated with different facial identities can be discriminated from each other outside the scope of these components, as early as 70ms following stimulus presentation. Next, electrodes associated with traditional ERP components as well as, critically, those not associated with such components are shown to contribute information to stimulus discriminability. And last, the levels of ERP-based pattern discrimination are found to correlate with recognition accuracy across subjects confirming the relevance of these methods for bridging brain and behavior data. Altogether, the current results shed new light on the fine-grained time course of neural face processing and showcase the value of novel methods for pattern analysis to investigating fundamental aspects of visual recognition.

Monocular advantage for face perception implicates subcortical mechanisms in adult humans.

The ability to recognize faces accurately and rapidly is an evolutionarily adaptive process. Most studies examining the neural correlates of face perception in adult humans have focused on a distributed cortical network of face-selective regions. There is, however, robust evidence from phylogenetic and ontogenetic studies that implicates subcortical structures, and recently, some investigations in adult humans indicate subcortical correlates of face perception as well. The questions addressed here are whether low-level subcortical mechanisms for face perception (in the absence of changes in expression) are conserved in human adults, and if so, what is the nature of these subcortical representations. In a series of four experiments, we presented pairs of images to the same or different eyes. Participants' performance demonstrated that subcortical mechanisms, indexed by monocular portions of the visual system, play a functional role in face perception. These mechanisms are sensitive to face-like configurations and afford a coarse representation of a face, comprised of primarily low spatial frequency information, which suffices for matching faces but not for more complex aspects of face perception such as sex differentiation. Importantly, these subcortical mechanisms are not implicated in the perception of other visual stimuli, such as cars or letter strings. These findings suggest a conservation of phylogenetically and ontogenetically lower-order systems in adult human face perception. The involvement of subcortical structures in face recognition provokes a reconsideration of current theories of face perception, which are reliant on cortical level processing, inasmuch as it bolsters the cross-species continuity of the biological system for face recognition.

The neural basis of visual word form processing: a multivariate investigation.

Current research on the neurobiological bases of reading points to the privileged role of a ventral cortical network in visual word processing. However, the properties of this network and, in particular, its selectivity for orthographic stimuli such as words and pseudowords remain topics of significant debate. Here, we approached this issue from a novel perspective by applying pattern-based analyses to functional magnetic resonance imaging data. Specifically, we examined whether, where and how, orthographic stimuli elicit distinct patterns of activation in the human cortex. First, at the category level, multivariate mapping found extensive sensitivity throughout the ventral cortex for words relative to false-font strings. Secondly, at the identity level, the multi-voxel pattern classification provided direct evidence that different pseudowords are encoded by distinct neural patterns. Thirdly, a comparison of pseudoword and face identification revealed that both stimulus types exploit common neural resources within the ventral cortical network. These results provide novel evidence regarding the involvement of the left ventral cortex in orthographic stimulus processing and shed light on its selectivity and discriminability profile. In particular, our findings support the existence of sublexical orthographic representations within the left ventral cortex while arguing for the continuity of reading with other visual recognition skills.

Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis.

Face individuation is one of the most impressive achievements of our visual system, and yet uncovering the neural mechanisms subserving this feat appears to elude traditional approaches to functional brain data analysis. The present study investigates the neural code of facial identity perception with the aim of ascertaining its distributed nature and informational basis. To this end, we use a sequence of multivariate pattern analyses applied to functional magnetic resonance imaging (fMRI) data. First, we combine information-based brain mapping and dynamic discrimination analysis to locate spatiotemporal patterns that support face classification at the individual level. This analysis reveals a network of fusiform and anterior temporal areas that carry information about facial identity and provides evidence that the fusiform face area responds with distinct patterns of activation to different face identities. Second, we assess the information structure of the network using recursive feature elimination. We find that diagnostic information is distributed evenly among anterior regions of the mapped network and that a right anterior region of the fusiform gyrus plays a central role within the information network mediating face individuation. These findings serve to map out and characterize a cortical system responsible for individuation. More generally, in the context of functionally defined networks, they provide an account of distributed processing grounded in information-based architectures.

Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities.

Personality traits and affective states are associated with biases in facial emotion perception. However, the precise personality impairments and affective states that underlie these biases remain largely unknown. To investigate how relevant factors influence facial emotion perception and recollection, Experiment 1 employed an image reconstruction approach in which community-dwelling adults (N = 89) rated the similarity of pairs of facial expressions, including those recalled from memory. Subsequently, perception- and memory-based expression representations derived from such ratings were assessed across participants and related to measures of personality impairment, state affect, and visual recognition abilities. Impairment in self-direction and level of positive affect accounted for the largest components of individual variability in perception and memory representations, respectively. Additionally, individual differences in these representations were impacted by face recognition ability. In Experiment 2, adult participants (N = 81) rated facial image reconstructions derived in Experiment 1, revealing that individual variability was associated with specific visual face properties, such as expressiveness, representation accuracy, and positivity/negativity. These findings highlight and clarify the influence of personality, affective state, and recognition abilities on individual differences in the perception and recollection of facial expressions.

Internal representations for face detection: an application of noise-based image classification to BOLD responses.

What basic visual structures underlie human face detection and how can we extract such structures directly from the amplitude of neural responses elicited by face processing? Here, we address these issues by investigating an extension of noise-based image classification to BOLD responses recorded in high-level visual areas. First, we assess the applicability of this classification method to such data and, second, we explore its results in connection with the neural processing of faces. To this end, we construct luminance templates from white noise fields based on the response of face-selective areas in the human ventral cortex. Using behaviorally and neurally-derived classification images, our results reveal a family of simple but robust image structures subserving face representation and detection. Thus, we confirm the role played by classical face selective regions in face detection and we help clarify the representational basis of this perceptual function. From a theory standpoint, our findings support the idea of simple but highly diagnostic neurally-coded features for face detection. At the same time, from a methodological perspective, our work demonstrates the ability of noise-based image classification in conjunction with fMRI to help uncover the structure of high-level perceptual representations.

Face-space architectures: evidence for the use of independent color-based features.

The concept of psychological face space lies at the core of many theories of face recognition and representation. To date, much of the understanding of face space has been based on principal component analysis (PCA); the structure of the psychological space is thought to reflect some important aspects of a physical face space characterized by PCA applications to face images. In the present experiments, we investigated alternative accounts of face space and found that independent component analysis provided the best fit to human judgments of face similarity and identification. Thus, our results challenge an influential approach to the study of human face space and provide evidence for the role of statistically independent features in face encoding. In addition, our findings support the use of color information in the representation of facial identity, and we thus argue for the inclusion of such information in theoretical and computational constructs of face space.

Using image reconstruction to investigate face perception in amnesia.

Damage to the medial temporal lobe (MTL), which is traditionally considered to subserve memory exclusively, has been reported to contribute to impaired face perception. However, it remains unknown how exactly such brain lesions may impact face representations and in particular facial shape and surface information, both of which are crucial for face perception. The present study employed a behavioral-based image reconstruction approach to reveal the pictorial representations of face perception in two amnesic patients: DA, who has an extensive bilateral MTL lesion that extends beyond the MTL in the right hemisphere, and BL, who has damage to the hippocampal dentate gyrus (DG). Both patients and their respective matched controls completed similarity judgments for pairs of faces, from which facial shape and surface features were subsequently derived and synthesized to create images of reconstructed facial appearance. Participants also completed a face oddity judgment task (FOJT) that has previously been shown to be sensitive to MTL cortical damage. While BL exhibited an impaired pattern of performance on the FOJT, DA demonstrated intact performance accuracy. Notably, the recovered pictorial content of faces was comparable between both patients and controls, although there was evidence for atypical face representations in BL particularly with regards to color. Our work provides novel insight into the face representations underlying face perception in two well-studied amnesic patients in the literature and demonstrates the applicability of the image reconstruction approach to individuals with brain damage.