The role of motion in the neural representation of social interactions in the posterior temporal cortex.

Humans are an inherently social species, with multiple focal brain regions sensitive to various visual social cues such as faces, bodies, and biological motion. More recently, research has begun to investigate how the brain responds to more complex, naturalistic social scenes, identifying a region in the posterior superior temporal sulcus (SI-pSTS; i.e., social interaction pSTS), amongst others, as an important region for processing social interaction. This research, however, has presented images or videos, and thus the contribution of motion to social interaction perception in these brain regions is not yet understood. In the current study, 22 participants viewed videos, image sequences, scrambled image sequences and static images of either social interactions or non-social independent actions. Combining univariate and multivariate analyses, we confirm that bilateral SI-pSTS plays a central role in dynamic social interaction perception but is much less involved when 'interactiveness' is conveyed solely with static cues. Regions in the social brain, including SI-pSTS and extrastriate body area (EBA), showed sensitivity to both motion and interactive content. While SI-pSTS is somewhat more tuned to video interactions than is EBA, both bilateral SI-pSTS and EBA showed a greater response to social interactions compared to non-interactions and both regions responded more strongly to videos than static images. Indeed, both regions showed higher responses to interactions than independent actions in videos and intact sequences, but not in other conditions. Exploratory multivariate regression analyses suggest that selectivity for simple visual motion does not in itself drive interactive sensitivity in either SI-pSTS or EBA. Rather, selectivity for interactions expressed in point-light animations, and selectivity for static images of bodies, make positive and independent contributions to this effect across the LOTC region. Our results strongly suggest that EBA and SI-pSTS work together during dynamic interaction perception, at least when interactive information is conveyed primarily via body information. As such, our results are also in line with proposals of a third visual stream supporting dynamic social scene perception.

Examining the value of body gestures in social reward contexts.

Brain regions associated with the processing of tangible rewards (such as money, food, or sex) are also involved in anticipating social rewards and avoiding social punishment. To date, studies investigating the neural underpinnings of social reward have presented feedback via static or dynamic displays of faces to participants. However, research demonstrates that participants find another type of social stimulus, namely, biological motion, rewarding as well, and exert effort to engage with this type of stimulus. Here we examine whether feedback presented via body gestures in the absence of facial cues also acts as a rewarding stimulus and recruits reward-related brain regions. To achieve this, we investigated the neural underpinnings of anticipating social reward and avoiding social disapproval presented via gestures alone, using a social incentive delay task. As predicted, the anticipation of social reward and avoidance of social disapproval engaged reward-related brain regions, including the nucleus accumbens, in a manner similar to previous studies' reports of feedback presented via faces and money. This study provides the first evidence that human body motion alone engages brain regions associated with reward processing in a similar manner to other social (i.e. faces) and non-social (i.e. money) rewards. The findings advance our understanding of social motivation in human perception and behavior.

Neural representations of haptic object size in the human brain revealed by multivoxel fMRI patterns.

The brain must interpret sensory input from diverse receptor systems to estimate object properties. Much has been learned about the brain mechanisms behind these processes in vision, but our understanding of haptic perception remains less clear. Here we examined haptic judgments of object size, which require integrating multiple cutaneous and proprioceptive afferent signals, as a model problem. To identify candidate human brain regions that support this process, participants (n = 16) in an event-related functional MRI experiment grasped objects to categorize them as one of four sizes. Object sizes were calibrated psychophysically to be equally distinct for each participant. We applied representational similarity logic to whole brain, multivoxel searchlight analyses to identify brain regions that exhibit size-relevant voxelwise activity patterns. Of particular interest was to identify regions for which more similar sizes produce more similar patterns of activity, which constitutes evidence of a metric size code. Regions of the intraparietal sulcus and the lateral prefrontal cortex met this criterion, both within hands and across hands. We suggest that these regions compute representations of haptic size that abstract over the specific peripheral afferent signals generated in a grasp. Results of a matched visual size task, performed by the same participants and analyzed in the same fashion, identified similar regions, indicating that these representations may be partly modality general. We consider these results with respect to perspectives on magnitude estimation in general and to computational views on perceptual signal integration.NEW & NOTEWORTHY Our understanding of the neural basis of haptics (perceiving the world through touch) remains incomplete. We used functional MRI to study human haptic judgments of object size, which require integrating multiple afferent signals. Multivoxel pattern analyses identified intraparietal and prefrontal regions that encode size haptically in a metric and hand-invariant fashion. Effector-independent haptic size estimates are useful on their own and in combination with other sensory estimates for a variety of perceptual and motor tasks.

Perceiving emotion and sex from the body: evidence from the Garner task for independent processes.

The appearance of the body signals socially relevant states and traits, but the how these cues are perceived is not well understood. Here we examined judgments of emotion and sex from the body's appearance. Understanding how we extract these cues is important because they are both salient and socially relevant. Participants viewed body images and either reported the emotion expressed by each body while ignoring its sex, or else reported the sex while ignoring its emotion. Following Garner's logic, two types of blocks were compared. In control blocks, the task-irrelevant dimension was fixed (e.g. all male in an emotion judgment task), whereas in orthogonal blocks it varied orthogonally to the task-relevant dimension (e.g. male-female). Where two dimensions draw on shared processes, interference results in relatively slower responses during orthogonal blocks. In contrast, a finding of no Garner interference - efficient selection of the task-relevant dimension - is taken to reflect independent processes. Bayesian analyses revealed evidence of no Garner interference between sex and emotion judgments, showing that extraction of these distinct signals from the body's appearance proceeds along largely parallel processing streams. These findings are informative about the mental architecture behind our perception of socially relevant characteristics of other people.

Evidence for Integrated Visual Face and Body Representations in the Anterior Temporal Lobes.

Research on visual face perception has revealed a region in the ventral anterior temporal lobes, often referred to as the anterior temporal face patch (ATFP), which responds strongly to images of faces. To date, the selectivity of the ATFP has been examined by contrasting responses to faces against a small selection of categories. Here, we assess the selectivity of the ATFP in humans with a broad range of visual control stimuli to provide a stronger test of face selectivity in this region. In Experiment 1, participants viewed images from 20 stimulus categories in an event-related fMRI design. Faces evoked more activity than all other 19 categories in the left ATFP. In the right ATFP, equally strong responses were observed for both faces and headless bodies. To pursue this unexpected finding, in Experiment 2, we used multivoxel pattern analysis to examine whether the strong response to face and body stimuli reflects a common coding of both classes or instead overlapping but distinct representations. On a voxel-by-voxel basis, face and whole-body responses were significantly positively correlated in the right ATFP, but face and body-part responses were not. This finding suggests that there is shared neural coding of faces and whole bodies in the right ATFP that does not extend to individual body parts. In contrast, the same approach revealed distinct face and body representations in the right fusiform gyrus. These results are indicative of an increasing convergence of distinct sources of person-related perceptual information proceeding from the posterior to the anterior temporal cortex.

A causal role for the extrastriate body area in detecting people in real-world scenes.

People are extremely efficient at detecting relevant objects in complex natural scenes. In three experiments, we used functional magnetic resonance imaging-guided transcranial magnetic stimulation (TMS) to investigate the role of the extrastriate body area (EBA) in the detection of people in scenes. In Experiment 1, participants reported, in different blocks, whether people or cars were present in a briefly presented scene. Detection (d-prime) of people, but not of cars, was impaired after TMS over right EBA (rEBA; five pulses at -200, -100, 0, 100, 200 ms) compared with sham stimulation. In Experiment 2, we applied TMS either before (-200, -100 ms) or after (+100, +200) the scene onset. Poststimulus EBA stimulation impaired people detection relative to prestimulus EBA stimulation, while timing had no effect during sham stimulation. In Experiment 3, we examined anatomical specificity by comparing TMS over EBA with TMS over scene-selective transverse occipital sulcus (TOS). Two scenes were presented side by side, and response times to detect which scene contained people (or cars) were measured. For people detection, but not for car detection, response times during EBA stimulation were significantly slower than during TOS stimulation. Furthermore, rEBA stimulation led to an equivalent slowing of response times to left and right lateralized targets. These findings are the first to demonstrate the causal involvement of a category-selective human brain region in detecting its preferred stimulus category in natural scenes. They shed light on the nature of such regions, and help us understand how we efficiently extract socially relevant information from a complex input.

A critical role for the hippocampus and perirhinal cortex in perceptual learning of scenes and faces: complementary findings from amnesia and FMRI.

It is debated whether subregions within the medial temporal lobe (MTL), in particular the hippocampus (HC) and perirhinal cortex (PrC), play domain-sensitive roles in learning. In the present study, two patients with differing degrees of MTL damage were first exposed to pairs of highly similar scenes, faces, and dot patterns and then asked to make repeated same/different decisions to preexposed and nonexposed (novel) pairs from the three categories (Experiment 1). We measured whether patients would show a benefit of prior exposure (preexposed > nonexposed) and whether repetition of nonexposed (and preexposed) pairs at test would benefit discrimination accuracy. Although selective HC damage impaired learning of scenes, but not faces and dot patterns, broader MTL damage involving the HC and PrC compromised discrimination learning of scenes and faces but left dot pattern learning unaffected. In Experiment 2, a similar task was run in healthy young participants in the MRI scanner. Functional region-of-interest analyses revealed that posterior HC and posterior parahippocampal gyrus showed greater activity during scene pattern learning, but not face and dot pattern learning, whereas PrC, anterior HC, and posterior fusiform gyrus were recruited during discrimination learning for faces, but not scenes and dot pattern learning. Critically, activity in posterior HC and PrC, but not the other functional region-of-interest analyses, was modulated by accuracy (correct > incorrect within a preferred category). Therefore, both approaches revealed a key role for the HC and PrC in discrimination learning, which is consistent with representational accounts in which subregions in these MTL structures store complex spatial and object representations, respectively.

Characterizing the discriminability of visual categorical information in strongly connected voxels.

Functional brain responses are strongly influenced by connectivity. Recently, we demonstrated a major example of this: category discriminability within occipitotemporal cortex (OTC) is enhanced for voxel sets that share strong functional connectivity to distal brain areas, relative to those that share lesser connectivity. That is, within OTC regions, sets of 'most-connected' voxels show improved multivoxel pattern discriminability for tool-, face-, and place stimuli relative to voxels with weaker connectivity to the wider brain. However, understanding whether these effects generalize to other domains (e.g. body perception network), and across different levels of the visual processing streams (e.g. dorsal as well as ventral stream areas) is an important extension of this work. Here, we show that this so-called connectivity-guided decoding (CGD) effect broadly generalizes across a wide range of categories (tools, faces, bodies, hands, places). This effect is robust across dorsal stream areas, but less consistent in earlier ventral stream areas. In the latter regions, category discriminability is generally very high, suggesting that extraction of category-relevant visual properties is less reliant on connectivity to downstream areas. Further, CGD effects are primarily expressed in a category-specific manner: For example, within the network of tool regions, discriminability of tool information is greater than non-tool information. The connectivity-guided decoding approach shown here provides a novel demonstration of the crucial relationship between wider brain connectivity and complex local-level functional responses at different levels of the visual processing streams. Further, this approach generates testable new hypotheses about the relationships between connectivity and local selectivity.

Converging evidence that left extrastriate body area supports visual sensitivity to social interactions.

Navigating our complex social world requires processing the interactions we observe. Recent psychophysical and neuroimaging studies provide parallel evidence that the human visual system may be attuned to efficiently perceive dyadic interactions. This work implies, but has not yet demonstrated, that activity in body-selective cortical regions causally supports efficient visual perception of interactions. We adopt a multi-method approach to close this important gap. First, using a large fMRI dataset (n = 92), we found that the left hemisphere extrastriate body area (EBA) responds more to face-to-face than non-facing dyads. Second, we replicated a behavioral marker of visual sensitivity to interactions: categorization of facing dyads is more impaired by inversion than non-facing dyads. Third, in a pre-registered experiment, we used fMRI-guided transcranial magnetic stimulation to show that online stimulation of the left EBA, but not a nearby control region, abolishes this selective inversion effect. Activity in left EBA, thus, causally supports the efficient perception of social interactions.

Facilitation and interference in spatial and body reference frames.

The observation of someone else's action facilitates similar actions in the observer. Such priming effects can be driven by alignment between the observer and the observed in body-centred or spatial coordinates (or both). The separate and joint contributions of these sources of priming remain to be fully characterised. Here, we compare spatial and body priming effects across the whole body "space", by using hand and foot responses. This allows a clearer separation of body priming from spatial priming than available from previous studies. In addition, we demonstrate two further features of these action priming effects. First, there are general interference and facilitation effects when the layout of viewed displays matches the participant's body (e.g. hand above the foot). These effects have not been considered in previous studies. Second, by taking these layout effects into account, we identify the facilitation and interference components of spatial and body priming effects. Both types of priming effect are observed, and facilitation and interference effects are only observed when both body and spatial frames of reference are working in the same direction. These findings show that in action perception, the behaviours of others are processed simultaneously in multiple frames of reference that have complex, interacting effects--both facilitating and interfering--on the motor system of the observer.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data.

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Asymmetric visual representation of sex from facial appearance.

We efficiently infer others' traits from their faces, and these inferences powerfully shape our social behaviour. Here, we investigated how sex is represented in facial appearance. Based on previous findings from sex-judgment tasks, we hypothesized that the perceptual encoding of sex is not balanced but rather polarized: for the processes that generate a sex percept, the default output is "male," and the representation of female faces extends that of the male, engaging activity over unique detectors that are not activated by male faces. We tested this hypothesis with the logic of Treisman's studies of visual search asymmetries, predicting that observers should more readily detect the presence of female faces amongst male distractors than vice versa. Across three experiments (N = 32 each), each using different face stimuli, we confirmed this prediction in response time and sensitivity measures. We apply GIST analyses to the face stimuli to exclude that the search asymmetry is explained by differences in image homogeneity. These findings demonstrate a property of the coding that links facial appearance with a significant social trait: the female face is coded as an extension of a male default. We offer a mechanistic description of perceptual detectors to account for our findings and posit that the origins of this polarized coding scheme are an outcome of biased early developmental experience.

Viewpoint (in)dependence of action representations: an MVPA study.

The discovery of mirror neurons-neurons that code specific actions both when executed and observed-in area F5 of the macaque provides a potential neural mechanism underlying action understanding. To date, neuroimaging evidence for similar coding of specific actions across the visual and motor modalities in human ventral premotor cortex (PMv)-the putative homologue of macaque F5-is limited to the case of actions observed from a first-person perspective. However, it is the third-person perspective that figures centrally in our understanding of the actions and intentions of others. To address this gap in the literature, we scanned participants with fMRI while they viewed two actions from either a first- or third-person perspective during some trials and executed the same actions during other trials. Using multivoxel pattern analysis, we found action-specific cross-modal visual-motor representations in PMv for the first-person but not for the third-person perspective. Additional analyses showed no evidence for spatial or attentional differences across the two perspective conditions. In contrast, more posterior areas in the parietal and occipitotemporal cortex did show cross-modal coding regardless of perspective. These findings point to a stronger role for these latter regions, relative to PMv, in supporting the understanding of others' actions with reference to one's own actions.

Visuo-motor imagery of specific manual actions: a multi-variate pattern analysis fMRI study.

An important human capacity is the ability to imagine performing an action, and its consequences, without actually executing it. Here we seek neural representations of specific manual actions that are common across visuo-motor performance and imagery. Participants were scanned with fMRI while they performed and observed themselves performing two different manual actions during some trials, and imagined performing and observing themselves performing the same actions during other trials. We used multi-variate pattern analysis to identify areas where representations of specific actions generalize across imagined and performed actions. The left anterior parietal cortex showed this property. In this region, we also found that activity patterns for imagined actions generalize better to performed actions than vice versa, and we provide simulation results that can explain this asymmetry. The present results are the first demonstration of action-specific representations that are similar irrespective of whether actions are actively performed or covertly imagined. Further, they demonstrate concretely how the apparent cross-modal visuo-motor coding of actions identified in studies of a human "mirror neuron system" could, at least partially, reflect imagery.

Mapping brain activation and information during category-specific visual working memory.

How is working memory for different visual categories supported in the brain? Do the same principles of cortical specialization that govern the initial processing and encoding of visual stimuli also apply to their short-term maintenance? We investigated these questions with a delayed discrimination paradigm for faces, bodies, flowers, and scenes and applied both univariate and multivariate analyses to functional magnetic resonance imaging (fMRI) data. Activity during encoding followed the well-known specialization in posterior areas. During the delay interval, activity shifted to frontal and parietal regions but was not specialized for category. Conversely, activity in visual areas returned to baseline during that interval but showed some evidence of category specialization on multivariate pattern analysis (MVPA). We conclude that principles of cortical activation differ between encoding and maintenance of visual material. Whereas perceptual processes rely on specialized regions in occipitotemporal cortex, maintenance involves the activation of a frontoparietal network that seems to require little specialization at the category level. We also confirm previous findings that MVPA can extract information from fMRI signals in the absence of suprathreshold activation and that such signals from visual areas can reflect the material stored in memory.

Representation of action in occipito-temporal cortex.

A fundamental question for social cognitive neuroscience is how and where in the brain the identities and actions of others are represented. Here we present a replication and extension of a study by Kable and Chatterjee [Kable, J. W., & Chatterjee, A. Specificity of action representations in the lateral occipito-temporal cortex. Journal of Cognitive Neuroscience, 18, 1498-1517, 2006] examining the role of occipito-temporal cortex in these processes. We presented full-cue movies of actors performing whole-body actions and used fMRI to test for action- and identity-specific adaptation effects. We examined a series of functionally defined regions, including the extrastriate and fusiform body areas, the fusiform face area, the parahippocampal place area, the lateral occipital complex, the right posterior superior temporal sulcus, and motion-selective area hMT+. These regions were analyzed with both standard univariate measures as well as multivoxel pattern analyses. Additionally, we performed whole-brain tests for significant adaptation effects. We found significant action-specific adaptation in many areas, but no evidence for identity-specific adaptation. We argue that this finding could be explained by differences in the familiarity of the stimuli presented: The actions shown were familiar but the actors performing the actions were unfamiliar. However, in contrast to previous findings, we found that the action adaptation effect could not be conclusively tied to specific functionally defined regions. Instead, our results suggest that the adaptation to previously seen actions across identities is a widespread effect, evident across lateral and ventral occipito-temporal cortex.

Division of labor between lateral and ventral extrastriate representations of faces, bodies, and objects.

The occipito-temporal cortex is strongly implicated in carrying out the high-level computations associated with vision. In human neuroimaging studies, focal regions are consistently found within this broad region that respond strongly and selectively to faces, bodies, or objects. A notable feature of these selective regions is that they are found in pairs. In the posterior-lateral occipito-temporal cortex, focal selectivity is found for faces (occipital face area), bodies (extrastriate body area), and objects (lateral occipital). These three areas are found bilaterally and at close quarters to each other. Likewise, in the ventro-medial occipito-temporal cortex, three similar category-selective regions are found, also in proximity to each other: for faces (fusiform face area), bodies (fusiform body area), and objects (posterior fusiform). Here we review some of the extensive evidence on the functional properties of these areas with two aims. First, we seek to identify principles that distinguish the posterior-lateral and ventro-medial clusters of selective regions but that apply generally within each cluster across the three stimulus kinds. Our review identifies and elaborates several principles by which these relationships hold. In brief, the posterior-lateral representations are more primitive, local, and stimulus-driven relative to the ventro-medial representations, which in contrast are more invariant to visual features, global, and linked to the subjective percept. Second, because the evidence base of studies that compare both posterior-lateral and ventro-medial representations of faces, bodies, and objects is still relatively small, we seek to provoke more cross-talk among the research strands that are traditionally separate. We identify several promising approaches for such future work.

A comparison of volume-based and surface-based multi-voxel pattern analysis.

For functional magnetic resonance imaging (fMRI), multi-voxel pattern analysis (MVPA) has been shown to be a sensitive method to detect areas that encode certain stimulus dimensions. By moving a searchlight through the volume of the brain, one can continuously map the information content about the experimental conditions of interest to the brain. Traditionally, the searchlight is defined as a volume sphere that does not take into account the anatomy of the cortical surface. Here we present a method that uses a cortical surface reconstruction to guide voxel selection for information mapping. This approach differs in two important aspects from a volume-based searchlight definition. First, it uses only voxels that are classified as grey matter based on an anatomical scan. Second, it uses a surface-based geodesic distance metric to define neighbourhoods of voxels, and does not select voxels across a sulcus. We study here the influence of these two factors onto classification accuracy and onto the spatial specificity of the resulting information map. In our example data set, participants pressed one of four fingers while undergoing fMRI. We used MVPA to identify regions in which local fMRI patterns can successfully discriminate which finger was moved. We show that surface-based information mapping is a more sensitive measure of local information content, and provides better spatial selectivity. This makes surface-based information mapping a useful technique for a data-driven analysis of information representation in the cerebral cortex.

Learning associations between action and perception: effects of incompatible training on body part and spatial priming.

Observation of another person executing an action primes the same action in the observer's motor system. Recent evidence has shown that these priming effects are flexible, where training of new associations, such as making a foot response when viewing a moving hand, can reduce standard action priming effects (Gillmeister, Catmur, Liepelt, Brass, & Heyes, 2008). Previously, these effects were obtained after explicit learning tasks in which the trained action was cued by the content of a visual stimulus. Here we report similar learning processes in an implicit task in which the participant's action is self-selected, and subsequent visual effects are determined by the nature of that action. Importantly, we show that these learning processes are specific to associations between actions and viewed body parts, in that incompatible spatial training did not influence body part or spatial priming effects. Our results are consistent with models of visuomotor learning that place particular emphasis on the repeated experience of watching oneself perform an action.

Patterns of fMRI activity dissociate overlapping functional brain areas that respond to biological motion.

Accurate perception of the actions and intentions of other people is essential for successful interactions in a social environment. Several cortical areas that support this process respond selectively in fMRI to static and dynamic displays of human bodies and faces. Here we apply pattern-analysis techniques to arrive at a new understanding of the neural response to biological motion. Functionally defined body-, face-, and motion-selective visual areas all responded significantly to "point-light" human motion. Strikingly, however, only body selectivity was correlated, on a voxel-by-voxel basis, with biological motion selectivity. We conclude that (1) biological motion, through the process of structure-from-motion, engages areas involved in the analysis of the static human form; (2) body-selective regions in posterior fusiform gyrus and posterior inferior temporal sulcus overlap with, but are distinct from, face- and motion-selective regions; (3) the interpretation of region-of-interest findings may be substantially altered when multiple patterns of selectivity are considered.