Multidimensional neural representations of social features during movie viewing.

The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others and talking about objects) were represented in the superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in the frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as key regions for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus.

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others' social interactions. However, despite the prominence of social interactions in memory, the neural basis of remembering social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others' social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that a subset of the STS activity evoked by viewing social interactions predicted neural responses in not only held-out movie data, but also during memory recall. These results provide the first evidence that activity in the STS is reinstated in response to specific social content and that its reactivation underlies our ability to remember others' interactions. These findings further suggest that the STS contains representations of social interactions that are not only perceptually driven, but also more abstract or conceptual in nature.

Rapid Processing of Observed Touch through Social Perceptual Brain Regions: An EEG-fMRI Fusion Study.

Seeing social touch triggers a strong social-affective response that involves multiple brain networks, including visual, social perceptual, and somatosensory systems. Previous studies have identified the specific functional role of each system, but little is known about the speed and directionality of the information flow. Is this information extracted via the social perceptual system or from simulation from somatosensory cortex? To address this, we examined the spatiotemporal neural processing of observed touch. Twenty-one human participants (seven males) watched 500-ms video clips showing social and nonsocial touch during electroencephalogram (EEG) recording. Visual and social-affective features were rapidly extracted in the brain, beginning at 90 and 150 ms after video onset, respectively. Combining the EEG data with functional magnetic resonance imaging (fMRI) data from our prior study with the same stimuli reveals that neural information first arises in early visual cortex (EVC), then in the temporoparietal junction and posterior superior temporal sulcus (TPJ/pSTS), and finally in the somatosensory cortex. EVC and TPJ/pSTS uniquely explain EEG neural patterns, while somatosensory cortex does not contribute to EEG patterns alone, suggesting that social-affective information may flow from TPJ/pSTS to somatosensory cortex. Together, these findings show that social touch is processed quickly, within the timeframe of feedforward visual processes, and that the social-affective meaning of touch is first extracted by a social perceptual pathway. Such rapid processing of social touch may be vital to its effective use during social interaction.SIGNIFICANCE STATEMENT Seeing physical contact between people evokes a strong social-emotional response. Previous research has identified the brain systems responsible for this response, but little is known about how quickly and in what direction the information flows. We demonstrated that the brain processes the social-emotional meaning of observed touch quickly, starting as early as 150 ms after the stimulus onset. By combining electroencephalogram (EEG) data with functional magnetic resonance imaging (fMRI) data, we show for the first time that the social-affective meaning of touch is first extracted by a social perceptual pathway and followed by the later involvement of somatosensory simulation. This rapid processing of touch through the social perceptual route may play a pivotal role in effective usage of touch in social communication and interaction.

Word-timestamped transcripts of two spoken narrative recall functional neuroimaging datasets.

After watching audiovisual movies, human participants produced spoken narrative recollections during functional magnetic resonance imaging (fMRI); presented here are word-level timestamps of their speech, temporally aligned to the publicly shared fMRI data. For the "FilmFestival" dataset, twenty participants watched ten short audiovisual movies, approximately 2-8 minutes each. For the "Sherlock" dataset, seventeen participants watched the first half of the first episode of BBC's Sherlock (48 minutes). After viewing, participants then verbally described what they remembered about the movies in their own words. Participants' speech was recorded using an MR-compatible microphone. The audio recordings were transcribed, then timestamped by a forced aligner; missing timestamps were filled in manually by human transcriptionists referencing the audio recording. Each file contains the participant's recall word by word, onset of each word in seconds with 1/10th-second precision, and the corresponding fMRI volume number (TR). This dataset can be used to investigate topics such as naturalistic memory and language production.

The unreliable influence of multivariate noise normalization on the reliability of neural dissimilarity.

Representational similarity analysis (RSA) is a key element in the multivariate pattern analysis toolkit. The central construct of the method is the representational dissimilarity matrix (RDM), which can be generated for datasets from different modalities (neuroimaging, behavior, and computational models) and directly correlated in order to evaluate their second-order similarity. Given the inherent noisiness of neuroimaging signals it is important to evaluate the reliability of neuroimaging RDMs in order to determine whether these comparisons are meaningful. Recently, multivariate noise normalization (NNM) has been proposed as a widely applicable method for boosting signal estimates for RSA, regardless of choice of dissimilarity metrics, based on evidence that the analysis improves the within-subject reliability of RDMs (Guggenmos et al. 2018; Walther et al. 2016). We revisited this issue with three fMRI datasets and evaluated the impact of NNM on within- and between-subject reliability and RSA effect sizes using multiple dissimilarity metrics. We also assessed its impact across regions of interest from the same dataset, its interaction with spatial smoothing, and compared it to GLMdenoise, which has also been proposed as a method that improves signal estimates for RSA (Charest et al. 2018). We found that across these tests the impact of NNM was highly variable, as also seems to be the case for other analysis choices. Overall, we suggest being conservative before adding steps and complexities to the (pre)processing pipeline for RSA.

Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing.

Recognizing others' social interactions is a crucial human ability. Using simple stimuli, previous studies have shown that social interactions are selectively processed in the superior temporal sulcus (STS), but prior work with movies has suggested that social interactions are processed in the medial prefrontal cortex (mPFC), part of the theory of mind network. It remains unknown to what extent social interaction selectivity is observed in real world stimuli when controlling for other covarying perceptual and social information, such as faces, voices, and theory of mind. The current study utilizes a functional magnetic resonance imaging (fMRI) movie paradigm and advanced machine learning methods to uncover the brain mechanisms uniquely underlying naturalistic social interaction perception. We analyzed two publicly available fMRI datasets, collected while both male and female human participants (n = 17 and 18) watched two different commercial movies in the MRI scanner. By performing voxel-wise encoding and variance partitioning analyses, we found that broad social-affective features predict neural responses in social brain regions, including the STS and mPFC. However, only the STS showed robust and unique selectivity specifically to social interactions, independent from other covarying features. This selectivity was observed across two separate fMRI datasets. These findings suggest that naturalistic social interaction perception recruits dedicated neural circuity in the STS, separate from the theory of mind network, and is a critical dimension of human social understanding.

Reduced task-dependent modulation of functional network architecture for positive versus negative affective touch processing in autism spectrum disorders.

Individuals with autism spectrum disorders (ASD) experience impairments in social communication and interaction, and often show difficulties with receiving and offering touch. Despite the high prevalence of abnormal reactions to touch in ASD, and the importance of touch communication in human relationships, the neural mechanisms underlying atypical touch processing in ASD remain largely unknown. To answer this question, we provided both pleasant and unpleasant touch stimulation to male adults with and without ASD during functional neuroimaging. By employing generalized psychophysiological interaction analysis combined with an independent component analysis approach, we characterize stimulus-dependent changes in functional connectivity patterns for processing two tactile stimuli that evoke different emotions (i.e., pleasant vs. unpleasant touch). Results reveal that neurotypical male adults showed extensive stimulus-sensitive modulations of the functional network architecture in response to the different types of touch, both at the level of brain regions and large-scale networks. Conversely, far fewer stimulus-sensitive modulations were observed in the ASD group. These aberrant functional connectivity profiles in the ASD group were marked by hypo-connectivity of the parietal operculum and major pain networks and hyper-connectivity between the semantic and limbic networks. Lastly, individuals presenting more social deficits and a more negative attitude towards social touch showed greater hyper-connectivity between the limbic and semantic networks. These findings suggest that reduced stimulus-related modulation of this functional network architecture is associated with abnormal processing of touch in ASD.

Task-dependent changes in functional connectivity during the observation of social and non-social touch interaction.

Previous studies have identified a collection of brain areas that show neural selectivity for the distinction between human-to-human and human-to-object interactions, including regions implicated in sensory and social processing. It remains largely unknown, however, how the functional communication between these areas changes with the type of interaction. Combining a generalized psychophysiological interaction (gPPI) analysis and independent component analysis (ICA), the current study sought to identify the context-sensitive modulation of the functional network architecture during touch observation. Thirty-seven participants watched 75 video clips displaying social and non-social touch events during a functional imaging scan. A gPPI analysis of pre-defined regions of interest revealed that social-cognitive brain regions show enhanced interregional coupling during social touch observation, both among social-cognitive brain regions and between social-cognitive regions and sensory regions. Conversely, during non-social touch observation, a significantly stronger coupling among brain areas within the system that processes the unimodal sensory information was observed. At the level of large-scale brain networks extracted with ICA, stronger connectivity between 11 pairs of networks, including default mode networks, was observed during social touch observation, while only three pairs of networks showed stronger connectivity during non-social touch observation. The current study identifies the presence of context-dependent changes in functional brain architecture based on whether the touch recipient is a person or an object, highlighting an increased exchange of neural information for social processing.

Intact neural representations of affective meaning of touch but lack of embodied resonance in autism: a multi-voxel pattern analysis study.

Background: Humans can easily grasp the affective meaning of touch when observing social interactions. Several neural systems support this ability, including the theory of mind (ToM) network and the somatosensory system linked to embodied resonance, but it is unclear how these systems are affected in autism spectrum disorder (ASD). Individuals with ASD exhibit impairments in the use of nonverbal communication such as social and reciprocal touch. Despite the importance of touch in social communication and the reported touch aversion in ASD, surprisingly little is known about the neural systems underlying impairments in touch communication in ASD. Methods: The present study applies a dynamic and socially meaningful stimulus set combined with functional magnetic resonance imaging (fMRI) to pinpoint atypicalities in the neural circuitry underlying socio-affective touch observation in adults with ASD. Twenty-one adults with ASD and 21 matched neurotypical adults evaluated the valence and arousal of 75 video fragments displaying touch interactions. Subsequently, they underwent fMRI while watching the same videos. Using multi-voxel pattern analysis (MVPA) and multiple regression analysis, we examined which brain regions represent the socio-affective meaning of observed touch. To further understand the brain-behavior relationship, we correlated the strength of affective representations in the somatosensory cortex with individuals' attitude towards social touch in general and with a quantitative index of autism traits as measured by the Social Responsiveness Scale. Results: Results revealed that the affective meaning of touch was well represented in the temporoparietal junction, a core mentalizing area, in both groups. Conversely, only the neurotypical group represented affective touch in the somatosensory cortex, a region involved in self-experienced touch. Lastly, irrespective of the group, individuals with a more positive attitude towards receiving, witnessing, and providing social touch and with a higher score on social responsivity showed more differentiated representations of the affective meaning of touch in these somatosensory areas. Conclusions: Together, our findings imply that male adults with ASD show intact cognitive understanding (i.e., "knowing") of observed socio-affective touch interactions, but lack of spontaneous embodied resonance (i.e., "feeling").

Distinct Activity Patterns of the Human Bed Nucleus of the Stria Terminalis and Amygdala during Fear Learning.

The amygdala and, more recently, also the bed nucleus of the stria terminalis, have been widely implicated in fear and anxiety. Much of our current knowledge is derived from animal studies and suggests an intricate convergence and divergence in functions related to defensive responding. In a recent paper, Klumpers and colleagues set out to examine these functions in a human fear learning procedure using functional magnetic resonance imaging. Their main findings were a role for the bed nucleus of the stria terminalis in threat anticipation, and for the amygdala in threat confrontation. Here, we provide a critical summary of this interesting study and point out some important issues that were not addressed by its authors. In particular, we first take a closer look at the striking differences between both samples that were combined for the study, and, secondly, we provide an in-depth discussion of their findings in relation to existing neurobehavioral models.

A Comparison of Functional Networks Derived From Representational Similarity, Functional Connectivity, and Univariate Analyses.

The invention of representational similarity analysis [RSA, following multi-voxel pattern analysis (MVPA)] has allowed cognitive neuroscientists to identify the representational structure of multiple brain regions, moving beyond functional localization. By comparing these structures, cognitive neuroscientists can characterize how brain areas form functional networks. Univariate analysis (UNIVAR) and functional connectivity analysis (FCA) are two other popular methods to identify functional networks. Despite their popularity, few studies have examined the relationship between networks from RSA with those from UNIVAR and FCA. Thus, the aim of the current study is to examine the similarities between neural networks derived from RSA with those from UNIVAR and FCA to explore how these methods relate to each other. We analyzed the data of a previously published study with the three methods and compared the results by performing (partial) correlation and multiple regression analysis. Our findings reveal that neural networks resulting from RSA, UNIVAR, and FCA methods are highly similar to each other even after ruling out the effect of anatomical proximity between the network nodes. Nevertheless, the neural network from each method shows unique organization that cannot be explained by any of the other methods. Thus, we conclude that the RSA, UNIVAR and FCA methods provide similar but not identical information on how brain regions are organized in functional networks.

A Multitude of Neural Representations Behind Multisensory "Social Norm" Processing.

Humans show a unique capacity to process complex information from multiple sources. Social perception in natural environment provides a good example of such capacity as it typically requires the integration of information from different sensory systems, and also from different levels of sensory processing. Here, instead of studying one isolate system and level of representation, we focused upon a neuroimaging paradigm which allows to capture multiple brain representations simultaneously, i.e., low and high-level processing in two different sensory systems, as well as abstract cognitive processing of congruency. Subjects performed social decisions based on the congruency between auditory and visual processing. Using multivoxel pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data, we probed a wide variety of representations. Our results confirmed the expected representations at each level and system according to the literature. Further, beyond the hierarchical organization of the visual, auditory and higher order neural systems, we provide a more nuanced picture of the brain functional architecture. Indeed, brain regions of the same neural system show similarity in their representations, but they also share information with regions from other systems. Further, the strength of neural information varied considerably across domains in a way that was not obviously related to task relevance. For instance, selectivity for task-irrelevant animacy of visual input was very strong. The present approach represents a new way to explore the richness of co-activated brain representations underlying the natural complexity in human cognition.

The multidimensional representational space of observed socio-affective touch experiences.

Observed touch interactions provide useful information on how others communicate with the external world. Previous studies revealed shared neural circuits between the direct experience and the passive observation of simple touch, such as being stroked/slapped. Here, we investigate the complexity of the neural representations underlying the understanding of others' socio-affective touch interactions. Importantly, we use a recently developed touch database that contains a larger range of more complex social and non-social touch interactions. Participants judged affective aspects of each touch event and were scanned while watching the same videos. Using correlational multivariate pattern analysis methods, we obtained neural similarity matrices in 18 regions of interest from five different networks: somatosensory, pain, the theory of mind, visual and motor regions. Among them, four networks except motor cortex represent the social nature of the touch, whereas fine-detailed affective information is reflected in more targeted areas such as social brain regions and somatosensory cortex. Lastly, individual social touch preference at the behavioral level was correlated with the involvement of somatosensory areas on representing affective information, suggesting that individuals with higher social touch preference exhibit stronger vicarious emotional responses to others' social touch experiences. Together, these results highlight the overall complexity and the individual modulation of the distributed neural representations underlying the processing of observed socio-affective touch.

Socio-affective touch expression database.

Socio-affective touch communication conveys a vast amount of information about emotions and intentions in social contexts. In spite of the complexity of the socio-affective touch expressions we use daily, previous studies addressed only a few aspects of social touch mainly focusing on hedonics, such as stroking, leaving a wide range of social touch behaviour unexplored. To overcome this limit, we present the Socio-Affective Touch Expression Database (SATED), which includes a large range of dynamic interpersonal socio-affective touch events varying in valence and arousal. The original database contained 26 different social touch expressions each performed by three actor pairs. To validate each touch expression, we conducted two behavioural experiments investigating perceived naturalness and affective values. Based on the rated naturalness and valence, 13 socio-affective touch expressions along with 12 corresponding non-social touch events were selected as a complete set, achieving 75 video clips in total. Moreover, we quantified motion energy for each touch expression to investigate its intrinsic correlations with perceived affective values and its similarity among actor- and action-pairs. As a result, the touch expression database is not only systematically defined and well-controlled, but also spontaneous and natural, while eliciting clear affective responses. This database will allow a fine-grained investigation of complex interpersonal socio-affective touch in the realm of social psychology and neuroscience along with potential application areas in affective computing and neighbouring fields.

Neuroanatomical correlates of haptic object processing: combined evidence from tractography and functional neuroimaging.

Touch delivers a wealth of information already from birth, helping infants to acquire knowledge about a variety of important object properties using their hands. Despite the fact that we are touch experts as much as we are visual experts, surprisingly, little is known how our perceptual ability in touch is linked to either functional or structural aspects of the brain. The present study, therefore, investigates and identifies neuroanatomical correlates of haptic perceptual performance using a novel, multi-modal approach. For this, participants' performance in a difficult shape categorization task was first measured in the haptic domain. Using a multi-modal functional magnetic resonance imaging and diffusion-weighted magnetic resonance imaging analysis pipeline, functionally defined and anatomically constrained white-matter pathways were extracted and their microstructural characteristics correlated with individual variability in haptic categorization performance. Controlling for the effects of age, total intracranial volume and head movements in the regression model, haptic performance was found to correlate significantly with higher axial diffusivity in functionally defined superior longitudinal fasciculus (fSLF) linking frontal and parietal areas. These results were further localized in specific sub-parts of fSLF. Using additional data from a second group of participants, who first learned the categories in the visual domain and then transferred to the haptic domain, haptic performance correlates were obtained in the functionally defined inferior longitudinal fasciculus. Our results implicate SLF linking frontal and parietal areas as an important white-matter track in processing touch-specific information during object processing, whereas ILF relays visually learned information during haptic processing. Taken together, the present results chart for the first time potential neuroanatomical correlates and interactions of touch-related object processing.

"Can touch this": Cross-modal shape categorization performance is associated with microstructural characteristics of white matter association pathways.

Previous studies on visuo-haptic shape processing provide evidence that visually learned shape information can transfer to the haptic domain. In particular, recent neuroimaging studies have shown that visually learned novel objects that were haptically tested recruited parts of the ventral pathway from early visual cortex to the temporal lobe. Interestingly, in such tasks considerable individual variation in cross-modal transfer performance was observed. Here, we investigate whether this individual variation may be reflected in microstructural characteristics of white-matter (WM) pathways. We first trained participants on a fine-grained categorization task of novel shapes in the visual domain, followed by a haptic categorization test. We then correlated visual training-performance and haptic test-performance, as well as performance on a symbol-coding task requiring visuo-motor dexterity with microstructural properties of WM bundles potentially involved in visuo-haptic processing (the inferior longitudinal fasciculus [ILF], the fronto-temporal part of the superior longitudinal fasciculus [SLFft ] and the vertical occipital fasciculus [VOF]). Behavioral results showed that haptic categorization performance was good on average but exhibited large inter-individual variability. Haptic performance also was correlated with performance in the symbol-coding task. WM analyses showed that fast visual learners exhibited higher fractional anisotropy (FA) in left SLFft and left VOF. Importantly, haptic test-performance (and symbol-coding performance) correlated with FA in ILF and with axial diffusivity in SLFft . These findings provide clear evidence that individual variation in visuo-haptic performance can be linked to microstructural characteristics of WM pathways. Hum Brain Mapp 38:842-854, 2017. © 2016 Wiley Periodicals, Inc.

Visual and Haptic Shape Processing in the Human Brain: Unisensory Processing, Multisensory Convergence, and Top-Down Influences.

Humans are highly adept at multisensory processing of object shape in both vision and touch. Previous studies have mostly focused on where visually perceived object-shape information can be decoded, with haptic shape processing receiving less attention. Here, we investigate visuo-haptic shape processing in the human brain using multivoxel correlation analyses. Importantly, we use tangible, parametrically defined novel objects as stimuli. Two groups of participants first performed either a visual or haptic similarity-judgment task. The resulting perceptual object-shape spaces were highly similar and matched the physical parameter space. In a subsequent fMRI experiment, objects were first compared within the learned modality and then in the other modality in a one-back task. When correlating neural similarity spaces with perceptual spaces, visually perceived shape was decoded well in the occipital lobe along with the ventral pathway, whereas haptically perceived shape information was mainly found in the parietal lobe, including frontal cortex. Interestingly, ventrolateral occipito-temporal cortex decoded shape in both modalities, highlighting this as an area capable of detailed visuo-haptic shape processing. Finally, we found haptic shape representations in early visual cortex (in the absence of visual input), when participants switched from visual to haptic exploration, suggesting top-down involvement of visual imagery on haptic shape processing.