Patterns of saliency and semantic features distinguish gaze of expert and novice viewers of surveillance footage.

When viewing the actions of others, we not only see patterns of body movements, but we also "see" the intentions and social relations of people. Experienced forensic examiners - Closed Circuit Television (CCTV) operators - have been shown to convey superior performance in identifying and predicting hostile intentions from surveillance footage than novices. However, it remains largely unknown what visual content CCTV operators actively attend to, and whether CCTV operators develop different strategies for active information seeking from what novices do. Here, we conducted computational analysis for the gaze-centered stimuli captured by experienced CCTV operators and novices' eye movements when viewing the same surveillance footage. Low-level image features were extracted by a visual saliency model, whereas object-level semantic features were extracted by a deep convolutional neural network (DCNN), AlexNet, from gaze-centered regions. We found that the looking behavior of CCTV operators differs from novices by actively attending to visual contents with different patterns of saliency and semantic features. Expertise in selectively utilizing informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions.

Major Depression and the Perception of Affective Instrumental and Expressive Gestures: An fMRI Investigation.

Major depressive disorder (MDD) is associated with biased perception of human movement. Gesture is important for communication and in this study we investigated neural correlates of gesture perception in MDD. We hypothesised different neural activity between individuals with MDD and typical individuals when viewing instrumental and expressive gestures that were negatively or positively valenced. Differences were expected in brain areas associated with gesture perception, including superior temporal, frontal, and emotion processing regions. We recruited 12 individuals with MDD and 12 typical controls matched on age, gender, and handedness. They viewed gestures displayed by stick figures while functional magnetic resonance imaging (fMRI) was performed. Results of a random effects three-way mixed ANOVA indicated that individuals with MDD had greater activity in the right claustrum compared to controls, regardless of gesture type or valence. Additionally, we observed main effects of gesture type and valence, regardless of group. Perceiving instrumental compared to expressive gestures was associated with greater activity in the left cuneus and left superior temporal gyrus, while perceiving negative compared to positive gestures was associated with greater activity in the right precuneus and right lingual gyrus. We also observed a two-way interaction between gesture type and valence in various brain regions.

A functional magnetic resonance imaging examination of audiovisual observation of a point-light string quartet using intersubject correlation and physical feature analysis.

We use functional Magnetic Resonance Imaging (fMRI) to explore synchronized neural responses between observers of audiovisual presentation of a string quartet performance during free viewing. Audio presentation was accompanied by visual presentation of the string quartet as stick figures observed from a static viewpoint. Brain data from 18 musical novices were obtained during audiovisual presentation of a 116 s performance of the allegro of String Quartet, No. 14 in D minor by Schubert played by the 'Quartetto di Cremona.' These data were analyzed using intersubject correlation (ISC). Results showed extensive ISC in auditory and visual areas as well as parietal cortex, frontal cortex and subcortical areas including the medial geniculate and basal ganglia (putamen). These results from a single fixed viewpoint of multiple musicians are greater than previous reports of ISC from unstructured group activity but are broadly consistent with related research that used ISC to explore listening to music or watching solo dance. A feature analysis examining the relationship between brain activity and physical features of the auditory and visual signals yielded findings of a large proportion of activity related to auditory and visual processing, particularly in the superior temporal gyrus (STG) as well as midbrain areas. Motor areas were also involved, potentially as a result of watching motion from the stick figure display of musicians in the string quartet. These results reveal involvement of areas such as the putamen in processing complex musical performance and highlight the potential of using brief naturalistic stimuli to localize distinct brain areas and elucidate potential mechanisms underlying multisensory integration.

Incongruence effects in cross-modal emotional processing in autistic traits: An fMRI study.

In everyday life, emotional information is often conveyed by both the face and the voice. Consequently, information presented by one source can alter the way in which information from the other source is perceived, leading to emotional incongruence. Here, we used functional magnetic resonance imaging (fMRI) to examine neutral correlates of two different types of emotional incongruence in audiovisual processing, namely incongruence of emotion-valence and incongruence of emotion-presence. Participants were in two groups, one group with a low Autism Quotient score (LAQ) and one with a high score (HAQ). Each participant experienced emotional (happy, fearful) or neutral faces or voices while concurrently being exposed to emotional (happy, fearful) or neutral voices or faces. They were instructed to attend to either the visual or auditory track. The incongruence effect of emotion-valence was characterized by activation in a wide range of brain regions in both hemispheres involving the inferior frontal gyrus, cuneus, superior temporal gyrus, and middle frontal gyrus. The incongruence effect of emotion-presence was characterized by activation in a set of temporal and occipital regions in both hemispheres, including the middle occipital gyrus, middle temporal gyrus and inferior temporal gyrus. In addition, the present study identified greater recruitment of the right inferior parietal lobule in perceiving audio-visual emotional expressions in HAQ individuals, as compared to the LAQ individuals. Depending on face or voice-to-be attended, different patterns of emotional incongruence were found between the two groups. Specifically, the HAQ group tend to show more incidental processing to visual information whilst the LAQ group tend to show more incidental processing to auditory information during the crossmodal emotional incongruence decoding. These differences might be attributed to different attentional demands and different processing strategies between the two groups.

Upregulation of Supplementary Motor Area Activation with fMRI Neurofeedback during Motor Imagery.

Functional magnetic resonance imaging (fMRI) neurofeedback (NF) is a promising tool to study the relationship between behavior and brain activity. It enables people to self-regulate their brain signal. Here, we applied fMRI NF to train healthy participants to increase activity in their supplementary motor area (SMA) during a motor imagery (MI) task of complex body movements while they received a continuous visual feedback signal. This signal represented the activity of participants' localized SMA regions in the NF group and a prerecorded signal in the control group (sham feedback). In the NF group only, results showed a gradual increase in SMA-related activity across runs. This upregulation was largely restricted to the SMA, while other regions of the motor network showed no, or only marginal NF effects. In addition, we found behavioral changes, i.e., shorter reaction times in a Go/No-go task after the NF training only. These results suggest that NF can assist participants to develop greater control over a specifically targeted motor region involved in motor skill learning. The results contribute to a better understanding of the underlying mechanisms of SMA NF based on MI with a direct implication for rehabilitation of motor dysfunctions.

Brain circuits signaling the absence of emotion in body language.

Adaptive social behavior and mental well-being depend on not only recognizing emotional expressions but also, inferring the absence of emotion. While the neurobiology underwriting the perception of emotions is well studied, the mechanisms for detecting a lack of emotional content in social signals remain largely unknown. Here, using cutting-edge analyses of effective brain connectivity, we uncover the brain networks differentiating neutral and emotional body language. The data indicate greater activation of the right amygdala and midline cerebellar vermis to nonemotional as opposed to emotional body language. Most important, the effective connectivity between the amygdala and insula predicts people's ability to recognize the absence of emotion. These conclusions extend substantially current concepts of emotion perception by suggesting engagement of limbic effective connectivity in recognizing the lack of emotion in body language reading. Furthermore, the outcome may advance the understanding of overly emotional interpretation of social signals in depression or schizophrenia by providing the missing link between body language reading and limbic pathways. The study thus opens an avenue for multidisciplinary research on social cognition and the underlying cerebrocerebellar networks, ranging from animal models to patients with neuropsychiatric conditions.

Hyperalignment of motor cortical areas based on motor imagery during action observation.

Multivariate Pattern Analysis (MVPA) has grown in importance due to its capacity to use both coarse and fine scale patterns of brain activity. However, a major limitation of multivariate analysis is the difficulty of aligning features across brains, which makes MVPA a subject specific analysis. Recent work by Haxby et al. (2011) introduced a method called Hyperalignment that explored neural activity in ventral temporal cortex during object recognition and demonstrated the ability to align individual patterns of brain activity into a common high dimensional space to facilitate Between Subject Classification (BSC). Here we examined BSC based on Hyperalignment of motor cortex during a task of motor imagery of three natural actions (lift, knock and throw). To achieve this we collected brain activity during the combined tasks of action observation and motor imagery to a parametric action space containing 25 stick-figure blends of the three natural actions. From these responses we derived Hyperalignment transformation parameters that were used to map subjects' representational spaces of the motor imagery task in the motor cortex into a common model representational space. Results showed that BSC of the neural response patterns based on Hyperalignment exceeded both BSC based on anatomical alignment as well as a standard Within Subject Classification (WSC) approach. We also found that results were sensitive to the order in which participants entered the Hyperalignment algorithm. These results demonstrate the effectiveness of Hyperalignment to align neural responses across subject in motor cortex to enable BSC of motor imagery.

Empathy to emotional voices and the use of real-time fMRI to enhance activation of the anterior insula.

The right anterior insula (AI), known to have a key role in the processing and understanding of social emotions, is activated during tasks that involve the act of empathising. Neurofeedback provides individuals with a visualisation of their own brain activity, enabling them to regulate and modify this activity. Following previous research investigating the ability of individuals to up-regulate right AI activity levels through neurofeedback, we investigated whether this could be similarly accomplished during an empathy task involving auditory stimuli of human positive and negative emotional expressions. Twenty participants, ten with feedback from right anterior insula and ten with feedback from a sham brain region, participated in two sessions that included sixteen neurofeedback runs and four transfer runs. Results showed that for the second session participants in the right AI neurofeedback group demonstrated better ability to up-regulate their right AI compared to the control group who received sham feedback. Examination of the relationship between individual participants' empathic traits and their ability to up-regulate right AI activity showed that participants low on empathic traits produced a greater increase in activation of right AI by the end of training. Moreover, the response to positively valenced audio stimuli was greater than for negatively valenced stimuli. These results have implications for therapeutic training of empathy in populations with limited empathic response.

Overlapping but Divergent Neural Correlates Underpinning Audiovisual Synchrony and Temporal Order Judgments.

Multisensory processing is a core perceptual capability, and the need to understand its neural bases provides a fundamental problem in the study of brain function. Both synchrony and temporal order judgments are commonly used to investigate synchrony perception between different sensory cues and multisensory perception in general. However, extensive behavioral evidence indicates that these tasks do not measure identical perceptual processes. Here we used functional magnetic resonance imaging to investigate how behavioral differences between the tasks are instantiated as neural differences. As these neural differences could manifest at either the sustained (task/state-related) and/or transient (event-related) levels of processing, a mixed block/event-related design was used to investigate the neural response of both time-scales. Clear differences in both sustained and transient BOLD responses were observed between the two tasks, consistent with behavioral differences indeed arising from overlapping but divergent neural mechanisms. Temporal order judgments, but not synchrony judgments, required transient activation in several left hemisphere regions, which may reflect increased task demands caused by an extra stage of processing. Our results highlight that multisensory integration mechanisms can be task dependent, which, in particular, has implications for the study of atypical temporal processing in clinical populations.

Exploring collective experience in watching dance through intersubject correlation and functional connectivity of fMRI brain activity.

How the brain contends with naturalistic viewing conditions when it must cope with concurrent streams of diverse sensory inputs and internally generated thoughts is still largely an open question. In this study, we used fMRI to record brain activity while a group of 18 participants watched an edited dance duet accompanied by a soundtrack. After scanning, participants performed a short behavioral task to identify neural correlates of dance segments that could later be recalled. Intersubject correlation (ISC) analysis was used to identify the brain regions correlated among observers, and the results of this ISC map were used to define a set of regions for subsequent analysis of functional connectivity. The resulting network was found to be composed of eight subnetworks and the significance of these subnetworks is discussed. While most subnetworks could be explained by sensory and motor processes, two subnetworks appeared related more to complex cognition. These results inform our understanding of the neural basis of common experience in watching dance and open new directions for the study of complex cognition.

Long-term music training modulates the recalibration of audiovisual simultaneity.

To overcome differences in physical transmission time and neural processing, the brain adaptively recalibrates the point of simultaneity between auditory and visual signals by adapting to audiovisual asynchronies. Here, we examine whether the prolonged recalibration process of passively sensed visual and auditory signals is affected by naturally occurring multisensory training known to enhance audiovisual perceptual accuracy. Hence, we asked a group of drummers, of non-drummer musicians and of non-musicians to judge the audiovisual simultaneity of musical and non-musical audiovisual events, before and after adaptation with two fixed audiovisual asynchronies. We found that the recalibration for the musicians and drummers was in the opposite direction (sound leading vision) to that of non-musicians (vision leading sound), and change together with both increased music training and increased perceptual accuracy (i.e. ability to detect asynchrony). Our findings demonstrate that long-term musical training reshapes the way humans adaptively recalibrate simultaneity between auditory and visual signals.

Affective responses to dance.

The objective of the present work was the characterization of mechanisms by which affective experiences are elicited in observers when watching dance movements. A total of 203 dance stimuli from a normed stimuli library were used in a series of independent experiments. The following measures were obtained: (i) subjective measures of 97 dance-naïve participants' affective responses (Likert scale ratings, interviews); and (ii) objective measures of the physical parameters of the stimuli (motion energy, luminance), and of the movements represented in the stimuli (roundedness, impressiveness). Results showed that (i) participants' ratings of felt and perceived affect differed, (ii) felt and perceived valence but not arousal ratings correlated with physical parameters of the stimuli (motion energy and luminance), (iii) roundedness in posture shape was related to the experience of more positive emotion than edgy shapes (1 of 3 assessed rounded shapes showed a clear effect on positiveness ratings while a second reached trend level significance), (iv) more impressive movements resulted in more positive affective responses, (v) dance triggered affective experiences through the imagery and autobiographical memories it elicited in some people, and (vi) the physical parameters of the video stimuli correlated only weakly and negatively with the aesthetics ratings of beauty, liking and interest. The novelty of the present approach was twofold; (i) the assessment of multiple affect-inducing mechanisms, and (ii) the use of one single normed stimulus set. The results from this approach lend support to both previous and present findings. Results are discussed with regards to current literature in the field of empirical aesthetics and affective neuroscience.

Differences in fMRI intersubject correlation while viewing unedited and edited videos of dance performance.

Intersubject correlation (ISC) analysis of functional magnetic resonance imaging (fMRI) data provides insight into how continuous streams of sensory stimulation are processed by groups of observers. Although edited movies are frequently used as stimuli in ISC studies, there has been little direct examination of the effect of edits on the resulting ISC maps. In this study we showed 16 observers two audiovisual movie versions of the same dance. In one experimental condition there was a continuous view from a single camera (Unedited condition) and in the other condition there were views from different cameras (Edited condition) that provided close up views of the feet or face and upper body. We computed ISC maps for each condition, as well as created a map that showed the difference between the conditions. The results from the Unedited and Edited maps largely overlapped in the occipital and temporal cortices, although more voxels were found for the Edited map. The difference map revealed greater ISC for the Edited condition in the Postcentral Gyrus, Lingual Gyrus, Precentral Gyrus and Medial Frontal Gyrus, while the Unedited condition showed greater ISC in only the Superior Temporal Gyrus. These findings suggest that the visual changes associated with editing provide a source of correlation in maps obtained from edited film, and highlight the utility of using maps to evaluate the difference in ISC between conditions.

"Some like it hot": spectators who score high on the personality trait openness enjoy the excitement of hearing dancers breathing without music.

Music is an integral part of dance. Over the last 10 years, however, dance stimuli (without music) have been repeatedly used to study action observation processes, increasing our understanding of the influence of observer's physical abilities on action perception. Moreover, beyond trained skills and empathy traits, very little has been investigated on how other observer or spectators' properties modulate action observation and action preference. Since strong correlations have been shown between music and personality traits, here we aim to investigate how personality traits shape the appreciation of dance when this is presented with three different music/sounds. Therefore, we investigated the relationship between personality traits and the subjective esthetic experience of 52 spectators watching a 24 min lasting contemporary dance performance projected on a big screen containing three movement phrases performed to three different sound scores: classical music (i.e., Bach), an electronic sound-score, and a section without music but where the breathing of the performers was audible. We found that first, spectators rated the experience of watching dance without music significantly different from with music. Second, we found that the higher spectators scored on the Big Five personality factor openness, the more they liked the no-music section. Third, spectators' physical experience with dance was not linked to their appreciation but was significantly related to high average extravert scores. For the first time, we showed that spectators' reported entrainment to watching dance movements without music is strongly related to their personality and thus may need to be considered when using dance as a means to investigate action observation processes and esthetic preferences.

Experience in judging intent to harm modulates parahippocampal activity: an fMRI study with experienced CCTV operators.

Does visual experience in judging intent to harm change our brain responses? And if it does, what are the mechanisms affected? We addressed these questions by studying the abilities of Closed Circuit Television (CCTV) operators, who must identify the presence of hostile intentions using only visual cues in complex scenes. We used functional magnetic resonance imaging to assess which brain processes are modulated by CCTV experience. To this end we scanned 15 CCTV operators and 15 age and gender matched novices while they watched CCTV videos of 16 sec, and asked them to report whether each clip would end in violence or not. We carried out four separate whole-brain analyses including 3 model-based analyses and one analysis of intersubject correlation to examine differences between the two groups. The three model analyses were based on 1) experimentally pre-defined clip activity labels of fight, confrontation, playful, and neutral behaviour, 2) participants' reports of violent outcomes during the scan, and 3) visual saliency within each clip, as pre-assessed using eye-tracking. The analyses identified greater activation in the right superior frontal gyrus for operators than novices when viewing playful behaviour, and reduced activity for operators in comparison with novices in the occipital and temporal regions, irrespective of the type of clips viewed. However, in the parahippocampal gyrus, all three model-based analyses consistently showed reduced activity for experienced CCTV operators. Activity in the anterior part of the parahippocampal gyrus (uncus) was found to increase with years of CCTV experience. The intersubject correlation analysis revealed a further effect of experience, with CCTV operators showing correlated activity in fewer brain regions (superior and middle temporal gyrus, inferior parietal lobule and the ventral striatum) than novices. Our results indicate that long visual experience in action observation, aimed to predict harmful behaviour, modulates brain mechanisms of intent recognition.

Look at those two!: The precuneus role in unattended third-person perspective of social interactions.

Human beings often observe other people's social interactions without being a part of them. Whereas the implications of some brain regions (e.g. amygdala) have been extensively examined, the implication of the precuneus remains yet to be determined. Here we examined the implication of the precuneus in third-person perspective of social interaction using functional magnetic resonance imaging (fMRI). Participants performed a socially irrelevant task while watching the biological motion of two agents acting in either typical (congruent to social conventions) or atypical (incongruent to social conventions) ways. When compared to typical displays, the atypical displays elicited greater activation in the central and posterior bilateral precuneus, and in frontoparietal and occipital regions. Whereas the right precuneus responded with greater activation also to upside down than upright displays, the left precuneus did not. Correlations and effective connectivity analysis added consistent evidence of an interhemispheric asymmetry between the right and left precuneus. These findings suggest that the precuneus reacts to violations of social expectations, and plays a crucial role in third-person perspective of others' interaction even when the social context is unattended.

Empirical evaluation of the uncanny valley hypothesis fails to confirm the predicted effect of motion.

The uncanny valley hypothesis states that the acceptability of an artificial character will not increase linearly in relation to its likeness to human form. Instead, after an initial rise in acceptability there will be a pronounced decrease when the character is similar, but not identical to human form (Mori, 1970/2012). Moreover, it has been claimed but never directly tested that movement would accentuate this dip and make moving characters less acceptable. We used a number of full-body animated computer characters along with a parametrically defined motion set to examine the effect of motion quality on the uncanny valley. We found that improving the motion quality systematically improved the acceptability of the characters. In particular, the character classified in the deepest location of the uncanny valley became more acceptable when it was animated. Our results showed that although an uncanny valley was found for static characters, the deepening of the valley with motion, originally predicted by Mori (1970/2012), was not obtained.

The role of kinematics in cortical regions for continuous human motion perception.

It has been proposed that we make sense of the movements of others by observing fluctuations in the kinematic properties of their actions. At the neural level, activity in the human motion complex (hMT+) and posterior superior temporal sulcus (pSTS) has been implicated in this relationship. However, previous neuroimaging studies have largely utilized brief, diminished stimuli, and the role of relevant kinematic parameters for the processing of human action remains unclear. We addressed this issue by showing extended-duration natural displays of an actor engaged in two common activities, to 12 participants in an fMRI study under passive viewing conditions. Our region-of-interest analysis focused on three neural areas (hMT+, pSTS, and fusiform face area) and was accompanied by a whole-brain analysis. The kinematic properties of the actor, particularly the speed of body part motion and the distance between body parts, were related to activity in hMT+ and pSTS. Whole-brain exploratory analyses revealed additional areas in posterior cortex, frontal cortex, and the cerebellum whose activity was related to these features. These results indicate that the kinematic properties of peoples' movements are continually monitored during everyday activity as a step to determining actions and intent.

Uni- and multisensory brain areas are synchronised across spectators when watching unedited dance recordings.

The superior temporal sulcus (STS) and gyrus (STG) are commonly identified to be functionally relevant for multisensory integration of audiovisual (AV) stimuli. However, most neuroimaging studies on AV integration used stimuli of short duration in explicit evaluative tasks. Importantly though, many of our AV experiences are of a long duration and ambiguous. It is unclear if the enhanced activity in audio, visual, and AV brain areas would also be synchronised over time across subjects when they are exposed to such multisensory stimuli. We used intersubject correlation to investigate which brain areas are synchronised across novices for uni- and multisensory versions of a 6-min 26-s recording of an unfamiliar, unedited Indian dance recording (Bharatanatyam). In Bharatanatyam, music and dance are choreographed together in a highly intermodal-dependent manner. Activity in the middle and posterior STG was significantly correlated between subjects and showed also significant enhancement for AV integration when the functional magnetic resonance signals were contrasted against each other using a general linear model conjunction analysis. These results extend previous studies by showing an intermediate step of synchronisation for novices: while there was a consensus across subjects' brain activity in areas relevant for unisensory processing and AV integration of related audio and visual stimuli, we found no evidence for synchronisation of higher level cognitive processes, suggesting these were idiosyncratic.

A psychophysical investigation of differences between synchrony and temporal order judgments.

BACKGROUND: Synchrony judgments involve deciding whether cues to an event are in synch or out of synch, while temporal order judgments involve deciding which of the cues came first. When the cues come from different sensory modalities these judgments can be used to investigate multisensory integration in the temporal domain. However, evidence indicates that that these two tasks should not be used interchangeably as it is unlikely that they measure the same perceptual mechanism. The current experiment further explores this issue across a variety of different audiovisual stimulus types. METHODOLOGY/PRINCIPAL FINDINGS: Participants were presented with 5 audiovisual stimulus types, each at 11 parametrically manipulated levels of cue asynchrony. During separate blocks, participants had to make synchrony judgments or temporal order judgments. For some stimulus types many participants were unable to successfully make temporal order judgments, but they were able to make synchrony judgments. The mean points of subjective simultaneity for synchrony judgments were all video-leading, while those for temporal order judgments were all audio-leading. In the within participants analyses no correlation was found across the two tasks for either the point of subjective simultaneity or the temporal integration window. CONCLUSIONS: Stimulus type influenced how the two tasks differed; nevertheless, consistent differences were found between the two tasks regardless of stimulus type. Therefore, in line with previous work, we conclude that synchrony and temporal order judgments are supported by different perceptual mechanisms and should not be interpreted as being representative of the same perceptual process.