Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama - a signal-analysis approach.

A mere co-presence of an unfamiliar person may modulate an individual's attentive engagement with specific events or situations to a significant degree. To understand better how such social presence affects experiences, we recorded a set of parallel multimodal facial and psychophysiological data with subjects (N = 36) who listened to dramatic audio scenes alone or when facing an unfamiliar person. Both a selection of 6 s affective sound clips (IADS-2) followed by a 27 min soundtrack extracted from a Finnish episode film depicted familiar and often intense social situations familiar from the everyday world. Considering the systemic complexity of both the chosen naturalistic stimuli and expected variations in the experimental social situation, we applied a novel combination of signal analysis methods using inter-subject correlation (ISC) analysis, Representational Similarity Analysis (RSA) and Recurrence Quantification Analysis (RQA) followed by gradient boosting classification. We report our findings concerning three facial signals, gaze, eyebrow and smile that can be linked to socially motivated facial movements. We found that ISC values of pairs, whether calculated on true pairs or any two individuals who had a partner, were lower than the group with single individuals. Thus, audio stimuli induced more unique responses in those subjects who were listening to it in the presence of another person, while individual listeners tended to yield a more uniform response as it was driven by dramatized audio stimulus alone. Furthermore, our classifiers models trained using recurrence properties of gaze, eyebrows and smile signals demonstrated distinctive differences in the recurrence dynamics of signals from paired subjects and revealed the impact of individual differences on the latter. We showed that the presence of an unfamiliar co-listener that modifies social dynamics of dyadic listening tasks can be detected reliably from visible facial modalities. By applying our analysis framework to a broader range of psycho-physiological data, together with annotations of the content, and subjective reports of participants, we expected more detailed dyadic dependencies to be revealed. Our work contributes towards modeling and predicting human social behaviors to specific types of audio-visually mediated, virtual, and live social situations.

An embodiment of the cinematographer: emotional and perceptual responses to different camera movement techniques.

We investigate the relationship between camera movement techniques and cognitive responses in audiences, reporting on an experiment exploring the effects of different camera movement methods on viewers' degree of immersion and emotional response. This follows directly from preceding experimental literature and is further motivated by accounts and experiences of practicing cinematographers (authors included), which indicates a correspondence between the two. We designed three different cinematic scenes with indifferent moods, and shot each one time with Steadicam, dolly, handheld, and static camera, resulting in 12 different clips. A total of 44 non-professional participants watched the clips and rated their reactions in terms of arousal and degree of involvement. Experimental results are mixed: movement affects the sense of involvement but not necessarily emotional response. We present and discuss some further explorative results and possible future directions to improve the design. We argue in this contribution that there is value in experimental approaches to cinematography, enabling the systematic study of creative intuitions and audience responses in controlled settings.

Narrative simulation of social experiences in naturalistic context - A neurocinematic approach.

Narratives may be regarded as simulations of everyday social situations. They are key to studying the human mind in socio-culturally determined contexts as they allow anchoring to the common ground of embodied and environmentally-engaged cognition. Here we review recent findings from naturalistic neuroscience on neural functions in conditions that mimic lifelike situations. We will focus particularly on neurocinematics, a research field that applies mediated narratives as stimuli for neuroimaging experiments. During the last two decades, this paradigm has contributed to an accumulation of insights about the neural underpinnings of behavior and sense-making in various narratively contextualized situations particularly pertaining to socio-emotional encounters. One of the key questions in neurocinematics is, how do intersubjectively synchronized brain activations relate to subjective experiences? Another question we address is how to bring natural contexts into experimental studies. Seeking to respond to both questions, we suggest neurocinematic studies to examine three manifestations of the same phenomenon side-by-side: subjective experiences of narrative situations, unfolding of narrative stimulus structure, and neural processes that co-constitute the experience. This approach facilitates identifying experientially meaningful activity patterns in the brain and points out what they may mean in relation to shared and communicable contents. Via rich-featured and temporally contextualized narrative stimuli, neurocinematics attempts to contribute to emerging holistic theories of neural dynamics and connectomics explaining typical and atypical interindividual variability.

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings.

Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human-robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human-computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.

Predictive modeling for trustworthiness and other subjective text properties in online nutrition and health communication.

While the internet has democratized and accelerated content creation and sharing, it has also made people more vulnerable to manipulation and misinformation. Also, the received information can be distorted by psychological biases. This is problematic especially in health-related communications which can greatly affect the quality of life of individuals. We assembled and analyzed 364 texts related to nutrition and health from Finnish online sources, such as news, columns and blogs, and asked non-experts to subjectively evaluate the texts. Texts were rated for their trustworthiness, sentiment, logic, information, clarity, and neutrality properties. We then estimated individual biases and consensus ratings that were used in training regression models. Firstly, we found that trustworthiness was significantly correlated to the information, neutrality and logic of the texts. Secondly, individual ratings for information and logic were significantly biased by the age and diet of the raters. Our best regression models explained up to 70% of the total variance of consensus ratings based on the low-level properties of texts, such as semantic embeddings, presence of key-terms and part-of-speech tags, references, quotes and paragraphs. With a novel combination of crowdsourcing, behavioral analysis, natural language processing and predictive modeling, our study contributes to the automated identification of reliable and high-quality online information. While critical evaluation of truthfulness cannot be surrendered to the machine only, our findings provide new insights into automated evaluation of subjective text properties and analysis of morphologically-rich languages in regards to trustworthiness.

Narrative comprehension beyond language: Common brain networks activated by a movie and its script.

Narratives surround us in our everyday life in different forms. In the sensory brain areas, the processing of narratives is dependent on the media of presentation, be that in audiovisual or written form. However, little is known of the brain areas that process complex narrative content mediated by various forms. To isolate these regions, we looked for the functional networks reacting in a similar manner to the same narrative content despite different media of presentation. We collected 3-T fMRI whole brain data from 31 healthy human adults during two separate runs when they were either viewing a movie or reading its screenplay text. The independent component analysis (ICA) was used to separate 40 components. By correlating the components' time-courses between the two different media conditions, we could isolate 5 functional networks that particularly related to the same narrative content. These TOP-5 components with the highest correlation covered fronto-temporal, parietal, and occipital areas with no major involvement of primary visual or auditory cortices. Interestingly, the top-ranked network with highest modality-invariance also correlated negatively with the dialogue predictor, thus pinpointing that narrative comprehension entails processes that are not language-reliant. In summary, our novel experiment design provided new insight into narrative comprehension networks across modalities.

Consistency and similarity of MEG- and fMRI-signal time courses during movie viewing.

Movie viewing allows human perception and cognition to be studied in complex, real-life-like situations in a brain-imaging laboratory. Previous studies with functional magnetic resonance imaging (fMRI) and with magneto- and electroencephalography (MEG and EEG) have demonstrated consistent temporal dynamics of brain activity across movie viewers. However, little is known about the similarities and differences of fMRI and MEG or EEG dynamics during such naturalistic situations. We thus compared MEG and fMRI responses to the same 15-min black-and-white movie in the same eight subjects who watched the movie twice during both MEG and fMRI recordings. We analyzed intra- and intersubject voxel-wise correlations within each imaging modality as well as the correlation of the MEG envelopes and fMRI signals. The fMRI signals showed voxel-wise within- and between-subjects correlations up to r = 0.66 and r = 0.37, respectively, whereas these correlations were clearly weaker for the envelopes of band-pass filtered (7 frequency bands below 100 Hz) MEG signals (within-subjects correlation r < 0.14 and between-subjects r < 0.05). Direct MEG-fMRI voxel-wise correlations were unreliable. Notably, applying a spatial-filtering approach to the MEG data uncovered consistent canonical variates that showed considerably stronger (up to r = 0.25) between-subjects correlations than the univariate voxel-wise analysis. Furthermore, the envelopes of the time courses of these variates up to about 10 Hz showed association with fMRI signals in a general linear model. Similarities between envelopes of MEG canonical variates and fMRI voxel time-courses were seen mostly in occipital, but also in temporal and frontal brain regions, whereas intra- and intersubject correlations for MEG and fMRI separately were strongest only in the occipital areas. In contrast to the conventional univariate analysis, the spatial-filtering approach was able to uncover associations between the MEG envelopes and fMRI time courses, shedding light on the similarities of hemodynamic and electromagnetic brain activities during movie viewing.

Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento.

How does the human brain recall and connect relevant memories with unfolding events? To study this, we presented 25 healthy subjects, during functional magnetic resonance imaging, the movie 'Memento' (director C. Nolan). In this movie, scenes are presented in chronologically reverse order with certain scenes briefly overlapping previously presented scenes. Such overlapping "key-frames" serve as effective memory cues for the viewers, prompting recall of relevant memories of the previously seen scene and connecting them with the concurrent scene. We hypothesized that these repeating key-frames serve as immediate recall cues and would facilitate reconstruction of the story piece-by-piece. The chronological version of Memento, shown in a separate experiment for another group of subjects, served as a control condition. Using multivariate event-related pattern analysis method and representational similarity analysis, focal fingerprint patterns of hemodynamic activity were found to emerge during presentation of key-frame scenes. This effect was present in higher-order cortical network with regions including precuneus, angular gyrus, cingulate gyrus, as well as lateral, superior, and middle frontal gyri within frontal poles. This network was right hemispheric dominant. These distributed patterns of brain activity appear to underlie ability to recall relevant memories and connect them with ongoing events, i.e., "what goes with what" in a complex story. Given the real-life likeness of cinematic experience, these results provide new insight into how the human brain recalls, given proper cues, relevant memories to facilitate understanding and prediction of everyday life events.

Haptic contents of a movie dynamically engage the spectator's sensorimotor cortex.

Observation of another person's actions and feelings activates brain areas that support similar functions in the observer, thereby facilitating inferences about the other's mental and bodily states. In real life, events eliciting this kind of vicarious brain activations are intermingled with other complex, ever-changing stimuli in the environment. One practical approach to study the neural underpinnings of real-life vicarious perception is to image brain activity during movie viewing. Here the goal was to find out how observed haptic events in a silent movie would affect the spectator's sensorimotor cortex. The functional state of the sensorimotor cortex was monitored by analyzing, in 16 healthy subjects, magnetoencephalographic (MEG) responses to tactile finger stimuli that were presented once per second throughout the session. Using canonical correlation analysis and spatial filtering, consistent single-trial responses across subjects were uncovered, and their waveform changes throughout the movie were quantified. The long-latency (85-175 ms) parts of the responses were modulated in concordance with the participants' average moment-by-moment ratings of own engagement in the haptic content of the movie (correlation r = 0.49; ratings collected after the MEG session). The results, obtained by using novel signal-analysis approaches, demonstrate that the functional state of the human sensorimotor cortex fluctuates in a fine-grained manner even during passive observation of temporally varying haptic events. Hum Brain Mapp 37:4061-4068, 2016. © 2016 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.

Brain-based decoding of mentally imagined film clips and sounds reveals experience-based information patterns in film professionals.

In the perceptual domain, it has been shown that the human brain is strongly shaped through experience, leading to expertise in highly-skilled professionals. What has remained unclear is whether specialization also shapes brain networks underlying mental imagery. In our fMRI study, we aimed to uncover modality-specific mental imagery specialization of film experts. Using multi-voxel pattern analysis we decoded from brain activity of professional cinematographers and sound designers whether they were imagining sounds or images of particular film clips. In each expert group distinct multi-voxel patterns, specific for the modality of their expertise, were found during classification of imagery modality. These patterns were mainly localized in the occipito-temporal and parietal cortex for cinematographers and in the auditory cortex for sound designers. We also found generalized patterns across perception and imagery that were distinct for the two expert groups: they involved frontal cortex for the cinematographers and temporal cortex for the sound designers. Notably, the mental representations of film clips and sounds of cinematographers contained information that went beyond modality-specificity. We were able to successfully decode the implicit presence of film genre from brain activity during mental imagery in cinematographers. The results extend existing neuroimaging literature on expertise into the domain of mental imagery and show that experience in visual versus auditory imagery can alter the representation of information in modality-specific association cortices.

Optimizing methods for linking cinematic features to fMRI data.

One of the challenges of naturalistic neurosciences using movie-viewing experiments is how to interpret observed brain activations in relation to the multiplicity of time-locked stimulus features. As previous studies have shown less inter-subject synchronization across viewers of random video footage than story-driven films, new methods need to be developed for analysis of less story-driven contents. To optimize the linkage between our fMRI data collected during viewing of a deliberately non-narrative silent film 'At Land' by Maya Deren (1944) and its annotated content, we combined the method of elastic-net regularization with the model-driven linear regression and the well-established data-driven independent component analysis (ICA) and inter-subject correlation (ISC) methods. In the linear regression analysis, both IC and region-of-interest (ROI) time-series were fitted with time-series of a total of 36 binary-valued and one real-valued tactile annotation of film features. The elastic-net regularization and cross-validation were applied in the ordinary least-squares linear regression in order to avoid over-fitting due to the multicollinearity of regressors, the results were compared against both the partial least-squares (PLS) regression and the un-regularized full-model regression. Non-parametric permutation testing scheme was applied to evaluate the statistical significance of regression. We found statistically significant correlation between the annotation model and 9 ICs out of 40 ICs. Regression analysis was also repeated for a large set of cubic ROIs covering the grey matter. Both IC- and ROI-based regression analyses revealed activations in parietal and occipital regions, with additional smaller clusters in the frontal lobe. Furthermore, we found elastic-net based regression more sensitive than PLS and un-regularized regression since it detected a larger number of significant ICs and ROIs. Along with the ISC ranking methods, our regression analysis proved a feasible method for ordering the ICs based on their functional relevance to the annotated cinematic features. The novelty of our method is - in comparison to the hypothesis-driven manual pre-selection and observation of some individual regressors biased by choice - in applying data-driven approach to all content features simultaneously. We found especially the combination of regularized regression and ICA useful when analyzing fMRI data obtained using non-narrative movie stimulus with a large set of complex and correlated features.

From naturalistic neuroscience to modeling radical embodiment with narrative enactive systems.

Mainstream cognitive neuroscience has begun to accept the idea of embodied mind, which assumes that the human mind is fundamentally constituted by the dynamical interactions of the brain, body, and the environment. In today's paradigm of naturalistic neurosciences, subjects are exposed to rich contexts, such as video sequences or entire films, under relatively controlled conditions, against which researchers can interpret changes in neural responses within a time window. However, from the point of view of radical embodied cognitive neuroscience, the increasing complexity alone will not suffice as the explanatory apparatus for dynamical embodiment and situatedness of the mind. We suggest that narrative enactive systems with dynamically adaptive content as stimuli, may serve better to account for the embodied mind engaged with the surrounding world. Among the ensuing challenges for neuroimaging studies is how to interpret brain data against broad temporal contexts of previous experiences that condition the unfolding experience of nowness. We propose means to tackle this issue, as well as ways to limit the exponentially growing combinatoria of narrative paths to a controllable number.

Spatial variability of functional brain networks in early-blind and sighted subjects.

To further the understanding how the human brain adapts to early-onset blindness, we searched in early-blind and normally-sighted subjects for functional brain networks showing the most and least spatial variabilities across subjects. We hypothesized that the functional networks compensating for early-onset blindness undergo cortical reorganization. To determine whether reorganization of functional networks affects spatial variability, we used functional magnetic resonance imaging to compare brain networks, derived by independent component analysis, of 7 early-blind and 7 sighted subjects while they rested or listened to an audio drama. In both conditions, the blind compared with sighted subjects showed more spatial variability in a bilateral parietal network (comprising the inferior parietal and angular gyri and precuneus) and in a bilateral auditory network (comprising the superior temporal gyri). In contrast, a vision-related left-hemisphere-lateralized occipital network (comprising the superior, middle and inferior occipital gyri, fusiform and lingual gyri, and the calcarine sulcus) was less variable in blind than sighted subjects. Another visual network and a tactile network were spatially more variable in the blind than sighted subjects in one condition. We contemplate whether our results on inter-subject spatial variability of brain networks are related to experience-dependent brain plasticity, and we suggest that auditory and parietal networks undergo a stronger experience-dependent reorganization in the early-blind than sighted subjects while the opposite is true for the vision-related occipital network.

Posterior parietal cortex activity reflects the significance of others' actions during natural viewing.

The posterior parietal cortex (PPC) has been associated with multiple stimulus-driven (e.g., processing stimulus movements, providing visual signals for the motor system), goal-directed (e.g., directing visual attention to a target, processing behavioral priority of intentions), and action-related functions in previous studies with non-naturalistic paradigms. Here, we examined how these functions reflect PPC activity during natural viewing. Fourteen healthy volunteers watched a re-edited movie during functional magnetic resonance imaging (fMRI). Participants separately annotated behavioral priority (accounting for percepts, thoughts, and emotions) they had experienced during movie episodes. Movements in the movie were quantified with computer vision and eye movements were recorded from a separate group of subjects. Our results show that while overlapping dorsomedial PPC areas respond to episodes with multiple types of stimulus content, ventrolateral PPC areas exhibit enhanced activity when viewing goal-directed human hand actions. Furthermore, PPC activity related to viewing goal-directed human hand actions was more accurately explained by behavioral priority than by movements of the stimulus or eye movements. Taken together, our results suggest that PPC participates in perception of goal-directed human hand actions, supporting the view that PPC has a special role in providing visual signals for the motor system ("how"), in addition to processing visual spatial movements ("where").

Listening to an audio drama activates two processing networks, one for all sounds, another exclusively for speech.

Earlier studies have shown considerable intersubject synchronization of brain activity when subjects watch the same movie or listen to the same story. Here we investigated the across-subjects similarity of brain responses to speech and non-speech sounds in a continuous audio drama designed for blind people. Thirteen healthy adults listened for ∼19 min to the audio drama while their brain activity was measured with 3 T functional magnetic resonance imaging (fMRI). An intersubject-correlation (ISC) map, computed across the whole experiment to assess the stimulus-driven extrinsic brain network, indicated statistically significant ISC in temporal, frontal and parietal cortices, cingulate cortex, and amygdala. Group-level independent component (IC) analysis was used to parcel out the brain signals into functionally coupled networks, and the dependence of the ICs on external stimuli was tested by comparing them with the ISC map. This procedure revealed four extrinsic ICs of which two-covering non-overlapping areas of the auditory cortex-were modulated by both speech and non-speech sounds. The two other extrinsic ICs, one left-hemisphere-lateralized and the other right-hemisphere-lateralized, were speech-related and comprised the superior and middle temporal gyri, temporal poles, and the left angular and inferior orbital gyri. In areas of low ISC four ICs that were defined intrinsic fluctuated similarly as the time-courses of either the speech-sound-related or all-sounds-related extrinsic ICs. These ICs included the superior temporal gyrus, the anterior insula, and the frontal, parietal and midline occipital cortices. Taken together, substantial intersubject synchronization of cortical activity was observed in subjects listening to an audio drama, with results suggesting that speech is processed in two separate networks, one dedicated to the processing of speech sounds and the other to both speech and non-speech sounds.

Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments.

We outline general theoretical and practical implications of what we promote as enactive cinema for the neuroscientific study of online socio-emotional interaction. In a real-time functional magnetic resonance imaging (rt-fMRI) setting, participants are immersed in cinematic experiences that simulate social situations. While viewing, their physiological reactions-including brain responses-are tracked, representing implicit and unconscious experiences of the on-going social situations. These reactions, in turn, are analyzed in real-time and fed back to modify the cinematic sequences they are viewing while being scanned. Due to the engaging cinematic content, the proposed setting focuses on living-by in terms of shared psycho-physiological epiphenomena of experience rather than active coping in terms of goal-oriented motor actions. It constitutes a means to parametrically modify stimuli that depict social situations and their broader environmental contexts. As an alternative to studying the variation of brain responses as a function of a priori fixed stimuli, this method can be applied to survey the range of stimuli that evoke similar responses across participants at particular brain regions of interest.

Functional subdivision of group-ICA results of fMRI data collected during cinema viewing.

Independent component analysis (ICA) can unravel functional brain networks from functional magnetic resonance imaging (fMRI) data. The number of the estimated components affects both the spatial pattern of the identified networks and their time-course estimates. Here group-ICA was applied at four dimensionalities (10, 20, 40, and 58 components) to fMRI data collected from 15 subjects who viewed a 15-min silent film ("At land" by Maya Deren). We focused on the dorsal attention network, the default-mode network, and the sensorimotor network. The lowest dimensionalities demonstrated most prominent activity within the dorsal attention network, combined with the visual areas, and in the default-mode network; the sensorimotor network only appeared with ICA comprising at least 20 components. The results suggest that even very low-dimensional ICA can unravel the most prominent functionally-connected brain networks. However, increasing the number of components gives a more detailed picture and functionally feasible subdivision of the major networks. These results improve our understanding of the hierarchical subdivision of brain networks during viewing of a movie that provides continuous stimulation embedded in an attention-directing narrative.

Stimulus-related independent component and voxel-wise analysis of human brain activity during free viewing of a feature film.

Understanding how the brain processes stimuli in a rich natural environment is a fundamental goal of neuroscience. Here, we showed a feature film to 10 healthy volunteers during functional magnetic resonance imaging (fMRI) of hemodynamic brain activity. We then annotated auditory and visual features of the motion picture to inform analysis of the hemodynamic data. The annotations were fitted to both voxel-wise data and brain network time courses extracted by independent component analysis (ICA). Auditory annotations correlated with two independent components (IC) disclosing two functional networks, one responding to variety of auditory stimulation and another responding preferentially to speech but parts of the network also responding to non-verbal communication. Visual feature annotations correlated with four ICs delineating visual areas according to their sensitivity to different visual stimulus features. In comparison, a separate voxel-wise general linear model based analysis disclosed brain areas preferentially responding to sound energy, speech, music, visual contrast edges, body motion and hand motion which largely overlapped the results revealed by ICA. Differences between the results of IC- and voxel-based analyses demonstrate that thorough analysis of voxel time courses is important for understanding the activity of specific sub-areas of the functional networks, while ICA is a valuable tool for revealing novel information about functional connectivity which need not be explained by the predefined model. Our results encourage the use of naturalistic stimuli and tasks in cognitive neuroimaging to study how the brain processes stimuli in rich natural environments.