Dissonant music engages early visual processing.

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

Modeling naturalistic face processing in humans with deep convolutional neural networks.

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.

Sense of own body shapes neural processes of memory encoding and reinstatement.

How is the fundamental sense of one's body, a basic aspect of selfhood, incorporated into memories for events? Disrupting bodily self-awareness during encoding impairs functioning of the left posterior hippocampus during retrieval, which implies weakened encoding. However, how changes in bodily self-awareness influence neural encoding is unknown. We investigated how the sense of body ownership, a core aspect of the bodily self, impacts encoding in the left posterior hippocampus and additional core memory regions including the angular gyrus. Furthermore, we assessed the degree to which memories are reinstated according to body ownership during encoding and vividness during retrieval as a measure of memory strength. We immersed participants in naturalistic scenes where events unfolded while we manipulated feelings of body ownership with a full-body-illusion during functional magnetic resonance imaging scanning. One week later, participants retrieved memories for the videos during functional magnetic resonance imaging scanning. A whole brain analysis revealed that patterns of activity in regions including the right hippocampus and angular gyrus distinguished between events encoded with strong versus weak body ownership. A planned region-of-interest analysis showed that patterns of activity in the left posterior hippocampus specifically could predict body ownership during memory encoding. Using the wider network of regions sensitive to body ownership during encoding and the left posterior hippocampus as separate regions-of-interest, we observed that patterns of activity present at encoding were reinstated more during the retrieval of events encoded with strong body ownership and high memory vividness. Our results demonstrate how the sense of physical self is bound within an event during encoding, which facilitates reactivation of a memory trace during retrieval.

Visuospatial processing in early brain-based visual impairment is associated with differential recruitment of dorsal and ventral visual streams.

Visuospatial processing impairments are prevalent in individuals with cerebral visual impairment (CVI) and are typically ascribed to "dorsal stream dysfunction" (DSD). However, the contribution of other cortical regions, including early visual cortex (EVC), frontal cortex, or the ventral visual stream, to such impairments remains unknown. Thus, here, we examined fMRI activity in these regions, while individuals with CVI (and neurotypicals) performed a visual search task within a dynamic naturalistic scene. First, behavioral performance was measured with eye tracking. Participants were instructed to search and follow a walking human target. CVI participants took significantly longer to find the target, and their eye gaze patterns were less accurate and less precise. Second, we used the same task in the MRI scanner. Along the dorsal stream, activation was reduced in CVI participants, consistent with the proposed DSD in CVI. Intriguingly, however, visual areas along the ventral stream showed the complete opposite pattern, with greater activation in CVI participants. In contrast, we found no differences in either EVC or frontal cortex between groups. These results suggest that the impaired visuospatial processing abilities in CVI are associated with differential recruitment of the dorsal and ventral visual streams, likely resulting from impaired selective attention.

Information flow across the cortical timescale hierarchy during narrative construction.

When listening to spoken narratives, we must integrate information over multiple, concurrent timescales, building up from words to sentences to paragraphs to a coherent narrative. Recent evidence suggests that the brain relies on a chain of hierarchically organized areas with increasing temporal receptive windows to process naturalistic narratives. We hypothesized that the structure of this cortical processing hierarchy should result in an observable sequence of response lags between networks comprising the hierarchy during narrative comprehension. This study uses functional MRI to estimate the response lags between functional networks during narrative comprehension. We use intersubject cross-correlation analysis to capture network connectivity driven by the shared stimulus. We found a fixed temporal sequence of response lags-on the scale of several seconds-starting in early auditory areas, followed by language areas, the attention network, and lastly the default mode network. This gradient is consistent across eight distinct stories but absent in data acquired during rest or using a scrambled story stimulus, supporting our hypothesis that narrative construction gives rise to internetwork lags. Finally, we build a simple computational model for the neural dynamics underlying the construction of nested narrative features. Our simulations illustrate how the gradual accumulation of information within the boundaries of nested linguistic events, accompanied by increased activity at each level of the processing hierarchy, can give rise to the observed lag gradient.

Deeper Than You Think: Partisanship-Dependent Brain Responses in Early Sensory and Motor Brain Regions.

Recent political polarization has illustrated how individuals with opposing political views often experience ongoing events in markedly different ways. In this study, we explored the neural mechanisms underpinning this phenomenon. We conducted fMRI scanning of 34 right- and left-wing participants (45% females) watching political videos (e.g., campaign ads and political speeches) just before the elections in Israel. As expected, we observed significant differences between left- and right-wing participants in their interpretation of the videos' content. Furthermore, neuroimaging results revealed partisanship-dependent differences in activation and synchronization in higher-order regions. Surprisingly, such differences were also revealed in early sensory, motor, and somatosensory regions. We found that the political content synchronized the responses of primary visual and auditory cortices in a partisanship-dependent manner. Moreover, right-wing (and not left-wing) individuals' sensorimotor cortex was involved in processing right-wing (and not left-wing) political content. These differences were pronounced to the extent that we could predict political orientation from the early brain-response alone. Importantly, no such differences were found with respect to neutral content. Therefore, these results uncover more fundamental neural mechanisms underlying processes of political polarization.SIGNIFICANCE STATEMENT The political sphere has become highly polarized in recent years. Would it be possible to identify the neural mechanisms underpinning such processes? In our study, left- and right-wing participants were scanned in fMRI while watching political video clips just before the elections in Israel. We found that political content was potent in synchronizing the brain responses of individuals holding similar views. This was far more pronounced in individuals holding right-wing views. Moreover, partisan-dependent differences in neural responses were identified already in early sensory, somatosensory, and motor regions, and only for political content. These results suggest that individuals' political views shape their neural responses at a very basic level.

Inter-subject correlation of electroencephalographic and behavioural responses reflects time-varying engagement with natural music.

Musical engagement can be conceptualized through various activities, modes of listening and listener states. Recent research has reported that a state of focused engagement can be indexed by the inter-subject correlation (ISC) of audience responses to a shared naturalistic stimulus. While statistically significant ISC has been reported during music listening, we lack insight into the temporal dynamics of engagement over the course of musical works-such as those composed in the Western classical style-which involve the formulation of expectations that are realized or derailed at subsequent points of arrival. Here, we use the ISC of electroencephalographic (EEG) and continuous behavioural (CB) responses to investigate the time-varying dynamics of engagement with functional tonal music. From a sample of adult musicians who listened to a complete cello concerto movement, we found that ISC varied throughout the excerpt for both measures. In particular, significant EEG ISC was observed during periods of musical tension that built to climactic highpoints, while significant CB ISC corresponded more to declarative entrances and points of arrival. Moreover, we found that a control stimulus retaining envelope characteristics of the intact music, but little other temporal structure, also elicited significantly correlated EEG and CB responses, though to lesser extents than the original version. In sum, these findings shed light on the temporal dynamics of engagement during music listening and clarify specific aspects of musical engagement that may be indexed by each measure.

Approaches to studying emotion using physiological responses to spoken narratives: A scoping review.

Narratives are effective tools for evoking emotions, and physiological measurements provide a means of objectively assessing emotional reactions - making them a potentially powerful pair of tools for studying emotional processes. However, extent research combining emotional narratives and physiological measurement varies widely in design and application, making it challenging to identify previous work, consolidate findings, and design effective experiments. Our scoping review explores the use of auditory emotional narratives and physiological measures in research, examining paradigms, study populations, and represented emotions. Following the PRISMA-ScR Checklist, we searched five databases for peer-reviewed experimental studies that used spoken narratives to induce emotion and reported autonomic physiological measures. Among 3466 titles screened and 653 articles reviewed, 110 studies were included. Our exploration revealed a variety of applications and experimental paradigms; emotional narratives paired with physiological measures have been used to study diverse topics and populations, including neurotypical and clinical groups. Although incomparable designs and sometimes contradictory results precluded general recommendations as regards which physiological measures to use when designing new studies, as a whole, the body of work suggests that these tools can be valuable to study emotions. Our review offers an overview of research employing narratives and physiological measures for emotion study, and highlights weaknesses in reporting practices and gaps in our knowledge concerning the robustness and specificity of physiological measures as indices of emotion. We discuss study design considerations and transparent reporting, to facilitate future using emotional narratives and physiological measures in studying emotions.

Neural unscrambling of temporal information during a nonlinear narrative.

Although we must experience our lives chronologically, storytellers often manipulate the order in which they relay events. How the brain processes temporal information while encoding a nonlinear narrative remains unclear. Here, we use functional magnetic resonance imaging during movie watching to investigate which brain regions are sensitive to information about time in a narrative and test whether the representation of temporal context across a narrative is more influenced by the order in which events are presented or their underlying chronological sequence. Results indicate that medial parietal regions are sensitive to cued jumps through time over and above other changes in context (i.e., location). Moreover, when processing non-chronological narrative information, the precuneus and posterior cingulate engage in on-the-fly temporal unscrambling to represent information chronologically. Specifically, days that are closer together in chronological time are represented more similarly regardless of when they are presented in the movie, and this representation is consistent across participants. Additional analyses reveal a strong spatial signature associated with higher magnitude jumps through time. These findings are consistent with prior theorizing on medial parietal regions as central to maintaining and updating narrative situation models, and suggest the priority of chronological information when encoding narrative events.

Individual differences in neural event segmentation of continuous experiences.

Event segmentation is a spontaneous part of perception, important for processing continuous information and organizing it into memory. Although neural and behavioral event segmentation show a degree of inter-subject consistency, meaningful individual variability exists atop these shared patterns. Here we characterized individual differences in the location of neural event boundaries across four short movies that evoked variable interpretations. Event boundary alignment across subjects followed a posterior-to-anterior gradient that was tightly correlated with the rate of segmentation: slower-segmenting regions that integrate information over longer time periods showed more individual variability in boundary locations. This relationship held irrespective of the stimulus, but the degree to which boundaries in particular regions were shared versus idiosyncratic depended on certain aspects of movie content. Furthermore, this variability was behaviorally significant in that similarity of neural boundary locations during movie-watching predicted similarity in how the movie was ultimately remembered and appraised. In particular, we identified a subset of regions in which neural boundary locations are both aligned with behavioral boundaries during encoding and predictive of stimulus interpretation, suggesting that event segmentation may be a mechanism by which narratives generate variable memories and appraisals of stimuli.

Diverse Face Images (DFI): Validated for racial representation and eye gaze.

Face processing is a central component of human communication and social engagement. The present investigation introduces a set of racially and ethnically inclusive faces created for researchers interested in perceptual and socio-cognitive processes linked to human faces. The Diverse Face Images (DFI) stimulus set includes high-quality still images of female faces that are racially and ethnically representative, include multiple images of direct and indirect gaze for each model and control for low-level perceptual variance between images. The DFI stimuli will support researchers interested in studying face processing throughout the lifespan as well as other questions that require a diversity of faces or gazes. This report includes a detailed description of stimuli development and norming data for each model. Adults completed a questionnaire rating each image in the DFI stimuli set on three major qualities relevant to face processing: (1) strength of race/ethnicity group associations, (2) strength of eye gaze orientation, and (3) strength of emotion expression. These validation data highlight the presence of rater variability within and between individual model images as well as within and between race and ethnicity groups.

GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database.

Since thoroughly validated naturalistic affective German speech stimulus databases are rare, we present here a novel validated database of speech sequences assembled with the purpose of emotion induction. The database comprises 37 audio speech sequences with a total duration of 92 minutes for the induction of positive, neutral, and negative emotion: comedian shows intending to elicit humorous and amusing feelings, weather forecasts, and arguments between couples and relatives from movies or television series. Multiple continuous and discrete ratings are used to validate the database to capture the time course and variabilities of valence and arousal. We analyse and quantify how well the audio sequences fulfil quality criteria of differentiation, salience/strength, and generalizability across participants. Hence, we provide a validated speech database of naturalistic scenarios suitable to investigate emotion processing and its time course with German-speaking participants. Information on using the stimulus database for research purposes can be found at the OSF project repository GAUDIE: https://osf.io/xyr6j/ .

Naturalistic Audio-Movies reveal common spatial organization across "visual" cortices of different blind individuals.

Occipital cortices of different sighted people contain analogous maps of visual information (e.g. foveal vs. peripheral). In congenital blindness, "visual" cortices respond to nonvisual stimuli. Do visual cortices of different blind people represent common informational maps? We leverage naturalistic stimuli and inter-subject pattern similarity analysis to address this question. Blindfolded sighted (n = 22) and congenitally blind (n = 22) participants listened to 6 sound clips (5-7 min each): 3 auditory excerpts from movies; a naturalistic spoken narrative; and matched degraded auditory stimuli (Backwards Speech, scrambled sentences), during functional magnetic resonance imaging scanning. We compared the spatial activity patterns evoked by each unique 10-s segment of the different auditory excerpts across blind and sighted people. Segments of meaningful naturalistic stimuli produced distinctive activity patterns in frontotemporal networks that were shared across blind and across sighted individuals. In the blind group only, segment-specific, cross-subject patterns emerged in visual cortex, but only for meaningful naturalistic stimuli and not Backwards Speech. Spatial patterns of activity within visual cortices are sensitive to time-varying information in meaningful naturalistic auditory stimuli in a broadly similar manner across blind individuals.

Vascular feature as a modulator of the aging brain.

The cerebral functional reorganization and declined cognitive function of aging might associate with altered vascular features. Here, we explored the altered cerebral hierarchical functional network of 2 conditions (task-free and naturalistic stimuli) in older adults and its relationship with vascular features (systemic microvascular and perfusion features, measured by magnetic resonance imaging) and behavior. Using cerebral gradient analysis, we found that compressive gradient of resting-state mainly located on the primary sensory-motor system and transmodal regions in aging, and further compress in these regions under the continuous naturalistic stimuli. Combining cerebral functional gradient, vascular features, and cognitive performance, the more compressive gradient in the resting-state, the worse vascular state, the lower cognitive function in older adults. Further modulation analysis demonstrated that both vascular features can regulate the relationship between gradient scores in the insula and behavior. Interestingly, systemic microvascular oxygenation also can modulate the relationship between cerebral gradient and cerebral perfusion. Furthermore, the less alteration of the compressive gradient with naturalistic stimuli came with lower cognitive function. Our findings demonstrated that the altered cerebral hierarchical functional structure in aging was linked with changed vascular features and behavior, offering a new framework for studying the physiological mechanism of functional connectivity in aging.

High-amplitude cofluctuations in cortical activity drive functional connectivity.

Resting-state functional connectivity is used throughout neuroscience to study brain organization and to generate biomarkers of development, disease, and cognition. The processes that give rise to correlated activity are, however, poorly understood. Here we decompose resting-state functional connectivity using a temporal unwrapping procedure to assess the contributions of moment-to-moment activity cofluctuations to the overall connectivity pattern. This approach temporally resolves functional connectivity at a timescale of single frames, which enables us to make direct comparisons of cofluctuations of network organization with fluctuations in the blood oxygen level-dependent (BOLD) time series. We show that surprisingly, only a small fraction of frames exhibiting the strongest cofluctuation amplitude are required to explain a significant fraction of variance in the overall pattern of connection weights as well as the network's modular structure. These frames coincide with frames of high BOLD activity amplitude, corresponding to activity patterns that are remarkably consistent across individuals and identify fluctuations in default mode and control network activity as the primary driver of resting-state functional connectivity. Finally, we demonstrate that cofluctuation amplitude synchronizes across subjects during movie watching and that high-amplitude frames carry detailed information about individual subjects (whereas low-amplitude frames carry little). Our approach reveals fine-scale temporal structure of resting-state functional connectivity and discloses that frame-wise contributions vary across time. These observations illuminate the relation of brain activity to functional connectivity and open a number of directions for future research.

The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition.

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

The role of low-frequency oscillations in three-dimensional perception with depth cues in virtual reality.

Currently, vision-related neuroscience studies are undergoing a trend from simplified image stimuli toward more naturalistic stimuli. Virtual reality (VR), as an emerging technology for visual immersion, provides more depth cues for three-dimensional (3D) presentation than two-dimensional (2D) image. It is still unclear whether the depth cues used to create 3D visual perception modulate specific cortical activation. Here, we constructed two visual stimuli presented by stereoscopic vision in VR and graphical projection with 2D image, respectively, and used electroencephalography to examine neural oscillations and their functional connectivity during 3D perception. We find that neural oscillations are specific to delta and theta bands in stereoscopic vision and the functional connectivity in the two bands increase in cortical areas related to visual pathways. These findings indicate that low-frequency oscillations play an important role in 3D perception with depth cues.

Act natural: Functional connectivity from naturalistic stimuli fMRI outperforms resting-state in predicting brain activity.

The search for an 'ideal' approach to investigate the functional connections in the human brain is an ongoing challenge for the neuroscience community. While resting-state functional magnetic resonance imaging (fMRI) has been widely used to study individual functional connectivity patterns, recent work has highlighted the benefits of collecting functional connectivity data while participants are exposed to naturalistic stimuli, such as watching a movie or listening to a story. For example, functional connectivity data collected during movie-watching were shown to predict cognitive and emotional scores more accurately than resting-state-derived functional connectivity. We have previously reported a tight link between resting-state functional connectivity and task-derived neural activity, such that the former successfully predicts the latter. In the current work we use data from the Human Connectome Project to demonstrate that naturalistic-stimulus-derived functional connectivity predicts task-induced brain activation maps more accurately than resting-state-derived functional connectivity. We then show that activation maps predicted using naturalistic stimuli are better predictors of individual intelligence scores than activation maps predicted using resting-state. We additionally examine the influence of naturalistic-stimulus type on prediction accuracy. Our findings emphasize the potential of naturalistic stimuli as a promising alternative to resting-state fMRI for connectome-based predictive modelling of individual brain activity and cognitive traits.

Discovering hidden brain network responses to naturalistic stimuli via tensor component analysis of multi-subject fMRI data.

The study of brain network interactions during naturalistic stimuli facilitates a deeper understanding of human brain function. To estimate large-scale brain networks evoked with naturalistic stimuli, a tensor component analysis (TCA) based framework was used to characterize shared spatio-temporal patterns across subjects in a purely data-driven manner. In this framework, a third-order tensor is constructed from the timeseries extracted from all brain regions from a given parcellation, for all participants, with modes of the tensor corresponding to spatial distribution, time series and participants. TCA then reveals spatially and temporally shared components, i.e., evoked networks with the naturalistic stimuli, their time courses of activity and subject loadings of each component. To enhance the reproducibility of the estimation with the adaptive TCA algorithm, a novel spectral clustering method, tensor spectral clustering, was proposed and applied to evaluate the stability of the TCA algorithm. We demonstrated the effectiveness of the proposed framework via simulations and real fMRI data collected during a motor task with a traditional fMRI study design. We also applied the proposed framework to fMRI data collected during passive movie watching to illustrate how reproducible brain networks are evoked by naturalistic movie viewing.

Asymmetrical cross-modal influence on neural encoding of auditory and visual features in natural scenes.

Natural scenes contain multi-modal information, which is integrated to form a coherent perception. Previous studies have demonstrated that cross-modal information can modulate neural encoding of low-level sensory features. These studies, however, mostly focus on the processing of single sensory events or rhythmic sensory sequences. Here, we investigate how the neural encoding of basic auditory and visual features is modulated by cross-modal information when the participants watch movie clips primarily composed of non-rhythmic events. We presented audiovisual congruent and audiovisual incongruent movie clips, and since attention can modulate cross-modal interactions, we separately analyzed high- and low-arousal movie clips. We recorded neural responses using electroencephalography (EEG), and employed the temporal response function (TRF) to quantify the neural encoding of auditory and visual features. The neural encoding of sound envelope is enhanced in the audiovisual congruent condition than the incongruent condition, but this effect is only significant for high-arousal movie clips. In contrast, audiovisual congruency does not significantly modulate the neural encoding of visual features, e.g., luminance or visual motion. In summary, our findings demonstrate asymmetrical cross-modal interactions during the processing of natural scenes that lack rhythmicity: Congruent visual information enhances low-level auditory processing, while congruent auditory information does not significantly modulate low-level visual processing.