Regularity and variability of functional brain connectivity characteristics between gyri and sulci under naturalistic stimulus.

The human cerebral cortex is folded into two fundamentally anatomical units: gyri and sulci. Previous studies have demonstrated the genetical, structural, and functional differences between gyri and sulci, providing a unique perspective for revealing the relationship among brain function, cognition, and behavior. While previous studies mainly focus on the functional differences between gyri and sulci under resting or task-evoked state, such characteristics under naturalistic stimulus (NS) which reflects real-world dynamic environments are largely unknown. To address this question, this study systematically investigates spatio-temporal functional connectivity (FC) characteristics between gyri and sulci under NS using a spatio-temporal graph convolutional network model. Based on the public Human Connectome Project dataset of 174 subjects with four different runs of both movie-watching NS and resting state 7T functional MRI data, we successfully identify unique FC features under NS, which are mainly involved in visual, auditory, emotional and cognitive control, and achieve high discriminative accuracy 93.06 % to resting state. Moreover, gyral regions as well as gyro-gyral connections consistently participate more as functional information exchange hubs than sulcal ones among these networks. This study provides novel insights into the functional brain mechanism under NS and lays a solid foundation for accurately mapping the brain anatomy-function relationship.

Prediction of cognitive scores by joint use of movie-watching fMRI connectivity and eye tracking via Attention-CensNet.

Background: Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states, such as rest and doing tasks. Nevertheless, the state-of-the-art methods have found it difficult to achieve desirable results from movie-watching paradigm functional magnetic resonance imaging (mfMRI) -induced brain functional connectivity, especially when there are fewer datasets. Incorporating other physical measurements into the prediction method may enhance accuracy. Eye tracking, becoming popular due to its portability and lower expense, can provide abundant behavioral features related to the output of human's cognition, and thus might supplement the mfMRI in observing participants' subconscious behaviors. However, there are very few studies on how to effectively integrate the multimodal information to strengthen the performance by a unified framework. Objective: A fusion approach with mfMRI and eye tracking, based on convolution with edge-node switching in graph neural networks (CensNet), is proposed in this article. Methods: In this graph model, participants are designated as nodes, mfMRI derived functional connectivity as node features, and different eye-tracking features are used to compute similarity between participants to construct heterogeneous graph edges. By taking multiple graphs as different channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. Results: The proposed model outperforms those using a single modality and single channel, and state-of-the-art methods. Conclusions: The results indicate that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes.

Altered Cerebral Processing of Videos in Children with Motor Dysfunction Suggests Broad Embodiment of Perceptual Cognitive Functions.

Embodied cognition theory suggests that motor dysfunctions affect cognition. We examined this hypothesis by inspecting whether cerebral processing of movies, featuring both goal-directed movements and content without humans, differ between children with congenital motor dysfunction and healthy controls. Electroencephalography was recorded from 23 healthy children and 23 children with limited or absent arm movement due to either arthrogryposis multiplex congenita or obstetric brachial plexus palsy. Each individual patient exhibited divergent neural responses, disclosed by significantly lower inter-subject correlation (ISC) of brain activity, during the videos compared to the healthy children. We failed to observe associations between this finding and the motor-related content of the various video scenes, suggesting that differences between the patients and controls reflect modulation of perceptual-cognitive processing of videos by upper-limb motor dysfunctions not limited to the watching-mirroring of motor actions. Thus, perceptual-cognitive processes in the brain seem to be more robustly embodied than has previously been thought.

How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness.

Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. The semantic information about any object or event can be highly complex and frequently draws on multiple sensory modalities, which makes it difficult to quantify. Past studies have primarily relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to further accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. An additional 140 participants provided word judgement to calculate similarity of word-word pairs. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.

Functional network connectivity and topology during naturalistic stimulus is altered in first-episode psychosis.

BACKGROUND: Psychotic disorders have been suggested to derive from dysfunctional integration of signaling between brain regions. Earlier studies have found several changes in functional network synchronization as well as altered network topology in patients with psychotic disorders. However, studies have used mainly resting-state that makes it more difficult to link functional alterations to any specific stimulus or experience. We set out to examine functional connectivity as well as graph (topological) measures and their association to symptoms in first-episode psychosis patients during movie viewing. Our goal was to understand whole-brain functional dynamics of complex naturalistic information processing in psychosis and changes in brain functional organization related to symptoms. METHODS: 71 first-episode psychosis patients and 57 control subjects watched scenes from the movie Alice in Wonderland during 3 T fMRI. We compared functional connectivity and graph measures indicating integration, segregation and centrality between groups, and examined the association between topology and symptom scores in the patient group. RESULTS: We identified a subnetwork with predominantly decreased links of functional connectivity in first-episode psychosis patients. The subnetwork was mainly comprised of nodes of and links between the cingulo-opercular, sensorimotor and default-mode networks. In topological measures, we observed between-group differences in properties of centrality. CONCLUSIONS: Functional brain networks are affected during naturalistic information processing already in the early stages of psychosis, concentrated in salience- and cognitive control-related hubs and subnetworks. Understanding these aberrant dynamics could add to better targeted cognitive and behavioral interventions in the early stages of psychotic disorders.

Time-varying measures of cerebral network centrality correlate with visual saliency during movie watching.

The extensive development of graph-theoretic analysis for functional connectivity has revealed the multifaceted characteristics of brain networks. Network centralities identify the principal functional regions, individual differences, and hub structure in brain networks. Neuroimaging studies using movie-watching have investigated brain function under naturalistic stimuli. Visual saliency is one of the promising measures for revealing cognition and emotions driven by naturalistic stimuli. This study investigated whether the visual saliency in movies was associated with network centrality. The study examined eigenvector centrality (EC), which is a measure of a region's influence in the brain network, and the participation coefficient (PC), which reflects the hub structure in the brain, was used for comparison. Static and time-varying EC and PC were analyzed by a parcel-based technique. While EC was correlated with brain activity in parcels in the visual and auditory areas during movie-watching, it was only correlated with parcels in the visual areas in the retinotopy task. In addition, high PC was consistently observed in parcels in the putative hub both during the tasks and the resting-state condition. Time-varying EC in the parietal parcels and time-varying PC in the primary sensory parcels significantly correlated with visual saliency in the movies. These results suggest that time-varying centralities in brain networks are distinctively associated with perceptual processing and subsequent higher processing of visual saliency.

A studyforrest extension, an annotation of spoken language in the German dubbed movie "Forrest Gump" and its audio-description.

Here we present an annotation of speech in the audio-visual movie "Forrest Gump" and its audio-description for a visually impaired audience, as an addition to a large public functional brain imaging dataset ( studyforrest.org). The annotation provides information about the exact timing of each of the more than 2500 spoken sentences, 16,000 words (including 202 non-speech vocalizations), 66,000 phonemes, and their corresponding speaker. Additionally, for every word, we provide lemmatization, a simple part-of-speech-tagging (15 grammatical categories), a detailed part-of-speech tagging (43 grammatical categories), syntactic dependencies, and a semantic analysis based on word embedding which represents each word in a 300-dimensional semantic space. To validate the dataset's quality, we build a model of hemodynamic brain activity based on information drawn from the annotation. Results suggest that the annotation's content and quality enable independent researchers to create models of brain activity correlating with a variety of linguistic aspects under conditions of near-real-life complexity.

Semantics-weighted lexical surprisal modeling of naturalistic functional MRI time-series during spoken narrative listening.

Probabilistic language models are increasingly used to provide neural representations of linguistic features under naturalistic settings. Word surprisal models can be applied to continuous fMRI recordings during task-free listening of narratives, to detect regions linked to language prediction and comprehension. Here, to this purpose, a novel semantics-weighted lexical surprisal is applied to naturalistic fMRI data. FMRI was performed at 3 Tesla in 31 subjects during task-free listening to a 12-minute audiobook played in both original and word-reversed (control) version. Lexical-only and semantics-weighted lexical surprisal models were estimated for the original and control word series. The two series were alternatively chosen to build the predictor of interest in the first-level general linear model and were compared in the second-level (group) analysis. The addition of the surprisal predictor to the stimulus-related predictors significantly improved the fitting of the neural signal. In average, the semantics-weighted model yielded lower surprisal values and, in some areas, better fitting of the fMRI data compared to the lexical-only model. The two models produced both overlapping and distinct activations: while lexical-only surprisal activated secondary auditory areas in the superior temporal gyri and the cerebellum, semantics-weighted surprisal additionally activated the left inferior frontal gyrus. These results confirm the usefulness of surprisal models in the naturalistic fMRI analysis of linguistic processes and suggest that the use of semantic information may increase the sensitivity of a probabilistic language model in higher-order language-related areas, with possible implications for future naturalistic fMRI studies of language under normal and (clinically or pharmacologically) modified conditions.

Connectivity of the precuneus-posterior cingulate cortex with the anterior cingulate cortex-medial prefrontal cortex differs consistently between control subjects and first-episode psychosis patients during a movie stimulus.

BACKGROUND: Functional connectivity is altered in psychotic disorders. Multiple findings concentrate on the default mode network, anchored on the precuneus-posterior cingulate cortex (PC-PCC). However, the nature of the alterations varies between studies and connectivity alterations have not been studied during an ecologically valid natural stimulus. In the present study, we investigated the functional and structural connectivity of a PC-PCC region, where functioning differentiated first-episode psychosis patients from control subjects during free viewing of a movie in our earlier study. METHODS: 14 first-episode psychosis patients and 12 control subjects were imaged with GE 3T, and 29 patients and 19 control subjects were imaged with a Siemens Skyra 3T scanner while watching scenes from the movie Alice in Wonderland. Group differences in functional connectivity were analysed for both scanners separately and results were compared to identify any overlap. Diffusion tensor measures of 26 patients and 19 control subjects were compared for the related white matter tracts, identified by deterministic tractography. RESULTS: Functional connectivity was increased in patients across scanners between the midline regions of the PC-PCC and the anterior cingulate cortex-medial prefrontal cortex (ACC-mPFC). We found no group differences in any of the diffusion tensor imaging measures. CONCLUSIONS: Already in the early stages of psychosis functional connectivity between the midline structures of the PC-PCC and the ACC-mPFC is consistently increased during naturalistic stimulus.

Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento.

How does the human brain recall and connect relevant memories with unfolding events? To study this, we presented 25 healthy subjects, during functional magnetic resonance imaging, the movie 'Memento' (director C. Nolan). In this movie, scenes are presented in chronologically reverse order with certain scenes briefly overlapping previously presented scenes. Such overlapping "key-frames" serve as effective memory cues for the viewers, prompting recall of relevant memories of the previously seen scene and connecting them with the concurrent scene. We hypothesized that these repeating key-frames serve as immediate recall cues and would facilitate reconstruction of the story piece-by-piece. The chronological version of Memento, shown in a separate experiment for another group of subjects, served as a control condition. Using multivariate event-related pattern analysis method and representational similarity analysis, focal fingerprint patterns of hemodynamic activity were found to emerge during presentation of key-frame scenes. This effect was present in higher-order cortical network with regions including precuneus, angular gyrus, cingulate gyrus, as well as lateral, superior, and middle frontal gyri within frontal poles. This network was right hemispheric dominant. These distributed patterns of brain activity appear to underlie ability to recall relevant memories and connect them with ongoing events, i.e., "what goes with what" in a complex story. Given the real-life likeness of cinematic experience, these results provide new insight into how the human brain recalls, given proper cues, relevant memories to facilitate understanding and prediction of everyday life events.

Representation of tactile scenes in the rodent barrel cortex.

After half a century of research, the sensory features coded by neurons of the rodent barrel cortex remain poorly understood. Still, views of the sensory representation of whisker information are increasingly shifting from a labeled line representation of single-whisker deflections to a selectivity for specific elements of the complex statistics of the multi-whisker deflection patterns that take place during spontaneous rodent behavior - so called natural tactile scenes. Here we review the current knowledge regarding the coding of patterns of whisker stimuli by barrel cortex neurons, from responses to single-whisker deflections to the representation of complex tactile scenes. A number of multi-whisker tunings have already been identified, including center-surround feature extraction, angular tuning during edge-like multi-whisker deflections, and even tuning to specific statistical properties of the tactile scene such as the level of correlation across whiskers. However, a more general model of the representation of multi-whisker information in the barrel cortex is still missing. This is in part because of the lack of a human intuition regarding the perception emerging from a whisker system, but also because in contrast to other primary sensory cortices such as the visual cortex, the spatial feature selectivity of barrel cortex neurons rests on highly nonlinear interactions that remained hidden to classical receptive field approaches.

Precuneus functioning differentiates first-episode psychosis patients during the fantasy movie Alice in Wonderland.

BACKGROUND: While group-level functional alterations have been identified in many brain regions of psychotic patients, multivariate machine-learning methods provide a tool to test whether some of such alterations could be used to differentiate an individual patient. Earlier machine-learning studies have focused on data collected from chronic patients during rest or simple tasks. We set out to unravel brain activation patterns during naturalistic stimulation in first-episode psychosis (FEP). METHOD: We recorded brain activity from 46 FEP patients and 32 control subjects viewing scenes from the fantasy film Alice in Wonderland. Scenes with varying degrees of fantasy were selected based on the distortion of the 'sense of reality' in psychosis. After cleaning the data with a novel maxCorr method, we used machine learning to classify patients and healthy control subjects on the basis of voxel- and time-point patterns. RESULTS: Most (136/194) of the voxels that best classified the groups were clustered in a bilateral region of the precuneus. Classification accuracies were up to 79.5% (p = 5.69 × 10-8), and correct classification was more likely the higher the patient's positive-symptom score. Precuneus functioning was related to the fantasy content of the movie, and the relationship was stronger in control subjects than patients. CONCLUSIONS: These findings are the first to show abnormalities in precuneus functioning during naturalistic information processing in FEP patients. Correlational findings suggest that these alterations are associated with positive psychotic symptoms and processing of fantasy. The results may provide new insights into the neuronal basis of reality distortion in psychosis.

A versatile software package for inter-subject correlation based analyses of fMRI.

In the inter-subject correlation (ISC) based analysis of the functional magnetic resonance imaging (fMRI) data, the extent of shared processing across subjects during the experiment is determined by calculating correlation coefficients between the fMRI time series of the subjects in the corresponding brain locations. This implies that ISC can be used to analyze fMRI data without explicitly modeling the stimulus and thus ISC is a potential method to analyze fMRI data acquired under complex naturalistic stimuli. Despite of the suitability of ISC based approach to analyze complex fMRI data, no generic software tools have been made available for this purpose, limiting a widespread use of ISC based analysis techniques among neuroimaging community. In this paper, we present a graphical user interface (GUI) based software package, ISC Toolbox, implemented in Matlab for computing various ISC based analyses. Many advanced computations such as comparison of ISCs between different stimuli, time window ISC, and inter-subject phase synchronization are supported by the toolbox. The analyses are coupled with re-sampling based statistical inference. The ISC based analyses are data and computation intensive and the ISC toolbox is equipped with mechanisms to execute the parallel computations in a cluster environment automatically and with an automatic detection of the cluster environment in use. Currently, SGE-based (Oracle Grid Engine, Son of a Grid Engine, or Open Grid Scheduler) and Slurm environments are supported. In this paper, we present a detailed account on the methods behind the ISC Toolbox, the implementation of the toolbox and demonstrate the possible use of the toolbox by summarizing selected example applications. We also report the computation time experiments both using a single desktop computer and two grid environments demonstrating that parallelization effectively reduces the computing time. The ISC Toolbox is available in https://code.google.com/p/isc-toolbox/

From Vivaldi to Beatles and back: predicting lateralized brain responses to music.

We aimed at predicting the temporal evolution of brain activity in naturalistic music listening conditions using a combination of neuroimaging and acoustic feature extraction. Participants were scanned using functional Magnetic Resonance Imaging (fMRI) while listening to two musical medleys, including pieces from various genres with and without lyrics. Regression models were built to predict voxel-wise brain activations which were then tested in a cross-validation setting in order to evaluate the robustness of the hence created models across stimuli. To further assess the generalizability of the models we extended the cross-validation procedure by including another dataset, which comprised continuous fMRI responses of musically trained participants to an Argentinean tango. Individual models for the two musical medleys revealed that activations in several areas in the brain belonging to the auditory, limbic, and motor regions could be predicted. Notably, activations in the medial orbitofrontal region and the anterior cingulate cortex, relevant for self-referential appraisal and aesthetic judgments, could be predicted successfully. Cross-validation across musical stimuli and participant pools helped identify a region of the right superior temporal gyrus, encompassing the planum polare and the Heschl's gyrus, as the core structure that processed complex acoustic features of musical pieces from various genres, with or without lyrics. Models based on purely instrumental music were able to predict activation in the bilateral auditory cortices, parietal, somatosensory, and left hemispheric primary and supplementary motor areas. The presence of lyrics on the other hand weakened the prediction of activations in the left superior temporal gyrus. Our results suggest spontaneous emotion-related processing during naturalistic listening to music and provide supportive evidence for the hemispheric specialization for categorical sounds with realistic stimuli. We herewith introduce a powerful means to predict brain responses to music, speech, or soundscapes across a large variety of contexts.