Audiovisual integration in the human brain: a coordinate-based meta-analysis.

People can seamlessly integrate a vast array of information from what they see and hear in the noisy and uncertain world. However, the neural underpinnings of audiovisual integration continue to be a topic of debate. Using strict inclusion criteria, we performed an activation likelihood estimation meta-analysis on 121 neuroimaging experiments with a total of 2,092 participants. We found that audiovisual integration is linked with the coexistence of multiple integration sites, including early cortical, subcortical, and higher association areas. Although activity was consistently found within the superior temporal cortex, different portions of this cortical region were identified depending on the analytical contrast used, complexity of the stimuli, and modality within which attention was directed. The context-dependent neural activity related to audiovisual integration suggests a flexible rather than fixed neural pathway for audiovisual integration. Together, our findings highlight a flexible multiple pathways model for audiovisual integration, with superior temporal cortex as the central node in these neural assemblies.

Evoked and induced power oscillations linked to audiovisual integration of affect.

Our affective experiences are influenced by combined multisensory information. Although the enhanced effects of congruent audiovisual information on our affective experiences have been well documented, the role of neural oscillations in the audiovisual integration of affective signals remains unclear. First, it is unclear whether oscillatory activity changes as a function of valence. Second, the function of phase-locked and non-phase-locked power changes in audiovisual integration of affect has not yet been clearly distinguished. To fill this gap, the present study performed time-frequency analyses on EEG data acquired while participants perceived positive, neutral and negative naturalistic video and music clips. A comparison between the congruent audiovisual condition and the sum of unimodal conditions was used to identify supra-additive (Audiovisual > Visual + Auditory) or sub-additive (Audiovisual < Visual + Auditory) integration effects. The results showed that early evoked sub-additive theta and sustained induced supra-additive delta and beta activities are linked to audiovisual integration of affect regardless of affective content.

Identification of task sets within and across stimulus modalities.

Previous studies have shown that task sets can be identified from functional magnetic resonance imaging (fMRI) data. However, these results may be partially confounded by differences in stimulus features associated with the different tasks. We disentangle stimulus modality and task features by presenting the same stimulus while varying task, and conversely, presenting different stimuli using the same task. Analyses were conducted on fMRI data previously collected on twenty participants who made either affective or semantic judgements of the same music pieces or the same silent video clips (Kim et al., 2017). Holding stimuli constant, task set was identified from fMRI data across individuals from both task activation data and functional connectivity data. Thus, we were able to identify whether participants made affective or semantic judgments when exposed to identical stimuli based on the task activation and functional connectivity data from other participants. Moreover, task set was successfully identified for cross-modal prediction in which stimuli in the training set bore no resemblance to those in the test set (e.g., using videos data to predict task for music data). Brain regions that were sensitive to tasks irrespective of sensory modality were identified by univariate and searchlight analyses of fMRI data. Consistent with a frontal-parietal network, middle frontal gyrus, inferior parietal gyrus, mid-cingulate cortex, and superior temporal sulcus were found to be key regions distinguishing the two task sets.

Modelling audiovisual integration of affect from videos and music.

Two experiments examined how affective values from visual and auditory modalities are integrated. Experiment 1 paired music and videos drawn from three levels of valence while holding arousal constant. Experiment 2 included a parallel combination of three levels of arousal while holding valence constant. In each experiment, participants rated their affective states after unimodal and multimodal presentations. Experiment 1 revealed a congruency effect in which stimulus combinations of the same extreme valence resulted in more extreme state ratings than component stimuli presented in isolation. An interaction between music and video valence reflected the greater influence of negative affect. Video valence was found to have a significantly greater effect on combined ratings than music valence. The pattern of data was explained by a five parameter differential weight averaging model that attributed greater weight to the visual modality and increased weight with decreasing values of valence. Experiment 2 revealed a congruency effect only for high arousal combinations and no interaction effects. This pattern was explained by a three parameter constant weight averaging model with greater weight for the auditory modality and a very low arousal value for the initial state. These results demonstrate key differences in audiovisual integration between valence and arousal.

Intrinsic brain activity as a diagnostic biomarker in children with benign epilepsy with centrotemporal spikes.

Benign epilepsy with centrotemporal spikes (BECTS) is often associated with neural circuit dysfunction, particularly during the transient active state characterized by interictal epileptiform discharges (IEDs). Little is known, however, about the functional neural circuit abnormalities in BECTS without IEDs, or if such abnormalities could be used to differentiate BECTS patients without IEDs from healthy controls (HCs) for early diagnosis. To this end, we conducted resting-state functional magnetic resonance imaging (RS-fMRI) and simultaneous Electroencephalogram (EEG) in children with BECTS (n = 43) and age-matched HC (n = 28). The simultaneous EEG recordings distinguished BECTS with IEDs (n = 20) from without IEDs (n = 23). Intrinsic brain activity was measured in all three groups using the amplitude of low frequency fluctuation at rest. Compared to HC, BECTS patients with IEDs exhibited an intrinsic activity abnormality in the thalamus, suggesting that thalamic dysfunction could contribute to IED emergence while patients without IEDs exhibited intrinsic activity abnormalities in middle frontal gyrus and superior parietal gyrus. Using multivariate pattern classification analysis, we were able to differentiate BECTS without IEDs from HCs with 88.23% accuracy. BECTS without epileptic transients can be distinguished from HC and BECTS with IEDs by unique regional abnormalities in resting brain activity. Both transient abnormalities as reflected by IEDs and chronic abnormalities as reflected by RS-fMRI may contribute to BECTS development and expression. Intrinsic brain activity and multivariate pattern classification techniques are promising tools to diagnose and differentiate BECTS syndromes. Hum Brain Mapp 36:3878-3889, 2015. © 2015 Wiley Periodicals, Inc.

Representations of modality-specific affective processing for visual and auditory stimuli derived from functional magnetic resonance imaging data.

There is converging evidence that people rapidly and automatically encode affective dimensions of objects, events, and environments that they encounter in the normal course of their daily routines. An important research question is whether affective representations differ with sensory modality. This research examined the nature of the dependency of affect and sensory modality at a whole-brain level of analysis in an incidental affective processing paradigm. Participants were presented with picture and sound stimuli that differed in positive or negative valence in an event-related functional magnetic resonance imaging experiment. Global statistical tests, applied at a level of the individual, demonstrated significant sensitivity to valence within modality, but not valence across modalities. Modality-general and modality-specific valence hypotheses predict distinctly different multidimensional patterns of the stimulus conditions. Examination of lower dimensional representation of the data demonstrated separable dimensions for valence processing within each modality. These results provide support for modality-specific valence processing in an incidental affective processing paradigm at a whole-brain level of analysis. Future research should further investigate how stimulus-specific emotional decoding may be mediated by the physical properties of the stimuli.

Neural representation of abstract and concrete concepts: a meta-analysis of neuroimaging studies.

A number of studies have investigated differences in neural correlates of abstract and concrete concepts with disagreement across results. A quantitative, coordinate-based meta-analysis combined data from 303 participants across 19 functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) studies to identify the differences in neural representation of abstract and concrete concepts. Studies that reported peak activations in standard space in contrast of abstract > concrete or concrete > abstract concepts at a whole brain level in healthy adults were included in this meta-analysis. Multilevel kernel density analysis (MKDA) was performed to identify the proportion of activated contrasts weighted by sample size and analysis type (fixed or random effects). Meta-analysis results indicated consistent and meaningful differences in neural representation for abstract and concrete concepts. Abstract concepts elicit greater activity in the inferior frontal gyrus and middle temporal gyrus compared to concrete concepts, while concrete concepts elicit greater activity in the posterior cingulate, precuneus, fusiform gyrus, and parahippocampal gyrus compared to abstract concepts. These results suggest greater engagement of the verbal system for processing of abstract concepts and greater engagement of the perceptual system for processing of concrete concepts, likely via mental imagery.