Listening beyond seeing: Event-related potentials to audiovisual processing in visual narrative.

Every day we integrate meaningful information coming from different sensory modalities, and previous work has debated whether conceptual knowledge is represented in modality-specific neural stores specialized for specific types of information, and/or in an amodal, shared system. In the current study, we investigated semantic processing through a cross-modal paradigm which asked whether auditory semantic processing could be modulated by the constraints of context built up across a meaningful visual narrative sequence. We recorded event-related brain potentials (ERPs) to auditory words and sounds associated to events in visual narratives-i.e., seeing images of someone spitting while hearing either a word (Spitting!) or a sound (the sound of spitting)-which were either semantically congruent or incongruent with the climactic visual event. Our results showed that both incongruent sounds and words evoked an N400 effect, however, the distribution of the N400 effect to words (centro-parietal) differed from that of sounds (frontal). In addition, words had an earlier latency N400 than sounds. Despite these differences, a sustained late frontal negativity followed the N400s and did not differ between modalities. These results support the idea that semantic memory balances a distributed cortical network accessible from multiple modalities, yet also engages amodal processing insensitive to specific modalities.

tDCS application over the STG improves the ability to recognize and appreciate elements involved in humor processing.

The superior temporal gyrus (STG) has been found to play a crucial role in the recognition of actions and facial expressions and may, therefore, be critical for the processing of humorous information. Here we investigated whether tDCS application to the STG would modulate the ability to recognize and appreciate the comic element in serious and comedic situations of misfortune. To this aim, the effects of different types of tDCS stimulation on the STG were analyzed during a task in which the participants were instructed to categorize various misfortunate situations as "comic" or "not comic". Participants underwent three different tDCS conditions: Anodal-right/Cathodal-left; Cathodal-right/Anodal-left; Sham. Images depicting people involved in accidents were grouped into three categories based on the facial expression of the victim: angry or painful (Affective); bewildered and funny (Comic); and images that did not contain the victim's face (No Face). An improvement in mean reaction times in response to both the Comic and No Face stimuli was observed following Anodal-left/Cathodal-right stimulation when compared to sham stimulation. This suggests that this stimulation type reduced the reaction times to socio-emotional complex scenes, regardless of facial expression. The Anodal-right/Cathodal-left stimulation reduced the mean reaction times for Comic stimuli only, suggesting that specifically the right STG may be involved in facial expression recognition and in the appreciation of the comic element in misfortunate situations. These results suggest a functional hemispheric asymmetry in STG response to social stimuli: the left STG might have a role in a general comprehension of social complex situations, while the right STG may be involved in the ability to recognize and integrate specific emotional aspects in a complex scene.