Imagining Sounds and Images: Decoding the Contribution of Unimodal and Transmodal Brain Regions to Semantic Retrieval in the Absence of Meaningful Input.

In the absence of sensory information, we can generate meaningful images and sounds from representations in memory. However, it remains unclear which neural systems underpin this process and whether tasks requiring the top-down generation of different kinds of features recruit similar or different neural networks. We asked people to internally generate the visual and auditory features of objects, either in isolation (car, dog) or in specific and complex meaning-based contexts (car/dog race). Using an fMRI decoding approach, in conjunction with functional connectivity analysis, we examined the role of auditory/visual cortex and transmodal brain regions. Conceptual retrieval in the absence of external input recruited sensory and transmodal cortex. The response in transmodal regions-including anterior middle temporal gyrus-was of equal magnitude for visual and auditory features yet nevertheless captured modality information in the pattern of response across voxels. In contrast, sensory regions showed greater activation for modality-relevant features in imagination (even when external inputs did not differ). These data are consistent with the view that transmodal regions support internally generated experiences and that they play a role in integrating perceptual features encoded in memory.

Distant from input: Evidence of regions within the default mode network supporting perceptually-decoupled and conceptually-guided cognition.

The default mode network supports a variety of mental operations such as semantic processing, episodic memory retrieval, mental time travel and mind-wandering, yet the commonalities between these functions remains unclear. One possibility is that this system supports cognition that is independent of the immediate environment; alternatively or additionally, it might support higher-order conceptual representations that draw together multiple features. We tested these accounts using a novel paradigm that separately manipulated the availability of perceptual information to guide decision-making and the representational complexity of this information. Using task based imaging we established regions that respond when cognition combines both stimulus independence with multi-modal information. These included left and right angular gyri and the left middle temporal gyrus. Although these sites were within the default mode network, they showed a stronger response to demanding memory judgements than to an easier perceptual task, contrary to the view that they support automatic aspects of cognition. In a subsequent analysis, we showed that these regions were located at the extreme end of a macroscale gradient, which describes gradual transitions from sensorimotor to transmodal cortex. This shift in the focus of neural activity towards transmodal, default mode, regions might reflect a process of where the functional distance from specific sensory enables conceptually rich and detailed cognitive states to be generated in the absence of input.

Fractionating the anterior temporal lobe: MVPA reveals differential responses to input and conceptual modality.

Words activate cortical regions in accordance with their modality of presentation (i.e., written vs. spoken), yet there is a long-standing debate about whether patterns of activity in any specific brain region capture modality-invariant conceptual information. Deficits in patients with semantic dementia highlight the anterior temporal lobe (ATL) as an amodal store of semantic knowledge but these studies do not permit precise localisation of this function. The current investigation used multiple imaging methods in healthy participants to examine functional dissociations within ATL. Multi-voxel pattern analysis identified spatially segregated regions: a response to input modality in anterior superior temporal gyrus (aSTG) and a response to meaning in more ventral anterior temporal lobe (vATL). This functional dissociation was supported by resting-state connectivity that found greater coupling for aSTG with primary auditory cortex and vATL with the default mode network. A meta-analytic decoding of these connectivity patterns implicated aSTG in processes closely tied to auditory processing (such as phonology and language) and vATL in meaning-based tasks (such as comprehension or social cognition). Thus we provide converging evidence for the segregation of meaning and input modality in the ATL.

"I know something you don't know": Discourse and social context effects on the N400 in adolescents.

Adolescence is a time of great cognitive and social development. Despite this, relatively few studies to date have investigated how perspective taking affects on-line language comprehension in adolescents. In the current study, we addressed this gap in the literature, making use of a Joint Comprehension Task in which two individuals with differing background knowledge jointly attend to linguistic stimuli. Using event-related potentials, we investigated adolescents' electrophysiological responses to (a) semantically anomalous sentence stimuli in discourse context and (b) semantically plausible sentence stimuli that the participants believe another individual finds semantically implausible. Our results demonstrate that a robust "N400 effect" (i.e., a well-established event-related potential, known to be sensitive to lexical-semantic integration difficulties)  is elicited by semantically anomalous sentences; this N400 effect is subsequently attenuated by discourse context. Lastly, a "social N400 effect" is elicited by sentences that are semantically plausible for the participants if they believe that another individual finds the sentences implausible. The results suggest that adolescents integrate the perspective of others during on-line language comprehension via simulation; that is, adolescents use their own language processing system to interpret language input from the perspective of other jointly attending individuals.

Oscillatory neuronal activity reflects lexical-semantic feature integration within and across sensory modalities in distributed cortical networks.

Research from the previous decade suggests that word meaning is partially stored in distributed modality-specific cortical networks. However, little is known about the mechanisms by which semantic content from multiple modalities is integrated into a coherent multisensory representation. Therefore we aimed to characterize differences between integration of lexical-semantic information from a single modality compared with two sensory modalities. We used magnetoencephalography in humans to investigate changes in oscillatory neuronal activity while participants verified two features for a given target word (e.g., "bus"). Feature pairs consisted of either two features from the same modality (visual: "red," "big") or different modalities (auditory and visual: "red," "loud"). The results suggest that integrating modality-specific features of the target word is associated with enhanced high-frequency power (80-120 Hz), while integrating features from different modalities is associated with a sustained increase in low-frequency power (2-8 Hz). Source reconstruction revealed a peak in the anterior temporal lobe for low-frequency and high-frequency effects. These results suggest that integrating lexical-semantic knowledge at different cortical scales is reflected in frequency-specific oscillatory neuronal activity in unisensory and multisensory association networks.

Observing, performing, and understanding actions: revisiting the role of cortical motor areas in processing of action words.

Language content and action/perception have been shown to activate common brain areas in previous neuroimaging studies. However, it is unclear whether overlapping cortical activation reflects a common neural source or adjacent, but distinct, sources. We address this issue by using multivoxel pattern analysis on fMRI data. Specifically, participants were instructed to engage in five tasks: (1) execute hand actions (AE), (2) observe hand actions (AO), (3) observe nonbiological motion (MO), (4) read action verbs, and (5) read nonaction verbs. A classifier was trained to distinguish between data collected from neural motor areas during (1) AE versus MO and (2) AO versus MO. These two algorithms were then used to test for a distinction between data collected during the reading of action versus nonaction verbs. The results show that the algorithm trained to distinguish between AE and MO distinguishes between word categories using signal recorded from the left parietal cortex and pre-SMA, but not from ventrolateral premotor cortex. In contrast, the algorithm trained to distinguish between AO and MO discriminates between word categories using the activity pattern in the left premotor and left parietal cortex. This shows that the sensitivity of premotor areas to language content is more similar to the process of observing others acting than to acting oneself. Furthermore, those parts of the brain that show comparable neural pattern for action execution and action word comprehension are high-level integrative motor areas rather than low-level motor areas.

Pragmatics in action: indirect requests engage theory of mind areas and the cortical motor network.

Research from the past decade has shown that understanding the meaning of words and utterances (i.e., abstracted symbols) engages the same systems we used to perceive and interact with the physical world in a content-specific manner. For example, understanding the word "grasp" elicits activation in the cortical motor network, that is, part of the neural substrate involved in planned and executing a grasping action. In the embodied literature, cortical motor activation during language comprehension is thought to reflect motor simulation underlying conceptual knowledge [note that outside the embodied framework, other explanations for the link between action and language are offered, e.g., Mahon, B. Z., & Caramazza, A. A critical look at the embodied cognition hypothesis and a new proposal for grouding conceptual content. Journal of Physiology, 102, 59-70, 2008; Hagoort, P. On Broca, brain, and binding: A new framework. Trends in Cognitive Sciences, 9, 416-423, 2005]. Previous research has supported the view that the coupling between language and action is flexible, and reading an action-related word form is not sufficient for cortical motor activation [Van Dam, W. O., van Dijk, M., Bekkering, H., & Rueschemeyer, S.-A. Flexibility in embodied lexical-semantic representations. Human Brain Mapping, doi: 10.1002/hbm.21365, 2011]. The current study goes one step further by addressing the necessity of action-related word forms for motor activation during language comprehension. Subjects listened to indirect requests (IRs) for action during an fMRI session. IRs for action are speech acts in which access to an action concept is required, although it is not explicitly encoded in the language. For example, the utterance "It is hot here!" in a room with a window is likely to be interpreted as a request to open the window. However, the same utterance in a desert will be interpreted as a statement. The results indicate (1) that comprehension of IR sentences activates cortical motor areas reliably more than comprehension of sentences devoid of any implicit motor information. This is true despite the fact that IR sentences contain no lexical reference to action. (2) Comprehension of IR sentences also reliably activates substantial portions of the theory of mind network, known to be involved in making inferences about mental states of others. The implications of these findings for embodied theories of language are discussed.

Context-dependent changes in functional connectivity of auditory cortices during the perception of object words.

Embodied theories hold that cognitive concepts are grounded in our sensorimotor systems. Specifically, a number of behavioral and neuroimaging studies have buttressed the idea that language concepts are represented in areas involved in perception and action [Pulvermueller, F. Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6, 576-582, 2005; Barsalou, L. W. Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577-660, 1999]. Proponents of a strong embodied account argue that activity in perception/action areas is triggered automatically upon encountering a word and reflect static semantic representations. In contrast to what would be expected if lexical semantic representations are automatically triggered upon encountering a word, a number of studies failed to find motor-related activity for words with a putative action-semantic component [Raposo, A., Moss, H. E., Stamatakis, E. A., & Tyler, L. K. Modulation of motor and premotor cortices by actions, action words and action sentences. Neuropsychologia, 47, 388-396, 2009; Rueschemeyer, S.-A., Brass, M., & Friederici, A. D. Comprehending prehending: Neural correlates of processing verbs with motor stems. Journal of Cognitive Neuroscience, 19, 855-865, 2007]. In a recent fMRI study, Van Dam and colleagues [Van Dam, W. O., Van Dijk, M., Bekkering, H., & Rueschemeyer, S.-A. Flexibility in embodied lexical-semantic representations. Human Brain Mapping, in press] showed that the degree to which a modality-specific region contributes to a representation considerably changes as a function of context. In the current study, we presented words for which both motor and visual properties (e.g., tennis ball, boxing glove) were important in constituting the concept. Our aim was to corroborate on earlier findings of flexible and context-dependent language representations by testing whether functional integration between auditory brain regions and perception/action areas is modulated by context. Functional connectivity was investigated by means of a psychophysiological interaction analysis, in which we found that bilateral superior temporal gyrus was more strongly connected with brain regions relevant for coding action information: (1) for Action Color words vs. Abstract words, and (2) for Action Color words presented in a context that emphasized action vs. a context that emphasized color properties.

Flexibility in embodied lexical-semantic representations.

According to an embodied view of language comprehension, language concepts are grounded in our perceptual systems. Evidence for the idea that concepts are grounded in areas involved in action and perception comes from both behavioral and neuroimaging studies (Glenberg [1997]: Behav Brain Sci 20:1-55; Barsalou [1999]: Behav Brain Sci 22:577-660; Pulvermueller [1999]: Behav Brain Sci 22:253-336; Barsalou et al. [2003]: Trends Cogn Sci 7:84-91). However, the results from several studies indicate that the activation of information in perception and action areas is not a purely automatic process (Raposo et al. [2009]: Neuropsychologia 47:388-396; Rueschemeyer et al. [2007]: J Cogn Neurosci 19:855-865). These findings suggest that embodied representations are flexible. In these studies, flexibility is characterized by the relative presence or absence of activation in our perceptual systems. However, even if the context in which a word is presented does not undermine a motor interpretation, it is possible that the degree to which a modality-specific region contributes to a representation depends on the context in which conceptual features are retrieved. In the present study, we investigated this issue by presenting word stimuli for which both motor and visual properties (e.g., Tennis ball, Boxing glove) were important in constituting the concept. Conform with the idea that language representations are flexible and context dependent, we demonstrate that the degree to which a modality-specific region contributes to a representation considerably changes as a function of context.

The function of words: distinct neural correlates for words denoting differently manipulable objects.

Recent research indicates that language processing relies on brain areas dedicated to perception and action. For example, processing words denoting manipulable objects has been shown to activate a fronto-parietal network involved in actual tool use. This is suggested to reflect the knowledge the subject has about how objects are moved and used. However, information about how to use an object may be much more central to the conceptual representation of an object than information about how to move an object. Therefore, there may be much more fine-grained distinctions between objects on the neural level, especially related to the usability of manipulable objects. In the current study, we investigated whether a distinction can be made between words denoting (1) objects that can be picked up to move (e.g., volumetrically manipulable objects: bookend, clock) and (2) objects that must be picked up to use (e.g., functionally manipulable objects: cup, pen). The results show that functionally manipulable words elicit greater levels of activation in the fronto-parietal sensorimotor areas than volumetrically manipulable words. This suggests that indeed a distinction can be made between different types of manipulable objects. Specifically, how an object is used functionally rather than whether an object can be displaced with the hand is reflected in semantic representations in the brain.

How specifically are action verbs represented in the neural motor system: an fMRI study.

Embodied accounts of language processing suggest that sensorimotor areas, generally dedicated to perception and action, are also involved in the processing and representation of word meaning. Support for such accounts comes from studies showing that language about actions selectively modulates the execution of congruent and incongruent motor responses (e.g., Glenberg & Kaschak, 2002), and from functional neuroimaging studies showing that understanding action-related language recruits sensorimotor brain areas (e.g. Hauk, Johnsrude, & Pulvermueller, 2004). In the current experiment we explored the basis of the neural motor system's involvement in representing words denoting actions. Specifically, we investigated whether the motor system's involvement is modulated by the specificity of the kinematics associated with a word. Previous research in the visual domain indicates that words denoting basic level category members lacking a specific form (e.g., bird) are less richly encoded within visual areas than words denoting subordinate level members (e.g., pelican), for which the visual form is better specified (Gauthier, Anderson, Tarr, Skudlarski, & Gore, 1997). In the present study we extend these findings to the motor domain. Modulation of the BOLD response elicited by verbs denoting a general motor program (e.g., to clean) was compared to modulation elicited by verbs denoting a more specific motor program (e.g., to wipe). Conform with our hypothesis, a region within the bilateral inferior parietal lobule, typically serving the representation of action plans and goals, was sensitive to the specificity of motor programs associated with the action verbs. These findings contribute to the growing body of research on embodied language representations by showing that the concreteness of an action-semantic feature is reflected in the neural response to action verbs.

Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions.

Iconic gestures are spontaneous hand movements that illustrate certain contents of speech and, as such, are an important part of face-to-face communication. This experiment targets the brain bases of how iconic gestures and speech are integrated during comprehension. Areas of integration were identified on the basis of two classic properties of multimodal integration, bimodal enhancement and inverse effectiveness (i.e., greater enhancement for unimodally least effective stimuli). Participants underwent fMRI while being presented with videos of gesture-supported sentences as well as their unimodal components, which allowed us to identify areas showing bimodal enhancement. Additionally, we manipulated the signal-to-noise ratio of speech (either moderate or good) to probe for integration areas exhibiting the inverse effectiveness property. Bimodal enhancement was found at the posterior end of the superior temporal sulcus and adjacent superior temporal gyrus (pSTS/STG) in both hemispheres, indicating that the integration of iconic gestures and speech takes place in these areas. Furthermore, we found that the left pSTS/STG specifically showed a pattern of inverse effectiveness, i.e., the neural enhancement for bimodal stimulation was greater under adverse listening conditions. This indicates that activity in this area is boosted when an iconic gesture accompanies an utterance that is otherwise difficult to comprehend. The neural response paralleled the behavioral data observed. The present data extends results from previous gesture-speech integration studies in showing that pSTS/STG plays a key role in the facilitation of speech comprehension through simultaneous gestural input.

Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials.

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (n = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain's electrophysiological index of semantic processing. A spatio-temporally fine-grained mixed-effect multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatio-temporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Activation of the language control network in bilingual visual word recognition.

Research into bilingual language production has identified a language control network that subserves control operations when bilinguals produce speech. Here we explore which brain areas are recruited for control purposes in bilingual language comprehension. In two experimental fMRI sessions, Dutch-English unbalanced bilinguals read words that differed in cross-linguistic form and meaning overlap across their two languages. The need for control operations was further manipulated by varying stimulus list composition across the two experimental sessions. We observed activation of the language control network in bilingual language comprehension as a function of both cross-linguistic form and meaning overlap and stimulus list composition. These findings suggest that the language control network is shared across bilingual language production and comprehension. We argue that activation of the language control network in language comprehension allows bilinguals to quickly and efficiently grasp the context-relevant meaning of words.

Bound together: Social binding leads to faster processing, spatial distortion, and enhanced memory of interacting partners.

The binding of features into perceptual wholes is a well-established phenomenon, which has previously only been studied in the context of early vision and low-level features, such as color or proximity. We hypothesized that a similar binding process, based on higher level information, could bind people into interacting groups, facilitating faster processing and enhanced memory of social situations. To investigate this possibility we used 3 experimental approaches to explore grouping effects in displays involving interacting people. First, using a visual search task we demonstrate more rapid processing for interacting (vs. noninteracting) pairs in an odd-quadrant paradigm (Experiments 1a and 1b). Second, using a spatial judgment task, we show that interacting individuals are remembered as physically closer than are noninteracting individuals (Experiments 2a and 2b). Finally, we show that memory retention of group-relevant and irrelevant features are enhanced when recalling interacting partners in a surprise memory task (Experiments 3a and 3b). Each of these results is consistent with the social binding hypothesis, and alternative explanations based on low level perceptual features and attentional effects are ruled out. We conclude that automatic midlevel grouping processes bind individuals into groups on the basis of their perceived interaction. Such social binding could provide the basis for more sophisticated social processing. Identifying the automatic encoding of social interactions in visual search, distortions of spatial working memory, and facilitated retrieval of object properties from longer-term memory, opens new approaches to studying social cognition with possible practical applications. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Large-scale replication study reveals a limit on probabilistic prediction in language comprehension.

Do people routinely pre-activate the meaning and even the phonological form of upcoming words? The most acclaimed evidence for phonological prediction comes from a 2005 Nature Neuroscience publication by DeLong, Urbach and Kutas, who observed a graded modulation of electrical brain potentials (N400) to nouns and preceding articles by the probability that people use a word to continue the sentence fragment ('cloze'). In our direct replication study spanning 9 laboratories (N=334), pre-registered replication-analyses and exploratory Bayes factor analyses successfully replicated the noun-results but, crucially, not the article-results. Pre-registered single-trial analyses also yielded a statistically significant effect for the nouns but not the articles. Exploratory Bayesian single-trial analyses showed that the article-effect may be non-zero but is likely far smaller than originally reported and too small to observe without very large sample sizes. Our results do not support the view that readers routinely pre-activate the phonological form of predictable words.

Neuronal interactions between mentalising and action systems during indirect request processing.

Human communication relies on the ability to process linguistic structure and to map words and utterances onto our environment. Furthermore, as what we communicate is often not directly encoded in our language (e.g. in the case of irony, jokes or indirect requests), we need to extract additional cues to infer the beliefs and desires of our conversational partners. Although the functional interplay between language and the ability to mentalise has been discussed in theoretical accounts in the past, the neurobiological underpinnings of these dynamics are currently not well understood. Here, we address this issue using functional imaging (fMRI). Participants listened to question-reply dialogues. In these dialogues, a reply is interpreted as a direct reply, an indirect reply or a request for action, depending on the question. We show that inferring meaning from indirect replies engages parts of the mentalising network (mPFC) while requests for action also activate the cortical motor system (IPL). Subsequent connectivity analysis using Dynamic Causal Modelling (DCM) revealed that this pattern of activation is best explained by an increase in effective connectivity from the mentalising network (mPFC) to the action system (IPL). These results are an important step towards a more integrative understanding of the neurobiological basis of indirect speech processing.

(How) observed eye-contact modulates gaze following. An fMRI study.

Humans are highly sensitive to directional gaze cues and rapidly shift attention in accordance with others' gaze (i.e., gaze following). Besides providing information about the physical environment, for instance, the location of an object, gaze direction can be used to extract information about the social environment, such as whether or not two people are interacting with each other. In the present fMRI study we investigated how these two different types of information conveyed by gaze direction interact with one another. Participants saw two faces that were either looking at each other or away from each other before jointly shifting gaze toward one of two target locations. Targets either appeared at the gazed at or the non-gazed at location. Behaviorally, gaze following (faster responses to congruent versus incongruent trials) was more prominent after observing eye contact than after observing no eye contact. In line with behavioral findings, neuroimaging results revealed enhanced activation in fronto-parietal and temporal areas in congruent trials when faces had looked at each other versus away from each other. These findings demonstrate that observing an attentional relation between others augments processing of their subsequent gaze cues.

The Social N400 effect: how the presence of other listeners affects language comprehension.

During conversation, it is necessary to keep track of what others can and cannot understand. Previous research has focused largely on understanding the time course along which knowledge about interlocutors influences language comprehension/production rather than the cognitive process by which interlocutors take each other's perspective. In addition, most work has looked at the effects of knowledge about a speaker on a listener's comprehension, and not on the possible effects of other listeners on a participant's comprehension process. In the current study, we introduce a novel joint comprehension paradigm that addresses the cognitive processes underlying perspective taking during language comprehension. Specifically, we show that participants who understand a language stimulus, but are simultaneously aware that someone sitting next to them does not understand the same stimulus, show an electrophysiological marker of semantic integration difficulty (i.e., an N400-effect). Crucially, in a second group of participants, we demonstrate that presenting exactly the same sentences to the participant alone (i.e. without a co-listener) results in no N400-effect. Our results suggest that (1) information about co-listeners as well as the speaker affect language comprehension, and (2) the cognitive process by which we understand what others comprehend mirrors our own language comprehension processes.

Feature activation during word recognition: action, visual, and associative-semantic priming effects.

Embodied theories of language postulate that language meaning is stored in modality-specific brain areas generally involved in perception and action in the real world. However, the temporal dynamics of the interaction between modality-specific information and lexical-semantic processing remain unclear. We investigated the relative timing at which two types of modality-specific information (action-based and visual-form information) contribute to lexical-semantic comprehension. To this end, we applied a behavioral priming paradigm in which prime and target words were related with respect to (1) action features, (2) visual features, or (3) semantically associative information. Using a Go/No-Go lexical decision task, priming effects were measured across four different inter-stimulus intervals (ISI = 100, 250, 400, and 1000 ms) to determine the relative time course of the different features. Notably, action priming effects were found in ISIs of 100, 250, and 1000 ms whereas a visual priming effect was seen only in the ISI of 1000 ms. Importantly, our data suggest that features follow different time courses of activation during word recognition. In this regard, feature activation is dynamic, measurable in specific time windows but not in others. Thus the current study (1) demonstrates how multiple ISIs can be used within an experiment to help chart the time course of feature activation and (2) provides new evidence for embodied theories of language.