Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation.

Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.

Neural dynamics of semantic composition.

Human speech comprehension is remarkable for its immediacy and rapidity. The listener interprets an incrementally delivered auditory input, millisecond by millisecond as it is heard, in terms of complex multilevel representations of relevant linguistic and nonlinguistic knowledge. Central to this process are the neural computations involved in semantic combination, whereby the meanings of words are combined into more complex representations, as in the combination of a verb and its following direct object (DO) noun (e.g., "eat the apple"). These combinatorial processes form the backbone for incremental interpretation, enabling listeners to integrate the meaning of each word as it is heard into their dynamic interpretation of the current utterance. Focusing on the verb-DO noun relationship in simple spoken sentences, we applied multivariate pattern analysis and computational semantic modeling to source-localized electro/magnetoencephalographic data to map out the specific representational constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of subsequent words in the utterance. Comparing context-independent semantic models of the DO noun with contextually constrained noun models reflecting the semantic properties of the preceding verb, we found that only the contextually constrained model showed a significant fit to the brain data. Pattern-based measures of directed connectivity across the left hemisphere language network revealed a continuous information flow among temporal, inferior frontal, and inferior parietal regions, underpinning the verb's modification of the DO noun's activated semantics. These results provide a plausible neural substrate for seamless real-time incremental interpretation on the observed millisecond time scales.

Balancing Prediction and Sensory Input in Speech Comprehension: The Spatiotemporal Dynamics of Word Recognition in Context.

Spoken word recognition in context is remarkably fast and accurate, with recognition times of ∼200 ms, typically well before the end of the word. The neurocomputational mechanisms underlying these contextual effects are still poorly understood. This study combines source-localized electroencephalographic and magnetoencephalographic (EMEG) measures of real-time brain activity with multivariate representational similarity analysis to determine directly the timing and computational content of the processes evoked as spoken words are heard in context, and to evaluate the respective roles of bottom-up and predictive processing mechanisms in the integration of sensory and contextual constraints. Male and female human participants heard simple (modifier-noun) English phrases that varied in the degree of semantic constraint that the modifier (W1) exerted on the noun (W2), as in pairs, such as "yellow banana." We used gating tasks to generate estimates of the probabilistic predictions generated by these constraints as well as measures of their interaction with the bottom-up perceptual input for W2. Representation similarity analysis models of these measures were tested against electroencephalographic and magnetoencephalographic brain data across a bilateral fronto-temporo-parietal language network. Consistent with probabilistic predictive processing accounts, we found early activation of semantic constraints in frontal cortex (LBA45) as W1 was heard. The effects of these constraints (at 100 ms after W2 onset in left middle temporal gyrus and at 140 ms in left Heschl's gyrus) were only detectable, however, after the initial phonemes of W2 had been heard. Within an overall predictive processing framework, bottom-up sensory inputs are still required to achieve early and robust spoken word recognition in context.SIGNIFICANCE STATEMENT Human listeners recognize spoken words in natural speech contexts with remarkable speed and accuracy, often identifying a word well before all of it has been heard. In this study, we investigate the brain systems that support this important capacity, using neuroimaging techniques that can track real-time brain activity during speech comprehension. This makes it possible to locate the brain areas that generate predictions about upcoming words and to show how these expectations are integrated with the evidence provided by the speech being heard. We use the timing and localization of these effects to provide the most specific account to date of how the brain achieves an optimal balance between prediction and sensory input in the interpretation of spoken language.

Predicting the Time Course of Individual Objects with MEG.

To respond appropriately to objects, we must process visual inputs rapidly and assign them meaning. This involves highly dynamic, interactive neural processes through which information accumulates and cognitive operations are resolved across multiple time scales. However, there is currently no model of object recognition which provides an integrated account of how visual and semantic information emerge over time; therefore, it remains unknown how and when semantic representations are evoked from visual inputs. Here, we test whether a model of individual objects--based on combining the HMax computational model of vision with semantic-feature information--can account for and predict time-varying neural activity recorded with magnetoencephalography. We show that combining HMax and semantic properties provides a better account of neural object representations compared with the HMax alone, both through model fit and classification performance. Our results show that modeling and classifying individual objects is significantly improved by adding semantic-feature information beyond ∼200 ms. These results provide important insights into the functional properties of visual processing across time.

Optimally efficient neural systems for processing spoken language.

Cognitive models claim that spoken words are recognized by an optimally efficient sequential analysis process. Evidence for this is the finding that nonwords are recognized as soon as they deviate from all real words (Marslen-Wilson 1984), reflecting continuous evaluation of speech inputs against lexical representations. Here, we investigate the brain mechanisms supporting this core aspect of word recognition and examine the processes of competition and selection among multiple word candidates. Based on new behavioral support for optimal efficiency in lexical access from speech, a functional magnetic resonance imaging study showed that words with later nonword points generated increased activation in the left superior and middle temporal gyrus (Brodmann area [BA] 21/22), implicating these regions in dynamic sound-meaning mapping. We investigated competition and selection by manipulating the number of initially activated word candidates (competition) and their later drop-out rate (selection). Increased lexical competition enhanced activity in bilateral ventral inferior frontal gyrus (BA 47/45), while increased lexical selection demands activated bilateral dorsal inferior frontal gyrus (BA 44/45). These findings indicate functional differentiation of the fronto-temporal systems for processing spoken language, with left middle temporal gyrus (MTG) and superior temporal gyrus (STG) involved in mapping sounds to meaning, bilateral ventral inferior frontal gyrus (IFG) engaged in less constrained early competition processing, and bilateral dorsal IFG engaged in later, more fine-grained selection processes.

From perception to conception: how meaningful objects are processed over time.

To recognize visual objects, our sensory perceptions are transformed through dynamic neural interactions into meaningful representations of the world but exactly how visual inputs invoke object meaning remains unclear. To address this issue, we apply a regression approach to magnetoencephalography data, modeling perceptual and conceptual variables. Key conceptual measures were derived from semantic feature-based models claiming shared features (e.g., has eyes) provide broad category information, while distinctive features (e.g., has a hump) are additionally required for more specific object identification. Our results show initial perceptual effects in visual cortex that are rapidly followed by semantic feature effects throughout ventral temporal cortex within the first 120 ms. Moreover, these early semantic effects reflect shared semantic feature information supporting coarse category-type distinctions. Post-200 ms, we observed the effects along the extent of ventral temporal cortex for both shared and distinctive features, which together allow for conceptual differentiation and object identification. By relating spatiotemporal neural activity to statistical feature-based measures of semantic knowledge, we demonstrate that qualitatively different kinds of perceptual and semantic information are extracted from visual objects over time, with rapid activation of shared object features followed by concomitant activation of distinctive features that together enable meaningful visual object recognition.

The Centre for Speech, Language and the Brain (CSLB) concept property norms.

Theories of the representation and processing of concepts have been greatly enhanced by models based on information available in semantic property norms. This information relates both to the identity of the features produced in the norms and to their statistical properties. In this article, we introduce a new and large set of property norms that are designed to be a more flexible tool to meet the demands of many different disciplines interested in conceptual knowledge representation, from cognitive psychology to computational linguistics. As well as providing all features listed by 2 or more participants, we also show the considerable linguistic variation that underlies each normalized feature label and the number of participants who generated each variant. Our norms are highly comparable with the largest extant set (McRae, Cree, Seidenberg, & McNorgan, 2005) in terms of the number and distribution of features. In addition, we show how the norms give rise to a coherent category structure. We provide these norms in the hope that the greater detail available in the Centre for Speech, Language and the Brain norms should further promote the development of models of conceptual knowledge. The norms can be downloaded at www.csl.psychol.cam.ac.uk/propertynorms.

Objects and categories: feature statistics and object processing in the ventral stream.

Recognizing an object involves more than just visual analyses; its meaning must also be decoded. Extensive research has shown that processing the visual properties of objects relies on a hierarchically organized stream in ventral occipitotemporal cortex, with increasingly more complex visual features being coded from posterior to anterior sites culminating in the perirhinal cortex (PRC) in the anteromedial temporal lobe (aMTL). The neurobiological principles of the conceptual analysis of objects remain more controversial. Much research has focused on two neural regions-the fusiform gyrus and aMTL, both of which show semantic category differences, but of different types. fMRI studies show category differentiation in the fusiform gyrus, based on clusters of semantically similar objects, whereas category-specific deficits, specifically for living things, are associated with damage to the aMTL. These category-specific deficits for living things have been attributed to problems in differentiating between highly similar objects, a process that involves the PRC. To determine whether the PRC and the fusiform gyri contribute to different aspects of an object's meaning, with differentiation between confusable objects in the PRC and categorization based on object similarity in the fusiform, we carried out an fMRI study of object processing based on a feature-based model that characterizes the degree of semantic similarity and difference between objects and object categories. Participants saw 388 objects for which feature statistic information was available and named the objects at the basic level while undergoing fMRI scanning. After controlling for the effects of visual information, we found that feature statistics that capture similarity between objects formed category clusters in fusiform gyri, such that objects with many shared features (typical of living things) were associated with activity in the lateral fusiform gyri whereas objects with fewer shared features (typical of nonliving things) were associated with activity in the medial fusiform gyri. Significantly, a feature statistic reflecting differentiation between highly similar objects, enabling object-specific representations, was associated with bilateral PRC activity. These results confirm that the statistical characteristics of conceptual object features are coded in the ventral stream, supporting a conceptual feature-based hierarchy, and integrating disparate findings of category responses in fusiform gyri and category deficits in aMTL into a unifying neurocognitive framework.

Bihemispheric foundations for human speech comprehension.

Emerging evidence from neuroimaging and neuropsychology suggests that human speech comprehension engages two types of neurocognitive processes: a distributed bilateral system underpinning general perceptual and cognitive processing, viewed as neurobiologically primary, and a more specialized left hemisphere system supporting key grammatical language functions, likely to be specific to humans. To test these hypotheses directly we covaried increases in the nonlinguistic complexity of spoken words [presence or absence of an embedded stem, e.g., claim (clay)] with variations in their linguistic complexity (presence of inflectional affixes, e.g., play+ed). Nonlinguistic complexity, generated by the on-line competition between the full word and its onset-embedded stem, was found to activate both right and left fronto-temporal brain regions, including bilateral BA45 and -47. Linguistic complexity activated left-lateralized inferior frontal areas only, primarily in BA45. This contrast reflects a differentiation between the functional roles of a bilateral system, which supports the basic mapping from sound to lexical meaning, and a language-specific left-lateralized system that supports core decompositional and combinatorial processes invoked by linguistically complex inputs. These differences can be related to the neurobiological foundations of human language and underline the importance of bihemispheric systems in supporting the dynamic processing and interpretation of spoken inputs.

Age-related neural reorganization during spoken word recognition: the interaction of form and meaning.

Research on language and aging typically shows that language comprehension is preserved across the life span. Recent neuroimaging results suggest that this good performance is underpinned by age-related neural reorganization [e.g., Tyler, L. K., Shafto, M. A., Randall, B., Wright, P., Marslen-Wilson, W. D., & Stamatakis, E. A. Preserving syntactic processing across the adult life span: The modulation of the frontotemporal language system in the context of age-related atrophy. Cerebral Cortex, 20, 352-364, 2010]. The current study examines how age-related reorganization affects the balance between component linguistic processes by manipulating semantic and phonological factors during spoken word recognition in younger and older adults. Participants in an fMRI study performed an auditory lexical decision task where words varied in their phonological and semantic properties as measured by degree of phonological competition and imageability. Older adults had a preserved lexicality effect, but compared with younger people, their behavioral sensitivity to phonological competition was reduced, as was competition-related activity in left inferior frontal gyrus. This was accompanied by increases in behavioral sensitivity to imageability and imageability-related activity in left middle temporal gyrus. These results support previous findings that neural compensation underpins preserved comprehension in aging and demonstrate that neural reorganization can affect the balance between semantic and phonological processing.

Left inferior frontal cortex and syntax: function, structure and behaviour in patients with left hemisphere damage.

For the past 150 years, neurobiological models of language have debated the role of key brain regions in language function. One consistently debated set of issues concern the role of the left inferior frontal gyrus in syntactic processing. Here we combine measures of functional activity, grey matter integrity and performance in patients with left hemisphere damage and healthy participants to ask whether the left inferior frontal gyrus is essential for syntactic processing. In a functional neuroimaging study, participants listened to spoken sentences that either contained a syntactically ambiguous or matched unambiguous phrase. Behavioural data on three tests of syntactic processing were subsequently collected. In controls, syntactic processing co-activated left hemisphere Brodmann areas 45/47 and posterior middle temporal gyrus. Activity in a left parietal cluster was sensitive to working memory demands in both patients and controls. Exploiting the variability in lesion location and performance in the patients, voxel-based correlational analyses showed that tissue integrity and neural activity-primarily in left Brodmann area 45 and posterior middle temporal gyrus-were correlated with preserved syntactic performance, but unlike the controls, patients were insensitive to syntactic preferences, reflecting their syntactic deficit. These results argue for the essential contribution of the left inferior frontal gyrus in syntactic analysis and highlight the functional relationship between left Brodmann area 45 and the left posterior middle temporal gyrus, suggesting that when this relationship breaks down, through damage to either region or to the connections between them, syntactic processing is impaired. On this view, the left inferior frontal gyrus may not itself be specialized for syntactic processing, but plays an essential role in the neural network that carries out syntactic computations.

Dissociating linguistic and task-related activity in the left inferior frontal gyrus.

The left inferior frontal gyrus (LIFG) has long been claimed to play a key role in language function. However, there is considerable controversy as to whether regions within LIFG have specific linguistic or domain-general functions. Using fMRI, we contrasted linguistic and task-related effects by presenting simple and morphologically complex words while subjects performed a lexical decision (LD) task or passively listened (PL) without making an overt response. LIFG Brodmann's area 47 showed greater activation in LD than PL, whereas LIFG Brodmann's area 44 showed greater activation to complex compared with simple words in both tasks. These results dissociate task-driven and obligatory language processing in LIFG and suggest that PL is the paradigm of choice for probing the core aspects of the neural language system.

The interaction of lexical semantics and cohort competition in spoken word recognition: an fMRI study.

Spoken word recognition involves the activation of multiple word candidates on the basis of the initial speech input--the "cohort"--and selection among these competitors. Selection may be driven primarily by bottom-up acoustic-phonetic inputs or it may be modulated by other aspects of lexical representation, such as a word's meaning [Marslen-Wilson, W. D. Functional parallelism in spoken word-recognition. Cognition, 25, 71-102, 1987]. We examined these potential interactions in an fMRI study by presenting participants with words and pseudowords for lexical decision. In a factorial design, we manipulated (a) cohort competition (high/low competitive cohorts which vary the number of competing word candidates) and (b) the word's semantic properties (high/low imageability). A previous behavioral study [Tyler, L. K., Voice, J. K., & Moss, H. E. The interaction of meaning and sound in spoken word recognition. Psychonomic Bulletin & Review, 7, 320-326, 2000] showed that imageability facilitated word recognition but only for words in high competition cohorts. Here we found greater activity in the left inferior frontal gyrus (BA 45, 47) and the right inferior frontal gyrus (BA 47) with increased cohort competition, an imageability effect in the left posterior middle temporal gyrus/angular gyrus (BA 39), and a significant interaction between imageability and cohort competition in the left posterior superior temporal gyrus/middle temporal gyrus (BA 21, 22). In words with high competition cohorts, high imageability words generated stronger activity than low imageability words, indicating a facilitatory role of imageability in a highly competitive cohort context. For words in low competition cohorts, there was no effect of imageability. These results support the behavioral data in showing that selection processes do not rely solely on bottom-up acoustic-phonetic cues but rather that the semantic properties of candidate words facilitate discrimination between competitors.

Reorganization of syntactic processing following left-hemisphere brain damage: does right-hemisphere activity preserve function?

The extent to which the human brain shows evidence of functional plasticity across the lifespan has been addressed in the context of pathological brain changes and, more recently, of the changes that take place during healthy ageing. Here we examine the potential for plasticity by asking whether a strongly left-lateralized system can successfully reorganize to the right-hemisphere following left-hemisphere brain damage. To do this, we focus on syntax, a key linguistic function considered to be strongly left-lateralized, combining measures of tissue integrity, neural activation and behavioural performance. In a functional neuroimaging study participants heard spoken sentences that differentially loaded on syntactic and semantic information. While healthy controls activated a left-hemisphere network of correlated activity including Brodmann areas 45/47 and posterior middle temporal gyrus during syntactic processing, patients activated Brodmann areas 45/47 bilaterally and right middle temporal gyrus. However, voxel-based morphometry analyses showed that only tissue integrity in left Brodmann areas 45/47 was correlated with activity and performance; poor tissue integrity in left Brodmann area 45 was associated with reduced functional activity and increased syntactic deficits. Activity in the right-hemisphere was not correlated with damage in the left-hemisphere or with performance. Reduced neural integrity in the left-hemisphere through brain damage or healthy ageing results in increased right-hemisphere activation in homologous regions to those left-hemisphere regions typically involved in the young. However, these regions do not support the same linguistic functions as those in the left-hemisphere and only indirectly contribute to preserved syntactic capacity. This establishes the unique role of the left hemisphere in syntax, a core component in human language.

Preserving syntactic processing across the adult life span: the modulation of the frontotemporal language system in the context of age-related atrophy.

Although widespread neural atrophy is an inevitable consequence of normal aging, not all cognitive abilities decline as we age. For example, spoken language comprehension tends to be preserved, despite atrophy in neural regions involved in language function. Here, we combined measures of behavior, functional activation, and gray matter (GM) change in a younger (19-34 years) and older group (49-86 years) of participants to identify the mechanisms leading to preserved language comprehension across the adult life span. We focussed primarily on syntactic functions because these are strongly left lateralized, providing the potential for contralateral recruitment. In an functional magnetic resonance imaging study, we used a word-monitoring task to minimize working memory demands, manipulating the availability of semantics and syntax to ask whether syntax is preserved in aging because of the functional recruitment of other brain regions, which successfully compensate for neural atrophy. Performance in the older group was preserved despite GM loss. This preservation was related to increased activity in right hemisphere frontotemporal regions, which was associated with age-related atrophy in the left hemisphere frontotemporal network activated in the young. We argue that preserved syntactic processing across the life span is due to the shift from a primarily left hemisphere frontotemporal system to a bilateral functional language network.

The processing of English regular inflections: Phonological cues to morphological structure.

Previous studies suggest that different neural and functional mechanisms are involved in the analysis of irregular (caught) and regular (filled) past tense forms in English. In particular, the comprehension and production of regular forms is argued to require processes of morpho-phonological assembly and disassembly, analysing these forms into a stem plus an inflectional affix (e.g., {fill}+{-ed}), as opposed to irregular forms, which do not have an overt stem+affix structure and must be analysed as full forms [Marslen-Wilson, W. D., & Tyler, L. K. (1997). Dissociating types of mental computation. Nature, 387, 592-594; Marslen-Wilson, W. D., & Tyler, L. K. (1998). Rules, representations, and the English past tense. Trends in Cognitive Science, 2, 428-435]. On this account, any incoming string that shows the critical diagnostic properties of an inflected form - a final coronal consonant (/t/, /d/, /s/, /z/) that agrees in voicing with the preceding segment as in filled, mild, or nilled - will automatically trigger an attempt at segmentation. We report an auditory speeded judgment experiment which explored the contribution of these critical morpho-phonological properties (labelled as the English inflectional rhyme pattern) to the processing of English regular inflections. The results show that any stimulus that can be interpreted as ending in a regular inflection, whether it is a real inflection (filled-fill), a pseudo-inflection (mild-mile) or a phonologically matched nonword (nilled-nill), is responded to more slowly than an unambiguously monomorphemic stimulus pair (e.g., belt-bell). This morpho-phonological effect was independent of phonological effects of voicing and syllabicity. The findings are interpreted as evidence for a basic morpho-phonological parsing process that applies to all items with the criterial phonological properties.

Clarifying the nature of the distinctiveness by domain interaction in conceptual structure: comment on Cree, McNorgan, and McRae (2006).

The conceptual structure account of semantic memory (CSA; L. K. Tyler & H. E. Moss, 2001) claims that feature correlation (the degree to which features co-occur) and feature distinctiveness (the number of concepts in which a feature occurs) interact with domains of knowledge (e.g., living vs. nonliving) such that the distinctive features of nonliving things are more highly correlated than the distinctive features of living things. Evidence for (B. Randall, H. E. Moss, J. M. Rodd, M. Greer, & L. K. Tyler, 2004) and against this claim (G. S. Cree, C. McNorgan, & K. McRae, 2006) has been reported. This comment outlines the CSA, discusses Cree et al.'s (2006) critiques of the Randall et al. (2004) experiments and the CSA, and reports new analyses of property norm and behavioral data, which replicate the results reported by Randall et al. (2004).

Feature Statistics Modulate the Activation of Meaning During Spoken Word Processing.

Understanding spoken words involves a rapid mapping from speech to conceptual representations. One distributed feature-based conceptual account assumes that the statistical characteristics of concepts' features--the number of concepts they occur in (distinctiveness/sharedness) and likelihood of co-occurrence (correlational strength)--determine conceptual activation. To test these claims, we investigated the role of distinctiveness/sharedness and correlational strength in speech-to-meaning mapping, using a lexical decision task and computational simulations. Responses were faster for concepts with higher sharedness, suggesting that shared features are facilitatory in tasks like lexical decision that require access to them. Correlational strength facilitated responses for slower participants, suggesting a time-sensitive co-occurrence-driven settling mechanism. The computational simulation showed similar effects, with early effects of shared features and later effects of correlational strength. These results support a general-to-specific account of conceptual processing, whereby early activation of shared features is followed by the gradual emergence of a specific target representation.

Temporal and frontal systems in speech comprehension: an fMRI study of past tense processing.

A prominent issue in cognitive neuroscience is whether language function is instantiated in the brain as a single undifferentiated process, or whether regions of relative specialisation can be demonstrated. The contrast between regular and irregular English verb inflection has been pivotal to this debate. Behavioural dissociations related to different lesion sites in brain-damaged patients suggest that processing regular and irregular past tenses involves different neural systems. Using event-related fMRI in a group of unimpaired young adults, we contrast processing of spoken regular and irregular past tense forms in a same-different judgement task, shown in earlier research with patients to engage left hemisphere language systems. An extensive fronto-temporal network, linking anterior cingulate (ACC), left inferior frontal cortex (LIFC) and bilateral superior temporal gyrus (STG), was preferentially activated for regularly inflected forms. Access to meaning from speech is supported by temporal cortex, but additional processing is required for forms that end in regular inflections, which differentially engage LIFC processes that support morpho-phonological segmentation and grammatical analysis.

The perirhinal cortex and conceptual processing: Effects of feature-based statistics following damage to the anterior temporal lobes.

The anterior temporal lobe (ATL) plays a prominent role in models of semantic knowledge, although it remains unclear how the specific subregions within the ATL contribute to semantic memory. Patients with neurodegenerative diseases, like semantic dementia, have widespread damage to the ATL thus making inferences about the relationship between anatomy and cognition problematic. Here we take a detailed anatomical approach to ask which substructures within the ATL contribute to conceptual processing, with the prediction that the perirhinal cortex (PRc) will play a critical role for concepts that are more semantically confusable. We tested two patient groups, those with and without damage to the PRc, across two behavioural experiments - picture naming and word-picture matching. For both tasks, we manipulated the degree of semantic confusability of the concepts. By contrasting the performance of the two groups, along with healthy controls, we show that damage to the PRc results in worse performance in processing concepts with higher semantic confusability across both experiments. Further by correlating the degree of damage across anatomically defined regions of interest with performance, we find that PRc damage is related to performance for concepts with increased semantic confusability. Our results show that the PRc supports a necessary and crucial neurocognitve function that enables fine-grained conceptual processes to take place through the resolution of semantic confusability.