In Spoken Word Recognition, the Future Predicts the Past.

Speech is an inherently noisy and ambiguous signal. To fluently derive meaning, a listener must integrate contextual information to guide interpretations of the sensory input. Although many studies have demonstrated the influence of prior context on speech perception, the neural mechanisms supporting the integration of subsequent context remain unknown. Using MEG to record from human auditory cortex, we analyzed responses to spoken words with a varyingly ambiguous onset phoneme, the identity of which is later disambiguated at the lexical uniqueness point. Fifty participants (both male and female) were recruited across two MEG experiments. Our findings suggest that primary auditory cortex is sensitive to phonological ambiguity very early during processing at just 50 ms after onset. Subphonemic detail is preserved in auditory cortex over long timescales and re-evoked at subsequent phoneme positions. Commitments to phonological categories occur in parallel, resolving on the shorter timescale of ∼450 ms. These findings provide evidence that future input determines the perception of earlier speech sounds by maintaining sensory features until they can be integrated with top-down lexical information.SIGNIFICANCE STATEMENT The perception of a speech sound is determined by its surrounding context in the form of words, sentences, and other speech sounds. Often, such contextual information becomes available later than the sensory input. The present study is the first to unveil how the brain uses this subsequent information to aid speech comprehension. Concretely, we found that the auditory system actively maintains the acoustic signal in auditory cortex while concurrently making guesses about the identity of the words being said. Such a processing strategy allows the content of the message to be accessed quickly while also permitting reanalysis of the acoustic signal to minimize parsing mistakes.

Lexical Preactivation in Basic Linguistic Phrases.

Many previous studies have shown that predictable words are read faster and lead to reduced neural activation, consistent with a model of reading in which words are activated in advance of being encountered. The nature of such preactivation, however, has typically been studied indirectly through its subsequent effect on word recognition. Here, we use magnetoencephalography to study the dynamics of prediction within serially presented adjective-noun phrases, beginning at the point at which the predictive information is first available to the reader. Using corpus transitional probability to estimate the predictability of a noun, we found an increase in activity in the left middle temporal gyrus in response to the presentation of highly predictive adjectives (i.e., adjectives that license a strong noun prediction). Moreover, we found that adjective predictivity and expected noun frequency interacted, such that the response to the highly predictive adjectives (e.g., stainless) was modulated by the frequency of the expected noun (steel). These results likely reflect preactivation of nouns in highly predictive contexts. The fact that the preactivation process was modulated by the frequency of the predicted item is argued to provide support for a frequency-sensitive lexicon.

Hierarchical dynamic coding coordinates speech comprehension in the brain.

Speech comprehension requires the human brain to transform an acoustic waveform into meaning. To do so, the brain generates a hierarchy of features that converts the sensory input into increasingly abstract language properties. However, little is known about how these hierarchical features are generated and continuously coordinated. Here, we propose that each linguistic feature is dynamically represented in the brain to simultaneously represent successive events. To test this 'Hierarchical Dynamic Coding' (HDC) hypothesis, we use time-resolved decoding of brain activity to track the construction, maintenance, and integration of a comprehensive hierarchy of language features spanning acoustic, phonetic, sub-lexical, lexical, syntactic and semantic representations. For this, we recorded 21 participants with magnetoencephalography (MEG), while they listened to two hours of short stories. Our analyses reveal three main findings. First, the brain incrementally represents and simultaneously maintains successive features. Second, the duration of these representations depend on their level in the language hierarchy. Third, each representation is maintained by a dynamic neural code, which evolves at a speed commensurate with its corresponding linguistic level. This HDC preserves the maintenance of information over time while limiting the interference between successive features. Overall, HDC reveals how the human brain continuously builds and maintains a language hierarchy during natural speech comprehension, thereby anchoring linguistic theories to their biological implementations.

Introducing MEG-MASC a high-quality magneto-encephalography dataset for evaluating natural speech processing.

The "MEG-MASC" dataset provides a curated set of raw magnetoencephalography (MEG) recordings of 27 English speakers who listened to two hours of naturalistic stories. Each participant performed two identical sessions, involving listening to four fictional stories from the Manually Annotated Sub-Corpus (MASC) intermixed with random word lists and comprehension questions. We time-stamp the onset and offset of each word and phoneme in the metadata of the recording, and organize the dataset according to the 'Brain Imaging Data Structure' (BIDS). This data collection provides a suitable benchmark to large-scale encoding and decoding analyses of temporally-resolved brain responses to speech. We provide the Python code to replicate several validations analyses of the MEG evoked responses such as the temporal decoding of phonetic features and word frequency. All code and MEG, audio and text data are publicly available to keep with best practices in transparent and reproducible research.

Neural dynamics of phoneme sequences reveal position-invariant code for content and order.

Speech consists of a continuously-varying acoustic signal. Yet human listeners experience it as sequences of discrete speech sounds, which are used to recognise discrete words. To examine how the human brain appropriately sequences the speech signal, we recorded two-hour magnetoencephalograms from 21 participants listening to short narratives. Our analyses show that the brain continuously encodes the three most recently heard speech sounds in parallel, and maintains this information long past its dissipation from the sensory input. Each speech sound representation evolves over time, jointly encoding both its phonetic features and the amount of time elapsed since onset. As a result, this dynamic neural pattern encodes both the relative order and phonetic content of the speech sequence. These representations are active earlier when phonemes are more predictable, and are sustained longer when lexical identity is uncertain. Our results show how phonetic sequences in natural speech are represented at the level of populations of neurons, providing insight into what intermediary representations exist between the sensory input and sub-lexical units. The flexibility in the dynamics of these representations paves the way for further understanding of how such sequences may be used to interface with higher order structure such as lexical identity.

Early Form-Based Morphological Decomposition in Tagalog: MEG Evidence from Reduplication, Infixation, and Circumfixation.

Neuro- and psycholinguistic experimentation supports the early decomposition of morphologically complex words within the ventral processing stream, which MEG has localized to the M170 response in the (left) visual word form area (VWFA). Decomposition into an exhaustive parse of visual morpheme forms extends beyond words like farmer to those imitating complexity (e.g., brother; Lewis et al., 2011), and to "unique" stems occurring in only one word but following the syntax and semantics of their affix (e.g., vulnerable; Gwilliams & Marantz, 2018). Evidence comes primarily from suffixation; other morphological processes have been under-investigated. This study explores circumfixation, infixation, and reduplication in Tagalog. In addition to investigating whether these are parsed like suffixation, we address an outstanding question concerning semantically empty morphemes. Some words in Tagalog resemble English winter as decomposition is not supported (wint-er); these apparently reduplicated pseudoreduplicates lack the syntactic and semantic features of reduplicated forms. However, unlike winter, these words exhibit phonological behavior predicted only if they involve a reduplicating morpheme. If these are decomposed, this provides evidence that words are analyzed as complex, like English vulnerable, when the grammar demands it. In a lexical decision task with MEG, we find that VWFA activity correlates with stem:word transition probability for circumfixed, infixed, and reduplicated words. Furthermore, a Bayesian analysis suggests that pseudoreduplicates with reduplicate-like phonology are also decomposed; other pseudoreduplicates are not. These findings are consistent with an interpretation that decomposition is modulated by phonology in addition to syntax and semantics.

Memory for affixes in a long-lag priming paradigm.

Psycholinguistic research on the processing of morphologically complex words has largely focused on debates about how/if lexical stems are recognized, stored, and retrieved. Comparatively little processing research has investigated similar issues for functional affixes. In Word or Lexeme Based Morphology (Aronoff 1994), affixes are not representational units on par with stems or roots. This view is in stark contrast to the claims of linguistic theories like Distributed Morphology (Halle & Marantz 1993), which assign rich representational content to affixes. We conducted a series of eight visual lexical decision studies, evaluating effects of derivational affix priming along with stem priming, identity priming, form priming, and semantic priming at long and short lags. We find robust and consistent affix priming (but not semantic or form priming) with lags up to 33 items, supporting the position that affixes are morphemes, i.e., representational units on par with stems. Intriguingly, we find only weaker evidence for the long-lag stem priming effect found in other studies. We interpret this potential asymmetry in terms of the salience of different morphological contexts for recollection memory.

Left posterior temporal cortex is sensitive to syntax within conceptually matched Arabic expressions.

During language comprehension, the brain processes not only word meanings, but also the grammatical structure-the "syntax"-that strings words into phrases and sentences. Yet the neural basis of syntax remains contentious, partly due to the elusiveness of experimental designs that vary structure independently of meaning-related variables. Here, we exploit Arabic's grammatical properties, which enable such a design. We collected magnetoencephalography (MEG) data while participants read the same noun-adjective expressions with zero, one, or two contiguously-written definite articles (e.g., 'chair purple'; 'the-chair purple'; 'the-chair the-purple'), representing equivalent concepts, but with different levels of syntactic complexity (respectively, indefinite phrases: 'a purple chair'; sentences: 'The chair is purple.'; definite phrases: 'the purple chair'). We expected regions processing syntax to respond differently to simple versus complex structures. Single-word controls ('chair'/'purple') addressed definiteness-based accounts. In noun-adjective expressions, syntactic complexity only modulated activity in the left posterior temporal lobe (LPTL), ~ 300 ms after each word's onset: indefinite phrases induced more MEG-measured positive activity. The effects disappeared in single-word tokens, ruling out non-syntactic interpretations. In contrast, left anterior temporal lobe (LATL) activation was driven by meaning. Overall, the results support models implicating the LPTL in structure building and the LATL in early stages of conceptual combination.

Adaptation to mis-pronounced speech: evidence for a prefrontal-cortex repair mechanism.

Speech is a complex and ambiguous acoustic signal that varies significantly within and across speakers. Despite the processing challenge that such variability poses, humans adapt to systematic variations in pronunciation rapidly. The goal of this study is to uncover the neurobiological bases of the attunement process that enables such fluent comprehension. Twenty-four native English participants listened to words spoken by a "canonical" American speaker and two non-canonical speakers, and performed a word-picture matching task, while magnetoencephalography was recorded. Non-canonical speech was created by including systematic phonological substitutions within the word (e.g. [s] → [sh]). Activity in the auditory cortex (superior temporal gyrus) was greater in response to substituted phonemes, and, critically, this was not attenuated by exposure. By contrast, prefrontal regions showed an interaction between the presence of a substitution and the amount of exposure: activity decreased for canonical speech over time, whereas responses to non-canonical speech remained consistently elevated. Grainger causality analyses further revealed that prefrontal responses serve to modulate activity in auditory regions, suggesting the recruitment of top-down processing to decode non-canonical pronunciations. In sum, our results suggest that the behavioural deficit in processing mispronounced phonemes may be due to a disruption to the typical exchange of information between the prefrontal and auditory cortices as observed for canonical speech.

Evidence for early morphological decomposition in visual word recognition.

We employ a single-trial correlational MEG analysis technique to investigate early processing in the visual recognition of morphologically complex words. Three classes of affixed words were presented in a lexical decision task: free stems (e.g., taxable), bound roots (e.g., tolerable), and unique root words (e.g., vulnerable, the root of which does not appear elsewhere). Analysis was focused on brain responses within 100-200 msec poststimulus onset in the previously identified letter string and visual word-form areas. MEG data were analyzed using cortically constrained minimum-norm estimation. Correlations were computed between activity at functionally defined ROIs and continuous measures of the words' morphological properties. ROIs were identified across subjects on a reference brain and then morphed back onto each individual subject's brain (n = 9). We find evidence of decomposition for both free stems and bound roots at the M170 stage in processing. The M170 response is shown to be sensitive to morphological properties such as affix frequency and the conditional probability of encountering each word given its stem. These morphological properties are contrasted with orthographic form features (letter string frequency, transition probability from one string to the next), which exert effects on earlier stages in processing ( approximately 130 msec). We find that effects of decomposition at the M170 can, in fact, be attributed to morphological properties of complex words, rather than to purely orthographic and form-related properties. Our data support a model of word recognition in which decomposition is attempted, and possibly utilized, for complex words containing bound roots as well as free word-stems.

Brain/behavior Asymmetry in Schizophrenia: A MEG Study of Cross-modal Semantic Priming.

Semantic priming has long been used to investigate how concepts and ideas are related at the level of language, and has become a convenient tool for assessing conceptual and semantic dysfunction in cognitive disorders, including schizophrenia. The study of semantic priming in schizophrenia has led to diverse results: enhanced priming, reduced priming, and priming equivalent to that found in nonpsychiatric comparison groups. A number of hypotheses have been proposed to explain some of the observed deficits in schizophrenia patients. For example, difficulties in word recognition may be due to hyperactivation of too many lexical representations or to a failure to inhibit lexical competitors. One way to distinguish between these possible explanations is to move beyond reliance on behavior alone and to examine the neural processes involved in lexical recognition. Here we present a magnetoencephalographic study of semantic priming in schizophrenia. Importantly, schizophrenia patients and healthy controls did not differ in performance on a priming task. We show that normal behavioral performance can occur in a context of aberrant neural responses. These findings suggest that normal behavioral responses in schizophrenia can be achieved through neural mechanisms that differ from those seen in the psychiatrically well brain.

Modeling Human Morphological Competence.

One of the central debates in the cognitive science of language has revolved around the nature of human linguistic competence. Whether syntactic competence should be characterized by abstract hierarchical structures or reduced to surface linear strings has been actively debated, but the nature of morphological competence has been insufficiently appreciated despite the parallel question in the cognitive science literature. In this paper, in order to investigate whether morphological competence should be characterized by abstract hierarchical structures, we conducted a crowdsourced acceptability judgment experiment on morphologically complex words and evaluated five computational models of morphological competence against human acceptability judgments: Character Markov Models (Character), Syllable Markov Models (Syllable), Morpheme Markov Models (Morpheme), Hidden Markov Models (HMM), and Probabilistic Context-Free Grammars (PCFG). Our psycholinguistic experimentation and computational modeling demonstrated that "morphous" computational models with morpheme units outperformed "amorphous" computational models without morpheme units and, importantly, PCFG with hierarchical structures most accurately explained human acceptability judgments on several evaluation metrics, especially for morphologically complex words with nested morphological structures. Those results strongly suggest that human morphological competence should be characterized by abstract hierarchical structures internally generated by the grammar, not reduced to surface linear strings externally attested in large corpora.

Left occipital and right frontal involvement in syntactic category prediction: MEG evidence from Standard Arabic.

Though recent years have seen a growth in research on predictive processes in language comprehension, their scope and mechanisms remain partially elusive. While mechanisms involved in predicting specific words are relatively well understood, those underlying syntactic prediction are still unclear. In part, this is because of the difficulty in designing experiments that manipulate syntactic predictability while controlling other variables. In this MEG study, we achieved this with a manipulation of syntactic category predictability within fully well-formed expressions of Standard Arabic. Participants read sentences beginning with a subject-adjective context, in which the presence of at least one of two possible cues (gender-incongruity and/or an intervening relative pronoun) was sufficient for predicting a target word's syntactic category. Absence of both cues (i.e., congruent subject-adjective context with no relative pronoun) increased uncertainty about the target's syntactic category. Using source analysis, we compared activity evoked by targets with predictable and unpredictable categories in the occipital lobe. We found an interaction effect consistent with previous findings: in the primary visual cortex, an early evoked component (visual M100) is enhanced only when the syntactic category was unpredictable. We also compared responses to pre-target predictive and unpredictive contexts across five bilateral frontal and temporal regions. In the right-hemispheric frontal region, we found a temporal cluster (~230 ms after adjective onset), where unpredictive contexts elicited more activation than predictive contexts. By hypothesis elimination, we conclude that the most likely variable driving this effect is syntactic entropy. Our results show that predictive mechanisms recruited during reading also involve predicting upcoming syntactic categories, implicating at least two cortical regions: the left visual cortex and the right frontal cortex.

Prefix Stripping Re-Re-Revisited: MEG Investigations of Morphological Decomposition and Recomposition.

We revisit a long-standing question in the psycholinguistic and neurolinguistic literature on comprehending morphologically complex words: are prefixes and suffixes processed using the same cognitive mechanisms? Recent work using Magnetoencephalography (MEG) to uncover the dynamic temporal and spatial responses evoked by visually presented complex suffixed single words provide us with a comprehensive picture of morphological processing in the brain, from early, form-based decomposition, through lexical access, grammatically constrained recomposition, and semantic interpretation. In the present study, we find that MEG responses to prefixed words reveal interesting early differences in the lateralization of the form-based decomposition response compared to the effects reported in the literature for suffixed words, but a very similar post-decomposition profile. These results not only address a question stretching back to the earliest days of modern psycholinguistics, but also add critical support and nuance to our much newer emerging understanding of spatial organization and temporal dynamics of morphological processing in the human brain.

The Form of Morphemes: MEG Evidence From Masked Priming of Two Hebrew Templates.

Studies of lexical access have benefited from comparisons between languages like English, which shows concatenative morphology, and Semitic languages showing non-concatenative morphology of roots and patterns. Morphological decomposition in Semitic has previously been probed using masked priming, originally developed to investigate concatenative morphology. However, studies conducted on Semitic languages have often targeted Semitic-specific questions, such as whether the root and the verbal template prime lexical access. The overall consequence of these studies for our understanding of lexical access remains unclear. In two experiments on Hebrew using MEG, we demonstrate that a verbal form which is orthographically and phonologically indistinguishable from non-verbal forms is primed by other verbs in the same template but not by similar nouns and adjectives. These results suggest that masked priming taps into more than just visual forms but reflects morphological content, even if this content is abstract, showing no distinct orthographic or phonological marking.

Building words and phrases in the left temporal lobe.

A central part of knowing a language is the ability to combine basic linguistic units to form complex representations. While our neurobiological understanding of how words combine into larger structures has significantly advanced in recent years, the combinatory operations that build words themselves remain unknown. Are complex words such as tombstone and starlet built with the same mechanisms that construct phrases from words, such as grey stone or bright star? Here we addressed this with two magnetoencephalography (MEG) experiments, which simultaneously varied demands associated with phrasal composition, and the processing of morphological complexity in compound and suffixed nouns. Replicating previous findings, we show that portions of the left anterior temporal lobe (LATL) are engaged in the combination of modifiers and monomorphemic nouns in phrases (e.g., brown rabbit). As regards compounding, we show that semantically transparent compounds (e.g., tombstone) also engage left anterior temporal cortex, though the spatiotemporal details of this effect differed from phrasal composition. Further, when a phrase was constructed from a modifier and a transparent compound (e.g., granite tombstone), the typical LATL phrasal composition response appeared at a delayed latency, which follows if an initial within-word operation (tomb + stone) must take place before the combination of the compound with the preceding modifier (granite + tombstone). In contrast to compounding, suffixation (i.e., star + let) did not engage the LATL in any consistent way, suggesting a distinct processing route. Finally, our results suggest an intriguing generalization that morpho-orthographic complexity that does not recruit the LATL may block the engagement of the LATL in subsequent phrase building. In sum, our findings offer a detailed spatiotemporal characterization of the lowest level combinatory operations that ultimately feed the composition of full sentences.

The effects of inventory on vowel perception in French and Spanish: an MEG study.

Production studies have shown that speakers of languages with larger phoneme inventories expand their acoustic space relative to languages with smaller inventories [Bradlow, A. (1995). A comparative acoustic study of English and Spanish vowels. Journal of the Acoustical Society of America, 97(3), 1916-1924; Jongman, A., Fourakis, M., & Sereno, J. (1989). The acoustic vowel space of Modern Greek and German. Language Speech, 32, 221-248]. In this study, we investigated whether this acoustic expansion in production has a perceptual correlate, that is, whether the perceived distance between pairs of sounds separated by equal acoustic distances varies as a function of inventory size or organization. We used magnetoencephalography, specifically the mismatch field response (MMF), and compared two language groups, French and Spanish, whose vowel inventories differ in size and organization. Our results show that the MMF is sensitive to inventory size but not organization, suggesting that speakers of languages with larger inventories perceive the same sounds as less similar than speakers with smaller inventories.

Syntactic Priming As a Test of Argument Structure: A Self-paced Reading Experiment.

Using data from a behavioral structural priming experiment, we test two competing theoretical approaches to argument structure, which attribute different configurations to (in)transitive structures. These approaches make different claims about the relationship between unergatives and transitive structures selecting either a DP complement or a small clause complement in structurally unambiguous sentences, thus making different predictions about priming relations between them. Using statistical tools that combine a factorial 6 × 6 within subjects ANOVA, a mixed effects ANCOVA and a linear mixed effects regression model, we report syntactic priming effects in comprehension, which suggest a stronger predictive contribution of a model that supports an interpretive semantics view of syntax, whereby syntactic structures do not necessarily reflect argument/event structure in semantically unambiguous configurations. They also contribute novel experimental evidence that correlate representational complexity with language processing in the mind and brain. Our study further upholds the validity of combining quantitative methods and theoretical approaches to linguistics for advancing our knowledge of syntactic phenomena.

Decomposition, lookup, and recombination: MEG evidence for the full decomposition model of complex visual word recognition.

There is much evidence that visual recognition of morphologically complex words (e.g., teacher) proceeds via a decompositional route, first involving recognition of their component morphemes (teach + -er). According to the Full Decomposition model, after the visual decomposition stage, followed by morpheme lookup, there is a final "recombination" stage, in which the decomposed morphemes are combined and the well-formedness of the complex form is evaluated. Here, we use MEG to provide evidence for the temporally-differentiated stages of this model. First, we demonstrate an early effect of derivational family entropy, corresponding to the stem lookup stage; this is followed by a surface frequency effect, corresponding to the later recombination stage. We also demonstrate a late effect of a novel statistical measure, semantic coherence, which quantifies the gradient semantic well-formedness of complex words. Our findings illustrate the usefulness of corpus measures in investigating the component processes within visual word recognition.

Neural correlates of the effects of morphological family frequency and family size: an MEG study.

Schreuder and Baayen (Schreuder. R., & Baayen, R. H. (1997). How complex simplex words can be. Journal of Memory and Language 37, 118-139) report that lexical decision times to nouns are not sensitive to the cumulative frequency of the noun's morphological derivatives in its "morphological family", even though such a cumulative frequency effect is obtained in the domain of inflection. Under a decomposition view of derivational morphology, this constitutes a puzzling exception to the robust finding that lexical frequency is one of the major determinants of behavioral response latencies. If morphologically complex words are decomposed, each occurrence of a member of a noun's morphological family should add to its root-frequency. We investigated the effects of morphological family frequency on the magnetoencephalographic response component M350, which shows sensitivity to factors affecting early stages of lexical processing, including lexical frequency. We hypothesized that high morphological family frequency should have a facilitory effect on the M350, even though no such effect can be seen in response time, presumably due to competition among possible root-affix combinations. Contrary to this hypothesis, we found that high family frequency elicits an M350 inhibition, suggesting that competition among morphological family members occurs at the M350. The result is significant, since there is evidence that competition among phonologically similar words occurs after, not at, the M350. Thus, our results suggest that competition within a morphological family precedes competition within a phonological similarity neighborhood.