An exploratory data analysis of word form prediction during word-by-word reading.

In 2005, we reported evidence indicating that upcoming phonological word forms-e.g., kite vs. airplane-were predicted during reading. We recorded brainwaves (electroencephalograms [EEGs]) as people read word-by-word and then correlated the predictability in context of indefinite articles that preceded nouns ( a kite vs. an airplane) with the average event-related brain potentials (ERPs) they elicited [K. A. DeLong, T. P. Urbach, M. Kutas, Nat. Neurosci. 8, 1117-1121 (2005)]. Amid a broader controversy about the role of word-form prediction in comprehension, those findings were recently challenged by a failed putative direct replication attempt [M. S. Nieuwland et al., eLife 7, e33468 (2018); nine labs, one experiment, and 2.6e4 observations]. To better understand the empirical justification for positing an association between prenominal article predictability and scalp potentials, we conducted a wide-ranging exploratory data analysis (EDA), pooling our original data with extant data from two followup studies (one lab, three experiments, and 1.2e4 observations). We modeled the time course of article predictability in the single-trial data by fitting linear mixed-effects regression (LMER) models at each time point and scalp location spanning a 3-s interval before, during, and after the article. Model comparisons based on Akaike information criteria (AIC) and slope-regression ERPs [rERPs; N. J. Smith, M. Kutas, Psychophysiology 52, 157-168 (2015)] provide substantial empirical support for a small positive association between article predictability and scalp potentials approximately 300 to 500 ms after article onset, predominantly over bilateral posterior scalp. We think this effect may reasonably be attributed to prediction of upcoming word forms.

Empirically grounding grounded cognition: the case of color.

Grounded cognition theories hold that the neural states involved in experiencing objects play a direct functional role in representing and accessing object knowledge from memory. However, extant data marshaled to support this view are also consistent with an opposing view that perceptuo-motor activations occur only following access to knowledge from amodal memory systems. We provide novel discriminating evidence for the functional involvement of visuo-perceptual states specifically in accessing knowledge about an object's color. We recorded event-related brain potentials (ERPs) while manipulating the visual contrast of monochromatic words ("lime") in a semantic decision task: responses were made for valid colors ("green") and locations ("kitchen") and withheld for invalid colors and locations. Low contrast delayed perceptual processing for both color and location. Critically, low contrast slowed access to color knowledge only. This finding reveals that the visual system plays a functional role in accessing object knowledge and uniquely supports grounded views of cognition.

Alive and grasping: stable and rapid semantic access to an object category but not object graspability.

How quickly do different kinds of conceptual knowledge become available following visual word perception? Resolving this question will inform neural and computational theories of visual word recognition and semantic memory use. We measured real-time brain activity using event-related brain potentials (ERPs) during a go/nogo task to determine the upper limit by which category-related knowledge (living/nonliving) and action-related knowledge (graspable/ungraspable) must have been accessed to influence a downstream decision process. We find that decision processes can be influenced by the living/nonliving distinction by 160ms after stimulus onset whereas information about (one-hand) graspability is not available before 300ms. We also provide evidence that rapid access to category-related knowledge occurs for all items, not just a subset of living, nonliving, graspable, or ungraspable ones, and for all participants regardless of their response speed. The latency of the N200 nogo effect by contrast is sensitive to decision speed. We propose a tentative hypothesis of the neural mechanisms underlying semantic access and a subsequent decision process.

Perceptual and motor attribute ratings for 559 object concepts.

To understand how and when object knowledge influences the neural underpinnings of language comprehension and linguistic behavior, it is critical to determine the specific kinds of knowledge that people have. To extend the normative data currently available, we report a relatively more comprehensive set of object attribute rating norms for 559 concrete object nouns, each rated on seven attributes corresponding to sensory and motor modalities-color, motion, sound, smell, taste, graspability, and pain-in addition to familiarity (376 raters, M = 23 raters per item). The mean ratings were subjected to principal-components analysis, revealing two primary dimensions plausibly interpreted as relating to survival. We demonstrate the utility of these ratings in accounting for lexical and semantic decision latencies. These ratings should prove useful for the design and interpretation of experimental tests of conceptual and perceptual object processing.

fitgrid: A Python package for multi-channel event-related time series regression modeling.

Electrical brain activity related to external stimulation and internal mental events can be measured at the scalp as tiny time-varying electric potential waveforms (electroencephalogram; EEG), typically a few tens of microvolts peak to peak (Berger, 1930). Even tinier brain responses, too small to be seen by naked eye in the EEG, can be detected by repeating the stimulation, aligning the EEG recordings to the triggering event and averaging them at each time point (Dawson, 1951, 1954). Under assumptions that the brain response (signal) is the same in each recording and the ongoing background EEG (noise) varies randomly, averaging improves the estimate of the "true" brain response at each time point as the random variation cancels. The average event-related brain potential (ERP) and its counterpart for event-related magnetic fields (ERFs) are cornerstones of experimental brain research in human sensation, perception, and cognition (Luck & Kappenman, 2013). Smith and Kutas pointed out that the average ERP at each time t is mathematically identical to the estimated constant ß ^ 0 ( t ) for the regression model y(t) = ß 0(t) + ε(t), fit by minimizing squared error (Smith & Kutas, 2015a). The average ERP can be viewed as a time series of model parameter estimates. Generalizing to more complex models such as multiple regression y = ß 0 + ß 1 X 1 + … + ß pXp + ε, likewise produces time series of estimates for the constant and each regressor coefficient, the ß ^ 0 ( t ) , ß ^ 1 ( t ) , … , ß ^ p ( t ) dubbed regression ERP (rERP) waveforms (see Smith & Kutas, 2015a, 2015b for discussion of related approaches). This holds for straight-line fits ("slope" rERPs) as well as models of curvilinear relationships such as spline regression (Smith & Kutas, 2015b). Besides the estimated coefficient rERPs, the approach also produces time series for all the basic and derived quantities of the fitted model: coefficient standard errors, residuals, likelihood, Akaike information criterion (AIC), and so forth. With the shift from averaging to regression modeling, however, comes a new problem: fitting, diagnosing, comparing, evaluating and interpreting large numbers of regression models.

Lumos!: Electrophysiological tracking of (wizarding) world knowledge use during reading.

In Troyer and Kutas (2018), individual differences in knowledge of the world of Harry Potter (HP) rapidly modulated individuals' average electrical brain potentials to contextually supported words in sentence endings. Using advances in single-trial electroencephalogram analysis, we examined whether this relationship is strictly a result of domain knowledge mediating the proportion of facts each participant knew; we find it is not. Participants read sentences ending in a contextually supported word, reporting online whether they had known each fact. Participants' reports correlated with HP domain knowledge and reliably modulated event-related brain potentials to sentence-final words within 250 ms. Critically, domain knowledge had a dissociable influence in the same time window for endings that participants reported not having known and/or were less likely to be known/remembered across participants. We hypothesize that knowledge impacts written word processing primarily by affecting the neural processes of (implicit) retrieval from long-term memory (LTM): Greater knowledge eases otherwise difficult retrieval processes. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Neural processing of vocal emotion and identity.

The voice is a marker of a person's identity which allows individual recognition even if the person is not in sight. Listening to a voice also affords inferences about the speaker's emotional state. Both these types of personal information are encoded in characteristic acoustic feature patterns analyzed within the auditory cortex. In the present study 16 volunteers listened to pairs of non-verbal voice stimuli with happy or sad valence in two different task conditions while event-related brain potentials (ERPs) were recorded. In an emotion matching task, participants indicated whether the expressed emotion of a target voice was congruent or incongruent with that of a (preceding) prime voice. In an identity matching task, participants indicated whether or not the prime and target voice belonged to the same person. Effects based on emotion expressed occurred earlier than those based on voice identity. Specifically, P2 (approximately 200 ms)-amplitudes were reduced for happy voices when primed by happy voices. Identity match effects, by contrast, did not start until around 300 ms. These results show an early task-specific emotion-based influence on the early stages of auditory sensory processing.

Probabilistic word pre-activation during language comprehension inferred from electrical brain activity.

Despite the numerous examples of anticipatory cognitive processes at micro and macro levels in many animal species, the idea that anticipation of specific words plays an integral role in real-time language processing has been contentious. Here we exploited a phonological regularity of English indefinite articles ('an' precedes nouns beginning with vowel sounds, whereas 'a' precedes nouns beginning with consonant sounds) in combination with event-related brain potential recordings from the human scalp to show that readers' brains can pre-activate individual words in a graded fashion to a degree that can be estimated from the probability that each word is given as a continuation for a sentence fragment offline. These findings are evidence that readers use the words in a sentence (as cues to their world knowledge) to estimate relative likelihoods for upcoming words.

Combined perception of emotion in pictures and musical sounds.

Evaluation of emotional scenes requires integration of information from different modality channels, most frequently from audition and vision. Neither the psychological nor neural basis of auditory-visual interactions during the processing of affect is well understood. In this study, possible interactions in affective processing were investigated via event-related potential (ERP) recordings during simultaneous presentation of affective pictures (from IAPS) and affectively sung notes that either matched or mismatched each other in valence. To examine the role of attention in multisensory affect-integration ERPs were recorded in two different rating tasks (voice affect rating, picture affect rating) as participants evaluated the affect communicated in one of the modalities, while that in the other modality was ignored. Both the behavioral and ERP data revealed some, although non-identical, patterns of cross-modal influences; modulation of the ERP-component P2 suggested a relatively early integration of affective information in the attended picture condition, though only for happy picture-voice pairs. In addition, congruent pairing of sad pictures and sad voice stimuli affected the late positive potential (LPP). Responses in the voice affect rating task were overall more likely to be modulated by the concomitant picture's affective valence than vice versa.

Interpreting event-related brain potential (ERP) distributions: implications of baseline potentials and variability with application to amplitude normalization by vector scaling.

Recent proposals regarding the purpose and validity of amplitude normalization by vector scaling including mitigation of baseline and noise problems in between-condition difference analyses are critically evaluated. In so doing, we elaborate on some of the points raised in regarding baselines and noise, especially as these impact amplitude normalization by vector scaling and discuss the motivation for measuring event-related brain potential (ERP) amplitudes relative to a pre-stimulus baseline and the implications of this for certain (but not all) inferences. Throughout, our focus is on the logic of interpreting ERP measurements with an emphasis on the importance of specific assumptions and consideration of what conclusions are and are not supported.

Hemispheric asymmetry in event knowledge activation during incremental language comprehension: A visual half-field ERP study.

During incremental language comprehension, the brain activates knowledge of described events, including knowledge elements that constitute semantic anomalies in their linguistic context. The present study investigates hemispheric asymmetries in this process, with the aim of advancing our understanding of the neural basis and functional properties of event knowledge activation during incremental comprehension. In a visual half-field event-related brain potential (ERP) experiment, participants read brief discourses in which the third sentence contained a word that was either highly expected, semantically anomalous but related to the described event (Event-Related), or semantically anomalous but unrelated to the described event (Event-Unrelated). For both visual fields of target word presentation, semantically anomalous words elicited N400 ERP components of greater amplitude than did expected words. Crucially, Event-Related anomalous words elicited a reduced N400 relative to Event-Unrelated anomalous words only with left visual field/right hemisphere presentation. This result suggests that right hemisphere processes are critical to the activation of event knowledge elements that violate the linguistic context, and in doing so informs existing theories of hemispheric asymmetries in semantic processing during language comprehension. Additionally, this finding coincides with past research suggesting a crucial role for the right hemisphere in elaborative inference generation, raises interesting questions regarding hemispheric coordination in generating event-specific linguistic expectancies, and more generally highlights the possibility of functional dissociation of event knowledge activation for the generation of elaborative inferences and for linguistic expectancies.

Quantifiers are incrementally interpreted in context, more than less.

Language interpretation is often assumed to be incremental. However, our studies of quantifier expressions in isolated sentences found N400 event-related brain potential (ERP) evidence for partial but not full immediate quantifier interpretation (Urbach & Kutas, 2010). Here we tested similar quantifier expressions in pragmatically supporting discourse contexts (Alex was an unusual toddler. Most/Few kids prefer sweets/vegetables…) while participants made plausibility judgments (Experiment 1) or read for comprehension (Experiment 2). Control Experiments 3A (plausibility) and 3B (comprehension) removed the discourse contexts. Quantifiers always modulated typical and/or atypical word N400 amplitudes. However, only the real-time N400 effects only in Experiment 2 mirrored offline quantifier and typicality crossover interaction effects for plausibility ratings and cloze probabilities. We conclude that quantifier expressions can be interpreted fully and immediately, though pragmatic and task variables appear to impact the speed and/or depth of quantifier interpretation.

Different mechanisms for role relations versus verb-action congruence effects: evidence from ERPs in picture-sentence verification.

Extant accounts of visually situated language processing do make general predictions about visual context effects on incremental sentence comprehension; these, however, are not sufficiently detailed to accommodate potentially different visual context effects (such as a scene-sentence mismatch based on actions versus thematic role relations, e.g., (Altmann & Kamide, 2007; Knoeferle & Crocker, 2007; Taylor & Zwaan, 2008; Zwaan & Radvansky, 1998)). To provide additional data for theory testing and development, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence (500 ms SOA in Experiment 1 and 300 ms SOA in Experiment 2), and post-sentence verification times indicating whether or not the verb and/or the thematic role relations matched a preceding picture (depicting two participants engaged in an action). Though incrementally processed, these two types of mismatch yielded different ERP effects. Role-relation mismatch effects emerged at the subject noun as anterior negativities to the mismatching noun, preceding action mismatch effects manifest as centro-parietal N400s greater to the mismatching verb, regardless of SOAs. These two types of mismatch manipulations also yielded different effects post-verbally, correlated differently with a participant's mean accuracy, verbal working memory and visual-spatial scores, and differed in their interactions with SOA. Taken together these results clearly implicate more than a single mismatch mechanism for extant accounts of picture-sentence processing to accommodate.

Generalized event knowledge activation during online sentence comprehension.

Recent research has demonstrated that knowledge of real-world eventsplays an important role inguiding online language comprehension. The present study addresses the scope of event knowledge activation during the course of comprehension, specifically investigating whether activation is limited to those knowledge elements that align with the local linguistic context.The present study addresses this issue by analyzing event-related brain potentials (ERPs) recorded as participants read brief scenariosdescribing typical real-world events. Experiment 1 demonstratesthat a contextually anomalous word elicits a reduced N400 if it is generally related to the described event, even when controlling for the degree of association of this word with individual words in the preceding context and with the expected continuation. Experiment 2 shows that this effect disappears when the discourse context is removed.These findings demonstrate that during the course of incremental comprehension, comprehenders activate general knowledge about the described event, even at points at which this knowledge would constitute an anomalous continuation of the linguistic stream. Generalized event knowledge activationcontributes to mental representations of described events, is immediately available to influence language processing, and likely drives linguistic expectancy generation.

Thinking ahead or not? Natural aging and anticipation during reading.

Despite growing evidence of young adults neurally pre-activating word features during sentence comprehension, less clear is the degree to which this generalizes to older adults. Using ERPs, we tested for linguistic prediction in younger and older readers by means of indefinite articles (a's and an's) preceding more and less probable noun continuations. Although both groups exhibited cloze probability-graded noun N400s, only the young showed significant article effects, indicating probabilistic sensitivity to the phonology of anticipated upcoming nouns. Additionally, both age groups exhibited prolonged increased frontal positivities to less probable nouns, although in older adults this effect was prominent only in a subset with high verbal fluency (VF). This ERP positivity to contextual constraint violations offers additional support for prediction in the young. For high VF older adults, the positivity may indicate they, too, engage in some form of linguistic pre-processing when implicitly cued, as may have occurred via the articles.

Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review.

Event-related potentials (ERPs) and magnetic fields (ERFs) are typically analyzed via ANOVAs on mean activity in a priori windows. Advances in computing power and statistics have produced an alternative, mass univariate analyses consisting of thousands of statistical tests and powerful corrections for multiple comparisons. Such analyses are most useful when one has little a priori knowledge of effect locations or latencies, and for delineating effect boundaries. Mass univariate analyses complement and, at times, obviate traditional analyses. Here we review this approach as applied to ERP/ERF data and four methods for multiple comparison correction: strong control of the familywise error rate (FWER) via permutation tests, weak control of FWER via cluster-based permutation tests, false discovery rate control, and control of the generalized FWER. We end with recommendations for their use and introduce free MATLAB software for their implementation.

Mass univariate analysis of event-related brain potentials/fields II: Simulation studies.

Mass univariate analysis is a relatively new approach for the study of ERPs/ERFs. It consists of many statistical tests and one of several powerful corrections for multiple comparisons. Multiple comparison corrections differ in their power and permissiveness. Moreover, some methods are not guaranteed to work or may be overly sensitive to uninteresting deviations from the null hypothesis. Here we report the results of simulations assessing the accuracy, permissiveness, and power of six popular multiple comparison corrections (permutation-based control of the familywise error rate [FWER], weak control of FWER via cluster-based permutation tests, permutation-based control of the generalized FWER, and three false discovery rate control procedures) using realistic ERP data. In addition, we look at the sensitivity of permutation tests to differences in population variance. These results will help researchers apply and interpret these procedures.

Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture-sentence verification.

To re-establish picture-sentence verification-discredited possibly for its over-reliance on post-sentence response time (RT) measures-as a task for situated comprehension, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence, and RTs indicating whether or not the verb matched a previously depicted action. For mismatches (vs. matches), speeded RTs were longer, verb N400s over centro-parietal scalp larger, and ERPs to the object noun more negative. RTs (congruence effect) correlated inversely with the centro-parietal verb N400s, and positively with the object ERP congruence effects. Verb N400s, object ERPs, and verbal working memory scores predicted more variance in RT effects (50%) than N400s alone. Thus, (1) verification processing is not all post-sentence; (2) simple priming cannot account for these results; and (3) verification tasks can inform studies of situated comprehension.

Overlapping dual ERP responses to low cloze probability sentence continuations.

In 2005, DeLong, Urbach, and Kutas took advantage of the a/an English indefinite article phonological alternation and the sensitivities of the N400 ERP component to show that readers can neurally preactivate individual words of a sentence (including nouns and their prenominal indefinite articles) in a graded fashion with a likelihood estimated from the words' offline probabilities as sentence continuations. Here we report an additional finding from that study: a prolonged ERP frontal positivity to less probable noun continuations. We suggest that this positivity is consistent with hypotheses that additional neural processing may be invoked when highly expected continuations are not encountered in the input and speculate briefly on possible functional correlates.

Quantifiers more or less quantify online: ERP evidence for partial incremental interpretation.

Event-related brain potentials were recorded during RSVP reading to test the hypothesis that quantifier expressions are incrementally interpreted fully and immediately. In sentences tapping general knowledge (Farmers grow crops/worms as their primary source of income), Experiment 1 found larger N400s for atypical (worms) than typical objects (crops). Experiment 2 crossed object typicality with non-logical subject-noun phrase quantifiers (most, few). Off-line plausibility ratings exhibited the crossover interaction predicted by full quantifier interpretation: Most farmers grow crops and Few farmers grow worms were rated more plausible than Most farmers grow worms and Few farmers grow crops. Object N400s, although modulated in the expected direction, did not reverse. Experiment 3 replicated these findings with adverbial quantifiers (Farmers often/rarely grow crops/worms). Interpretation of quantifier expressions thus is neither fully immediate nor fully delayed. Furthermore, object atypicality was associated with a frontal slow positivity in few-type/rarely quantifier contexts, suggesting systematic processing differences among quantifier types.