The CABB dataset: A multimodal corpus of communicative interactions for behavioural and neural analyses.

We present a dataset of behavioural and fMRI observations acquired in the context of humans involved in multimodal referential communication. The dataset contains audio/video and motion-tracking recordings of face-to-face, task-based communicative interactions in Dutch, as well as behavioural and neural correlates of participants' representations of dialogue referents. Seventy-one pairs of unacquainted participants performed two interleaved interactional tasks in which they described and located 16 novel geometrical objects (i.e., Fribbles) yielding spontaneous interactions of about one hour. We share high-quality video (from three cameras), audio (from head-mounted microphones), and motion-tracking (Kinect) data, as well as speech transcripts of the interactions. Before and after engaging in the face-to-face communicative interactions, participants' individual representations of the 16 Fribbles were estimated. Behaviourally, participants provided a written description (one to three words) for each Fribble and positioned them along 29 independent conceptual dimensions (e.g., rounded, human, audible). Neurally, fMRI signal evoked by each Fribble was measured during a one-back working-memory task. To enable functional hyperalignment across participants, the dataset also includes fMRI measurements obtained during visual presentation of eight animated movies (35 min total). We present analyses for the various types of data demonstrating their quality and consistency with earlier research. Besides high-resolution multimodal interactional data, this dataset includes different correlates of communicative referents, obtained before and after face-to-face dialogue, allowing for novel investigations into the relation between communicative behaviours and the representational space shared by communicators. This unique combination of data can be used for research in neuroscience, psychology, linguistics, and beyond.

Idiosyncratic grammars: syntactic processing in second language comprehension uses subjective feature representations.

Learning the syntax of a second language (L2) often represents a big challenge to L2 learners. Previous research on syntactic processing in L2 has mainly focused on how L2 speakers respond to "objective" syntactic violations, that is, phrases that are incorrect by native standards. In this study, we investigate how L2 learners, in particular those of less than near-native proficiency, process phrases that deviate from their own, "subjective," and often incorrect syntactic representations, that is, whether they use these subjective and idiosyncratic representations during sentence comprehension. We study this within the domain of grammatical gender in a population of German learners of Dutch, for which systematic errors of grammatical gender are well documented. These L2 learners as well as a control group of Dutch native speakers read Dutch sentences containing gender-marked determiner-noun phrases in which gender agreement was either (objectively) correct or incorrect. Furthermore, the noun targets were selected such that, in a high proportion of nouns, objective and subjective correctness would differ for German learners. The ERP results show a syntactic violation effect (P600) for objective gender agreement violations for native, but not for nonnative speakers. However, when the items were re-sorted for the L2 speakers according to subjective correctness (as assessed offline), the P600 effect emerged as well. Thus, rather than being insensitive to violations of gender agreement, L2 speakers are similarly sensitive as native speakers but base their sensitivity on their subjective-sometimes incorrect-representations.

Repeated naming affects the accessibility of nonselected words: Evidence from picture-word interference experiments.

This study traced different types of distractor effects in the picture-word interference (PWI) task across repeated naming. Starting point was a PWI study by Kurtz et al. (2018). It reported that naming a picture (e.g., of a duck) was slowed down by a distractor word phonologically related to an alternative picture name from a different taxonomic level ("birch" related to "bird") when compared to an unrelated control, indicating that the alternative name was (phonologically) coactivated. Importantly, the effect was stable across repeated naming. The authors argued that this stability challenges adaptive models of word production according to which coactivated but nonselected words become less accessible for future retrieval; such a change in lexical accessibility should have been reflected in reduced interference. Using a similar experimental protocol as Kurtz et al., our study looked at the stability of different distractor effects in PWI. Interference from a distractor word phonologically related to an alternative name was stable across repeated naming, replicating Kurtz et al. In contrast, interference from a distractor word denoting a semantic category coordinate ("stork") was reduced across repeated naming. A similar pattern was found for a distractor word corresponding to an alternative name ("bird"). Facilitation from a distractor word phonologically related to the target name ("dust"), in contrast, was stable across repeated naming. We discuss the implications of these findings with respect to the issue of changes in the accessibility of lexical representations, the use of the PWI task in this context, and the use of this task in general. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Adaptive processing in word production: Evidence from picture-word interference studies.

Adaptive models of word production hold that lexical processing is shaped by recent production episodes. In particular, the models proposed by Howard et al. (2006) and Oppenheim et al. (2010) assume that the connection strength between semantic and lexical representations is updated continuously, on each use of a word. These changes make semantically related competitor words to a selected target word (e.g., "cherry" or "grape" for the target word "apple") less accessible for future retrieval. We put this assumption to a novel test in two picture-word interference experiments. The experiments looked at semantic interference (longer picture naming latencies in the presence of a semantically related distractor word compared to an unrelated distractor word; the effect is assumed to reflect the activation of competitor words to a target word) and its development across repeated naming. Across two times of measurement, semantically related distractor words (and their unrelated controls) were either identical (fixed target-distractor mapping, e.g., either distractor "cherry" or distractor "grape" for the target word "apple" at both times of measurement) or different (variable target-distractor mapping, e.g., distractor "cherry" at first time of measurement and distractor "grape" at second time of measurement or vice versa). Semantic interference was reduced at the second time of measurement, and this reduction was found with both fixed and variable target-distractor mapping. This generalization to new target-distractor combinations provides strong novel support for the notion of production-dependent changes in lexical accessibility as captured by adaptive models of word production. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Adaptive lexical processing of semantic competitors extends to alternative names: Evidence from blocked-cyclic picture naming.

Naming a picture (e.g., "duck") in the context of semantically related pictures (e.g., "eagle," "stork," "parrot") takes longer than naming it in the context of unrelated pictures (e.g., "knave," "toast," "atlas"). Adaptive models of word production attribute this semantic interference effect in blocked-cyclic naming (BCN) to an adaptive mechanism that makes competitor words, (e.g., the semantically related word "eagle" for the target word "duck") which are activated but not selected for production, less accessible for future retrieval. Results from a recent picture-word-interference study, however, suggested that alternative names (e.g., "bird" for "duck") might be exempt from this mechanism, challenging adaptive lexical processing as a general mechanism. We tested whether converging evidence is obtained in BCN. In Experiment 1, we embedded pictures responded to with alternative (category) names (e.g., "bird") into contexts composed of pictures responded to with specific (exemplar) names (e.g., "duck," "eagle," "stork," and "parrot"). If alternative names are exempt from adaptive lexical processing, interference in the homogeneous context should be found for specific name items but not for alternative name items. In contrast to this prediction, there was similar-sized interference for both types of items. In Experiment 2, we replaced the alternative name items with unrelated items. For these items, interference was largely diminished, ruling out that the effect found in Experiment 1 is a general set effect. Overall, our data suggest that alternative names are not special with respect to adaptive lexical processing.

Is beta in agreement with the relatives? Using relative clause sentences to investigate MEG beta power dynamics during sentence comprehension.

There remains some debate about whether beta power effects observed during sentence comprehension reflect ongoing syntactic unification operations (beta-syntax hypothesis), or instead reflect maintenance or updating of the sentence-level representation (beta-maintenance hypothesis). In this study, we used magnetoencephalography to investigate beta power neural dynamics while participants read relative clause sentences that were initially ambiguous between a subject- or an object-relative reading. An additional condition included a grammatical violation at the disambiguation point in the relative clause sentences. The beta-maintenance hypothesis predicts a decrease in beta power at the disambiguation point for unexpected (and less preferred) object-relative clause sentences and grammatical violations, as both signal a need to update the sentence-level representation. While the beta-syntax hypothesis also predicts a beta power decrease for grammatical violations due to a disruption of syntactic unification operations, it instead predicts an increase in beta power for the object-relative clause condition because syntactic unification at the point of disambiguation becomes more demanding. We observed decreased beta power for both the agreement violation and object-relative clause conditions in typical left hemisphere language regions, which provides compelling support for the beta-maintenance hypothesis. Mid-frontal theta power effects were also present for grammatical violations and object-relative clause sentences, suggesting that violations and unexpected sentence interpretations are registered as conflicts by the brain's domain-general error detection system.

Distractor strength and selective attention in picture-naming performance.

Whereas it has long been assumed that competition plays a role in lexical selection in word production (e.g., Levelt, Roelofs, & Meyer, 1999), recently Finkbeiner and Caramazza (2006) argued against the competition assumption on the basis of their observation that visible distractors yield semantic interference in picture naming, whereas masked distractors yield semantic facilitation. We examined an alternative account of these findings that preserves the competition assumption. According to this account, the interference and facilitation effects of distractor words reflect whether or not distractors are strong enough to exceed a threshold for entering the competition process. We report two experiments in which distractor strength was manipulated by means of coactivation and visibility. Naming performance was assessed in terms of mean response time (RT) and RT distributions. In Experiment 1, with low coactivation, semantic facilitation was obtained from clearly visible distractors, whereas poorly visible distractors yielded no semantic effect. In Experiment 2, with high coactivation, semantic interference was obtained from both clearly and poorly visible distractors. These findings support the competition threshold account of the polarity of semantic effects in naming.

Processing symbolic magnitude information conveyed by number words and by scalar adjectives.

Humans not only process and compare magnitude information such as size, duration, and number perceptually, but they also communicate about these properties using language. In this respect, a relevant class of lexical items are so-called scalar adjectives like "big," "long," "loud," and so on which refer to magnitude information. It has been proposed that humans use an amodal and abstract representation format shared by different dimensions, called the generalised magnitude system (GMS). In this paper, we test the hypothesis that scalar adjectives are symbolic references to GMS representations, and, therefore, GMS gets involved in processing their meaning. Previously, a parallel hypothesis on the relation between number symbols and GMS representations has been tested with the size congruity paradigm. The results of these experiments showed interference between the processing of number symbols and the processing of physical (font-) size. In the first three experiments of the present study (total N = 150), we used the size congruity paradigm and the same/different task to look at the potential interaction between physical size magnitude and numerical magnitude expressed by number words. In the subsequent three experiments (total N = 149), we looked at a parallel potential interaction between physical size magnitude and scalar adjective meaning. In the size congruity paradigm, we observed interference between the processing of the numerical value of number words and the meaning of scalar adjectives, on the one hand, and physical (font-) size, on the other hand, when participants had to judge the number words or the adjectives (while ignoring physical size). No interference was obtained for the reverse situation, i.e., when participants judged the physical font size (while ignoring numerical value or meaning). The results of the same/different task for both number words and scalar adjectives strongly suggested that the interference that was observed in the size congruity paradigm was likely due to a response conflict at the decision stage of processing rather than due to the recruitment of GMS representations. Taken together, it can be concluded that the size congruity paradigm does not provide evidence in support the hypothesis that GMS representations are used in the processing of number words or scalar adjectives. Nonetheless, the hypothesis we put forward about scalar adjectives is still is a promising potential line of research. We make a number of suggestions for how this hypothesis can be explored in future studies.

A laboratory study of naturalistic second language learning: Acquiring grammatical gender from simple dialogue.

The aim of this study was twofold: first, to develop an experimental technique as a tool to investigate learning outcomes of spontaneous, naturalistic second language (L2) learning under controlled laboratory conditions; and second, to explore how this technique can be used to understand the basic conditions and limits of this learning. Two variants of a dialogue game were tested in which corrective L2 input was provided to the learners, but the learning aspect was camouflaged. Participants were German learners of Dutch who are known to display persistent grammatical gender errors in Dutch owing to incorrect first language (L1)-L2 transfer. In Experiment 1, the participant and a 'virtual partner' (audio-recordings) took turns in describing cards using gender-marked article-noun phrases. However, the majority of the participants became aware of learning articles as goal of the experiment, either because of the way we asked participants about the goal of the experiment or because of the task used. Therefore, we changed both aspects and used a dialogue-memory game in Experiment 2, which indeed led only a minority (28%) to suspect the real goal of the study. In both experiments, participants showed substantial learning of word gender (on average 13.8 percentage points increase in accuracy) after only one instance of correct input. A manipulation of the number of trials (lag) between correct input and production did not affect results. Furthermore, the 72% of 'naïve' participants in Experiment 2 showed as much learning as the full sample. Thus, the new paradigm offers important insights into the determinants of naturalistic learning. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The role of prosodic breaks and pitch accents in grouping words during on-line sentence processing.

The present study addresses the question whether accentuation and prosodic phrasing can have a similar function, namely, to group words in a sentence together. Participants listened to locally ambiguous sentences containing object- and subject-control verbs while ERPs were measured. In Experiment 1, these sentences contained a prosodic break, which can create a certain syntactic grouping of words, or no prosodic break. At the disambiguation, an N400 effect occurred when the disambiguation was in conflict with the syntactic grouping created by the break. We found a similar N400 effect without the break, indicating that the break did not strengthen an already existing preference. This pattern held for both object- and subject-control items. In Experiment 2, the same sentences contained a break and a pitch accent on the noun following the break. We argue that the pitch accent indicates a broad focus covering two words [see Gussenhoven, C. On the limits of focus projection in English. In P. Bosch & R. van der Sandt (Eds.), Focus: Linguistic, cognitive, and computational perspectives. Cambridge: University Press, 1999], thus grouping these words together. For object-control items, this was semantically possible, which led to a "good-enough" interpretation of the sentence. Therefore, both sentences were interpreted equally well and the N400 effect found in Experiment 1 was absent. In contrast, for subject-control items, a corresponding grouping of the words was impossible, both semantically and syntactically, leading to processing difficulty in the form of an N400 effect and a late positivity. In conclusion, accentuation can group words together on the level of information structure, leading to either a semantically "good-enough" interpretation or a processing problem when such a semantic interpretation is not possible.

Semantic and syntactic composition of minimal adjective-noun phrases in Dutch: An MEG study.

The possibility to combine smaller units of meaning (e.g., words) to create new and more complex meanings (e.g., phrases and sentences) is a fundamental feature of human language. In the present project, we investigated how the brain supports the semantic and syntactic composition of two-word adjective-noun phrases in Dutch, using magnetoencephalography (MEG). The present investigation followed up on previous studies reporting a composition effect in the left anterior temporal lobe (LATL) when comparing neural activity at nouns combined with adjectives, as opposed to nouns in a non-compositional context. The first aim of the present study was to investigate whether this effect, as well as its modulation by noun specificity and adjective class, can also be observed in Dutch. A second aim was to investigate to what extent these effects may be driven by syntactic composition rather than primarily by semantic composition as was previously proposed. To this end, a novel condition was administered in which participants saw nouns combined with pseudowords lacking meaning but agreeing with the nouns in terms of grammatical gender, as real adjectives would. We failed to observe a composition effect or its modulation in both a confirmatory analysis (focused on the cortical region and time-window where it has previously been reported) and in exploratory analyses (where we tested multiple regions and an extended potential time-window of the effect). A syntactically driven composition effect was also not observed in our data. We do, however, successfully observe an independent, previously reported effect on single word processing in our data, confirming that our MEG data processing pipeline does meaningfully capture language processing activity by the brain. The failure to observe the composition effect in LATL is surprising given that it has been previously reported in multiple studies. Reviewing all previous studies investigating this effect, we propose that materials and a task involving imagery might be necessary for this effect to be observed. In addition, we identified substantial variability in the regions of interest analyzed in previous studies, which warrants additional checks of robustness of the effect. Further research should identify limits and conditions under which this effect can be observed. The failure to observe specifically a syntactic composition effect in such minimal phrases is less surprising given that it has not been previously reported in MEG data.

The interplay between prosody and syntax in sentence processing: the case of subject- and object-control verbs.

This study addresses the question whether prosodic information can affect the choice for a syntactic analysis in auditory sentence processing. We manipulated the prosody (in the form of a prosodic break; PB) of locally ambiguous Dutch sentences to favor one of two interpretations. The experimental items contained two different types of so-called control verbs (subject and object control) in the matrix clause and were syntactically disambiguated by a transitive or by an intransitive verb. In Experiment 1, we established the default off-line preference of the items for a transitive or an intransitive disambiguating verb with a visual and an auditory fragment completion test. The results suggested that subject- and object-control verbs differently affect the syntactic structure that listeners expect. In Experiment 2, we investigated these two types of verbs separately in an on-line ERP study. Consistent with the literature, the PB elicited a closure positive shift. Furthermore, in subject-control items, an N400 effect for intransitive relative to transitive disambiguating verbs was found, both for sentences with and for sentences without a PB. This result suggests that the default preference for subject-control verbs goes in the same direction as the effect of the PB. In object-control items, an N400 effect for intransitive relative to transitive disambiguating verbs was found for sentences with a PB but no effect in the absence of a PB. This indicates that a PB can affect the syntactic analysis that listeners pursue.

Syntactic processing in L2 depends on perceived reliability of the input: Evidence from P600 responses to correct input.

In 3 ERP experiments, we investigated how experienced L2 speakers process natural and correct syntactic input that deviates from their own, sometimes incorrect, syntactic representations. Our previous study (Lemhöfer, Schriefers, & Indefrey, 2014) had shown that L2 speakers do engage in native-like syntactic processing of gender agreement but base this processing on their own idiosyncratic (and sometimes incorrect) grammars. However, as in other standard ERP studies, but different from realistic L2 input, the materials in that study contained a large proportion of incorrect sentences. In the present study, German speakers of Dutch read exclusively objectively correct Dutch sentences that did or did not contain subjective determiner "errors" (e.g., de boot "the boat," which conflicts with the intuition of many German speakers that the correct phrase should be het boot). During reading for comprehension (Experiment 1), no syntax-related ERP responses for subjectively incorrect compared to correct phrases were observed. The same was true even when participants explicitly attended to and learned from the determiners in the sentences (Experiment 2). Only when participants judged the correctness of determiners in each sentence (Experiment 3) did a clear P600 appear. These results suggest that the full and native-like use of subjective grammars, as reflected in the P600 to subjective violations, occurs only when speakers have reason to mistrust the grammaticality of the input, either because of the nature of the task (grammaticality judgments) or because of the salient presence of incorrect sentences. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Language conflict in the bilingual brain.

The large majority of humankind is more or less fluent in 2 or even more languages. This raises the fundamental question how the language network in the brain is organized such that the correct target language is selected at a particular occasion. Here we present behavioral and functional magnetic resonance imaging data showing that bilingual processing leads to language conflict in the bilingual brain even when the bilinguals' task only required target language knowledge. This finding demonstrates that the bilingual brain cannot avoid language conflict, because words from the target and nontarget languages become automatically activated during reading. Importantly, stimulus-based language conflict was found in brain regions in the LIPC associated with phonological and semantic processing, whereas response-based language conflict was only found in the pre-supplementary motor area/anterior cingulate cortex when language conflict leads to response conflicts.

Do perceived context pictures automatically activate their phonological code?

Morsella and Miozzo (Morsella, E., & Miozzo, M. (2002). Evidence for a cascade model of lexical access in speech production. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 555-563) have reported that the to-be-ignored context pictures become phonologically activated when participants name a target picture, and took this finding as support for cascaded models of lexical retrieval in speech production. In a replication and extension of their experiment in German, we failed to obtain priming effects from context pictures phonologically related to a to-be-named target picture. By contrast, corresponding context words (i.e., the names of the respective pictures) and the same context pictures, when used in an identity condition, did reliably facilitate the naming process. This pattern calls into question the generality of the claim advanced by Morsella and Miozzo that perceptual processing of pictures in the context of a naming task automatically leads to the activation of corresponding lexical-phonological codes.

Sentence processing in the visual and auditory modality: do comma and prosodic break have parallel functions?

Two Event-Related Potential (ERP) studies contrast the processing of locally ambiguous sentences in the visual and the auditory modality. These sentences are disambiguated by a lexical element. Before this element appears in a sentence, the sentence can also be disambiguated by a boundary marker: a comma in the visual modality, or a prosodic break in the auditory modality. Previous studies have shown that a specific ERP component, the Closure Positive Shift (CPS), can be elicited by these markers. The results of the present studies show that both the comma and the prosodic break disambiguate the ambiguous sentences before the critical lexical element, despite the fact that a clear CPS is only found in the auditory modality. Comma and prosodic break thus have parallel functions irrespective of whether they do or do not elicit a CPS.

Conceptual coherence affects phonological activation of context objects during object naming.

In 4 picture-word interference experiments, speakers named a target object that was presented with a context object. Using auditory distractors that were phonologically related or unrelated either to the target object or the context object, the authors assessed whether phonological processing was confined to the target object or not. Phonological activation of the context objects was reliably observed if the target and context objects were embedded in a conceptually coherent scene (e.g., if the picture showed a mouse eating some cheese), regardless of whether the target was cued by its thematic role (agent vs. patient) or by color. However, this activation dissipated if the two objects were presented in an arbitrary object array (e.g., if the cheese was presented along with a finger). These findings suggest that conceptual coherence among multiple objects affects the information flow in the conceptual-lexical system during speech planning.

Native language influences on word recognition in a second language: a megastudy.

Many studies have reported that word recognition in a second language (L2) is affected by the native language (L1). However, little is known about the role of the specific language combination of the bilinguals. To investigate this issue, the authors administered a word identification task (progressive demasking) on 1,025 monosyllabic English (L2) words to native speakers of French, German, and Dutch. A regression approach was adopted, including a large number of within- and between-language variables as predictors. A substantial overlap of reaction time patterns was found across the groups of bilinguals, showing that word recognition results obtained for one group of bilinguals generalize to bilinguals with different mother tongues. Moreover, among the set of significant predictors, only one between-language variable was present (cognate status); all others reflected characteristics of the target language. Thus, although influences across languages exist, word recognition in L2 by proficient bilinguals is primarily determined by within-language factors, whereas cross-language effects appear to be limited. An additional comparison of the bilingual data with a native control group showed that there are subtle but significant differences between L1 and L2 processing.

Incremental learning in word production: Tracing the fate of non-selected alternative picture names.

Picture naming studies have shown that alternative picture names become phonologically coactivated even when they are eventually not produced (e.g., dog when poodle is produced and vice versa). The authors investigated whether this pattern is shaped by recent experience. Specifically, they tested whether the phonological coactivation of an alternative name is attenuated, when speakers consistently only use one particular name in a large number of naming episodes. In 3 picture-word interference experiments, the authors measured the phonological coactivation of basic-level alternative names during subordinate-level naming (Experiments 1 and 3) and of subordinate-level alternative names during basic-level naming (Experiment 2). They measured the coactivation of the alternative name at different points in time with distractor words that were phonologically related or unrelated to that name. If the pattern of lexical activation is shaped by previous naming episodes and the phonological coactivation of the nonproduced alternative name decreases, interference from related distractors should be reduced in the course of the experiment. Contrary to this prediction, the interference effect from distractors phonologically related to the alternative name remained stable. This was also true when participants were not familiarized with the pictures, more naming episodes were implemented, and a consolidation phase was introduced (Experiment 3). Overall, these results indicate some limitation of incremental learning in word production. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Assessing the utility of frequency tagging for tracking memory-based reactivation of word representations.

Reinstatement of memory-related neural activity measured with high temporal precision potentially provides a useful index for real-time monitoring of the timing of activation of memory content during cognitive processing. The utility of such an index extends to any situation where one is interested in the (relative) timing of activation of different sources of information in memory, a paradigm case of which is tracking lexical activation during language processing. Essential for this approach is that memory reinstatement effects are robust, so that their absence (in the average) definitively indicates that no lexical activation is present. We used electroencephalography to test the robustness of a reported subsequent memory finding involving reinstatement of frequency-specific entrained oscillatory brain activity during subsequent recognition. Participants learned lists of words presented on a background flickering at either 6 or 15 Hz to entrain a steady-state brain response. Target words subsequently presented on a non-flickering background that were correctly identified as previously seen exhibited reinstatement effects at both entrainment frequencies. Reliability of these statistical inferences was however critically dependent on the approach used for multiple comparisons correction. We conclude that effects are not robust enough to be used as a reliable index of lexical activation during language processing.