The Multilingual Picture Database.

The growing interdisciplinary research field of psycholinguistics is in constant need of new and up-to-date tools which will allow researchers to answer complex questions, but also expand on languages other than English, which dominates the field. One type of such tools are picture datasets which provide naming norms for everyday objects. However, existing databases tend to be small in terms of the number of items they include, and have also been normed in a limited number of languages, despite the recent boom in multilingualism research. In this paper we present the Multilingual Picture (Multipic) database, containing naming norms and familiarity scores for 500 coloured pictures, in thirty-two languages or language varieties from around the world. The data was validated with standard methods that have been used for existing picture datasets. This is the first dataset to provide naming norms, and translation equivalents, for such a variety of languages; as such, it will be of particular value to psycholinguists and other interested researchers. The dataset has been made freely available.

Retrospective confidence judgments: Meta-analysis of functional magnetic resonance imaging studies.

Confidence in our retrieved memories, that is, retrospective confidence, is a metamemory process we perform daily. There is an abundance of applied research focusing on the metamemory judgments and very diverse studies including a wide range of clinical populations. However, the neural correlates that support its functioning are not well defined impeding the implementation of noninvasive neuromodulatory clinical interventions. To address the neural basis of metamemory judgments, we ran a meta-analysis, where we used the activation likelihood estimation method on the 19 eligible functional magnetic resonance imaging studies. The main analysis of retrospective confidence revealed concordant bilateral activation in the parahippocampal gyrus, left middle frontal gyrus, and right amygdala. We also run an analysis between the two extreme levels of confidence, namely, high and low. This additional analysis was exploratory, since the minimum amount of articles reporting these two levels was not reached. Activations for the exploratory high > low confidence subtraction analysis were the same as observed in the main analysis on retrospective confidence, whereas the exploratory low > high subtraction showed distinctive activations of the right precuneus. The involvement of the right precuneus emphasizes its role in the evaluation of low confidence memories, as suggested by previous studies. Overall, our study contributes to a better understanding of the specific brain structures involved in confidence evaluations. Better understanding of the neural basis of metamemory might eventually lead to designing more precise neuromodulatory interventions, significantly improving treatment of patients suffering from metamemory problems.

Breaking down the cocktail party: Attentional modulation of cerebral audiovisual speech processing.

Recent studies utilizing electrophysiological speech envelope reconstruction have sparked renewed interest in the cocktail party effect by showing that auditory neurons entrain to selectively attended speech. Yet, the neural networks of attention to speech in naturalistic audiovisual settings with multiple sound sources remain poorly understood. We collected functional brain imaging data while participants viewed audiovisual video clips of lifelike dialogues with concurrent distracting speech in the background. Dialogues were presented in a full-factorial design, comprising task (listen to the dialogues vs. ignore them), audiovisual quality and semantic predictability. We used univariate analyses in combination with multivariate pattern analysis (MVPA) to study modulations of brain activity related to attentive processing of audiovisual speech. We found attentive speech processing to cause distinct spatiotemporal modulation profiles in distributed cortical areas including sensory and frontal-control networks. Semantic coherence modulated attention-related activation patterns in the earliest stages of auditory cortical processing, suggesting that the auditory cortex is involved in high-level speech processing. Our results corroborate views that emphasize the dynamic nature of attention, with task-specificity and context as cornerstones of the underlying neuro-cognitive mechanisms.

Modulation of Brain Activity by Selective Attention to Audiovisual Dialogues.

In real-life noisy situations, we can selectively attend to conversations in the presence of irrelevant voices, but neurocognitive mechanisms in such natural listening situations remain largely unexplored. Previous research has shown distributed activity in the mid superior temporal gyrus (STG) and sulcus (STS) while listening to speech and human voices, in the posterior STS and fusiform gyrus when combining auditory, visual and linguistic information, as well as in left-hemisphere temporal and frontal cortical areas during comprehension. In the present functional magnetic resonance imaging (fMRI) study, we investigated how selective attention modulates neural responses to naturalistic audiovisual dialogues. Our healthy adult participants (N = 15) selectively attended to video-taped dialogues between a man and woman in the presence of irrelevant continuous speech in the background. We modulated the auditory quality of dialogues with noise vocoding and their visual quality by masking speech-related facial movements. Both increased auditory quality and increased visual quality were associated with bilateral activity enhancements in the STG/STS. In addition, decreased audiovisual stimulus quality elicited enhanced fronto-parietal activity, presumably reflecting increased attentional demands. Finally, attention to the dialogues, in relation to a control task where a fixation cross was attended and the dialogue ignored, yielded enhanced activity in the left planum polare, angular gyrus, the right temporal pole, as well as in the orbitofrontal/ventromedial prefrontal cortex and posterior cingulate gyrus. Our findings suggest that naturalistic conversations effectively engage participants and reveal brain networks related to social perception in addition to speech and semantic processing networks.

Quick reorganization of memory traces for morphologically complex words in young children.

Formation of neural mechanisms for morphosyntactic processing in young children is still poorly understood. Here, we addressed neural processing and rapid online acquisition of familiar and unfamiliar combinations of morphemes. Three different types of morphologically complex words - derived, inflected, and novel (pseudostem + real suffix) - were presented in a passive listening setting to 16 typically developing 3-4-year old children (as part of a longitudinal Helsinki SLI follow-up study). The mismatch negativity (MMN) component of event-related potentials (ERP), an established index of long-term linguistic memory traces in the brain, was analysed separately for the initial and final periods of the exposure to these items. We found MMN response enhancement for the inflected words towards the end of the recording session, whereas no response change was observed for the derived or novel complex forms. This enhancement indicates rapid build-up of a new memory trace for the combination of real morphemes, suggesting a capacity for online formation of whole-form lexicalized representations as one of the morphological mechanisms in the developing brain. Furthermore, this enhancement increased with age, suggesting the development of automatic morphological processing circuits in the age range of 3-4 years.

Acquisition of L2 morphology by adult language learners.

Learning a new language requires the acquisition of morphological units that enable the fluent use of words in different grammatical contexts. While accumulating research has elucidated the neural processing of native morphology, much less is known about how second-language (L2) learners acquire and process morphology in their L2. To address this question, we presented native speakers as well as beginning and advanced learners of Finnish with spoken (1) derived words, (2) inflected words, (3) novel derivations (novel combinations of existing stem + suffix), and (4) pseudo-suffixed words (existing stem + pseudo-suffix) in a passive listening EEG experiment. An early (60 msec after suffix deviation point) positive ERP response showed no difference between inflections and derivations, suggesting similar early parsing of these complex words. At 130 msec, derivations elicited a lexical ERP pattern of full-form memory-trace activation, present in the L2 beginners and advanced speakers to different degrees, implying a shift from lexical processing to more dual parsing and lexical activation of the complex forms with increasing proficiency. Pseudo-suffixed words produced a syntactic pattern in a later, 170-240 msec time-window, exhibiting enhanced ERPs compared to well-formed inflections, indicating second-pass syntactic parsing. Overall, the L2 learners demonstrated a gradual effect of proficiency towards L1-like responses. Advanced L2 learners seem to have developed memory traces for derivations and their neurolinguistic system is capable of early automatic parsing. This suggests that advanced learners have already developed sensitivity to morphological information, while such knowledge is weak in beginners. Discrepancies in ERP dynamics and topographies indicate partially differing recruitment of the language network in L1 and L2. In beginners, response differences between existing and novel morphology were scarce, implying that representations for complex forms are not yet well-established. The results suggest successful development of brain mechanisms for automatic processing of L2 morphology, capable of gradually attaining L1-like functionality with increasing proficiency.

Morphological processing in the brain: The good (inflection), the bad (derivation) and the ugly (compounding).

There is considerable behavioral evidence that morphologically complex words such as 'tax-able' and 'kiss-es' are processed and represented combinatorially. In other words, they are decomposed into their constituents 'tax' and '-able' during comprehension (reading or listening), and producing them might also involve on-the-spot combination of these constituents (especially for inflections). However, despite increasing amount of neurocognitive research, the neural mechanisms underlying these processes are still not fully understood. The purpose of this critical review is to offer a comprehensive overview on the state-of-the-art of the research on the neural mechanisms of morphological processing. In order to take into account all types of complex words, we include findings on inflected, derived, and compound words presented both visually and aurally. More specifically, we cover a wide range of electro- and magnetoencephalography (EEG and MEG, respectively) as well as structural/functional magnetic resonance imaging (s/fMRI) studies that focus on morphological processing. We present the findings with respect to the temporal course and localization of morphologically complex word processing. We summarize the observed findings, their interpretations with respect to current psycholinguistic models, and discuss methodological approaches as well as their possible limitations.

Formation of neocortical memory circuits for unattended written word forms: neuromagnetic evidence.

To master linguistic communication, humans must acquire large vocabularies quickly and effortlessly. Efficient word learning might be facilitated by the ability to rapidly acquire novel word forms even outside the focus of attention, occurring within minutes of repetitive exposure and suggesting fast and automatic lexicon acquisition. However, this phenomenon has been studied in the auditory modality only, and it is unknown whether similar mechanisms also exist in the visual domain. We tested this by presenting participants with novel written word forms while the focus of their attention was on a non-linguistic dual colour-detection task. Matched familiar word forms served as a control. Using magnetoencephalography (MEG), we scrutinised changes in neuromagnetic responses to familiar and to novel word forms over approximately 15 minutes of exposure. We found, for the first time, a visual analogue of automatic rapid build-up of neural memory circuits for unattended novel lexical items, seen as a rapid enhancement of early (~100 ms post-onset) activation in the left anterior-superior temporal lobe. Our results suggest that the brain quickly forms cortical representations for new written forms, and indicate that the automatic neural mechanisms subserving rapid online acquisition of novel linguistic information might be shared by both auditory and visual modalities.

Task-free auditory EEG paradigm for probing multiple levels of speech processing in the brain.

While previous studies on language processing highlighted several ERP components in relation to specific stages of sound and speech processing, no study has yet combined them to obtain a comprehensive picture of language abilities in a single session. Here, we propose a novel task-free paradigm aimed at assessing multiple levels of speech processing by combining various speech and nonspeech sounds in an adaptation of a multifeature passive oddball design. We recorded EEG in healthy adult participants, who were presented with these sounds in the absence of sound-directed attention while being engaged in a primary visual task. This produced a range of responses indexing various levels of sound processing and language comprehension: (a) P1-N1 complex, indexing obligatory auditory processing; (b) P3-like dynamics associated with involuntary attention allocation for unusual sounds; (c) enhanced responses for native speech (as opposed to nonnative phonemes) from ∼50 ms from phoneme onset, indicating phonological processing; (d) amplitude advantage for familiar real words as opposed to meaningless pseudowords, indexing automatic lexical access; (e) topographic distribution differences in the cortical activation of action verbs versus concrete nouns, likely linked with the processing of lexical semantics. These multiple indices of speech-sound processing were acquired in a single attention-free setup that does not require any task or subject cooperation; subject to future research, the present protocol may potentially be developed into a useful tool for assessing the status of auditory and linguistic functions in uncooperative or unresponsive participants, including a range of clinical or developmental populations.

Beyond volume: A surface-based approach to bilingualism-induced grey matter changes.

Bilingualism is a sustained experience associated with structural changes in cortical grey matter (GM) morphology. Apart from a few studies, a dominant method used to assess bilingualism-induced GM changes has been the voxel-based morphometry (VBM) analysis. While VBM is sensitive to GM volume/density differences in general, it cannot be used to identify whether the observed difference is due to relative changes in, e.g., cortical thickness, area or folding, as it uses a single combined measure of them all. Here, we used surface-based analysis (SBA) approach to investigate whether early acquisition of a second language (L2) affects the cortical GM morphology relative to late L2 acquisition. More specifically, our aim was to test a hypothesis that early acquisition of two languages induces GM changes that are predominantly surface area-driven, while late acquisition is supposedly characterised with primarily thickness-driven changes. To this end, several surface-based measures were concurrently compared between the groups. In line with the hypothesis, the results revealed that early bilingual experience is associated with significantly extended cortical surface area over the left pars opercularis and the right superior temporal gyrus. Contrary to our expectations, however, we found no evidence supporting the postulated association between late L2 acquisition and increased cortical thickness. Nevertheless, our study highlights the importance of including cortical surface measures when investigating bilingualism-related GM modulations.

Effect of language experience on selective auditory attention: An event-related potential study.

Dual language experience has typically been shown to improve various executive control functions. We investigated with event-related brain potentials (ERPs) recorded from early (natively) bilingual speakers and control participants whether it also affects auditory selective attention. We delivered to our participants two tone streams, one to the left and one to the right ear. Both streams consisted of standard tones and two types of infrequent deviant tones which had either an enhanced duration or intensity. The participants were instructed to attend either to the right or left stream and to detect longer-duration deviants in the attended stream. The results showed that the early bilinguals did not outperform the controls in target detection accuracy or speed. However, the late portion of the attention-related ERP modulation (the negative difference, Nd) was larger over the left hemisphere in the early bilinguals than in the controls, suggesting that the maintenance of selective attention or further processing of selectively attended sounds is enhanced in the bilinguals. Moreover, the late reorienting negativity (RON) in response to intensity-deviant tones was larger in the bilinguals, suggesting more efficient disengagement of attention from distracting auditory events. Hence, our results demonstrate that brain responses associated with certain aspects of auditory attention are enhanced in the bilingual adults, indicating that early dual language exposure modulates the neuronal responsiveness of auditory modality.

TMS uncovers details about sub-regional language-specific processing networks in early bilinguals.

Despite numerous functional neuroimaging and intraoperative electrical cortical mapping studies aimed at investigating the cortical organisation of native (L1) and second (L2) language processing, the neural underpinnings of bilingualism remain elusive. We investigated whether the neural network engaged in speech production over the bilateral posterior inferior frontal gyrus (pIFG) is the same (i.e., shared) or different (i.e., language-specific) for the two languages of bilingual speakers. Navigated transcranial magnetic stimulation (TMS) was applied over the left and right posterior inferior gyrus (pIFG), while early simultaneous bilinguals performed a picture naming task with their native languages. An ex-Gaussian distribution was fitted to the naming latencies and the resulting parameters were compared between languages and across stimulation conditions. The results showed that although the naming performance in general was highly comparable between the languages, TMS produced a language-specific effect when the pulses were delivered to the left pIFG at 200 ms poststimulus. We argue that this result causally demonstrates, for the first time, that even within common language-processing areas, there are distinct language-specific neural populations for the different languages in early simultaneous bilinguals.

Language control mechanisms differ for native languages: Neuromagnetic evidence from trilingual language switching.

How does the brain process and control languages that are learned at a different age, when proficiency in all these languages is high? Early acquired strong languages are likely to have higher baseline activation levels than later learned less-dominant languages. However, it is still largely unknown how the activation levels of these different languages are controlled, and how interference from an irrelevant language is prevented. In this magnetoencephalography (MEG) study on language switching during auditory perception, early Finnish-Swedish bilinguals (N = 18) who mastered English with high proficiency after childhood were presented with spoken words in each of the three languages, while performing a simple semantic categorisation task. Switches from the later learned English to either of the native languages resulted in increased neural activation in the superior temporal gyrus (STG) 400-600ms after word onset (N400m response), whereas such increase was not detected for switches from native languages to English or between the native languages. In an earlier time window of 350-450ms, English non-switch trials showed higher activation levels in the inferior frontal gyrus (IFG), pointing to ongoing inhibition of the native languages during the use of English. Taken together, these asymmetric switch costs suggest that native languages are suppressed during the use of a non-native language, despite the receptive nature of the language task. This effect seems to be driven mostly by age of acquisition or language exposure, rather than proficiency. Our results indicate that mechanisms of control between two native languages differ from those of a later learned language, as upbringing in an early bilingual environment has likely promoted automatiation of language control specifically for the native languages.

Shaving Bridges and Tuning Kitaraa: The Effect of Language Switching on Semantic Processing.

Language switching has been repeatedly found to be costly. Yet, there are reasons to believe that switches in language might benefit language comprehension in some groups of people, such as less proficient language learners. This study therefore investigated the interplay between language switching and semantic processing in groups with varying language proficiency. EEG was recorded while L2 learners of English with intermediate and high proficiency levels read semantically congruent or incongruent sentences in L2. Translations of congruent and incongruent target words were additionally presented in L1 to create intrasentential language switches. A control group of English native speakers was tested in order to compare responses to non-switched stimuli with those of L2 learners. An omnibus ANOVA including all groups revealed larger N400 responses for non-switched incongruent stimuli compared to congruent stimuli. Additionally, despite switches to L1 at target word position, semantic N400 responses were still elicited in both L2 learner groups. Further switching effects were reflected by an N400-like effect and a late positivity complex, pointing to possible parsing efforts after language switches. Our results therefore show that although language switches are associated with increased mental effort, switches may not necessarily be costly on the semantic level. This finding contributes to the ongoing discussion on language inhibition processes, and shows that, in these intermediate and high proficient L2 learners, semantic processes look similar to those of native speakers of English.

Flexible, rapid and automatic neocortical word form acquisition mechanism in children as revealed by neuromagnetic brain response dynamics.

Children learn new words and word forms with ease, often acquiring a new word after very few repetitions. Recent neurophysiological research on word form acquisition in adults indicates that novel words can be acquired within minutes of repetitive exposure to them, regardless of the individual's focused attention on the speech input. Although it is well-known that children surpass adults in language acquisition, the developmental aspects of such rapid and automatic neural acquisition mechanisms remain unexplored. To address this open question, we used magnetoencephalography (MEG) to scrutinise brain dynamics elicited by spoken words and word-like sounds in healthy monolingual (Danish) children throughout a 20-min repetitive passive exposure session. We found rapid neural dynamics manifested as an enhancement of early (~100ms) brain activity over the short exposure session, with distinct spatiotemporal patterns for different novel sounds. For novel Danish word forms, signs of such enhancement were seen in the left temporal regions only, suggesting reliance on pre-existing language circuits for acquisition of novel word forms with native phonology. In contrast, exposure both to novel word forms with non-native phonology and to novel non-speech sounds led to activity enhancement in both left and right hemispheres, suggesting that more wide-spread cortical networks contribute to the build-up of memory traces for non-native and non-speech sounds. Similar studies in adults have previously reported more sluggish (~15-25min, as opposed to 4min in the present study) or non-existent neural dynamics for non-native sound acquisition, which might be indicative of a higher degree of plasticity in the children's brain. Overall, the results indicate a rapid and highly plastic mechanism for a dynamic build-up of memory traces for novel acoustic information in the children's brain that operates automatically and recruits bilateral temporal cortical circuits.

Bilingualism modulates the white matter structure of language-related pathways.

Learning and speaking a second language (L2) may result in profound changes in the human brain. Here, we investigated local structural differences along two language-related white matter trajectories, the arcuate fasciculus and the inferior fronto-occipital fasciculus (IFOF), between early simultaneous bilinguals and late sequential bilinguals. We also examined whether early exposure to two languages might lead to a more bilateral structural organization of the arcuate fasciculus. Fractional anisotropy, mean and radial diffusivities (FA, MD, and RD respectively) were extracted to analyse tract-specific changes. Additionally, global voxel-wise effects were investigated with Tract-Based Spatial Statistics (TBSS). We found that relative to late exposure, early exposure to L2 leads to increased FA along a phonology-related segment of the arcuate fasciculus, but induces no modulations along the IFOF, associated to semantic processing. Late sequential bilingualism, however, was associated with decreased MD along the bilateral IFOF. Our results suggest that early vs. late bilingualism may lead to qualitatively different kind of changes in the structural language-related network. Furthermore, we show that early bilingualism contributes to the structural laterality of the arcuate fasciculus, leading to a more bilateral organization of these perisylvian language-related tracts.

Acquisition and consolidation of novel morphology in human neocortex: A neuromagnetic study.

Research into neurobiological mechanisms of morphosyntactic processing of language has suggested specialised systems for decomposition and storage, which are used flexibly during the processing of complex polymorphemic words (such as those formed through affixation, e.g., boy + s = noun + plural marker or boy + ish = noun plus attenuator). However, neural underpinnings of acquisition of novel morphology are still unknown. We implicitly trained our participants with new derivational affixes through a word-picture association task and investigated the neural processes underlying formation of neural memory traces for new affixes. The participants' brain activity was recorded using magnetoencephalography (MEG), as they passively listened to the newly trained and untrained suffixes combined with real word and pseudoword stems. The MEG recording was repeated after a night's sleep using the same stimuli, to test the effects of overnight consolidation. The newly trained suffixes combined with real stems elicited stronger source activity in the left inferior frontal gyrus (LIFG) at ∼50 msec after the suffix onset than untrained suffixes, suggesting memory trace formation for the newly learned suffixes already on the same day. The following day, the suffix learning effect spread to the left superior temporal gyrus (STG) where it was again manifest as a response enhancement, particularly at ∼200-300 msec after the suffix onset, which might reflect an additional effect of overnight consolidation. Overall, the results demonstrate the rapid and dynamic processes of both immediate build-up and longer-term consolidation of neocortical memory traces for novel morphology, taking place after a short period of exposure to novel morphology and involving fronto-temporal perisylvian language circuitry.

Auditory evoked potentials to speech and nonspeech stimuli are associated with verbal skills in preschoolers.

Children's obligatory auditory event-related potentials (ERPs) to speech and nonspeech sounds have been shown to associate with reading performance in children at risk or with dyslexia and their controls. However, very little is known of the cognitive processes these responses reflect. To investigate this question, we recorded ERPs to semisynthetic syllables and their acoustically matched nonspeech counterparts in 63 typically developed preschoolers, and assessed their verbal skills with an extensive set of neurocognitive tests. P1 and N2 amplitudes were larger for nonspeech than speech stimuli, whereas the opposite was true for N4. Furthermore, left-lateralized P1s were associated with better phonological and prereading skills, and larger P1s to nonspeech than speech stimuli with poorer verbal reasoning performance. Moreover, left-lateralized N2s, and equal-sized N4s to both speech and nonspeech stimuli were associated with slower naming. In contrast, children with equal-sized N2 amplitudes at left and right scalp locations, and larger N4s for speech than nonspeech stimuli, performed fastest. We discuss the possibility that children’s ERPs reflect not only neural encoding of sounds, but also sound quality processing, memory-trace construction, and lexical access. The results also corroborate previous findings that speech and nonspeech sounds are processed by at least partially distinct neural substrates.

Rapid and automatic speech-specific learning mechanism in human neocortex.

A unique feature of human communication system is our ability to rapidly acquire new words and build large vocabularies. However, its neurobiological foundations remain largely unknown. In an electrophysiological study optimally designed to probe this rapid formation of new word memory circuits, we employed acoustically controlled novel word-forms incorporating native and non-native speech sounds, while manipulating the subjects' attention on the input. We found a robust index of neurolexical memory-trace formation: a rapid enhancement of the brain's activation elicited by novel words during a short (~30min) perceptual exposure, underpinned by fronto-temporal cortical networks, and, importantly, correlated with behavioural learning outcomes. Crucially, this neural memory trace build-up took place regardless of focused attention on the input or any pre-existing or learnt semantics. Furthermore, it was found only for stimuli with native-language phonology, but not for acoustically closely matching non-native words. These findings demonstrate a specialised cortical mechanism for rapid, automatic and phonology-dependent formation of neural word memory circuits.