Language development beyond the here-and-now: Iconicity and displacement in child-directed communication.

Most language use is displaced, referring to past, future, or hypothetical events, posing the challenge of how children learn what words refer to when the referent is not physically available. One possibility is that iconic cues that imagistically evoke properties of absent referents support learning when referents are displaced. In an audio-visual corpus of caregiver-child dyads, English-speaking caregivers interacted with their children (N = 71, 24-58 months) in contexts in which the objects talked about were either familiar or unfamiliar to the child, and either physically present or displaced. The analysis of the range of vocal, manual, and looking behaviors caregivers produced suggests that caregivers used iconic cues especially in displaced contexts and for unfamiliar objects, using other cues when objects were present.

Prosodic modulations in child-directed language and their impact on word learning.

Child-directed language can support language learning, but how? We addressed two questions: (1) how caregivers prosodically modulated their speech as a function of word familiarity (known or unknown to the child) and accessibility of referent (visually present or absent from the immediate environment); (2) whether such modulations affect children's unknown word learning and vocabulary development. We used data from 38 English-speaking caregivers (from the ECOLANG corpus) talking about toys (both known and unknown to their children aged 3-4 years) both when the toys are present and when absent. We analyzed prosodic dimensions (i.e., speaking rate, pitch and intensity) of caregivers' productions of 6529 toy labels. We found that unknown labels were spoken with significantly slower speaking rate, wider pitch and intensity range than known labels, especially in the first mentions, suggesting that caregivers adjust their prosody based on children's lexical knowledge. Moreover, caregivers used slower speaking rate and larger intensity range to mark the first mentions of toys that were physically absent. After the first mentions, they talked about the referents louder with higher mean pitch when toys were present than when toys were absent. Crucially, caregivers' mean pitch of unknown words and the degree of mean pitch modulation for unknown words relative to known words (pitch ratio) predicted children's immediate word learning and vocabulary size 1 year later. In conclusion, caregivers modify their prosody when the learning situation is more demanding for children, and these helpful modulations assist children in word learning. RESEARCH HIGHLIGHTS: In naturalistic interactions, caregivers use slower speaking rate, wider pitch and intensity range when introducing new labels to 3-4-year-old children, especially in first mentions. Compared to when toys are present, caregivers speak more slowly with larger intensity range to mark the first mentions of toys that are physically absent. Mean pitch to mark word familiarity predicts children's immediate word learning and future vocabulary size.

Mouth and facial informativeness norms for 2276 English words.

Mouth and facial movements are part and parcel of face-to-face communication. The primary way of assessing their role in speech perception has been by manipulating their presence (e.g., by blurring the area of a speaker's lips) or by looking at how informative different mouth patterns are for the corresponding phonemes (or visemes; e.g., /b/ is visually more salient than /g/). However, moving beyond informativeness of single phonemes is challenging due to coarticulation and language variations (to name just a few factors). Here, we present mouth and facial informativeness (MaFI) for words, i.e., how visually informative words are based on their corresponding mouth and facial movements. MaFI was quantified for 2276 English words, varying in length, frequency, and age of acquisition, using phonological distance between a word and participants' speechreading guesses. The results showed that MaFI norms capture well the dynamic nature of mouth and facial movements per word, with words containing phonemes with roundness and frontness features, as well as visemes characterized by lower lip tuck, lip rounding, and lip closure being visually more informative. We also showed that the more of these features there are in a word, the more informative it is based on mouth and facial movements. Finally, we demonstrated that the MaFI norms generalize across different variants of English language. The norms are freely accessible via Open Science Framework ( https://osf.io/mna8j/ ) and can benefit any language researcher using audiovisual stimuli (e.g., to control for the effect of speech-linked mouth and facial movements).

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension.

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Linking language to sensory experience: Onomatopoeia in early language development.

A key question in developmental research concerns how children learn associations between words and meanings in their early language development. Given a vast array of possible referents, how does the child know what a word refers to? We contend that onomatopoeia (e.g. knock, meow), where a word's sound evokes the sound properties associated with its meaning, are particularly useful in children's early vocabulary development, offering a link between word and sensory experience not present in arbitrary forms. We suggest that, because onomatopoeia evoke imagery of the referent, children can draw from sensory experience to easily link onomatopoeic words to meaning, both when the referent is present as well as when it is absent. We use two sources of data: naturalistic observations of English-speaking caregiver-child interactions from 14 up to 54 months, to establish whether these words are present early in caregivers' speech to children, and experimental data to test whether English-speaking children can learn from onomatopoeia when it is present. Our results demonstrate that onomatopoeia: (a) are most prevalent in early child-directed language and in children's early productions, (b) are learnt more easily by children compared with non-iconic forms and (c) are used by caregivers in contexts where they can support communication and facilitate word learning.

The left inferior frontal gyrus: A neural crossroads between abstract and concrete knowledge.

Evidence from both neuropsychology and neuroimaging suggests that different types of information are necessary for representing and processing concrete and abstract word meanings. Both abstract and concrete concepts, however, conjointly rely on perceptual, verbal and contextual knowledge, with abstract concepts characterized by low values of imageability (IMG) (low sensory-motor grounding) and low context availability (CA) (more difficult to contextualize). Imaging studies supporting differences between abstract and concrete concepts show a greater recruitment of the left inferior frontal gyrus (LIFG) for abstract concepts, which has been attributed either to the representation of abstract-specific semantic knowledge or to the request for more executive control than in the case of concrete concepts. We conducted an fMRI study on 27 participants, using a lexical decision task involving both abstract and concrete words, whose IMG and CA values were explicitly modelled in separate parametric analyses. The LIFG was significantly more activated for abstract than for concrete words, and a conjunction analysis showed a common activation for words with low IMG or low CA only in the LIFG, in the same area reported for abstract words. A regional template map of brain activations was then traced for words with low IMG or low CA, and BOLD regional time-series were extracted and correlated with the specific LIFG neural activity elicited for abstract words. The regions associated to low IMG, which were functionally correlated with LIFG, were mainly in the left hemisphere, while those associated with low CA were in the right hemisphere. Finally, in order to reveal which LIFG-related network increased its connectivity with decreases of IMG or CA, we conducted generalized psychophysiological interaction analyses. The connectivity strength values extracted from each region connected with the LIFG were correlated with specific LIFG neural activity for abstract words, and a regression analysis was conducted to highlight which areas recruited by low IMG or low CA predicted the greater activation of the IFG for abstract concepts. Only the left middle temporal gyrus/angular gyrus, known to be involved in semantic processing, was a significant predictor of LIFG activity differentiating abstract from concrete words. The results show that the abstract conceptual processing requires the interplay of multiple brain regions, necessary for both the intrinsic and extrinsic properties of abstract knowledge. The LIFG can be thus identified as the neural crossroads between different types of information equally necessary for representing processing and differentiating abstract concepts from concrete ones.

Acquisition of abstract concepts is influenced by emotional valence.

There is considerable lack of evidence concerning the linguistic and cognitive skills underpinning abstract vocabulary acquisition. The present study considers the role of emotional valence in providing an embodied learning experience in which to anchor abstract meanings. First, analyses of adult ratings of age-of-acquisition, concreteness and valence demonstrate that abstract words acquired early tend to be emotionally valenced. Second, auditory Lexical Decision accuracies of children aged 6-7, 8-9, and 10-11 years (n = 20 per group) complement these analyses, demonstrating that emotional valence facilitates processing of abstract words, but not concrete. These findings provide the first evidence that young, school-aged children are sensitive to emotional valence and that this facilitates acquisition of abstract words.

Mapping language to the world: the role of iconicity in the sign language input.

Most research on the mechanisms underlying referential mapping has assumed that learning occurs in ostensive contexts, where label and referent co-occur, and that form and meaning are linked by arbitrary convention alone. In the present study, we focus on iconicity in language, that is, resemblance relationships between form and meaning, and on non-ostensive contexts, where label and referent do not co-occur. We approach the question of language learning from the perspective of the language input. Specifically, we look at child-directed language (CDL) in British Sign Language (BSL), a language rich in iconicity due to the affordances of the visual modality. We ask whether child-directed signing exploits iconicity in the language by highlighting the similarity mapping between form and referent. We find that CDL modifications occur more often with iconic signs than with non-iconic signs. Crucially, for iconic signs, modifications are more frequent in non-ostensive contexts than in ostensive contexts. Furthermore, we find that pointing dominates in ostensive contexts, and suggest that caregivers adjust the semiotic resources recruited in CDL to context. These findings offer first evidence for a role of iconicity in the language input and suggest that iconicity may be involved in referential mapping and language learning, particularly in non-ostensive contexts.

Impaired Comprehension of Speed Verbs in Parkinson's Disease.

OBJECTIVES: A wealth of studies provide evidence for action simulation during language comprehension. Recent research suggests such action simulations might be sensitive to fine-grained information, such as speed. Here, we present a crucial test for action simulation of speed in language by assessing speed comprehension in patients with Parkinson's disease (PD). Based on the patients' motor deficits, we hypothesized that the speed of motion described in language would modulate their performance in semantic tasks. Specifically, they would have more difficulty processing language about relatively fast speed than language about slow speed. METHODS: We conducted a semantic similarity judgment task on fast and slow action verbs in patients with PD and age-matched healthy controls. Participants had to decide which of two verbs most closely matched a target word. RESULTS: Compared to controls, PD patients were slower making judgments about fast action verbs, but not for judgments about slow action verbs, suggesting impairment in processing language about fast action. Moreover, this impairment was specific to verbs describing fast action performed with the hand. CONCLUSIONS: Problems moving quickly lead to difficulties comprehending language about moving quickly. This study provides evidence that speed is an important part of action representations. (JINS, 2017, 23, 412-420).

The neural representation of abstract words: the role of emotion.

It is generally assumed that abstract concepts are linguistically coded, in line with imaging evidence of greater engagement of the left perisylvian language network for abstract than concrete words (Binder JR, Desai RH, Graves WW, Conant LL. 2009. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex. 19:2767-2796; Wang J, Conder JA, Blitzer DN, Shinkareva SV. 2010. Neural representation of abstract and concrete concepts: A meta-analysis of neuroimaging studies. Hum Brain Map. 31:1459-1468). Recent behavioral work, which used tighter matching of items than previous studies, however, suggests that abstract concepts also entail affective processing to a greater extent than concrete concepts (Kousta S-T, Vigliocco G, Vinson DP, Andrews M, Del Campo E. The representation of abstract words: Why emotion matters. J Exp Psychol Gen. 140:14-34). Here we report a functional magnetic resonance imaging experiment that shows greater engagement of the rostral anterior cingulate cortex, an area associated with emotion processing (e.g., Etkin A, Egner T, Peraza DM, Kandel ER, Hirsch J. 2006. Resolving emotional conflict: A role for the rostral anterior cingulate cortex in modulating activity in the amygdala. Neuron. 52:871), in abstract processing. For abstract words, activation in this area was modulated by the hedonic valence (degree of positive or negative affective association) of our items. A correlation analysis of more than 1,400 English words further showed that abstract words, in general, receive higher ratings for affective associations (both valence and arousal) than concrete words, supporting the view that engagement of emotional processing is generally required for processing abstract words. We argue that these results support embodiment views of semantic representation, according to which, whereas concrete concepts are grounded in our sensory-motor experience, affective experience is crucial in the grounding of abstract concepts.

How does emotional content affect lexical processing?

Even single words in isolation can evoke emotional reactions, but the mechanisms by which emotion is involved in automatic lexical processing are unclear. Previous studies using extremely similar materials and methods have yielded apparently incompatible patterns of results. In much previous work, however, words' emotional content is entangled with other non-emotional characteristics such as frequency of occurrence, familiarity and age of acquisition, all of which have potential consequences for lexical processing themselves. In the present study, the authors compare different models of emotion using the British Lexicon Project, a large-scale freely available lexical decision database. After controlling for the potentially confounding effects of non-emotional variables, a variety of statistical approaches revealed that emotional words, whether positive or negative, are processed faster than neutral words. This effect appears to be categorical rather than graded; is not modulated by emotional arousal; and is not limited to words explicitly referring to emotions. The authors suggest that emotional connotations facilitate processing due to the grounding of words' meanings in emotional experience.

Communicative efficiency in multimodal language directed at children and adults.

The ecology of human communication is face to face. In these contexts, speakers dynamically modify their communication across vocal (e.g., speaking rate) and gestural (e.g., cospeech gestures related in meaning to the content of speech) channels while speaking. What is the function of these adjustments? Here we ask whether speakers dynamically make these adjustments to increase communicative success, and decrease cognitive effort while speaking. We assess whether speakers modulate word durations and produce iconic (i.e., imagistically evoking properties of referents) gestures depending on the predictability of each word they utter. Predictability is operationalized as surprisal and computed from computational language models trained on corpora of child-directed, or adult-directed language. Using data from a novel corpus (Ecological Language Corpus) of naturalistic interactions between adult-child (aged 3-4), and adult-adult, we show that surprisal predicts speakers' multimodal adjustments and that some of these effects are modulated by whether the comprehender is a child or an adult. Thus, communicative efficiency applies generally across vocal and gestural communicative channels not being limited to structural properties of language or vocal modality. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Reading time data for evaluating broad-coverage models of English sentence processing.

We make available word-by-word self-paced reading times and eye-tracking data over a sample of English sentences from narrative sources. These data are intended to form a gold standard for the evaluation of computational psycholinguistic models of sentence comprehension in English. We describe stimuli selection and data collection and present descriptive statistics, as well as comparisons between the two sets of reading times.

I don't see what you're saying: The maluma/takete effect does not depend on the visual appearance of phonemes as they are articulated.

In contrast to the principle of arbitrariness, recent work has shown that language can iconically depict referents being talked about. One such example is the maluma/takete effect: an association between certain phonemes (e.g., those in maluma) and round shapes, and other phonemes (e.g., those in takete and spiky shapes). An open question has been whether this association is crossmodal (arising from phonemes' sound or kinesthetics) or unimodal (arising from phonemes' visual appearance). In the latter case, individuals may associate a person's rounded lips as they pronounce the /u/ in maluma with round shapes. We examined this hypothesis by having participants pair nonwords with shapes in either an audio-only condition (they only heard nonwords) or an audiovisual condition (they both heard nonwords and saw them articulated). We found no evidence that seeing nonwords articulated enhanced the maluma/takete effect. In fact, there was evidence that it decreased it in some cases. This was confirmed with a Bayesian analysis. These results eliminate a plausible explanation for the maluma/takete effect, as an instance of visual matching. We discuss the alternate possibility that it involves crossmodal associations.

Effects of Iconicity in Recognition Memory.

Iconicity refers to a resemblance between word form and meaning. Previous work has shown that iconic words are learned earlier and processed faster. Here, we examined whether iconic words are recognized better on a recognition memory task. We also manipulated the level at which items were encoded-with a focus on either their meaning or their form-in order to gain insight into the mechanism by which iconicity would affect memory. In comparison with non-iconic words, iconic words were associated with a higher false alarm rate, a lower d' score, and a lower response criterion in Experiment 1. We did not observe any interaction between iconicity and encoding condition. To test the generalizability of these findings, we examined effects of iconicity in a recognition memory megastudy across 3880 items. After controlling for a variety of lexical and semantic variables, iconicity was predictive of more hits and false alarms, and a lower response criterion in this dataset. In Experiment 2, we examined whether these effects were due to increased feelings of familiarity for iconic items by including a familiar versus recollect decision. This experiment replicated the overall results of Experiment 1 and found that participants were more likely to categorize words that they had seen before as familiar (vs. recollected) if they were iconic. Together, these results demonstrate that iconicity has an effect on memory. We discuss implications for theories of iconicity.

Learning from others is good, with others is better: the role of social interaction in human acquisition of new knowledge.

Learning in humans is highly embedded in social interaction: since the very early stages of our lives, we form memories and acquire knowledge about the world from and with others. Yet, within cognitive science and neuroscience, human learning is mainly studied in isolation. The focus of past research in learning has been either exclusively on the learner or (less often) on the teacher, with the primary aim of determining developmental trajectories and/or effective teaching techniques. In fact, social interaction has rarely been explicitly taken as a variable of interest, despite being the medium through which learning occurs, especially in development, but also in adulthood. Here, we review behavioural and neuroimaging research on social human learning, specifically focusing on cognitive models of how we acquire semantic knowledge from and with others, and include both developmental as well as adult work. We then identify potential cognitive mechanisms that support social learning, and their neural correlates. The aim is to outline key new directions for experiments investigating how knowledge is acquired in its ecological niche, i.e. socially, within the framework of the two-person neuroscience approach. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.

Benefit of visual speech information for word comprehension in post-stroke aphasia.

Aphasia is a language disorder that often involves speech comprehension impairments affecting communication. In face-to-face settings, speech is accompanied by mouth and facial movements, but little is known about the extent to which they benefit aphasic comprehension. This study investigated the benefit of visual information accompanying speech for word comprehension in people with aphasia (PWA) and the neuroanatomic substrates of any benefit. Thirty-six PWA and 13 neurotypical matched control participants performed a picture-word verification task in which they indicated whether a picture of an animate/inanimate object matched a subsequent word produced by an actress in a video. Stimuli were either audiovisual (with visible mouth and facial movements) or auditory-only (still picture of a silhouette) with audio being clear (unedited) or degraded (6-band noise-vocoding). We found that visual speech information was more beneficial for neurotypical participants than PWA, and more beneficial for both groups when speech was degraded. A multivariate lesion-symptom mapping analysis for the degraded speech condition showed that lesions to superior temporal gyrus, underlying insula, primary and secondary somatosensory cortices, and inferior frontal gyrus were associated with reduced benefit of audiovisual compared to auditory-only speech, suggesting that the integrity of these fronto-temporo-parietal regions may facilitate cross-modal mapping. These findings provide initial insights into our understanding of the impact of audiovisual information on comprehension in aphasia and the brain regions mediating any benefit.

The role of multimodal cues in second language comprehension.

In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues - meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal - while using multimodal cues to a lesser extent than L1 comprehenders overall.

Consensus Paper: Situated and Embodied Language Acquisition.

Theories of embodied cognition postulate that perceptual, sensorimotor, and affective properties of concepts support language learning and processing. In this paper, we argue that language acquisition, as well as processing, is situated in addition to being embodied. In particular, first, it is the situated nature of initial language development that affords for the developing system to become embodied. Second, the situated nature of language use changes across development and adulthood. We provide evidence from empirical studies for embodied effects of perception, action, and valence as they apply to both embodied cognition and situated cognition across developmental stages. Although the evidence is limited, we urge researchers to consider differentiating embodied cognition within situated context, in order to better understand how these separate mechanisms interact for learning to occur. This delineation also provides further clarity to the study of classroom-based applications and the role of embodied and situated cognition in the study of developmental disorders. We argue that theories of language acquisition need to address for the complex situated context of real-world learning by completing a "circular notion": observing experimental paradigms in real-world settings and taking these observations to later refine lab-based experiments.

The road to language learning is iconic: evidence from British Sign Language.

An arbitrary link between linguistic form and meaning is generally considered a universal feature of language. However, iconic (i.e., nonarbitrary) mappings between properties of meaning and features of linguistic form are also widely present across languages, especially signed languages. Although recent research has shown a role for sign iconicity in language processing, research on the role of iconicity in sign-language development has been mixed. In this article, we present clear evidence that iconicity plays a role in sign-language acquisition for both the comprehension and production of signs. Signed languages were taken as a starting point because they tend to encode a higher degree of iconic form-meaning mappings in their lexicons than spoken languages do, but our findings are more broadly applicable: Specifically, we hypothesize that iconicity is fundamental to all languages (signed and spoken) and that it serves to bridge the gap between linguistic form and human experience.