Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language.

When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers' visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers' latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

Predicting Social Competence in Autistic and Non-Autistic Children: Effects of Prosody and the Amount of Speech Input.

PURPOSE: Autistic individuals often face challenges perceiving and expressing emotions, potentially stemming from differences in speech prosody. Here we explore how autism diagnoses between groups, and measures of social competence within groups may be related to, first, children's speech characteristics (both prosodic features and amount of spontaneous speech), and second, to these two factors in mothers' speech to their children. METHODS: Autistic (n = 21) and non-autistic (n = 18) children, aged 7-12 years, participated in a Lego-building task with their mothers, while conversational speech was recorded. Mean F0, pitch range, pitch variability, and amount of spontaneous speech were calculated for each child and their mother. RESULTS: The results indicated no differences in speech characteristics across autistic and non-autistic children, or across their mothers, suggesting that conversational context may have large effects on whether differences between autistic and non-autistic populations are found. However, variability in social competence within the group of non-autistic children (but not within autistic children) was predictive of children's mean F0, pitch range and pitch variability. The amount of spontaneous speech produced by mothers (but not their prosody) predicted their autistic children's social competence, which may suggest a heightened impact of scaffolding for mothers of autistic children. CONCLUSION: Together, results suggest complex interactions between context, social competence, and adaptive parenting strategies in driving prosodic differences in children's speech.

Sensorimotor foundations of speech perception in infancy.

The perceptual system for speech is highly organized from early infancy. This organization bootstraps young human learners' ability to acquire their native speech and language from speech input. Here, we review behavioral and neuroimaging evidence that perceptual systems beyond the auditory modality are also specialized for speech in infancy, and that motor and sensorimotor systems can influence speech perception even in infants too young to produce speech-like vocalizations. These investigations complement existing literature on infant vocal development and on the interplay between speech perception and production systems in adults. We conclude that a multimodal speech and language network is present before speech-like vocalizations emerge.

Contextual effects on spoken word processing: An eye-tracking study of the time course of tone and vowel activation in Mandarin.

Lexical access is highly contextual. For example, vowel (rime) information is prioritized over tone in the lexical access of isolated words in Mandarin Chinese, but these roles are flipped in constraining contexts. The time course of these contextual effects remains unclear, and so here we tracked the real-time eye gaze of native Mandarin speakers in a visual-world paradigm. While listening to a noun classifier, before the target noun was even uttered, gaze to the target noun was already greater than looking to phonologically unrelated distractors. Critically, there was also more distraction from a cohort competitor (tone information) than a segmental competitor (vowel information) in more semantically constraining contexts. Results confirm that phonological activation in Mandarin lexical access is highly sensitive to context, with tone taking priority over vowel information even before a target word is heard. Results suggest that phonological activation in real-time lexical access may be highly context-specific across languages. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Developmental change in children's speech processing of auditory and visual cues: An eyetracking study.

This study investigates how children aged two to eight years (N = 129) and adults (N = 29) use auditory and visual speech for word recognition. The goal was to bridge the gap between apparent successes of visual speech processing in young children in visual-looking tasks, with apparent difficulties of speech processing in older children from explicit behavioural measures. Participants were presented with familiar words in audio-visual (AV), audio-only (A-only) or visual-only (V-only) speech modalities, then presented with target and distractor images, and looking to targets was measured. Adults showed high accuracy, with slightly less target-image looking in the V-only modality. Developmentally, looking was above chance for both AV and A-only modalities, but not in the V-only modality until 6 years of age (earlier on /k/-initial words). Flexible use of visual cues for lexical access develops throughout childhood.

Rethinking the phonetics of baby-talk: Differences across Canada and Vanuatu in the articulation of mothers' speech to infants.

Infant-directed speech (IDS) is phonetically distinct from adult-directed speech (ADS): It is typically considered to have special prosody-like higher pitch and slower speaking rates-as well as unique speech sound properties, for example, more breathy, hyperarticulated, and/or variable consonant and vowel articulation. These phonetic features are widely observed in the IDS of caregivers from urbanized contexts who speak a handful of very well-researched languages. Yet studies with more diverse socio-cultural and linguistic samples show that this "typical" IDS prosody is not consistently observed across cultures. We extended cross-cultural work by examining IDS speech segment articulation, which-like prosody-is also thought to be a characteristic phonetic feature of IDS that might aid speech and language development. Here we asked whether IDS vowels have different articulatory features compared to ADS vowels in two distinct linguistic and socio-cultural contexts: urban English-speaking Canadian mothers, and rural Lenakel- and Southwest Tanna-speaking ni-Vanuatu mothers (n = 57, 20-46 years of age). Replicating prior work, Canadian mothers had more variable vowels in IDS compared to ADS, but also did not show clear register differences for breathiness or hyperarticulation. Vowels spoken by ni-Vanuatu mothers showed very distinct articulatory tendencies, using less variable (and less breathy) IDS vowels. Along with other work showing diversity in IDS phonetics across populations, this paper suggests that any understanding of how IDS might aid speech and language development are best examined through a culturally- and linguistically-specific lens.

Postural control of the vocal tract affects auditory speech perception.

Many researchers have proposed that sensorimotor information about the dynamic production of speech gestures can supplement the auditory perception of speech. Here we show that information about postural, nonspeech control of the vocal tract-such as breathing through the nose or mouth-also affects speech perception. Experimental participants breathed either through the nose or the mouth while identifying categories of speech sounds differing in nasal versus oral airflow. Participants showed an increased tendency to hear speech sounds as having nasal articulation when breathing through the nose, relative to when breathing through the mouth. These results suggest that postural information about the state of the vocal tract, like the motor configuration of the speech articulators while breathing, can modulate the perceptual processing of speech sounds. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Tone slips in Cantonese: Evidence for early phonological encoding.

This article examines speech errors in Cantonese with the aim of fleshing out a larger speech production architecture for encoding phonological tone. A corpus was created by extracting 2462 speech errors, including 668 tone errors, from audio recordings of natural conversations. The structure of these errors was then investigated in order to distinguish two contemporary approaches to tone in speech production. In the tonal frames account, tone is encoded like metrical stress, represented in abstract structural frames for a word. Because tone cannot be mis-selected in tonal frames, tone errors are expected to be rare and non-contextual, as observed with stress. An alternative is that tone is actively selected in phonological encoding like phonological segments. This approach predicts that tone errors will be relatively common and exhibit the contextual patterns observed with segments, like perseveration and anticipation. In our corpus, tone errors are the second most common type of error, and the majority of errors exhibit contextual patterns that parallel segmental errors. Building on prior research, a two-stage model of phonological tone encoding is proposed, following the patterns seen in tone errors: Tone is phonologically selected concurrently with segments, but then sequentially assigned after segments to a syllable.

Visual scanning of a talking face in preterm and full-term infants.

Preterm birth (< 37 gestational weeks) is associated with long-term risks for health and neurodevelopment, but recently, studies have also started exploring how preterm birth affects early language development in the 1st year of life. Because the timing and quality of auditory and visual input is very different for preterm versus full-term infants, audiovisual speech perception in early development may be particularly sensitive to preterm birth. We tested extremely preterm to late preterm infants at 8 months postnatal age (28 to 36 weeks of gestation), as well as 2 full-term comparison groups with similar postnatal (8 months) and maturational (6 months) ages, on visual scanning of a video showing a French-English bilingual woman speaking in the infants' native language (French) and a nonnative language (English). Preterm infants showed similar scanning patterns for both languages, failing to differentiate between native and nonnative languages in their looking, unlike both groups of full-term infants, who looked more to the eyes than the mouth for the native language compared with the nonnative language. No clear relationship between scanning patterns and degree of prematurity was found. These findings are the first to show that audiovisual speech perception is affected in even later-born preterm infants, thus identifying a particularly sensitive deficit in early speech processing. Further research will need to investigate how preterms' special vulnerability in audiovisual speech processing may contribute to the other language difficulties found in these populations. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Infants' statistical word segmentation in an artificial language is linked to both parental speech input and reported production abilities.

Individual variability in infant's language processing is partly explained by environmental factors, like the quantity of parental speech input, as well as by infant-specific factors, like speech production. Here, we explore how these factors affect infant word segmentation. We used an artificial language to ensure that only statistical regularities (like transitional probabilities between syllables) could cue word boundaries, and then asked how the quantity of parental speech input and infants' babbling repertoire predict infants' abilities to use these statistical cues. We replicated prior reports showing that 8-month-old infants use statistical cues to segment words, with a preference for part-words over words (a novelty effect). Crucially, 8-month-olds with larger novelty effects had received more speech input at 4 months and had greater production abilities at 8 months. These findings establish for the first time that the ability to extract statistical information from speech correlates with individual factors in infancy, like early speech experience and language production. Implications of these findings for understanding individual variability in early language acquisition are discussed.

Perceptual Reorganization of Lexical Tones: Effects of Age and Experimental Procedure.

Findings on the perceptual reorganization of lexical tones are mixed. Some studies report good tone discrimination abilities for all tested age groups, others report decreased or enhanced discrimination with increasing age, and still others report U-shaped developmental curves. Since prior studies have used a wide range of contrasts and experimental procedures, it is unclear how specific task requirements interact with discrimination abilities at different ages. In the present work, we tested German and Cantonese adults on their discrimination of Cantonese lexical tones, as well as German-learning infants between 6 and 18 months of age on their discrimination of two specific Cantonese tones using two different types of experimental procedures. The adult experiment showed that German native speakers can discriminate between lexical tones, but native Cantonese speakers show significantly better performance. The results from German-learning infants suggest that 6- and 18-month-olds discriminate tones, while 9-month-olds do not, supporting a U-shaped developmental curve. Furthermore, our results revealed an effect of methodology, with good discrimination performance at 6 months after habituation but not after familiarization. These results support three main conclusions. First, habituation can be a more sensitive procedure for measuring infants' discrimination than familiarization. Second, the previous finding of a U-shaped curve in the discrimination of lexical tones is further supported. Third, discrimination abilities at 18 months appear to reflect mature perceptual sensitivity to lexical tones, since German adults also discriminated the lexical tones with high accuracy.

The many facets of speech production and its complex effects on phonological processing.

In our commentary, we discuss two additional points about developmental speech production. First, we suggest that more precision is needed to accurately describe 'speech production' processes, and we suggest that hierarchical constructs from the adult literatures on articulatory phonology and speech motor control may be applicable to infants as well. Second, we discuss the implications from data that indicate that the effects of production are subject to task-, attentional-, linguistic-, and experience-related demands.

Infants' Looking to Surprising Events: When Eye-Tracking Reveals More than Looking Time.

Research on infants' reasoning abilities often rely on looking times, which are longer to surprising and unexpected visual scenes compared to unsurprising and expected ones. Few researchers have examined more precise visual scanning patterns in these scenes, and so, here, we recorded 8- to 11-month-olds' gaze with an eye tracker as we presented a sampling event whose outcome was either surprising, neutral, or unsurprising: A red (or yellow) ball was drawn from one of three visible containers populated 0%, 50%, or 100% with identically colored balls. When measuring looking time to the whole scene, infants were insensitive to the likelihood of the sampling event, replicating failures in similar paradigms. Nevertheless, a new analysis of visual scanning showed that infants did spend more time fixating specific areas-of-interest as a function of the event likelihood. The drawn ball and its associated container attracted more looking than the other containers in the 0% condition, but this pattern was weaker in the 50% condition, and even less strong in the 100% condition. Results suggest that measuring where infants look may be more sensitive than simply how much looking there is to the whole scene. The advantages of eye tracking measures over traditional looking measures are discussed.

Sound symbolism facilitates word learning in 14-month-olds.

Sound symbolism, or the nonarbitrary link between linguistic sound and meaning, has often been discussed in connection with language evolution, where the oral imitation of external events links phonetic forms with their referents (e.g., Ramachandran & Hubbard, 2001). In this research, we explore whether sound symbolism may also facilitate synchronic language learning in human infants. Sound symbolism may be a useful cue particularly at the earliest developmental stages of word learning, because it potentially provides a way of bootstrapping word meaning from perceptual information. Using an associative word learning paradigm, we demonstrated that 14-month-old infants could detect Köhler-type (1947) shape-sound symbolism, and could use this sensitivity in their effort to establish a word-referent association.

Foreign language learning in French speakers is associated with rhythm perception, but not with melody perception.

There has been increasing interest in links between language and music. Here, we investigate the relation between foreign language learning and music perception. We administered tests measuring melody and rhythm perception as well as a questionnaire on musical and foreign language experience to 147 monolingual French speakers. As expected, we found that musicians had better melody and rhythm perception than nonmusicians and that, among musicians, there was a positive correlation between the total number of years of music training and test scores. Crucially, we also found a positive correlation between the total number of years learning foreign languages and rhythm perception, but we found no such relation with melody perception. Moreover, the degree to which participants were better at rhythm than melody perception was also related to foreign language experience. Results suggest that both music training and learning foreign languages (primarily English, Spanish, and German in our sample) are related to French speakers' perception of rhythm, but not to their perception of melody. These results are discussed with respect to the rhythmic properties of French and suggest a common perceptual basis for rhythm in language and music.

The development of sensorimotor influences in the audiovisual speech domain: some critical questions.

Speech researchers have long been interested in how auditory and visual speech signals are integrated, and the recent work has revived interest in the role of speech production with respect to this process. Here, we discuss these issues from a developmental perspective. Because speech perception abilities typically outstrip speech production abilities in infancy and childhood, it is unclear how speech-like movements could influence audiovisual speech perception in development. While work on this question is still in its preliminary stages, there is nevertheless increasing evidence that sensorimotor processes (defined here as any motor or proprioceptive process related to orofacial movements) affect developmental audiovisual speech processing. We suggest three areas on which to focus in future research: (i) the relation between audiovisual speech perception and sensorimotor processes at birth, (ii) the pathways through which sensorimotor processes interact with audiovisual speech processing in infancy, and (iii) developmental change in sensorimotor pathways as speech production emerges in childhood.

Referential labeling can facilitate phonetic learning in infancy.

All languages employ certain phonetic contrasts when distinguishing words. Infant speech perception is rapidly attuned to these contrasts before many words are learned, thus phonetic attunement is thought to proceed independently of lexical and referential knowledge. Here, evidence to the contrary is provided. Ninety-eight 9-month-old English-learning infants were trained to perceive a non-native Cantonese tone contrast.Two object­tone audiovisual pairings were consistently presented, which highlighted the target contrast (Object A with Tone X; Object B with Tone Y). Tone discrimination was then assessed. Results showed improved tone discrimination if object­tone pairings were perceived as being referential word labels, although this effect was modulated by vocabulary size. Results suggest how lexical and referential knowledge could play a role in phonetic attunement.

Object labeling influences infant phonetic learning and generalization.

Different kinds of speech sounds are used to signify possible word forms in every language. For example, lexical stress is used in Spanish (/'be.be/, 'he/she drinks' versus /be.'be/, 'baby'), but not in French (/'be.be/ and /be.'be/ both mean 'baby'). Infants learn many such native language phonetic contrasts in their first year of life, likely using a number of cues from parental speech input. One such cue could be parents' object labeling, which can explicitly highlight relevant contrasts. Here we ask whether phonetic learning from object labeling is abstract-that is, if learning can generalize to new phonetic contexts. We investigate this issue in the prosodic domain, as the abstraction of prosodic cues (like lexical stress) has been shown to be particularly difficult. One group of 10-month-old French-learners was given consistent word labels that contrasted on lexical stress (e.g., Object A was labeled /'ma.bu/, and Object B was labeled /ma.'bu/). Another group of 10-month-olds was given inconsistent word labels (i.e., mixed pairings), and stress discrimination in both groups was measured in a test phase with words made up of new syllables. Infants trained with consistently contrastive labels showed an earlier effect of discrimination compared to infants trained with inconsistent labels. Results indicate that phonetic learning from object labeling can indeed generalize, and suggest one way infants may learn the sound properties of their native language(s).

Inner speech captures the perception of external speech.

Talking silently to ourselves occupies much of our mental lives, yet the mechanisms underlying this experience remain unclear. The following experiments provide behavioral evidence that the auditory content of inner speech is provided by corollary discharge. Corollary discharge is the motor system's prediction of the sensory consequences of its actions. This prediction can bias perception of other sensations, pushing percepts to match with prediction. The two experiments below show this bias induced by inner speech, demonstrating that inner speech causes external sounds to be heard as similar to the imagined speech, and that this bias operates on subphonemic content.

Lip movements affect infants' audiovisual speech perception.

Speech is robustly audiovisual from early in infancy. Here we show that audiovisual speech perception in 4.5-month-old infants is influenced by sensorimotor information related to the lip movements they make while chewing or sucking. Experiment 1 consisted of a classic audiovisual matching procedure, in which two simultaneously displayed talking faces (visual [i] and [u]) were presented with a synchronous vowel sound (audio /i/ or /u/). Infants' looking patterns were selectively biased away from the audiovisual matching face when the infants were producing lip movements similar to those needed to produce the heard vowel. Infants' looking patterns returned to those of a baseline condition (no lip movements, looking longer at the audiovisual matching face) when they were producing lip movements that did not match the heard vowel. Experiment 2 confirmed that these sensorimotor effects interacted with the heard vowel, as looking patterns differed when infants produced these same lip movements while seeing and hearing a talking face producing an unrelated vowel (audio /a/). These findings suggest that the development of speech perception and speech production may be mutually informative.