Selective attention to the mouth of a talker in Japanese-learning infants and toddlers: Its relationship with vocabulary and compensation for noise.

Infants increasingly gaze at the mouth of talking faces during the latter half of the first postnatal year. This study investigated mouth-looking behavior of 120 full-term infants and toddlers (6 months-3 years) and 12 young adults (21-24 years) from Japanese monolingual families. The purpose of the study included: (1) Is such an attentional shift to the mouth in infancy similarly observed in Japanese environment where contribution of visual speech is known to be relatively weak? (2) Can noisy conditions increase mouth-looking behavior of Japanese young children? (3) Is the mouth-looking behavior related to language acquisition? To this end, movies of a talker speaking short phrases were presented while manipulating signal-to-noise ratio (SNR: Clear, SN+4, and SN-4). Expressive vocabulary of toddlers was obtained through their parents. The results indicated that Japanese infants initially have a strong preference for the eyes to mouth which is weakened toward 10 months, but the shift was later and in a milder fashion compared to known results for English-learning infants. Even after 10 months, no clear-cut preference for the mouth was observed even in linguistically challenging situations with strong noise until 3 years of age. In the Clear condition, there was a return of the gaze to the eyes as early as 3 years of age, where they showed increasing attention to the mouth with increasing noise level. In addition, multiple regression analyses revealed a tendency that 2- and 3-year-olds with larger vocabulary increasingly look at the eyes. Overall, the gaze of Japanese-learning infants and toddlers was more biased to the eyes in various aspects compared to known results of English-learning infants. The present findings shed new light on our understanding of the development of selective attention to the mouth in non-western populations.

Language/Culture Modulates Brain and Gaze Processes in Audiovisual Speech Perception.

Several behavioural studies have shown that the interplay between voice and face information in audiovisual speech perception is not universal. Native English speakers (ESs) are influenced by visual mouth movement to a greater degree than native Japanese speakers (JSs) when listening to speech. However, the biological basis of these group differences is unknown. Here, we demonstrate the time-varying processes of group differences in terms of event-related brain potentials (ERP) and eye gaze for audiovisual and audio-only speech perception. On a behavioural level, while congruent mouth movement shortened the ESs' response time for speech perception, the opposite effect was observed in JSs. Eye-tracking data revealed a gaze bias to the mouth for the ESs but not the JSs, especially before the audio onset. Additionally, the ERP P2 amplitude indicated that ESs processed multisensory speech more efficiently than auditory-only speech; however, the JSs exhibited the opposite pattern. Taken together, the ESs' early visual attention to the mouth was likely to promote phonetic anticipation, which was not the case for the JSs. These results clearly indicate the impact of language and/or culture on multisensory speech processing, suggesting that linguistic/cultural experiences lead to the development of unique neural systems for audiovisual speech perception.

Event-related potentials for better speech perception in noise by cochlear implant users.

Speech perception in noise is still difficult for cochlear implant (CI) users even with many years of CI use. This study aimed to investigate neurophysiological and behavioral foundations for CI-dependent speech perception in noise. Seventeen post-lingual CI users and twelve age-matched normal hearing adults participated in two experiments. In Experiment 1, CI users' auditory-only word perception in noise (white noise, two-talker babble; at 10 dB SNR) degraded by about 15%, compared to that in quiet (48% accuracy). CI users' auditory-visual word perception was generally better than auditory-only perception. Auditory-visual word perception was degraded under information masking by the two-talker noise (69% accuracy), compared to that in quiet (77%). Such degradation was not observed for white noise (77%), suggesting that the overcoming of information masking is an important issue for CI users' speech perception improvement. In Experiment 2, event-related cortical potentials were recorded in an auditory oddball task in quiet and noise (white noise only). Similarly to the normal hearing participants, the CI users showed the mismatch negative response (MNR) to deviant speech in quiet, indicating automatic speech detection. In noise, the MNR disappeared in the CI users, and only the good CI performers (above 66% accuracy) showed P300 (P3) like the normal hearing participants. P3 amplitude in the CI users was positively correlated with speech perception scores. These results suggest that CI users' difficulty in speech perception in noise is associated with the lack of automatic speech detection indicated by the MNR. Successful performance in noise may begin with attended auditory processing indicated by P3.