Multi-modal cross-linguistic perception of Mandarin tones in clear speech.

Clearly enunciated speech (relative to conversational, plain speech) involves articulatory and acoustic modifications that enhance auditory-visual (AV) segmental intelligibility. However, little research has explored clear-speech effects on the perception of suprasegmental properties such as lexical tone, particularly involving visual (facial) perception. Since tone production does not primarily rely on vocal tract configurations, tones may be less visually distinctive. Questions thus arise as to whether clear speech can enhance visual tone intelligibility, and if so, whether any intelligibility gain can be attributable to tone-specific category-enhancing (code-based) clear-speech cues or tone-general saliency-enhancing (signal-based) cues. The present study addresses these questions by examining the identification of clear and plain Mandarin tones with visual-only, auditory-only, and AV input modalities by native (Mandarin) and nonnative (English) perceivers. Results show that code-based visual and acoustic clear tone modifications, although limited, affect both native and nonnative intelligibility, with category-enhancing cues increasing intelligibility and category-blurring cues decreasing intelligibility. In contrast, signal-based cues, which are extensively available, do not benefit native intelligibility, although they contribute to nonnative intelligibility gain. These findings demonstrate that linguistically relevant visual tonal cues are existent. In clear speech, such tone category-enhancing cues are incorporated with saliency-enhancing cues across AV modalities for intelligibility improvements.

Intonation processing of interrogative words in Mandarin: an event-related potential study.

Intonation is the variation in pitch used in speech, which forms the premise of tonal and non-tonal languages. Interrogative words are words that introduce questions. Previous research lacks clarity regarding the specific cues used in the processing of word intonation. To address this gap, this study used the event-related potential electroencephalogram (EEG) research method to explore the intonation processing of tone two (mid-rising) interrogative words in Mandarin. For this, the word "shui," meaning "who," was selected as the experimental material. To avoid the influence of the environment, gender, and semantics, the Hum version, corresponding to the stimulus material, was also adopted for the experiment. This study used a passive oddball paradigm to examine the clues of intonation information processing in automatic cognitive processing through amplitude, latency, time window, and evoked location potential mismatch negativity. The standard stimulus was the declarative intonation with a high probability of occurrence (90%), and the deviant stimulus was the interrogative intonation with a low probability of occurrence (10%). In the time window of 370-450 ms, the mismatch negativity was found at the F3, F4, C3, Cz, and C4 channels. The findings show that, in the passive oddball paradigm, lexical semantics are essential for intonation processing at the pre-attentive level, which is dominated by the frontal and central areas of the brain. The results support the functional and comprehensive hypotheses that the processing of intonation is based on the function of language and that bilateral regions are involved in this processing. This study makes an important contribution by providing event-related potential evidence that lexical semantics plays a key role in the pre-attentive processing of intonation, as shown by the significant differences between semantic and non-semantic conditions.

Cochlear-implant Mandarin tone recognition with a disyllabic word corpus.

Despite pitch being considered the primary cue for discriminating lexical tones, there are secondary cues such as loudness contour and duration, which may allow some cochlear implant (CI) tone discrimination even with severely degraded pitch cues. To isolate pitch cues from other cues, we developed a new disyllabic word stimulus set (Di) whose primary (pitch) and secondary (loudness) cue varied independently. This Di set consists of 270 disyllabic words, each having a distinct meaning depending on the perceived tone. Thus, listeners who hear the primary pitch cue clearly may hear a different meaning from listeners who struggle with the pitch cue and must rely on the secondary loudness contour. A lexical tone recognition experiment was conducted, which compared Di with a monosyllabic set of natural recordings. Seventeen CI users and eight normal-hearing (NH) listeners took part in the experiment. Results showed that CI users had poorer pitch cues encoding and their tone recognition performance was significantly influenced by the "missing" or "confusing" secondary cues with the Di corpus. The pitch-contour-based tone recognition is still far from satisfactory for CI users compared to NH listeners, even if some appear to integrate multiple cues to achieve high scores. This disyllabic corpus could be used to examine the performance of pitch recognition of CI users and the effectiveness of pitch cue enhancement based Mandarin tone enhancement strategies. The Di corpus is freely available online: https://github.com/BetterCI/DiTone.

N400 Evidence That the Early Stages of Lexical Access Ignore Knowledge About Phonological Alternations.

Hearing a word that was already expected often facilitates comprehension, attenuating the amplitude of the N400 event-related brain potential component. On the other hand, hearing a word that was not expected elicits a larger N400. In the present study, we examined whether the N400 would be attenuated when a person hears something that is not exactly what they expected but is a viable alternative pronunciation of the morpheme they expected. This was done using Mandarin syllables, some of which can be pronounced with different lexical tones depending on the context. In two large-sample experiments (total n = 160) testing syllables in isolation and in phonologically viable contexts, we found little evidence that hearing an alternative pronunciation of the expected word attenuates the N400. These results suggest that comprehenders do not take advantage of their knowledge about systematic phonological alternations during the early stages of prediction or discrimination.

Processing Mandarin Tone 3 Sandhi at the Morphosyntactic Interface: Reduplication and Lexical Compounds.

Mandarin tone 3 sandhi is a phonological alternation in which the initial tone 3 (i.e., low tone) syllable changes to a tone 2 (i.e., rising tone) when followed by another tone 3. The present study used a cross-modal syllable-morpheme matching experiment to examine how native speakers process the sandhi sequences derived from verb reduplication and compounding, respectively. Embedded in a visually-presented sentential context, a disyllabic sequence containing a sandhi target was displayed simultaneously with a monosyllabic audio, either a tone 1 (i.e., high-level tone), tone 2 (i.e., rising tone) or tone 3 (i.e., low tone), and participants judged whether the audio syllable matched the visual morpheme. Results showed that the tone 3 sandhi was processed differently in the two constructions. The underlying tone and the surface tone were co-activated and competed with each other in sandhi compounds whereas predominant activation of the underlying tone, over the surface tone, was observed in reduplication. The processing of tone 3 sandhi offers support for distinctive morphological structures: a lexical compound is represented both as a whole-word unit and as a combination of two individual morphemes whereas a verb reduplication is represented and accessed as a monomorphemic unit in the mental lexicon.

Emerging native-similar neural representations underlie non-native speech category learning success.

Learning non-native phonetic categories in adulthood is an exceptionally challenging task, characterized by large inter-individual differences in learning speed and outcomes. The neurobiological mechanisms underlying the inter-individual differences in the learning efficacy are not fully understood. Here we examined the extent to which training-induced neural representations of non-native Mandarin tone categories in English listeners (n = 53) are increasingly similar to those of the native listeners (n = 33) who acquired these categories early in infancy. We particularly assessed whether the neural similarities in representational structure between non-native learners and native listeners are robust neuromarkers of inter-individual differences in learning success. Using inter-subject neural representational similarity (IS-NRS) analysis and predictive modeling on two functional magnetic resonance imaging (fMRI) datasets, we examined the neural representational mechanisms underlying speech category learning success. Learners' neural representations that were significantly similar to the native listeners emerged in brain regions mediating speech perception following training; the extent of the emerging neural similarities with native listeners significantly predicted the learning speed and outcome in learners. The predictive power of IS-NRS outperformed models with other neural representational measures. Furthermore, neural representations underlying successful learning are multidimensional but cost-efficient in nature. The degree of the emergent native-similar neural representations was closely related to the robust neural sensitivity to feedback in the frontostriatal network. These findings provide important insights on experience-dependent representational neuroplasticity underlying successful speech learning in adulthood and could be leveraged in designing individualized feedback-based training paradigms that maximize learning efficiency.

Protracted Development on Native Tone Interpretation: Evidence From Mandarin-Learning Infants' Novel Word Learning.

Studies have shown that infants from cultures with tone languages develop categorical perception of their native lexical tone before their first birthday, but few studies have explored whether, and when, they interpret the phonemic function of lexical tone in word learning. Two habituation-switch experiments were conducted to explore whether Mandarin-learning infants could exploit tonal cues during their word learning, and detect a change when the association of two word-object pairs was switched. In Experiment 1, two words were solely differentiated by their lexical tones (/fai/ vs. /fai/), and Mandarin-learning infants failed to detect the switch of tones at 14 months, but succeeded at 18 months. In Experiment 2, two words were markedly distinct (/fai/ vs. /bǒu/), and infants could detect the change of words as early as 14 months. The results indicate that infants may not refer to the lexical function of tone during their novel word learning until 18 months, even though infants from birth are able to distinguish the Tone 1 vs. Tone 3 contrast. Given that lexical tone is expressed by variations of the pitch contours, which are also related to intonation, infants' increasing knowledge of both tone and intonation may contribute to their misinterpretation of pitch contours in word learning at 14 months and, further, to their development of a sophisticated use of the phonemic function of lexical tone at 18 months of age.

The impact of spectral resolution on the mismatch response to Mandarin Chinese tones: an ERP study of cochlear implant simulations.

OBJECTIVE: This study examined the impact of spectral resolution on the processing of lexical tones and the number of frequency channels required for a cochlear implant (CI) to transmit Chinese tonal information to the brain. METHODS: ERPs were recorded in an auditory oddball task. Normal-hearing participants listened to speech sounds of two tones and their CI simulations in 1, 4, 8, or 32 channels. The mismatch response elicited by speech sounds and CI simulations in different numbers of channels were compared. RESULTS: The mismatch negativity (MMN) was observed for speech sounds. For the 1-channel CI simulations, deviants elicited a more positive waveform than standard stimuli. No MMN response was observed with the 4-channel simulations. A reliable MMN response was observed for the 8- and 32-channel simulations. The MMN responses elicited by the 8- and 32-channel simulations were equivalent in magnitudes and smaller than that elicited by speech sounds. CONCLUSIONS: More than eight frequency channels are required for a CI to transmit Chinese tonal information. The presence of both positive and negative mismatch responses suggests multiple mechanisms underlying auditory mismatch responses. SIGNIFICANCE: The current findings of spectral resolution constraints on the transmission of tonal information should be taken into account in the design of the CI devices.

Development and validation of a new Mandarin tone identification test.

OBJECTIVES: The objectives of this study were to develop a new Mandarin tone identification test (MTIT) to assess the Mandarin tone identification ability of children with hearing impairment (HI) and at age around 7 years; and to evaluate the reliability and sensitivity of the MTIT. METHODS: The word materials to be used in the MTIT were developed in Phase I. Monosyllables were chosen to represent the daily repertoire of young children and to avoid the influence of co-articulation and intonation. Each test stimulus set contained four words, with one target, one containing contrastive tone, and two unrelated distracters. All words were depicted using simple pictures, and the test targets in quiet or in noise were presented using recorded stimuli on a custom software. Phase II evaluated the reliability and sensitivity of the MTIT. Participants were 50 normal-hearing native-Mandarin speakers around 7 years of age. RESULTS: In Phase I, the MTIT was developed as described above. The final test consists of 51 words that are within the vocabulary repertoire of children aged 7 years. In Phase II, with the Mandarin tone identification scores collected from 50 children, the repeated measure ANOVA showed a significant main effect of S/N on MTIT performance (p<0.001). Pairwise comparisons revealed a significant difference in performance across the five S/N conditions (p<0.01) when S/N varied from -30 to -10dB. Cronbach's alpha at -15dB S/N was 0.66, suggesting satisfactory internal consistency reliability. A paired-samples t-test showed that there was no significant difference between the test-retest scores across the five S/N conditions (p>0.05). CONCLUSIONS: Compared with the available Mandarin tone identification tools, MTIT systematically evaluated the tone identification performance in noisy environment for normal hearing children at age around 7 years. Results also showed satisfactory internal consistency reliability, good test-retest reliability and good sensitivity. In the near future, MTIT could be used to evaluate tone perception ability of children with hearing impairment and help to design hearing rehabilitation strategies for this population at the age critical for their language learning.

Tone identification in Mandarin-speaking children with profound hearing impairment.

OBJECTIVES: Little is known about the tone identification ability of Mandarin-speaking children with profound hearing impairment (HI) and fitted with hearing aids (HAs). The present study aimed to evaluate the Mandarin tone identification ability in children with profound HI and fitted with HAs, and explore the effects of hearing thresholds, the age of first HA fitting and the duration of HA use on Mandarin tone identification ability. METHODS: Subjects were Mandarin-speaking children aged 5;4-12;6 years with profound HI (n=41). The Mandarin Tone Identification Test was administered in five test conditions: in -10, -5, 0 and 5dB signal-to-noise ratios (S/Ns) and quiet. Hearing aids were in the usual user's settings, optimized for the best speech reception, and the volume was set to comfortable listening level. RESULTS: Two-way repeated measures ANOVA showed significant effects of test conditions and tone contrasts in Mandarin tone identification. Post-hoc pairwise comparisons showed significant difference in performance among the five test conditions. Results also indicated that, among the six tone contrasts, the Tone 1/Tone 2 and Tone 2/Tone 3 contrasts were the most difficult tone contrasts in quiet. No significant difference in performance was found among the six tone contrasts in noise. Pearson product-moment correlation showed that the age of first HA fitting, the duration of HA use, aided and unaided average thresholds as well as the aided hearing thresholds in the low frequency region were not significantly correlated with tone identification ability in the five test conditions. CONCLUSIONS: Consistent with previous findings, the Tone 1/Tone 2 and Tone 2/Tone 3 contrasts were the most difficult to identify in quiet; but the presence of noise resulted in these tone contrasts being equally difficult. Findings also suggest that the tone identification ability of children with profound HI needs to be improved. Although the age of first HA fitting and the duration of HA use were not significantly correlated with tone identification performance in children with profound HI, this finding does not preclude the importance of early HA fitting.