Multi-modal cross-linguistic perception of Mandarin tones in clear speech.

Clearly enunciated speech (relative to conversational, plain speech) involves articulatory and acoustic modifications that enhance auditory-visual (AV) segmental intelligibility. However, little research has explored clear-speech effects on the perception of suprasegmental properties such as lexical tone, particularly involving visual (facial) perception. Since tone production does not primarily rely on vocal tract configurations, tones may be less visually distinctive. Questions thus arise as to whether clear speech can enhance visual tone intelligibility, and if so, whether any intelligibility gain can be attributable to tone-specific category-enhancing (code-based) clear-speech cues or tone-general saliency-enhancing (signal-based) cues. The present study addresses these questions by examining the identification of clear and plain Mandarin tones with visual-only, auditory-only, and AV input modalities by native (Mandarin) and nonnative (English) perceivers. Results show that code-based visual and acoustic clear tone modifications, although limited, affect both native and nonnative intelligibility, with category-enhancing cues increasing intelligibility and category-blurring cues decreasing intelligibility. In contrast, signal-based cues, which are extensively available, do not benefit native intelligibility, although they contribute to nonnative intelligibility gain. These findings demonstrate that linguistically relevant visual tonal cues are existent. In clear speech, such tone category-enhancing cues are incorporated with saliency-enhancing cues across AV modalities for intelligibility improvements.

Plain-to-clear speech video conversion for enhanced intelligibility.

Clearly articulated speech, relative to plain-style speech, has been shown to improve intelligibility. We examine if visible speech cues in video only can be systematically modified to enhance clear-speech visual features and improve intelligibility. We extract clear-speech visual features of English words varying in vowels produced by multiple male and female talkers. Via a frame-by-frame image-warping based video generation method with a controllable parameter (displacement factor), we apply the extracted clear-speech visual features to videos of plain speech to synthesize clear speech videos. We evaluate the generated videos using a robust, state of the art AI Lip Reader as well as human intelligibility testing. The contributions of this study are: (1) we successfully extract relevant visual cues for video modifications across speech styles, and have achieved enhanced intelligibility for AI; (2) this work suggests that universal talker-independent clear-speech features may be utilized to modify any talker's visual speech style; (3) we introduce "displacement factor" as a way of systematically scaling the magnitude of displacement modifications between speech styles; and (4) the high definition generated videos make them ideal candidates for human-centric intelligibility and perceptual training studies.

Production and Perception Evidence of a Merger: [l] and [n] in Fuzhou Min.

The current study investigated the merger-in-progress between word-initial nasal and lateral consonants in Fuzhou Min, examining the linguistic and social factors that modulate the merger. First, the acoustic cues to the l-n distinction were examined in Fuzhou Min. Acoustic analyses suggested a collapse of phonemic contrast between prescriptive L and N (phonemes in the unmerged system), with none of the six acoustic cues showing any difference across L and N. Linear discriminant analysis did identify acoustically distinct [l] and [n] tokens, although the mapping onto the phonetic space of prescriptive L and N substantially overlapped. Speakers of all ages and both genders tended to produce [l], and low vowels correlated with more [n]-like classification. In perception, AX discrimination data showed Fuzhou Min listeners confused both prescriptive L and N and acoustic [l] and [n]. Greater sensitivity to the acoustic differences occurred in the context of low vowels and a nasal coda, supported by the acoustics of the stimuli, and younger listeners were more sensitive to the difference between [l] and [n] than older listeners. In two-alternative forced choice (2AFC) identification, Fuzhou Min listeners also identified the merged form as L more frequently than N, with more L responses elicited in the context of low vowels and in the absence of nasal codas. Overall, although Fuzhou Min speakers produced some acoustically distinct [l] and [n] tokens in the context of a sound merger, these productions did not map onto prescriptive L and N. In addition, younger listeners were more sensitive to the acoustic distinction than older listeners, suggesting an emerging acoustic contrast possibly arising due to contact with Mandarin.

The influence of inter-dialect contact on the Korean three-way laryngeal distinction: An acoustic comparison among Seoul Korean speakers and Gyeongsang speakers with limited and extended residence in Seoul.

This exploratory study investigates the acoustic correlates of the Korean three-way laryngeal stop distinction in Gyeongsang long-term (LT) transplants who were born in the Gyeongsang region but moved to Seoul to pursue higher education. Acoustic data were collected from eight LT transplants, five short-term (ST) transplants, and 11 Seoul speakers to examine whether exposure to Seoul Korean (SK) affects Gyeongsang speakers' cue-weighting in distinguishing stops in production. LT transplants produced stimuli in both Gyeongsang and Seoul dialects. A cue-weighting model based on the acoustic data reveals that voice onset time (VOT) is less important to distinguish lenis from aspirated stops for Seoul speakers and for LT transplants' SK, as compared to ST transplants and LT transplants' Gyeongsang Korean (GK). In addition, fundamental frequency (F0) is more important for the lenis-aspirated distinction for Seoul speakers and LT transplants' SK, as compared to ST and LT transplants' GK, showing that LT transplants rely less on VOT and more on F0 to distinguish lenis from aspirated stops compared to ST transplants. LT transplants' SK reveals that they rely more on VOT and less on F0 compared to SK speakers. The cue-weighting model of the LT transplants provide empirical evidence that a series of sound changes in GK is due to inter-dialect contact.

The contrast between clear and plain speaking style for Mandarin tones.

We examine the acoustic characteristics of clear and plain conversational productions of Mandarin tones. Twenty-one native Mandarin speakers were asked to produce a selection of Mandarin words in both plain and clear speaking styles. Several tokens were gathered for each of the four tones giving a total of 2045 productions. Six critical tonal cues were computed for each production: fundamental frequency (F0) mean, slope, and second derivative, duration, mean intensity, and a binary variable coding whether the production involved creaky voice. A linear mixed-effects regression model was used to explore how these cues changed with respect to the clear versus plain distinction for each tone, with speaking style as the fixed effect and speaker being a random effect. The strongest effects detected were that duration and mean intensity increased in clear speech across speakers and tones. Tones 2 and 3 increased in mean F0 and Tone 4 increased its slope. An additional finding was that, for contour tones, speakers accomplished the increase in duration by stretching out the tone contours in time while largely not changing the F0 range. These results are discussed in terms of signal-based (affecting all tones) and code-based (enhancing contrast between tones) change.

ADFAC: Automatic detection of facial articulatory features.

Using computer-vision and image processing techniques, we aim to identify specific visual cues as induced by facial movements made during monosyllabic speech production. The method is named ADFAC: Automatic Detection of Facial Articulatory Cues. Four facial points of interest were detected automatically to represent head, eyebrow and lip movements: nose tip (proxy for head movement), medial point of left eyebrow, and midpoints of the upper and lower lips. The detected points were then automatically tracked in the subsequent video frames. Critical features such as the distance, velocity, and acceleration describing local facial movements with respect to the resting face of each speaker were extracted from the positional profiles of each tracked point. In this work, a variant of random forest is proposed to determine which facial features are significant in classifying speech sound categories. The method takes in both video and audio as input and extracts features from any video with a plain or simple background. The method is implemented in MATLAB and scripts are made available on GitHub for easy access.•Using innovative computer-vision and image processing techniques to automatically detect and track keypoints on the face during speech production in videos, thus allowing more natural articulation than previous sensor-based approaches.•Measuring multi-dimensional and dynamic facial movements by extracting time-related, distance-related and kinematics-related features in speech production.•Adopting the novel random forest classification approach to determine and rank the significance of facial features toward accurate speech sound categorization.

Multi-modal cross-linguistic perception of fricatives in clear speech.

Research shows that acoustic modifications in clearly enunciated fricative consonants (relative to the plain, conversational productions) facilitate auditory fricative perception, particularly for auditorily salient sibilant fricatives and for native perception. However, clear-speech effects on visual fricative perception have received less attention. A comparison of auditory and visual (facial) clear-fricative perception is particularly interesting since sibilant fricatives in English are more auditorily salient while non-sibilants are more visually salient. This study thus examines clear-speech effects on multi-modal perception of English sibilant and non-sibilant fricatives. Native English perceivers and non-native (Mandarin, Korean) perceivers with different fricative inventories in their native languages (L1s) identified clear and conversational fricative-vowel syllables in audio-only, visual-only, and audio-visual (AV) modes. The results reveal an overall positive clear-speech effect when visual information is involved. Considering the factor of AV saliency, clear speech benefits sibilants more in the auditory domain and non-sibilants more in the visual domain. With respect to language background, non-native (Mandarin and Korean) perceivers benefit from visual as well as auditory information, even for fricatives non-existent in their respective L1s, but the patterns of clear-speech gains are affected by the relative AV weighting and "nativeness" of the fricatives. These findings are discussed in terms of how saliency-enhancing and category-distinctive cues of speech sounds are adopted in AV perception to improve intelligibility.

Characterizing the distinctive acoustic cues of Mandarin tones.

This study aims to characterize distinctive acoustic features of Mandarin tones based on a corpus of 1025 monosyllabic words produced by 21 native Mandarin speakers. For each tone, 22 acoustic cues were extracted. Besides standard F0, duration, and intensity measures, further cues were determined by fitting two mathematical functions to the pitch contours. The first function is a parabola, which gives three parameters: a mean F0, an F0 slope, and an F0 second derivative. The second is a broken-line function, which models the contour as a continuous curve consisting of two lines with a single breakpoint. Cohen's d, sparse Principal Component Analysis, and other statistical measures are used to identify which of the cues, and which combinations of the cues, are important for distinguishing each tone from each other among all the speakers. Although the specific cues that best characterize the tone contours depend on the particular tone and the statistical measure used, this paper shows that the three cues obtained by fitting a parabola to the tone contour are broadly effective. This research suggests using these three cues as a canonical choice for defining tone characteristics.

Effects of Sound Change on the Weighting of Acoustic Cues to the Three-Way Laryngeal Stop Contrast in Korean: Diachronic and Dialectal Comparisons.

Both segmental and suprasegmental properties of the South Kyungsang dialect of Korean have changed under the influence of standard Seoul Korean. This study examines how such sound change affects acoustic cues to the three-way laryngeal contrast among Korean stops across Kyungsang generations through a comparison with Seoul Korean. Thirty-nine female Korean speakers differing in dialect (Kyungsang, Seoul) and age (older, younger) produced words varying in initial stops and lexical accent patterns, for which voice onset time and fundamental frequency (F0) at vowel onset were measured. This study first confirms previous findings regarding age and dialectal variation in distinguishing the three Korean stops. In addition, we report age variation in the use of voice onset time and F0 for the stops in Kyungsang Korean, with younger speakers using F0 more than older speakers as a cue to the stop distinction. This age variation is accounted for by the reduced lexical tonal properties of Kyungsang Korean and the increased influence of Seoul Korean. A comparison of the specific cue weighting across speaker groups also reveals that younger Kyungsang speakers pattern with Seoul speakers who arguably follow the enhancing F0 role of the innovative younger Seoul speakers. The shared cue weighting pattern across generations and dialects suggests that each speaker group changes the acoustic cue weighting in a similar direction.

Tonal Neutralization of Taiwanese Checked and Smooth Syllables: An Acoustic Study.

Taiwanese tonal alternation is realized in a circular chain shift fashion for both smooth and checked syllables. Debate regarding the processes of less productive Taiwanese tonal alternation has centered on whether a surface tone is derived from an underlying tone, or whether a surface tone is selected without undergoing any derivation. The current study investigates this controversial issue by examining Taiwanese checked tone and smooth tone sandhi neutralization in production. In particular, we analyzed whether checked citation and sandhi tone 53 (C21→C53), checked citation and sandhi tone 21 (C53→C21), smooth citation and sandhi tone 55 (S51→S55), and smooth citation and sandhi tone 21 (S33→S21) are acoustically completely neutralized in fundamental frequency (F0) height, contour, and duration. A non-sandhi exception was also included to evaluate the effect of position-in-word on F0 height and duration given that citation tones always appear in phrase-final position. Any trace of influence from the underlying representation would indicate a computational mechanism, whereas the absence of any trace would suggest a lexical mechanism for the production of Taiwanese tonal alternation. Results did not show any influence of F0 height, F0 contour, or tone duration from the underlying representation for both checked and smooth tones, supporting a lexical mechanism in speech production for less productive tonal alternations.

Training Children to Perceive Non-native Lexical Tones: Tone Language Background, Bilingualism, and Auditory-Visual Information.

This study investigates the role of language background and bilingual status in the perception of foreign lexical tones. Eight groups of participants, consisting of children of 6 and 8 years from one of four language background (tone or non-tone) × bilingual status (monolingual or bilingual)-Thai monolingual, English monolingual, English-Thai bilingual, and English-Arabic bilingual were trained to perceive the four Mandarin lexical tones. Half the children in each of these eight groups were given auditory-only (AO) training and half auditory-visual (AV) training. In each group Mandarin tone identification was tested before and after (pre- and post-) training with both auditory-only test (ao-test) and auditory-visual test (av test). The effect of training on Mandarin tone identification was minimal for 6-year-olds. On the other hand, 8-year-olds, particularly those with tone language experience showed greater pre- to post-training improvement, and this was best indexed by ao-test trials. Bilingual vs. monolingual background did not facilitate overall improvement due to training, but it did modulate the efficacy of the Training mode: for bilinguals both AO and AV training, and especially AO, resulted in performance gain; but for monolinguals training was most effective with AV stimuli. Again this effect was best indexed by ao-test trials. These results suggest that tone language experience, be it monolingual or bilingual, is a strong predictor of learning unfamiliar tones; that monolinguals learn best from AV training trials and bilinguals from AO training trials; and that there is no metalinguistic advantage due to bilingualism in learning to perceive lexical tones.

Source characteristics of voiceless dorsal fricatives.

Aerodynamic and acoustic data on voiceless dorsal fricatives [x/χ] in Arabic, Persian, and Spanish were recorded to measure the extent to which such productions involve trilling of the uvula, thus exhibiting a sound source which, contrary to assumptions for voiceless fricatives, is mixed rather than aperiodic. Oscillation in airflow indicative of uvular vibration was present more often than not in Arabic (63%) and Persian (75%), while Spanish dorsal fricatives were more commonly produced with unimodal flow indicative of an aperiodic source. When present, uvular vibration frequencies averaged 68 Hz in Arabic and 67 Hz in Persian. Rates of uvular vibration were highly variable, however, and ranged between 40 and 116 Hz, with oscillatory periods averaging 4-5 cycles in duration, with a range of 1-12. The effect of these source characteristics on dorsal fricative acoustics was to significantly skew the spectral shape parameters (M1-M4) commonly used to characterize properties of the anterior filter; however, spectral peak frequency was found to be stable to changes in source characteristics, suggesting the occurrence of trilled tokens is not due to velar-uvular allophony, but rather is more fundamental to dorsal fricative production.

Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers.

Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers' performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.

Just noticeable differences for pitch direction, height, and slope for Mandarin and English listeners.

Previous studies on tones suggest that Mandarin listeners are more sensitive to pitch direction and slope while English listeners primarily attend to pitch height. In this study, just noticeable differences were established for pitch discrimination using a three-interval, forced-choice procedure with a two-down, one-up staircase design. A high rising and a high falling Mandarin tone were manipulated in terms of pitch direction, height, and slope. Results indicate that, overall, Mandarin listeners are more sensitive to pitch slope and English listeners to pitch height. However, these effects are modulated by both the direction (falling/rising) and slope of the pitch contours.

Fetal rhythm-based language discrimination: a biomagnetometry study.

Using fetal biomagnetometry, this study measured changes in fetal heart rate to assess discrimination of two rhythmically different languages (English and Japanese). Two-minute passages in English and Japanese were read by the same female bilingual speaker. Twenty-four mother-fetus pairs (mean gestational age=35.5 weeks) participated. Fetal magnetocardiography was recorded while the participants were presented first with passage 1, a passage in English, and then, following an 18 min interval, with passage 2, either a different passage in English (English-English condition: N=12) or in Japanese (English-Japanese condition: N=12). The fetal magnetocardiogram was reconstructed following independent components analysis decomposition. The mean interbeat intervals were calculated for a 30 s baseline interval directly preceding each passage and for the first 30 s of each passage. We then subtracted the mean interbeat interval of the 30 s baseline interval from that of the first 30 s interval, yielding an interbeat interval change value for each passage. A significant interaction between condition and passage indicated that the English-Japanese condition elicited a more robust interbeat interval change for passage 2 (novelty phase) than for passage 1 (familiarity phase), reflecting a faster heart rate during passage 2, whereas the English-English condition did not. This effect indicates that fetuses are sensitive to the change in language from English to Japanese. These findings provide the first evidence for fetal language discrimination as assessed by fetal biomagnetometry and support the hypothesis that rhythm constitutes a prenatally available building block in language acquisition.

Acoustic characteristics of clearly spoken English tense and lax vowels.

Clearly produced vowels exhibit longer duration and more extreme spectral properties than plain, conversational vowels. These features also characterize tense relative to lax vowels. This study explored the interaction of clear-speech and tensity effects by comparing clear and plain productions of three English tense-lax vowel pairs (/i-ɪ/, /ɑ-ʌ/, /u-ʊ/ in /kVd/ words). Both temporal and spectral acoustic features were examined, including vowel duration, vowel-to-word duration ratio, formant frequency, and dynamic spectral characteristics. Results revealed that the tense-lax vowel difference was generally enhanced in clear relative to plain speech, but clear-speech modifications for tense and lax vowels showed a trade-off in the use of temporal and spectral cues. While plain-to-clear vowel lengthening was greater for tense than lax vowels, clear-speech modifications in spectral change were larger for lax than tense vowels. Moreover, peripheral tense vowels showed more consistent clear-speech modifications in the temporal than spectral domain. Presumably, articulatory constraints limit the spectral variation of these extreme vowels, so clear-speech modifications resort to temporal features and reserve the primary spectral features for tensity contrasts. These findings suggest that clear-speech and tensity interactions involve compensatory modifications in different acoustic domains.

What Comes After /f/? Prediction in Speech Derives From Data-Explanatory Processes.

Acoustic cues are short-lived and highly variable, which makes speech perception a difficult problem. However, most listeners solve this problem effortlessly. In the present experiment, we demonstrated that part of the solution lies in predicting upcoming speech sounds and that predictions are modulated by high-level expectations about the current sound. Participants heard isolated fricatives (e.g., "s," "sh") and predicted the upcoming vowel. Accuracy was above chance, which suggests that fine-grained detail in the signal can be used for prediction. A second group performed the same task but also saw a still face and a letter corresponding to the fricative. This group performed markedly better, which suggests that high-level knowledge modulates prediction by helping listeners form expectations about what the fricative should have sounded like. This suggests a form of data explanation operating in speech perception: Listeners account for variance due to their knowledge of the talker and current phoneme, and they use what is left over to make more accurate predictions about the next sound.

Does Second Language Experience Modulate Perception of Tones in a Third Language?

It is unclear what roles native language (L1) and second language (L2) play in the perception of lexical tones in a third language (L3). In tone perception, listeners with different language backgrounds use different fundamental frequency (F0). While English listeners use F0 height, Mandarin listeners rely more on F0 direction. The present study addresses whether knowledge of Mandarin, particularly as an L2, results in speakers' reliance on F0 direction in their perception of L3 (Cantonese) tones. Fifteen English-speaking L2 learners of Mandarin constituted the target group, and 15 English monolinguals and 15 native Mandarin speakers, with no background in other tonal languages, were included as control groups. All groups had to discriminate Cantonese tones either by distinguishing a contour tone from a level tone (F0 direction pair) or a level tone from another level tone (F0 height pair). The results showed that L2 learners patterned differently from both control groups by using F0 direction as well as F0 height under the influence of L1 and L2 experience. The acoustics of the tones also affected all listeners' discrimination. When L2 and L3 are similar in terms of the presence of lexical tone, L2 experience modulates the perception of L3 tones.

Contingent categorization in speech perception.

The speech signal is notoriously variable, with the same phoneme realized differently depending on factors like talker and phonetic context. Variance in the speech signal has led to a proliferation of theories of how listeners recognize speech. A promising approach, supported by computational modeling studies, is contingent categorization, wherein incoming acoustic cues are computed relative to expectations. We tested contingent encoding empirically. Listeners were asked to categorize fricatives in CV syllables constructed by splicing the fricative from one CV syllable with the vowel from another CV syllable. The two spliced syllables always contained the same fricative, providing consistent bottom-up cues; however on some trials, the vowel and/or talker mismatched between these syllables, giving conflicting contextual information. Listeners were less accurate and slower at identifying the fricatives in mismatching splices. This suggests that listeners rely on context information beyond bottom-up acoustic cues during speech perception, providing support for contingent categorization.

Speaker sex influences processing of grammatical gender.

Spoken words carry linguistic and indexical information to listeners. Abstractionist models of spoken word recognition suggest that indexical information is stripped away in a process called normalization to allow processing of the linguistic message to proceed. In contrast, exemplar models of the lexicon suggest that indexical information is retained in memory, and influences the process of spoken word recognition. In the present study native Spanish listeners heard Spanish words that varied in grammatical gender (masculine, ending in -o, or feminine, ending in -a) produced by either a male or a female speaker. When asked to indicate the grammatical gender of the words, listeners were faster and more accurate when the sex of the speaker "matched" the grammatical gender than when the sex of the speaker and the grammatical gender "mismatched." No such interference was observed when listeners heard the same stimuli, but identified whether the speaker was male or female. This finding suggests that indexical information, in this case the sex of the speaker, influences not just processes associated with word recognition, but also higher-level processes associated with grammatical processing. This result also raises questions regarding the widespread assumption about the cognitive independence and automatic nature of grammatical processes.