The effect of native language and bilingualism on multimodal perception in speech: A study of audio-aerotactile integrationa).

Previous studies of speech perception revealed that tactile sensation can be integrated into the perception of stop consonants. It remains uncertain whether such multisensory integration can be shaped by linguistic experience, such as the listener's native language(s). This study investigates audio-aerotactile integration in phoneme perception for English and French monolinguals as well as English-French bilingual listeners. Six step voice onset time continua of alveolar (/da/-/ta/) and labial (/ba/-/pa/) stops constructed from both English and French end points were presented to listeners who performed a forced-choice identification task. Air puffs were synchronized to syllable onset and randomly applied to the back of the hand. Results show that stimuli with an air puff elicited more "voiceless" responses for the /da/-/ta/ continuum by both English and French listeners. This suggests that audio-aerotactile integration can occur even though the French listeners did not have an aspiration/non-aspiration contrast in their native language. Furthermore, bilingual speakers showed larger air puff effects compared to monolinguals in both languages, perhaps due to bilinguals' heightened receptiveness to multimodal information in speech.

On the two rhotic schwas in Southwestern Mandarin: when homophony meets morphology in articulation.

This is an acoustic and articulatory study of the two rhotic schwas in Southwestern Mandarin (SWM), i.e., the er-suffix (a functional morpheme) and the rhotic schwa phoneme. Electromagnetic Articulography (EMA) and ultrasound results from 10 speakers show that the two rhotic schwas were both produced exclusively with the bunching of the tongue body. No retroflex versions of the two rhotic schwas were found, nor was retraction of the tongue root into the pharynx observed. On the other hand, the er-suffix and the rhotic schwa, though homophonous, significantly differ in certain types of acoustic and articulatory measurements. In particular, more pronounced lip protrusion is involved in the production of the rhotic schwa phoneme than in the er-suffix. It is equally remarkable that contrast preservation is not an issue because the two rhotic schwas are in complementary distribution. Taken together, the present results suggest that while morphologically-induced phonetic variation can be observed in articulation, gestural economy may act to constrain articulatory variability, resulting in the absence of retroflex tongue variants in the two rhotic schwas, the only two remaining r-colored sounds in SWM.

Comparing metrics for quantification of children's tongue shape complexity using ultrasound imaging.

Speech sound disorders can pose a challenge to communication in children that may persist into adulthood. As some speech sounds are known to require differential control of anterior versus posterior regions of the tongue body, valid measurement of the degree of differentiation of a given tongue shape has the potential to shed light on development of motor skill in typical and disordered speakers. The current study sought to compare the success of multiple techniques in quantifying tongue shape complexity as an index of degree of lingual differentiation in child and adult speakers. Using a pre-existing data set of ultrasound images of tongue shapes from adult speakers producing a variety of phonemes, we compared the extent to which three metrics of tongue shape complexity differed across phonemes/phoneme classes that were expected to differ in articulatory complexity. We then repeated this process with ultrasound tongue shapes produced by a sample of young children. The results of these comparisons suggested that a modified curvature index and a metric representing the number of inflection points best reflected small changes in tongue shapes across individuals differing in vocal tract size. Ultimately, these metrics have the potential to reveal delays in motor skill in young children, which could inform assessment procedures and treatment decisions for children with speech delays and disorders.

Comparison of auto-contouring and hand-contouring of ultrasound images of the tongue surface.

Contours traced by trained phoneticians have been considered to be the most accurate way to identify the midsagittal tongue surface from ultrasound video frames. In this study, inter-measurer reliability was evaluated using measures that quantified both how closely human-placed contours approximated each other as well as how consistent measurers were in defining the start and end points of contours. High reliability across three measurers was found for all measures, consistent with treating contours placed by trained phoneticians as the 'gold standard.' However, due to the labour-intensive nature of hand-placing contours, automatic algorithms that detect the tongue surface are increasingly being used to extract tongue-surface data from ultrasound videos. Contours placed by six automatic algorithms (SLURP, EdgeTrak, EPCS, and three different configurations of the algorithm provided in Articulate Assistant Advanced) were compared to human-placed contours, with the same measures used to evaluate the consistency of the trained phoneticians. We found that contours defined by SLURP, EdgeTrak, and two of the AAA configurations closely matched the hand-placed contours along sections of the image where the algorithms and humans agreed that there was a discernible contour. All of the algorithms were much less reliable than humans in determining the anterior (tongue-tip) edge of tongue contours. Overall, the contours produced by SLURP, EdgeTrak, and AAA should be useable in a variety of clinical applications, subject to spot-checking. Additional practical considerations of these algorithms are also discussed.

Auditory-perceptual acuity in rhotic misarticulation: baseline characteristics and treatment response.

The rhotic sound /r/ is one of the latest-emerging sounds in English, and many children receive treatment for residual errors affecting /r/ that persist past the age of 9. Auditory-perceptual abilities of children with residual speech errors are thought to be different from their typically developing peers. This study examined auditory-perceptual acuity in children with residual speech errors affecting /r/ and the relation of these skills to production accuracy, both before and after a period of treatment incorporating visual biofeedback. Identification of items along an /r/-/w/ continuum was assessed prior to treatment. Production accuracy for /r/ was acoustically measured from standard/r/stimulability probes elicited before and after treatment. Fifty-nine children aged 9-15 with residual speech errors (RSE) affecting /r/ completed treatment, and forty-eight age-matched controls who completed the same auditory-perceptual task served as a comparison group. It was hypothesized that children with RSE would show lower auditory-perceptual acuity than typically developing speakers and that higher auditory-perceptual acuity would be associated with more accurate production before treatment. It was also hypothesized that auditory-perceptual acuity would serve as a mediator of treatment response. Results indicated that typically developing children have more acute perception of the /r/-/w/ contrast than children with RSE. Contrary to hypothesis, baseline auditory-perceptual acuity for /r/ did not predict baseline production severity. For baseline auditory-perceptual acuity in relation to biofeedback efficacy, there was an interaction between auditory-perceptual acuity and gender, such that higher auditory-perceptual acuity was associated with greater treatment response in female, but not male, participants.

Unsupervised speaker adaptation for speaker independent acoustic to articulatory speech inversion.

Speech inversion is a well-known ill-posed problem and addition of speaker differences typically makes it even harder. Normalizing the speaker differences is essential to effectively using multi-speaker articulatory data for training a speaker independent speech inversion system. This paper explores a vocal tract length normalization (VTLN) technique to transform the acoustic features of different speakers to a target speaker acoustic space such that speaker specific details are minimized. The speaker normalized features are then used to train a deep feed-forward neural network based speech inversion system. The acoustic features are parameterized as time-contextualized mel-frequency cepstral coefficients. The articulatory features are represented by six tract-variable (TV) trajectories, which are relatively speaker invariant compared to flesh point data. Experiments are performed with ten speakers from the University of Wisconsin X-ray microbeam database. Results show that the proposed speaker normalization approach provides an 8.15% relative improvement in correlation between actual and estimated TVs as compared to the system where speaker normalization was not performed. To determine the efficacy of the method across datasets, cross speaker evaluations were performed across speakers from the Multichannel Articulatory-TIMIT and EMA-IEEE datasets. Results prove that the VTLN approach provides improvement in performance even across datasets.

Towards the Quantification of Peggy Babcock: Speech Errors and Their Position within the Word.

Sequences of similar (i.e., partially identical) words can be hard to say, as indicated by error frequencies, longer reaction and execution times. This study investigates the role of the location of this partial identity and the accompanying differences, i.e. whether errors are more frequent with mismatches in word onsets (top cop), codas (top tock) or both (pop tot). Number of syllables (tippy ticky) and empty positions (top ta) were also varied. Since the gradient nature of errors can be difficult to determine acoustically, articulatory data were investigated. Articulator movements were recorded using electromagnetic articulography, for up to 9 speakers of American English repeatedly producing 2-word sequences to an accelerating metronome. Most word pairs showed more intrusions and greater variability in coda than in onset position, in contrast to the predominance of onset position errors in corpora from perceptual observation.

Tongue shapes for rhotics in school-age children with and without residual speech errors.

Speakers of North American English use variable tongue shapes for rhotic sounds. However, quantifying tongue shapes for rhotics can be challenging, and little is known about how tongue shape complexity corresponds to perceptual ratings of rhotic accuracy in children with residual speech sound errors (RSE). In this study, 16 children aged 9-16 with RSE and 14 children with typical speech (TS) development made multiple productions of 'Let Robby cross Church Street'. Midsagittal ultrasound images were collected once for children with TS and twice for children in the RSE group (once after 7 h of speech therapy, then again after another 7 h of therapy). Tongue contours for the rhotics in the four words were traced and quantified using a new metric of tongue shape complexity: the number of inflections. Rhotics were also scored for accuracy by four listeners. During the first assessment, children with RSE had fewer tongue inflections than children with TS. Following 7 h of therapy, there were increases in the number of inflections for the RSE group, with the cluster items cross and Street reaching tongue complexity levels of those with TS. Ratings of rhotic accuracy were correlated with the number of inflections. Therefore, the number of inflections in the tongue, an index of tongue shape complexity, was associated with perceived accuracy of rhotic productions.

Using Electromagnetic Articulography to Measure Denture Micromovement during Chewing with and without Denture Adhesive.

PURPOSE: To study the effects of denture adhesive upon denture micromovements in three dimensions during the chewing of hard, sticky, and tough food items observed using a novel method involving an electromagnetic articulograph (EMA) speech research system. MATERIALS AND METHODS: Ten volunteers (mean age 60.9 ± 10.4 years) with fair- or poor-fitting complete maxillary dentures were enrolled. Chewing experiments were conducted using two treatments (adhesive or no-adhesive control) and three foods: carrots (hard), raisins (sticky), and processed meat stick (tough). Denture micromovement was measured through a novel application of a Northern Digital Wave EMA System. Three-dimensional denture position was captured during mastication using three sensors embedded into a replica denture for each subject. Following individual characterization of a "home" reference position, the Euclidean Distances from Home (DfH) were calculated for each recorded sample of the chewing experiments. The DfH at each sample represented the denture movement for that 1/100th of a second of the activity. The DfH data were then summarized as the mean DfH, the maximum DfH, and total distance traveled by the denture. Several thresholds were also analyzed, including the percent of time that the DfH ≥1.5 mm, ≥2.0 mm, and ≥2.5 mm. RESULTS: With adhesive treatment, the mean DfH of dentures during chewing was reduced by 26.8% for carrot, 30.3% for raisin, and 31.0% for meat stick, when compared with no-adhesive treatment (p < 0.001 for all comparisons). Similar results were also seen for the maximum DfH and total distance travelled endpoints across foods. For the threshold endpoints, adhesive treatment was associated with a statistically significant reduction in denture micromovements at all three thresholds across foods. At the threshold of DfH ≥ 1.5 mm, adhesive treatment was associated with a reduction in micromovement by 61.6%, 56.2%, and 70.0% with carrot, raisin, and meat stick, respectively (p ≤ 0.004 for all comparisons). CONCLUSIONS: Observations of denture movement using the Wave EMA System were able to differentiate systematically between adhesive treatment and no-adhesive treatment for denture micromovements during different chewing challenges. Use of adhesive was associated with statistically significant reductions in denture micromovements for hard, sticky, and tough foods as measured with both distance and threshold endpoints.

Quantitative identification of dialect-specific articulatory settings.

The purpose of this study was to quantitatively contrast the articulatory settings of two Dutch dialects. Tongue movement data during speech were collected on site at two high schools (34 speakers) in the Netherlands using a portable electromagnetic articulography device. Comparing the tongue positions during pauses in speech between the two groups revealed a clear difference in the articulatory settings, with significantly more frontal tongue positions for the speakers from Ubbergen in the Southeast of the Netherlands compared to those from Ter Apel in the North of the Netherlands. These results provide quantitative evidence for differences in articulatory settings at the dialect level.

Indexing head movement during speech production using optical markers.

Optical marker tracking integrated with electromagnetic articulometry was used to assess the movement extent of various points on (a) forehead skin and (b) points on a head-mounted apparatus, relative to a fixed point just above the upper incisors, and to compare the accuracy of the two different approaches to indexing head position during speech production. Both methods can provide a satisfactory index of head position. If skin-affixed markers are used, a minimum of 4 is recommended. Locations for optimal marker placement are identified.

Methods for quantifying tongue shape and complexity using ultrasound imaging.

Quantification of tongue shape is potentially useful for indexing articulatory strategies arising from intervention, therapy and development. Tongue shape complexity is a parameter that can be used to reflect regional functional independence of the tongue musculature. This paper considers three different shape quantification methods - based on Procrustes analysis, curvature inflections and Fourier coefficients - and uses a linear discriminant analysis to test how well each method is able to classify tongue shapes from different phonemes. Test data are taken from six native speakers of American English producing 15 phoneme types. Results classify tongue shapes accurately when combined across quantification methods. These methods hold promise for extending the use of ultrasound in clinical assessments of speech deficits.

A real-time articulatory visual feedback approach with target presentation for second language pronunciation learning.

Articulatory information can support learning or remediating pronunciation of a second language (L2). This paper describes an electromagnetic articulometer-based visual-feedback approach using an articulatory target presented in real-time to facilitate L2 pronunciation learning. This approach trains learners to adjust articulatory positions to match targets for a L2 vowel estimated from productions of vowels that overlap in both L1 and L2. Training of Japanese learners for the American English vowel /æ/ that included visual training improved its pronunciation regardless of whether audio training was also included. Articulatory visual feedback is shown to be an effective method for facilitating L2 pronunciation learning.

Voice Register in Mon: Acoustics and Electroglottography.

Mon is spoken in villages in Thailand and Myanmar. The dialect of Ban Nakhonchum, Thailand, has 2 voice registers, modal and breathy; these phonation types, along with other phonetic properties, distinguish minimal pairs. Four native speakers of this dialect recorded repetitions of 14 randomized words (7 minimal pairs) for acoustic analysis. We used a subset of these pairs in a listening test to verify the perceptual robustness of the register distinction. Acoustic analysis found significant differences in noise component, spectral slope and fundamental frequency. In a subsequent session 4 speakers were also recorded using electroglottography, which showed systematic differences in the contact quotient. The salience of these properties in maintaining the register distinction is discussed in the context of possible tonogenesis for this language.

Effects of compatible versus competing rhythmic grouping on errors and timing variability in speech.

In typical speech words are grouped into prosodic constituents. This study investigates how such grouping interacts with segmental sequencing patterns in the production of repetitive word sequences. We experimentally manipulated grouping behavior using a rhythmic repetition task to elicit speech for perceptual and acoustic analysis to test the hypothesis that prosodic structure and patterns of segmental alternation can interact in the production planning process. Talkers produced alternating sequences of two words (top cop) and non-alternating controls (top top and cop cop), organized into six-word sequences. These sequences were further organized into prosodic groupings of three two-word pairs or two three-word triples by means of visual cues and audible metronome clicks. Results for six speakers showed more speech errors in triples, that is, when pairwise word alternation was mismatched with prosodic subgrouping in triples. This result suggests that the planning process for the segmental units of an utterance interacts with the planning process for the prosodic grouping of its words. It also highlights the importance of extending commonly used experimental speech elicitation methods to include more complex prosodic patterns, in order to evoke the kinds of interaction between prosodic structure and planning that occur in the production of lexical forms in continuous communicative speech.

Exposure to Canadian French Cued Speech Improves Consonant Articulation in Children With Cochlear Implants: Acoustic and Articulatory Data.

PURPOSE: One of the strategies that can be used to support speech communication in deaf children is cued speech, a visual code in which manual gestures are used as additional phonological information to supplement the acoustic and labial speech information. Cued speech has been shown to improve speech perception and phonological skills. This exploratory study aims to assess whether and how cued speech reading proficiency may also have a beneficial effect on the acoustic and articulatory correlates of consonant production in children. METHOD: Eight children with cochlear implants (from 5 to 11 years of age) and with different receptive proficiency in Canadian French Cued Speech (three children with low receptive proficiency vs. five children with high receptive proficiency) are compared to 10 children with typical hearing (from 4 to 11 years of age) on their production of stop and fricative consonants. Articulation was assessed with ultrasound measurements. RESULTS: The preliminary results reveal that cued speech proficiency seems to sustain the development of speech production in children with cochlear implants and to improve their articulatory gestures, particularly for the place contrast in stops as well as fricatives. CONCLUSION: This work highlights the importance of studying objective data and comparing acoustic and articulatory measurements to better characterize speech production in children.

Assessing ultrasound probe stabilization for quantifying speech production contrasts using the Adjustable Laboratory Probe Holder for UltraSound (ALPHUS).

Ultrasound imaging of the tongue is biased by the probe movements relative to the speaker's head. Two common remedies are restricting or algorithmically compensating for such movements, each with its own challenges. We describe these challenges in details and evaluate an open-source, adjustable probe stabilizer for ultrasound (ALPHUS), specifically designed to address these challenges by restricting uncorrectable probe movements while allowing for correctable ones (e.g., jaw opening) to facilitate naturalness. The stabilizer is highly modular and adaptable to different users (e.g., adults and children) and different research/clinical needs (e.g., imaging in both midsagittal and coronal orientations). The results of three experiments show that probe movement over uncorrectable degrees of freedom was negligible, while movement over correctable degrees of freedom that could be compensated through post-processing alignment was relatively large, indicating unconstrained articulation over parameters relevant for natural speech. Results also showed that probe movements as small as 5 mm or 2 degrees can neutralize phonemic contrasts in ultrasound tongue positions. This demonstrates that while stabilized but uncorrected ultrasound imaging can provide reliable tongue shape information (e.g., curvature or complexity), accurate tongue position (e.g., height or backness) with respect to vocal tract hard structure needs correction for probe displacement relative to the head.

Anticipatory lip gestures: a validation of the Movement Expansion Model in congenitally blind speakers.

In this paper, anticipatory co-articulation of the lip protrusion and constriction gestures is investigated in speakers with visual deprivation. Audio-visual recordings of 11 congenitally blind French speakers producing [V-roundC-roundV+round] sequences were measured with a lip shape tracking system. Lip protrusion and constriction values and their relative timings were analyzed. Results show that despite the reduced magnitude of lip protrusion and constriction area in blind speakers, the timing of the anticipatory gestures can be appropriately modeled by the Movement Expansion Model [from Abry and Lallouache (1995a). Bul. de la Comm. Parlée 3, 85-99; (1995b). Proceedings of ICPHS, pp. 152-155; Noiray et al. (2011). J. Acoust. Soc. Am. 129, 340-349], which predicts lawful anticipatory behavior expanding linearly as the intervocalic consonant interval increases.

Acoustic and articulatory analysis of French vowels produced by congenitally blind adults and sighted adults.

In a previous paper [Ménard et al., J. Acoust. Soc. Am. 126, 1406-1414 (2009)], it was demonstrated that, despite enhanced auditory discrimination abilities for synthesized vowels, blind adult French speakers produced vowels that were closer together in the acoustic space than those produced by sighted adult French speakers, suggesting finer control of speech production in the sighted speakers. The goal of the present study is to further investigate the articulatory effects of visual deprivation on vowels produced by 11 blind and 11 sighted adult French speakers. Synchronous ultrasound, acoustic, and video recordings of the participants articulating the ten French oral vowels were made. Results show that sighted speakers produce vowels that are spaced significantly farther apart in the acoustic vowel space than blind speakers. Furthermore, blind speakers use smaller differences in lip protrusion but larger differences in tongue position and shape than their sighted peers to produce rounding and place of articulation contrasts. Trade-offs between lip and tongue positions were examined. Results are discussed in the light of the perception-for-action control theory.

Neural and visual processing of social gaze cueing in typical and ASD adults.

Atypical eye gaze in joint attention is a clinical characteristic of autism spectrum disorder (ASD). Despite this documented symptom, neural processing of joint attention tasks in real-life social interactions is not understood. To address this knowledge gap, functional-near infrared spectroscopy (fNIRS) and eye-tracking data were acquired simultaneously as ASD and typically developed (TD) individuals engaged in a gaze-directed joint attention task with a live human and robot partner. We test the hypothesis that face processing deficits in ASD are greater for interactive faces than for simulated (robot) faces. Consistent with prior findings, neural responses during human gaze cueing modulated by face visual dwell time resulted in increased activity of ventral frontal regions in ASD and dorsal parietal systems in TD participants. Hypoactivity of the right dorsal parietal area during live human gaze cueing was correlated with autism spectrum symptom severity: Brief Observations of Symptoms of Autism (BOSA) scores (r = âˆ'0.86). Contrarily, neural activity in response to robot gaze cueing modulated by visual acquisition factors activated dorsal parietal systems in ASD, and this neural activity was not related to autism symptom severity (r = 0.06). These results are consistent with the hypothesis that altered encoding of incoming facial information to the dorsal parietal cortex is specific to live human faces in ASD. These findings open new directions for understanding joint attention difficulties in ASD by providing a connection between superior parietal lobule activity and live interaction with human faces. Lay Summary: Little is known about why it is so difficult for autistic individuals to make eye contact with other people. We find that in a live face-to-face viewing task with a robot, the brains of autistic participants were similar to typical participants but not when the partner was a live human. Findings suggest that difficulties in real-life social situations for autistic individuals may be specific to difficulties with live social interaction rather than general face gaze.