Speech production and perception data collection in R: A tutorial for web-based methods using speechcollectr.

This tutorial is designed for speech scientists familiar with the R programming language who wish to construct experiment interfaces in R. We begin by discussing some of the benefits of building experiment interfaces in R-including R's existing tools for speech data analysis, platform independence, suitability for web-based testing, and the fact that R is open source. We explain basic concepts of reactive programming in R, and we apply these principles by detailing the development of two sample experiments. The first of these experiments comprises a speech production task in which participants are asked to read words with different emotions. The second sample experiment involves a speech perception task, in which participants listen to recorded speech and identify the emotion the talker expressed with forced-choice questions and confidence ratings. Throughout this tutorial, we introduce the new R package speechcollectr, which provides functions uniquely suited to web-based speech data collection. The package streamlines the code required for speech experiments by providing functions for common tasks like documenting participant consent, collecting participant demographic information, recording audio, checking the adequacy of a participant's microphone or headphones, and presenting audio stimuli. Finally, we describe some of the difficulties of remote speech data collection, along with the solutions we have incorporated into speechcollectr to meet these challenges.

Effects of Spectral Envelope and Fundamental Frequency Shifts on the Perception of Foreign-Accented Speech.

To investigate the role of spectral pattern information in the perception of foreign-accented speech, we measured the effects of spectral shifts on judgments of talker discrimination, perceived naturalness, and intelligibility when listening to Mandarin-accented English and native-accented English sentences. In separate conditions, the spectral envelope and fundamental frequency (F0) contours were shifted up or down in three steps using coordinated scale factors (multiples of 8% and 30%, respectively). Experiment 1 showed that listeners perceive spectrally shifted sentences as coming from a different talker for both native-accented and foreign-accented speech. Experiment 2 demonstrated that downward shifts applied to male talkers and the largest upward shifts applied to all talkers reduced the perceived naturalness, regardless of accent. Overall, listeners rated foreign-accented speech as sounding less natural even for unshifted speech. In Experiment 3, introducing spectral shifts further lowered the intelligibility of foreign-accented speech. When speech from the same foreign-accented talker was shifted to simulate five different talkers, increased exposure failed to produce an improvement in intelligibility scores, similar to the pattern observed when listeners actually heard five foreign-accented talkers. Intelligibility of spectrally shifted native-accented speech was near ceiling performance initially, and no further improvement or decrement was observed. These experiments suggest a mechanism that utilizes spectral envelope and F0 cues in a talker-dependent manner to support the perception of foreign-accented speech.

Perception of gender in children's voices.

To investigate the perception of gender from children's voices, adult listeners were presented with /hVd/ syllables, in isolation and in sentence context, produced by children between 5 and 18 years. Half the listeners were informed of the age of the talker during trials, while the other half were not. Correct gender identifications increased with talker age; however, performance was above chance even for age groups where the cues most often associated with gender differentiation (i.e., average fundamental frequency and formant frequencies) were not consistently different between boys and girls. The results of acoustic models suggest that cues were used in an age-dependent manner, whether listeners were explicitly told the age of the talker or not. Overall, results are consistent with the hypothesis that talker age and gender are estimated jointly in the process of speech perception. Furthermore, results show that the gender of individual talkers can be identified accurately well before reliable anatomical differences arise in the vocal tracts of females and males. In general, results support the notion that the transmission of gender information from voice depends substantially on gender-dependent patterns of articulation, rather than following deterministically from anatomical differences between male and female talkers.

Unsupervised repetition enables rapid perceptual learning.

This study examined how listeners disambiguate an auditory scene comprising multiple competing unknown sources and determine a salient source. Experiment 1 replicated findings from McDermott, Wrobleski, and Oxenham. [(2011). Proc. Natl. Acad. Sci. U. S. A. 108(3), 1188-1193] using a multivariate Gaussian model to generate mixtures of two novel sounds. The results showed that listeners were unable to identify either sound in the mixture despite repeated exposure unless one sound was repeated several times while being mixed with a different distractor each time. The results support the idea that repetition provides a basis for segregating a single source from competing novel sounds. In subsequent experiments, the previous identification task was extended to a recognition task and the results were modeled. To confirm the repetition benefit, experiment 2 asked listeners to recognize a temporal ramp in either a repeating sound or non-repeating sounds. The results showed that perceptual salience of the repeating sound allowed robust recognition of its temporal ramp, whereas similar features were ignored in the non-repeating sounds. The response from two neural models of learning, generalized Hebbian learning and anti-Hebbian learning, were compared with the human listener results from experiment 2. The Hebbian network showed a similar response pattern as for the listener results, whereas the opposite pattern was observed for the anti-Hebbian output.

Rapid adaptation to non-native speech is impaired in cochlear implant users.

To examine difficulties experienced by cochlear implant (CI) users when perceiving non-native speech, intelligibility of non-native speech was compared in conditions with single and multiple alternating talkers. Compared to listeners with normal hearing, no rapid talker-dependent adaptation was observed and performance was approximately 40% lower for CI users following increased exposure in both talker conditions. Results suggest that lower performance for CI users may stem from combined effects of limited spectral resolution, which diminishes perceptible differences across accents, and limited access to talker-specific acoustic features of speech, which reduces the ability to adapt to non-native speech in a talker-dependent manner.

Constraints on ideal binary masking for the perception of spectrally-reduced speech.

This study investigated recognition of sentences processed using ideal binary masking (IBM) with limited spectral resolution. Local thresholds (LCs) of -12, 0, and 5 dB were applied which altered the target and masker power following IBM. Recognition was reduced due to persistence of the masker and limited target recovery, thus preventing IBM from ideal target-masker segregation. Linear regression and principal component analyses showed that, regardless of masker type and number of spectral channels, higher LCs were associated with poorer recognition. In addition, limitations on target recovery led to more detrimental effects on speech recognition compared to persistence of the masker.

Modeling the perception of children's age from speech acoustics.

Adult listeners were presented with /hVd/ syllables spoken by boys and girls ranging from 5 to 18 years of age. Half of the listeners were informed of the sex of the speaker; the other half were not. Results indicate that veridical age in children can be predicted accurately based on the acoustic characteristics of the talker's voice and that listener behavior is highly predictable on the basis of speech acoustics. Furthermore, listeners appear to incorporate assumptions about talker sex into their estimates of talker age, even when information about the talker's sex is not explicitly provided for them.

Perceiving foreign-accented speech with decreased spectral resolution in single- and multiple-talker conditions.

To determine the effect of reduced spectral resolution on the intelligibility of foreign-accented speech, vocoder-processed sentences from native and Mandarin-accented English talkers were presented to listeners in single- and multiple-talker conditions. Reduced spectral resolution had little effect on native speech but lowered performance for foreign-accented speech, with a further decrease in multiple-talker conditions. Following the initial exposure, foreign-accented speech with reduced spectral resolution was less intelligible than unprocessed speech in both single- and multiple-talker conditions. Intelligibility improved with extended exposure, but only for single-talker conditions. Results indicate a perceptual impairment when perceiving foreign-accented speech with reduced spectral resolution.

Production and perception of emotional prosody by adults with autism spectrum disorder.

This study examined production and perception of affective prosody by adults with autism spectrum disorder (ASD). Previous research has reported increased pitch variability in talkers with ASD compared to typically developing (TD) controls in grammatical speaking tasks (e.g., comparing interrogative vs. declarative sentences), but it is unclear whether this pattern extends to emotional speech. In this study, speech recordings in five emotion contexts (angry, happy, interested, sad, and neutral) were obtained from 15 adult males with ASD and 15 controls (Experiment 1), and were later presented to 52 listeners (22 with ASD) who were asked to identify the emotion expressed and rate the level of naturalness of the emotion in each recording (Experiment 2). Compared to the TD group, talkers with ASD produced phrases with greater intensity, longer durations, and increased pitch range for all emotions except neutral, suggesting that their greater pitch variability was specific to emotional contexts. When asked to identify emotion from speech, both groups of listeners were more accurate at identifying the emotion context from speech produced by ASD speakers compared to TD speakers, but rated ASD emotional speech as sounding less natural. Collectively, these results reveal differences in emotional speech production in talkers with ASD that provide an acoustic basis for reported perceptions of oddness in the speech presentation of adults with ASD. Autism Res 2017, 10: 1991-2001. © 2017 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: This study examined emotional speech communication produced and perceived by adults with autism spectrum disorder (ASD) and typically-developing (TD) controls. Compared to the TD group, talkers with ASD produced emotional phrases that were louder, longer, and more variable in pitch. Both ASD and TD listeners were more accurate at identifying emotion in speech produced by ASD speakers compared to TD speakers, but rated ASD emotional speech as sounding less natural.

Voice gender and the segregation of competing talkers: Perceptual learning in cochlear implant simulations.

Two experiments explored the role of differences in voice gender in the recognition of speech masked by a competing talker in cochlear implant simulations. Experiment 1 confirmed that listeners with normal hearing receive little benefit from differences in voice gender between a target and masker sentence in four- and eight-channel simulations, consistent with previous findings that cochlear implants deliver an impoverished representation of the cues for voice gender. However, gender differences led to small but significant improvements in word recognition with 16 and 32 channels. Experiment 2 assessed the benefits of perceptual training on the use of voice gender cues in an eight-channel simulation. Listeners were assigned to one of four groups: (1) word recognition training with target and masker differing in gender; (2) word recognition training with same-gender target and masker; (3) gender recognition training; or (4) control with no training. Significant improvements in word recognition were observed from pre- to post-test sessions for all three training groups compared to the control group. These improvements were maintained at the late session (one week following the last training session) for all three groups. There was an overall improvement in masked word recognition performance provided by gender mismatch following training, but the amount of benefit did not differ as a function of the type of training. The training effects observed here are consistent with a form of rapid perceptual learning that contributes to the segregation of competing voices but does not specifically enhance the benefits provided by voice gender cues.

Precedence based speech segregation in bilateral cochlear implant users.

The precedence effect (PE) enables the perceptual dominance by a source (lead) over an echo (lag) in reverberant environments. In addition to facilitating sound localization, the PE can play an important role in spatial unmasking of speech. Listeners attending to binaural vocoder simulations with identical channel center frequencies and phase demonstrated PE-based benefits in a closed-set speech segregation task. When presented with the same stimuli, bilateral cochlear implant users did not derive such benefits. These findings suggest that envelope extraction in itself may not lead to a breakdown of the PE benefits, and that other factors may play a role.

Perceptual adaptation to gender and expressive properties in speech: the role of fundamental frequency.

Recent studies have demonstrated perceptual adaptation to nonlinguistic properties of speech involving voice gender and emotional expression. The present study extends this work by examining the contribution of fundamental frequency (F0) to these effects. Voice recordings of vowel-consonant-vowel (VCV) syllables from six talkers were processed using the STRAIGHT vocoder and an auditory morphing technique to synthesize gender (experiment 1) and expressive (experiment 2) speech sound continua ranging from one category endpoint to the other (female to male; angry to happy). Continuum endpoints served as adaptors for F0 present and F0 removed conditions. F0 removed stimuli were created by replacing the periodic excitation source with broadband noise. Confirming previous findings, aftereffects were found in the F0 present condition, resulting in a decreased likelihood to identify test stimuli as belonging to the adaptor category. No aftereffects appeared when F0 was removed, highlighting the importance of F0 in adaptation. However, in an identification test listeners were still able to categorize F0 removed stimuli at better-than-chance levels, indicating that residual cues for gender and emotion were available even when F0 was not present.

Auditory training of speech recognition with interrupted and continuous noise maskers by children with hearing impairment.

Previous studies have indicated that individuals with normal hearing (NH) experience a perceptual advantage for speech recognition in interrupted noise compared to continuous noise. In contrast, adults with hearing impairment (HI) and younger children with NH receive a minimal benefit. The objective of this investigation was to assess whether auditory training in interrupted noise would improve speech recognition in noise for children with HI and perhaps enhance their utilization of glimpsing skills. A partially-repeated measures design was used to evaluate the effectiveness of seven 1-h sessions of auditory training in interrupted and continuous noise. Speech recognition scores in interrupted and continuous noise were obtained from pre-, post-, and 3 months post-training from 24 children with moderate-to-severe hearing loss. Children who participated in auditory training in interrupted noise demonstrated a significantly greater improvement in speech recognition compared to those who trained in continuous noise. Those who trained in interrupted noise demonstrated similar improvements in both noise conditions while those who trained in continuous noise only showed modest improvements in the interrupted noise condition. This study presents direct evidence that auditory training in interrupted noise can be beneficial in improving speech recognition in noise for children with HI.

Identification of frequency-shifted vowels.

Within certain limits, speech intelligibility is preserved with upward or downward scaling of the spectral envelope. To study these limits and assess their interaction with fundamental frequency (F0), vowels in /hVd/ syllables were processed using the STRAIGHT vocoder and presented to listeners for identification. Identification accuracy showed a gradual decline when the spectral envelope was scaled up or down in vowels spoken by men, women, and children. Upward spectral envelope shifts led to poorer identification of children's vowels compared to adults, while downward shifts had a greater impact on men's vowels compared to women and children. Coordinated shifts (F0 and spectral envelope shifted in the same direction) generally produced higher accuracy than conditions with F0 and spectral envelope shifted in opposite directions. Vowel identification was poorest in conditions with very high F0, consistent with suggestions from the literature that sparse sampling of the spectral envelope may be a factor in vowel identification. However, the gradual decline in accuracy as a function of both upward and downward spectral envelope shifts and the interaction between spectral envelope shifts and F0 suggests the additional operation of perceptual mechanisms sensitive to the statistical covariation of F0 and formant frequencies in natural speech.

Relationship between fundamental and formant frequencies in voice preference.

Covariation in the size of laryngeal and vocal tract structures leads to a moderate correlation between fundamental frequency (F0) and formant frequencies (FFs) in natural speech. A method of adjustment procedure was used to test whether listeners prefer combinations of F0 and FFs that reflect this covariation. Vowel sequences spoken by two men and two women were processed by the STRAIGHT vocoder to construct three sets of frequency-shifted continua. The distributions of "best choice" responses in all three experiments confirm that listeners prefer coordinated patterns of F0 and FF similar to those of natural speech.

Effects of cochlear implant processing and fundamental frequency on the intelligibility of competing sentences.

Speech perception in the presence of another competing voice is one of the most challenging tasks for cochlear implant users. Several studies have shown that (1) the fundamental frequency (F0) is a useful cue for segregating competing speech sounds and (2) the F0 is better represented by the temporal fine structure than by the temporal envelope. However, current cochlear implant speech processing algorithms emphasize temporal envelope information and discard the temporal fine structure. In this study, speech recognition was measured as a function of the F0 separation of the target and competing sentence in normal-hearing and cochlear implant listeners. For the normal-hearing listeners, the combined sentences were processed through either a standard implant simulation or a new algorithm which additionally extracts a slowed-down version of the temporal fine structure (called Frequency-Amplitude-Modulation-Encoding). The results showed no benefit of increasing F0 separation for the cochlear implant or simulation groups. In contrast, the new algorithm resulted in gradual improvements with increasing F0 separation, similar to that found with unprocessed sentences. These results emphasize the importance of temporal fine structure for speech perception and demonstrate a potential remedy for difficulty in the perceptual segregation of competing speech sounds.

Effects of auditory feedback deprivation length on the vowel /epsilon/ produced by pediatric cochlear-implant users.

Effects of auditory deprivation on speech production by ten cochlear-implanted children were investigated by turning off the implant for durations ranging from 0.3 to 5.0 s and measuring the formant frequencies (F1 and F2) of the vowel /epsilon/. In five of the ten talkers, F1 and/or F2 shifted when auditory feedback was eliminated. Without feedback, F2 frequency lowered consistently, suggesting vowel centralization. Phonetic transcription indicated that some of these acoustic changes led to perceptible shifts in phonetic quality. The results provide evidence that brief periods of auditory deprivation can produce perceptible changes in vowels produced by some cochlear-implanted children.

Effects of auditory feedback on fricatives produced by cochlear-implanted adults and children: acoustic and perceptual evidence.

Acoustic analyses and perception experiments were conducted to determine the effects of brief deprivation of auditory feedback on fricatives produced by cochlear implant users. The words /si/ and /Si/ were recorded by four children and four adults with their cochlear implant speech processor turned on or off. In the processor-off condition, word durations increased significantly for a majority of talkers. These increases were greater for children compared to adults, suggesting that children may rely on auditory feedback to a greater extent than adults. Significant differences in spectral measures of /S/ were found between processor-on and processor-off conditions for two of the four children and for one of the four adults. These talkers also demonstrated a larger /s/-/S/ contrast in centroid values compared to the other talkers within their respective groups. This finding may indicate that talkers who produce fine spectral distinctions are able to perceive these distinctions through their implants and to use this feedback to fine tune their speech. Two listening experiments provided evidence that some of the acoustic changes were perceptible to normal-hearing listeners. Taken together, these experiments indicate that for certain cochlear-implant users the brief absence of auditory feedback may lead to perceptible modifications in fricative consonants.

Effects of electrode design and configuration on channel interactions.

A potential shortcoming of existing multichannel cochlear implants is electrical-field summation during simultaneous electrode stimulation. Electrical-field interactions can disrupt the stimulus waveform prior to neural activation. To test whether speech intelligibility can be degraded by electrical-field interaction, speech recognition performance and interaction were examined for three Clarion electrode arrays: the pre-curved, enhanced bipolar electrode array, the enhanced bipolar electrode with an electrode positioner, and the Hi-Focus electrode with a positioner. Channel interaction was measured by comparing stimulus detection thresholds for a probe signal in the presence of a sub-threshold perturbation signal as a function of the separation between the two simultaneously stimulated electrodes. Correct identification of vowels, consonants, and words in sentences was measured with two speech strategies: one which used simultaneous stimulation and another which used sequential stimulation. Speech recognition scores were correlated with measured electrical-field interaction for the strategy which used simultaneous stimulation but not the strategy which used sequential stimulation. Higher speech recognition scores with the simultaneous strategy were generally associated with lower levels of electrical-field interaction. Electrical-field interaction accounted for as much as 70% of the variance in speech recognition scores, suggesting that electrical-field interaction is a significant contributor to the variability found across patients who use simultaneous strategies.

Synthesis fidelity and time-varying spectral change in vowels.

Recent studies have shown that synthesized versions of American English vowels are less accurately identified when the natural time-varying spectral changes are eliminated by holding the formant frequencies constant over the duration of the vowel. A limitation of these experiments has been that vowels produced by formant synthesis are generally less accurately identified than the natural vowels after which they are modeled. To overcome this limitation, a high-quality speech analysis-synthesis system (STRAIGHT) was used to synthesize versions of 12 American English vowels spoken by adults and children. Vowels synthesized with STRAIGHT were identified as accurately as the natural versions, in contrast with previous results from our laboratory showing identification rates 9%-12% lower for the same vowels synthesized using the cascade formant model. Consistent with earlier studies, identification accuracy was not reduced when the fundamental frequency was held constant across the vowel. However, elimination of time-varying changes in the spectral envelope using STRAIGHT led to a greater reduction in accuracy (23%) than was previously found with cascade formant synthesis (11%). A statistical pattern recognition model, applied to acoustic measurements of the natural and synthesized vowels, predicted both the higher identification accuracy for vowels synthesized using STRAIGHT compared to formant synthesis, and the greater effects of holding the formant frequencies constant over time with STRAIGHT synthesis. Taken together, the experiment and modeling results suggest that formant estimation errors and incorrect rendering of spectral and temporal cues by cascade formant synthesis contribute to lower identification accuracy and underestimation of the role of time-varying spectral change in vowels.