Intelligibility as a measure of speech perception: Current approaches, challenges, and recommendations.

Intelligibility measures, which assess the number of words or phonemes a listener correctly transcribes or repeats, are commonly used metrics for speech perception research. While these measures have many benefits for researchers, they also come with a number of limitations. By pointing out the strengths and limitations of this approach, including how it fails to capture aspects of perception such as listening effort, this article argues that the role of intelligibility measures must be reconsidered in fields such as linguistics, communication disorders, and psychology. Recommendations for future work in this area are presented.

Perceptual consequences of native and non-native clear speech.

Native talkers are able to enhance acoustic characteristics of their speech in a speaking style known as "clear speech," which is better understood by listeners than "plain speech." However, despite substantial research in the area of clear speech, it is less clear whether non-native talkers of various proficiency levels are able to adopt a clear speaking style and if so, whether this style has perceptual benefits for native listeners. In the present study, native English listeners evaluated plain and clear speech produced by three groups: native English talkers, non-native talkers with lower proficiency, and non-native talkers with higher proficiency. Listeners completed a transcription task (i.e., an objective measure of the speech intelligibility). We investigated intelligibility as a function of language background and proficiency and also investigated the acoustic modifications that are associated with these perceptual benefits. The results of the study suggest that both native and non-native talkers modulate their speech when asked to adopt a clear speaking style, but that the size of the acoustic modifications, as well as consequences of this speaking style for perception differ as a function of language background and language proficiency.

The nature of non-native speech sound representations.

Most current theories and models of second language speech perception are grounded in the notion that learners acquire speech sound categories in their target language. In this paper, this classic idea in speech perception is revisited, given that clear evidence for formation of such categories is lacking in previous research. To understand the debate on the nature of speech sound representations in a second language, an operational definition of "category" is presented, and the issues of categorical perception and current theories of second language learning are reviewed. Following this, behavioral and neuroimaging evidence for and against acquisition of categorical representations is described. Finally, recommendations for future work are discussed. The paper concludes with a recommendation for integration of behavioral and neuroimaging work and theory in this area.

Investigating a bias for cue preservation in loanword adaptation.

Loanword adaptation exhibits a bias favoring sound cue preservation, possibly due to a conservative caution against deleting cues of unsure expendability in a foreign language. This study tests whether listeners are biased to preserve an acoustically ambiguous sound cue in a nonce word framed as originating from a foreign language. Results show the opposite: Listeners are less likely to transcribe an ambiguous sound cue as a phonological segment when the word containing it is framed as a loanword. However, listeners who identify as more open and accommodating to foreign people and languages show relatively more preservation in the loanword condition.

The maintenance of clear speech in naturalistic conversations.

Clear speech is a style that speakers adopt when talking with listeners whom these speakers anticipate may have a problem understanding speech. This study examines whether native English speakers use clear speech in conversations with non-native English speakers when native speakers are not explicitly asked to use clear speech (i.e., clear speech elicited with naturalistic methods). The results of the study suggest that native English speakers use clear speech in conversations with non-native English speakers even when native speakers are not explicitly asked to. Native English speakers' speech is more intelligible in the early portions of the conversations than in the late portions of each conversation. Further, the speakers "reset" to clearer speech at the start of each Diapix picture. Additionally, acoustic properties of the speech are examined to complement the intelligibility results. These findings suggest the instigation of clear speech may be listener-driven but the maintenance of clear speech is likely more speaker-driven.

Rapid adaptation to non-native speech is impaired in cochlear implant users.

To examine difficulties experienced by cochlear implant (CI) users when perceiving non-native speech, intelligibility of non-native speech was compared in conditions with single and multiple alternating talkers. Compared to listeners with normal hearing, no rapid talker-dependent adaptation was observed and performance was approximately 40% lower for CI users following increased exposure in both talker conditions. Results suggest that lower performance for CI users may stem from combined effects of limited spectral resolution, which diminishes perceptible differences across accents, and limited access to talker-specific acoustic features of speech, which reduces the ability to adapt to non-native speech in a talker-dependent manner.

Perceptual Consequences of Variability in Native and Non-Native Speech.

BACKGROUND/AIMS: Native speakers often have a difficult time understanding non-native speech, and this challenge is frequently attributed to a more variable signal. While theories and models of general speech perception are grounded in issues of variability, they rarely consider non-native speech. Here, we ask how a specific type of variability (speaking rate) impacts two measures of perception for both native and non-native speech. METHODS: In the present study, one group of listeners transcribed speech, providing a measure of intelligibility. A second group of listeners rated how fluent the speaker was, providing a measure of fluency. RESULTS: The results show that variability in speaking rate correlates with a non-native speaker's intelligibility. However, perceived fluency measures are not predicted by this variability measure. CONCLUSIONS: These results, taken with studies of the range of variability in non-native speech, suggest that variability in non-native speech is not a monolithic construct. Current theories and models of perception can be enhanced by examining non-native speech and how variability in that speech impacts perception.

Speaking rate consistency in native and non-native speakers of English.

Non-native speech differs from native speech in multiple ways. Previous research has described segmental and suprasegmental differences between native and non-native speech in terms of group averages. For example, average speaking rate for non-natives is slower than for natives. However, it is unknown whether non-native speech is also more variable than native speech. This study introduces a method of comparing rate change across utterances, demonstrating that non-native speaking rate is more variable than native speech. These results suggest that future work examining non-native speech perception and production should investigate both mean differences and variability in the signal.

Enhancing speech learning by combining task practice with periods of stimulus exposure without practice.

Language acquisition typically involves periods when the learner speaks and listens to the new language, and others when the learner is exposed to the language without consciously speaking or listening to it. Adaptation to variants of a native language occurs under similar conditions. Here, speech learning by adults was assessed following a training regimen that mimicked this common situation of language immersion without continuous active language processing. Experiment 1 focused on the acquisition of a novel phonetic category along the voice-onset-time continuum, while Experiment 2 focused on adaptation to foreign-accented speech. The critical training regimens of each experiment involved alternation between periods of practice with the task of phonetic classification (Experiment 1) or sentence recognition (Experiment 2) and periods of stimulus exposure without practice. These practice and exposure periods yielded little to no improvement separately, but alternation between them generated as much or more improvement as did practicing during every period. Practice appears to serve as a catalyst that enables stimulus exposures encountered both during and outside of the practice periods to contribute to quite distinct cases of speech learning. It follows that practice-plus-exposure combinations may tap a general learning mechanism that facilitates language acquisition and speech processing.

Long-term temporal tracking of speech rate affects spoken-word recognition.

Humans unconsciously track a wide array of distributional characteristics in their sensory environment. Recent research in spoken-language processing has demonstrated that the speech rate surrounding a target region within an utterance influences which words, and how many words, listeners hear later in that utterance. On the basis of hypotheses that listeners track timing information in speech over long timescales, we investigated the possibility that the perception of words is sensitive to speech rate over such a timescale (e.g., an extended conversation). Results demonstrated that listeners tracked variation in the overall pace of speech over an extended duration (analogous to that of a conversation that listeners might have outside the lab) and that this global speech rate influenced which words listeners reported hearing. The effects of speech rate became stronger over time. Our findings are consistent with the hypothesis that neural entrainment by speech occurs on multiple timescales, some lasting more than an hour.

The Effects of Acoustic and Semantic Enhancements on Perception of Native and Non-Native Speech.

Previous research has shown that native listeners benefit from clearly produced speech, as well as from predictable semantic context when these enhancements are delivered in native speech. However, it is unclear whether native listeners benefit from acoustic and semantic enhancements differently when listening to other varieties of speech, including non-native speech. The current study examines to what extent native English listeners benefit from acoustic and semantic cues present in native and non-native English speech. Native English listeners transcribed sentence final words that were of different levels of semantic predictability, produced in plain- or clear-speaking styles by Native English talkers and by native Mandarin talkers of higher- and lower-proficiency in English. The perception results demonstrated that listeners benefited from semantic cues in higher- and lower-proficiency talkers' speech (i.e., transcribed speech more accurately), but not from acoustic cues, even though higher-proficiency talkers did make substantial acoustic enhancements from plain to clear speech. The current results suggest that native listeners benefit more robustly from semantic cues than from acoustic cues when those cues are embedded in non-native speech.

Selectivity to acoustic features of human speech in the auditory cortex of the mouse.

A better understanding of the neural mechanisms of speech processing can have a major impact in the development of strategies for language learning and in addressing disorders that affect speech comprehension. Technical limitations in research with human subjects hinder a comprehensive exploration of these processes, making animal models essential for advancing the characterization of how neural circuits make speech perception possible. Here, we investigated the mouse as a model organism for studying speech processing and explored whether distinct regions of the mouse auditory cortex are sensitive to specific acoustic features of speech. We found that mice can learn to categorize frequency-shifted human speech sounds based on differences in formant transitions (FT) and voice onset time (VOT). Moreover, neurons across various auditory cortical regions were selective to these speech features, with a higher proportion of speech-selective neurons in the dorso-posterior region. Last, many of these neurons displayed mixed-selectivity for both features, an attribute that was most common in dorsal regions of the auditory cortex. Our results demonstrate that the mouse serves as a valuable model for studying the detailed mechanisms of speech feature encoding and neural plasticity during speech-sound learning.

Passive exposure to task-relevant stimuli enhances categorization learning.

Learning to perform a perceptual decision task is generally achieved through sessions of effortful practice with feedback. Here, we investigated how passive exposure to task-relevant stimuli, which is relatively effortless and does not require feedback, influences active learning. First, we trained mice in a sound-categorization task with various schedules combining passive exposure and active training. Mice that received passive exposure exhibited faster learning, regardless of whether this exposure occurred entirely before active training or was interleaved between active sessions. We next trained neural-network models with different architectures and learning rules to perform the task. Networks that use the statistical properties of stimuli to enhance separability of the data via unsupervised learning during passive exposure provided the best account of the behavioral observations. We further found that, during interleaved schedules, there is an increased alignment between weight updates from passive exposure and active training, such that a few interleaved sessions can be as effective as schedules with long periods of passive exposure before active training, consistent with our behavioral observations. These results provide key insights for the design of efficient training schedules that combine active learning and passive exposure in both natural and artificial systems.

Examining the Relationship Between Multiple Tests of Receptive Vocabulary.

PURPOSE: Numerous tasks have been developed to measure receptive vocabulary, many of which were designed to be administered in person with a trained researcher or clinician. The purpose of the current study is to compare a common, in-person test of vocabulary with other vocabulary assessments that can be self-administered. METHOD: Fifty-three participants completed the Peabody Picture Vocabulary Test (PPVT) via online video call to mimic in-person administration, as well as four additional fully automated, self-administered measures of receptive vocabulary. Participants also completed three control tasks that do not measure receptive vocabulary. RESULTS: Pearson correlations indicated moderate correlations among most of the receptive vocabulary measures (approximately r = .50-.70). As expected, the control tasks revealed only weak correlations to the vocabulary measures. However, subsets of items of the four self-administered measures of receptive vocabulary achieved high correlations with the PPVT (r > .80). These subsets were found through a repeated resampling approach. CONCLUSIONS: Measures of receptive vocabulary differ in which items are included and in the assessment task (e.g., lexical decision, picture matching, synonym matching). The results of the current study suggest that several self-administered tasks are able to achieve high correlations with the PPVT when a subset of items are scored, rather than the full set of items. These data provide evidence that subsets of items on one behavioral assessment can more highly correlate to another measure. In practical terms, these data demonstrate that self-administered, automated measures of receptive vocabulary can be used as reasonable substitutes of at least one test (PPVT) that requires human interaction. That several of the fully automated measures resulted in high correlations with the PPVT suggests that different tasks could be selected depending on the needs of the researcher. It is important to note the aim was not to establish clinical relevance of these measures, but establish whether researchers could use an experimental task of receptive vocabulary that probes a similar construct to what is captured by the PPVT, and use these measures of individual differences.

Accent-independent adaptation to foreign accented speech.

Foreign-accented speech can be difficult to understand but listeners can adapt to novel talkers and accents with appropriate experience. Previous studies have demonstrated talker-independent but accent-dependent learning after training on multiple talkers from a single language background. Here, listeners instead were exposed to talkers from five language backgrounds during training. After training, listeners generalized their learning to novel talkers from language backgrounds both included and not included in the training set. These findings suggest that generalization of foreign-accent adaptation is the result of exposure to systematic variability in accented speech that is similar across talkers from multiple language backgrounds.

Selectivity to acoustic features of human speech in the auditory cortex of the mouse.

A better understanding of the neural mechanisms of speech processing can have a major impact in the development of strategies for language learning and in addressing disorders that affect speech comprehension. Technical limitations in research with human subjects hinder a comprehensive exploration of these processes, making animal models essential for advancing the characterization of how neural circuits make speech perception possible. Here, we investigated the mouse as a model organism for studying speech processing and explored whether distinct regions of the mouse auditory cortex are sensitive to specific acoustic features of speech. We found that mice can learn to categorize frequency-shifted human speech sounds based on differences in formant transitions (FT) and voice onset time (VOT). Moreover, neurons across various auditory cortical regions were selective to these speech features, with a higher proportion of speech-selective neurons in the dorso-posterior region. Last, many of these neurons displayed mixed-selectivity for both features, an attribute that was most common in dorsal regions of the auditory cortex. Our results demonstrate that the mouse serves as a valuable model for studying the detailed mechanisms of speech feature encoding and neural plasticity during speech-sound learning.

Passive exposure to task-relevant stimuli enhances categorization learning.

Learning to perform a perceptual decision task is generally achieved through sessions of effortful practice with feedback. Here, we investigated how passive exposure to task-relevant stimuli, which is relatively effortless and does not require feedback, influences active learning. First, we trained mice in a sound-categorization task with various schedules combining passive exposure and active training. Mice that received passive exposure exhibited faster learning, regardless of whether this exposure occurred entirely before active training or was interleaved between active sessions. We next trained neural-network models with different architectures and learning rules to perform the task. Networks that use the statistical properties of stimuli to enhance separability of the data via unsupervised learning during passive exposure provided the best account of the behavioral observations. We further found that, during interleaved schedules, there is an increased alignment between weight updates from passive exposure and active training, such that a few interleaved sessions can be as effective as schedules with long periods of passive exposure before active training, consistent with our behavioral observations. These results provide key insights for the design of efficient training schedules that combine active learning and passive exposure in both natural and artificial systems.

Just give it time: Differential effects of disruption and delay on perceptual learning.

Speech perception and production are critical skills when acquiring a new language. However, the nature of the relationship between these two processes is unclear, particularly for non-native speech sound contrasts. Although it has been assumed that perception and production are supportive, recent evidence has demonstrated that, under some circumstances, production can disrupt perceptual learning. Specifically, producing the to-be-learned contrast on each trial can disrupt perceptual learning of that contrast. Here, we treat speech perception and speech production as separate tasks. From this perspective, perceptual learning studies that include a production component on each trial create a task switch. We report two experiments that test how task switching can disrupt perceptual learning. One experiment demonstrates that the disruption caused by switching to production is sensitive to time delays: Increasing the delay between perception and production on a trial can reduce and even eliminate disruption of perceptual learning. The second experiment shows that if a task other than producing the to-be-learned contrast is imposed, the task-switching component of disruption is not influenced by a delay. These experiments provide a new understanding of the relationship between speech perception and speech production, and clarify conditions under which the two cooperate or compete.

Effects of Spectral Envelope and Fundamental Frequency Shifts on the Perception of Foreign-Accented Speech.

To investigate the role of spectral pattern information in the perception of foreign-accented speech, we measured the effects of spectral shifts on judgments of talker discrimination, perceived naturalness, and intelligibility when listening to Mandarin-accented English and native-accented English sentences. In separate conditions, the spectral envelope and fundamental frequency (F0) contours were shifted up or down in three steps using coordinated scale factors (multiples of 8% and 30%, respectively). Experiment 1 showed that listeners perceive spectrally shifted sentences as coming from a different talker for both native-accented and foreign-accented speech. Experiment 2 demonstrated that downward shifts applied to male talkers and the largest upward shifts applied to all talkers reduced the perceived naturalness, regardless of accent. Overall, listeners rated foreign-accented speech as sounding less natural even for unshifted speech. In Experiment 3, introducing spectral shifts further lowered the intelligibility of foreign-accented speech. When speech from the same foreign-accented talker was shifted to simulate five different talkers, increased exposure failed to produce an improvement in intelligibility scores, similar to the pattern observed when listeners actually heard five foreign-accented talkers. Intelligibility of spectrally shifted native-accented speech was near ceiling performance initially, and no further improvement or decrement was observed. These experiments suggest a mechanism that utilizes spectral envelope and F0 cues in a talker-dependent manner to support the perception of foreign-accented speech.

Semantic predictability and adaptation to nonnative speech.

Listeners improve their ability to understand nonnative speech through exposure. The present study examines the role of semantic predictability during adaptation. Listeners were trained on high-predictability, low-predictability, or semantically anomalous sentences. Results demonstrate that trained participants improve their perception of nonnative speech compared to untrained participants. Adaptation is most robust for the types of sentences participants heard during training; however, semantic predictability during exposure did not impact the amount of adaptation overall. Results show advantages in adaptation specific to the type of speech material, a finding similar to the specificity of adaptation previously demonstrated for individual talkers or accents.