Effects of acoustic and semantic cues on listening effort during native and non-native speech perception.

Relative to native listeners, non-native listeners who are immersed in a second language environment experience increased listening effort and a reduced ability to successfully perform an additional task while listening. Previous research demonstrated that listeners can exploit a variety of intelligibility-enhancing cues to cope with adverse listening conditions. However, little is known about the implications of those speech perception strategies for listening effort. The current research aims to investigate by means of pupillometry how listening effort is modulated in native and non-native listeners by the availability of semantic context and acoustic enhancements during the comprehension of spoken sentences. For this purpose, semantic plausibility and speaking style were manipulated both separately and in combination during a speech perception task in noise. The signal to noise ratio was individually adjusted for each participant in order to target 50% intelligibility level. Behavioural results indicated that native and non-native listeners were equally able to fruitfully exploit both semantic and acoustic cues to aid their comprehension. Pupil data indicated that listening effort was reduced for both groups of listeners when acoustic enhancements were available, while the presence of a plausible semantic context did not lead to a reduction in listening effort.

The relationship between talker acoustics, intelligibility, and effort in degraded listening conditions.

Listening to degraded speech is associated with decreased intelligibility and increased effort. However, listeners are generally able to adapt to certain types of degradations. While intelligibility of degraded speech is modulated by talker acoustics, it is unclear whether talker acoustics also affect effort and adaptation. Moreover, it has been demonstrated that talker differences are preserved across spectral degradations, but it is not known whether this effect extends to temporal degradations and which acoustic-phonetic characteristics are responsible. In a listening experiment combined with pupillometry, participants were presented with speech in quiet as well as in masking noise, time-compressed, and noise-vocoded speech by 16 Southern British English speakers. Results showed that intelligibility, but not adaptation, was modulated by talker acoustics. Talkers who were more intelligible under noise-vocoding were also more intelligible under masking and time-compression. This effect was linked to acoustic-phonetic profiles with greater vowel space dispersion (VSD) and energy in mid-range frequencies, as well as slower speaking rate. While pupil dilation indicated increasing effort with decreasing intelligibility, this study also linked reduced effort in quiet to talkers with greater VSD. The results emphasize the relevance of talker acoustics for intelligibility and effort in degraded listening conditions.

Intelligibility of conversational and clear speech in young and older talkers as perceived by young and older listeners.

A related paper [Hazan, Tuomainen, Tu, Kim, Davis, Brungart, and Sheffield. (2018b). Hear. Res. 369, 33-41] showed that, for young adult listeners, speech produced by older adults was less intelligible than the speech of young adults but both talker groups improved the intelligibility of their speech via clear speech modifications. Here, this study was extended to include older listeners with/without presbycusis. The results showed that for older listeners, speech produced by older adults was less intelligible than the speech of young adults and scores improved in the clear speech condition. The best predictor of intelligibility was the amount of energy in the mid-frequency range of the spectrum.

Clear speech adaptations in spontaneous speech produced by young and older adults.

The study investigated the speech adaptations by older adults (OA) with and without age-related hearing loss made to communicate effectively in challenging communicative conditions. Acoustic analyses were carried out on spontaneous speech produced during a problem-solving task (diapix) carried out by talker pairs in different listening conditions. There were 83 talkers of Southern British English. Fifty-seven talkers were OAs aged 65-84, 30 older adults with normal hearing (OANH), and 27 older adults with hearing loss (OAHL) [mean pure tone average (PTA) 0.250-4 kHz: 27.7 dB HL]. Twenty-six talkers were younger adults (YA) aged 18-26 with normal hearing. Participants were recorded while completing the diapix task with a conversational partner (YA of the same sex) when (a) both talkers heard normally (NORM), (b) the partner had a simulated hearing loss, and (c) both talkers heard babble noise. Irrespective of hearing status, there were age-related differences in some acoustic characteristics of YA and OA speech produced in NORM, most likely linked to physiological factors. In challenging conditions, while OANH talkers typically patterned with YA talkers, OAHL talkers made adaptations more consistent with an increase in vocal effort. The study suggests that even mild presbycusis in healthy OAs can affect the speech adaptations made to maintain effective communication.

Do talkers produce less dispersed phoneme categories in a clear speaking style?

This study investigated whether adaptations made in clear speaking styles result in more discriminable phonetic categories than in a casual style. Multiple iterations of keywords with word-initial /s/-/ʃ/ were obtained from 40 adults in casual and clear speech via picture description. For centroids, cross-category distance increased in clear speech but with no change in within-category dispersion and no effect on discriminability. However, talkers produced fewer tokens with centroids in the ambiguous region for the /s/-/ʃ/ distinction. These results suggest that, whereas interlocutor feedback regarding communicative success may promote greater segmental adaptations, it is not necessary for some adaptation to occur.

Partner-directed gaze and co-speech hand gestures: effects of age, hearing loss and noise.

Research on the adaptations talkers make to different communication conditions during interactive conversations has primarily focused on speech signals. We extended this type of investigation to two other important communicative signals, i.e., partner-directed gaze and iconic co-speech hand gestures with the aim of determining if the adaptations made by older adults differ from younger adults across communication conditions. We recruited 57 pairs of participants, comprising 57 primary talkers and 57 secondary ones. Primary talkers consisted of three groups: 19 older adults with mild Hearing Loss (older adult-HL); 17 older adults with Normal Hearing (older adult-NH); and 21 younger adults. The DiapixUK "spot the difference" conversation-based task was used to elicit conversions in participant pairs. One easy (No Barrier: NB) and three difficult communication conditions were tested. The three conditions consisted of two in which the primary talker could hear clearly, but the secondary talkers could not, due to multi-talker babble noise (BAB1) or a less familiar hearing loss simulation (HLS), and a condition in which both the primary and secondary talkers heard each other in babble noise (BAB2). For primary talkers, we measured mean number of partner-directed gazes; mean total gaze duration; and the mean number of co-speech hand gestures. We found a robust effects of communication condition that interacted with participant group. Effects of age were found for both gaze and gesture in BAB1, i.e., older adult-NH looked and gestured less than younger adults did when the secondary talker experienced babble noise. For hearing status, a difference in gaze between older adult-NH and older adult-HL was found for the BAB1 condition; for gesture this difference was significant in all three difficult communication conditions (older adult-HL gazed and gestured more). We propose the age effect may be due to a decline in older adult's attention to cues signaling how well a conversation is progressing. To explain the hearing status effect, we suggest that older adult's attentional decline is offset by hearing loss because these participants have learned to pay greater attention to visual cues for understanding speech.

Developmental and gender-related trends of intra-talker variability in consonant production.

This study investigates the effect of age and gender on the internal structure, cross-category distance, and discriminability of phonemic categories for two contrasts varying in fricative place of articulation (/s/-/∫/) and stop voicing (/b/-/p/) in word-initial tokens spoken by adults and normally developing children aged 9-14 yr. Vast between- and within-talker variability was observed with 16% of speakers exhibiting some degree of overlap between phonemic categories-a possible contribution to the range of talker intelligibility found in the literature. Females of all ages produced farther and thus more discriminable categories than males, although gender-marking for fricative between-category distance did not emerge until approximately 11 yr of age. Children produced farther yet also much more dispersed categories than adults with increasing discriminability with age, such that by age 13, children's categories were no less discriminable than those of adults. However, children's ages did not predict category distance or dispersion, indicating that convergence on adult-like category structure must occur later in adolescence.

Is clear speech tailored to counter the effect of specific adverse listening conditions?

The study investigated the perception of speech produced to counter the effects of adverse listening conditions. Participants completed a problem-solving task with an interlocutor in good listening conditions (NB) or with the interlocutor hearing them via a vocoder (VOC) or babble (BAB). Keywords extracted from recordings were presented in babble for initial consonant identification. BAB tokens were identified faster than VOC or NB tokens even though VOC and BAB tokens were rated as similarly clear. Acoustic measures showed clarifications to be global rather than specifically enhancing phonological contrasts. These results suggest that clear speaking styles are tailored to listeners' needs.

Impact of talker variability on word recognition in non-native listeners.

This study investigates the perception of English words produced by 45 native talkers presented in moderate noise to native Norwegian listeners. The relative intelligibility of individual talkers is compared with that obtained for native listeners in order to determine whether intrinsic talker clarity is determined by global acoustic-phonetic characteristics. Talker intelligibility was strongly correlated across native and non-native listeners although the acoustic-phonetic characteristics that correlated with intelligibility varied across the two groups. For both groups, intelligibility was correlated with an amount of energy in the mid-frequency region, but whereas mean word duration was another relevant factor for the natives it was the F2 range in the vowel space for the non-natives. There was also a strong correlation across groups as to the lexical items most often misperceived. Results for two different listening conditions (recognition of isolated words vs triplets) suggested that non-native performance was to a certain extent hampered by increased cognitive load in the triplet condition.

Speech modifications in interactive speech: effects of age, sex and noise type.

When attempting to maintain conversations in noisy communicative settings, talkers typically modify their speech to make themselves understood by the listener. In this study, we investigated the impact of background interference type and talker age on speech adaptations, vocal effort and communicative success. We measured speech acoustics (articulation rate, mid-frequency energy, fundamental frequency), vocal effort (correlation between mid-frequency energy and fundamental frequency) and task completion time in 114 participants aged 8-80 years carrying out an interactive problem-solving task in good and noisy listening conditions (quiet, non-speech noise, background speech). We found greater changes in fundamental frequency and mid-frequency energy in non-speech noise than in background speech and similar reductions in articulation rate in both. However, older participants (50+ years) increased vocal effort in both background interference types, whereas younger children (less than 13 years) increased vocal effort only in background speech. The presence of background interference did not lead to longer task completion times. These results suggest that when the background interference involves a higher cognitive load, as in the case of other speech of other talkers, children and older talkers need to exert more vocal effort to ensure successful communication. We discuss these findings within the communication effort framework. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part II)'.

Acoustic-phonetic characteristics of speech produced with communicative intent to counter adverse listening conditions.

This study investigated whether speech produced in spontaneous interactions when addressing a talker experiencing actual challenging conditions differs in acoustic-phonetic characteristics from speech produced (a) with communicative intent under more ideal conditions and (b) without communicative intent under imaginary challenging conditions (read, clear speech). It also investigated whether acoustic-phonetic modifications made to counteract the effects of a challenging listening condition are tailored to the condition under which communication occurs. Forty talkers were recorded in pairs while engaged in "spot the difference" picture tasks in good and challenging conditions. In the challenging conditions, one talker heard the other (1) via a three-channel noise vocoder (VOC); (2) with simultaneous babble noise (BABBLE). Read, clear speech showed more extreme changes in median F0, F0 range, and speaking rate than speech produced to counter the effects of a challenging listening condition. In the VOC condition, where F0 and intensity enhancements are unlikely to aid intelligibility, talkers did not change their F0 median and range; mean energy and vowel F1 increased less than in the BABBLE condition. This suggests that speech production is listener-focused, and that talkers modulate their speech according to their interlocutors' needs, even when not directly experiencing the challenging listening condition.

DiapixUK: task materials for the elicitation of multiple spontaneous speech dialogs.

The renewed focus of attention on investigating spontaneous speech samples in speech and language research has increased the need for recordings of speech in interactive settings. The DiapixUK task is a new and extended set of picture materials based on the Diapix task by Van Engen et al. (Language and Speech, 53, 510-540, 2010), where two people are recorded while conversing to solve a 'spot the difference' task. The new task materials allow for multiple recordings of the same speaker pairs due to a larger set of picture pairs that have a number of tested features: equal difficulty across all 12 picture pairs, no learning effect of completing more than one picture task and balanced contributions from both speakers. The new materials also provide extra flexibility, making them useful in a wide range of research projects; they are multi-layered electronic images that can be adapted to suit different research needs. This article presents details of the development of the DiapixUK materials, along with data taken from a large corpus of spontaneous speech that are used to demonstrate its new features. Current and potential applications of the task are also discussed.

The effect of native vowel processing ability and frequency discrimination acuity on the phonetic training of English vowels for native speakers of Greek.

The perception and production of nonnative phones in second language (L2) learners can be improved via auditory training, but L2 learning is often characterized by large differences in performance across individuals. This study examined whether success in learning L2 vowels, via five sessions of high-variability phonetic training, related to the learners' native (L1) vowel processing ability or their frequency discrimination acuity. A group of native speakers of Greek received training, while another completed the pre-/post-tests but without training. Pre-/post-tests assessed different aspects of their L2 and L1 vowel processing and frequency acuity. L2 and L1 vowel processing were assessed via: (a) Natural English (L2) vowel identification in quiet and in multi-talker babble, and natural Greek (L1) vowel identification in babble; (b) the categorization of synthetic English and Greek vowel continua; and (c) discrimination of the same continua. Frequency discrimination acuity was assessed for a nonspeech continuum. Frequency discrimination acuity was related to measures of both L1 and L2 vowel processing, a finding that favors an auditory processing over a speech-specific explanation for individual variability in L2 vowel learning. The most efficient frequency discriminators at pre-test were also the most accurate both in English vowel perception and production after training.

Developmental factors and the non-native speaker effect in auditory-visual speech perception.

The study investigated language and developmental factors in the use of visual information in audiovisual speech perception in speakers from different language backgrounds. Mandarin-Chinese and English adults and 8- to 9-year-old children were presented with /ba/, /da/, and /ga/ tokens spoken by two English and two Mandarin-Chinese speakers. A syllable identification task was presented in auditory-only, visual-only, and audiovisual (congruent and incongruent) conditions in clear and in noise. The results showed an increase in the use of visual information in adults relative to children in both the Chinese and English groups. In addition, a positive correlation between the total visual effect and speechreading performance was found, suggesting that the smaller visual influence in the bimodal condition for children might be accounted by their less sophisticated speechreading ability. In regard to the language factor, it was found that Chinese perceivers use visual information in their audiovisual speech processing to the same extent as English perceivers. Finally, there was evidence for a "non-native speaker effect" (i.e., stronger visual effect for non-native speech stimuli), but only for the English participants. Results from the current study suggest that the visual appearance of individual speakers and the acoustic-phonetic properties of specific languages should be considered in future cross-language studies.

The effects of high variability training on voice identity learning.

High variability training has been shown to benefit the learning of new face identities. In three experiments, we investigated whether this is also the case for voice identity learning. In Experiment 1a, we contrasted high variability training sets - which included stimuli extracted from a number of different recording sessions, speaking environments and speaking styles - with low variability stimulus sets that only included a single speaking style (read speech) extracted from one recording session (see Ritchie & Burton, 2017 for faces). Listeners were tested on an old/new recognition task using read sentences (i.e. test materials fully overlapped with the low variability training stimuli) and we found a high variability disadvantage. In Experiment 1b, listeners were trained in a similar way, however, now there was no overlap in speaking style or recording session between training sets and test stimuli. Here, we found a high variability advantage. In Experiment 2, variability was manipulated in terms of the number of unique items as opposed to number of unique speaking styles. Here, we contrasted the high variability training sets used in Experiment 1a with low variability training sets that included the same breadth of styles, but fewer unique items; instead, individual items were repeated (see Murphy, Ipser, Gaigg, & Cook, 2015 for faces). We found only weak evidence for a high variability advantage, which could be explained by stimulus-specific effects. We propose that high variability advantages may be particularly pronounced when listeners are required to generalise from trained stimuli to different-sounding, previously unheard stimuli. We discuss these findings in the context of mechanisms thought to underpin advantages for high variability training.

Listening Effort During Sentence Processing Is Increased for Non-native Listeners: A Pupillometry Study.

Current evidence demonstrates that even though some non-native listeners can achieve native-like performance for speech perception tasks in quiet, the presence of a background noise is much more detrimental to speech intelligibility for non-native compared to native listeners. Even when performance is equated across groups, it is likely that greater listening effort is required for non-native listeners. Importantly, the added listening effort might result in increased fatigue and a reduced ability to successfully perform multiple tasks simultaneously. Task-evoked pupil responses have been demonstrated to be a reliable measure of cognitive effort and can be useful in clarifying those aspects. In this study we compared the pupil response for 23 native English speakers and 27 Italian speakers of English as a second language. Speech intelligibility was tested for sentences presented in quiet and in background noise at two performance levels that were matched across groups. Signal-to-noise levels corresponding to these sentence intelligibility levels were pre-determined using an adaptive intelligibility task. Pupil response was significantly greater in non-native compared to native participants across both intelligibility levels. Therefore, for a given intelligibility level, a greater listening effort is required when listening in a second language in order to understand speech in noise. Results also confirmed that pupil response is sensitive to speech intelligibility during language comprehension, in line with previous research. However, contrary to our predictions, pupil response was not differentially modulated by intelligibility levels for native and non-native listeners. The present study corroborates that pupillometry can be deemed as a valid measure to be used in speech perception investigation, because it is sensitive to differences both across participants, such as listener type, and across conditions, such as variations in the level of speech intelligibility. Importantly, pupillometry offers us the possibility to uncover differences in listening effort even when those do not emerge in the performance level of individuals.

Children's Acoustic and Linguistic Adaptations to Peers With Hearing Impairment.

Purpose: This study aims to examine the clear speaking strategies used by older children when interacting with a peer with hearing loss, focusing on both acoustic and linguistic adaptations in speech. Method: The Grid task, a problem-solving task developed to elicit spontaneous interactive speech, was used to obtain a range of global acoustic and linguistic measures. Eighteen 9- to 14-year-old children with normal hearing (NH) performed the task in pairs, once with a friend with NH and once with a friend with a hearing impairment (HI). Results: In HI-directed speech, children increased their fundamental frequency range and midfrequency intensity, decreased the number of words per phrase, and expanded their vowel space area by increasing F1 and F2 range, relative to NH-directed speech. However, participants did not appear to make changes to their articulation rate, the lexical frequency of content words, or lexical diversity when talking to their friend with HI compared with their friend with NH. Conclusions: Older children show evidence of listener-oriented adaptations to their speech production; although their speech production systems are still developing, they are able to make speech adaptations to benefit the needs of a peer with HI, even without being given a specific instruction to do so. Supplemental Material: https://doi.org/10.23641/asha.6118817.

How do aging and age-related hearing loss affect the ability to communicate effectively in challenging communicative conditions?

This study investigated the relation between the intelligibility of conversational and clear speech produced by older and younger adults and (a) the acoustic profile of their speech (b) communication effectiveness. Speech samples from 30 talkers from the elderLUCID corpus were used: 10 young adults (YA), 10 older adults with normal hearing (OANH) and 10 older adults with presbycusis (OAHL). Samples were extracted from recordings made while participants completed a problem-solving cooperative task (diapix) with a conversational partner who could either hear them easily (NORM) or via a simulated hearing loss (HLS), which led talkers to naturally adopt a clear speaking style. In speech-in-noise listening experiments involving 21 young adult listeners, speech samples by OANH and OAHL were rated and perceived as less intelligible than those of YA talkers. HLS samples were more intelligible than NORM samples, with greater improvements in intelligibility across conditions seen for OA speech. The presence of presbycusis affected (a) the clear speech strategies adopted by OAHL talkers and (b) task effectiveness: OAHL talkers showed some adaptations consistent with an increase in vocal effort, and it took them significantly longer than the YA group to complete the diapix task. The relative energy in the 1-3 kHz frequency region of the long-term average spectrum was the feature that best predicted: (a) the intelligibility of speech samples, and (b) task transaction time in the HLS condition. Overall, our study suggests that spontaneous speech produced by older adults is less intelligible in babble noise, probably due to less energy present in the 1-3 kHz frequency range rich in acoustic cues. Even mild presbycusis in 'healthy aged' adults can affect the dynamic adaptations in speech that are beneficial for effective communication.

Suprasegmental Characteristics of Spontaneous Speech Produced in Good and Challenging Communicative Conditions by Talkers Aged 9-14 Years.

Purpose: This study investigated the acoustic characteristics of spontaneous speech by talkers aged 9-14 years and their ability to adapt these characteristics to maintain effective communication when intelligibility was artificially degraded for their interlocutor. Method: Recordings were made for 96 children (50 female participants, 46 male participants) engaged in a problem-solving task with a same-sex friend; recordings for 20 adults were used as reference. The task was carried out in good listening conditions (normal transmission) and in degraded transmission conditions. Articulation rate, median fundamental frequency (f0), f0 range, and relative energy in the 1- to 3-kHz range were analyzed. Results: With increasing age, children significantly reduced their median f0 and f0 range, became faster talkers, and reduced their mid-frequency energy in spontaneous speech. Children produced similar clear speech adaptations (in degraded transmission conditions) as adults, but only children aged 11-14 years increased their f0 range, an unhelpful strategy not transmitted via the vocoder. Changes made by children were consistent with a general increase in vocal effort. Conclusion: Further developments in speech production take place during later childhood. Children use clear speech strategies to benefit an interlocutor facing intelligibility problems but may not be able to attune these strategies to the same degree as adults.

The effect of talker- and listener-related factors on intelligibility for a real-word, open-set perception test.

The aims of this study were to evaluate whether talker intelligibility is consistent across listeners differing in age and gender and to investigate the process of attunement to talker characteristics in children and adults. Word intelligibility rates were obtained from 135 listeners (adults, 11-12-year-olds, and 7-8-year-olds) for 45 talkers from a homogeneous accent group. There were 2 test conditions, each containing multiple talkers. Both test conditions contained multiple talkers. In the single-word condition, key words were presented in isolation, whereas in the triplet condition, triplets of key words were preceded by a precursor sentence by the same talker. For identical word materials, word intelligibility at a signal-to-noise ratio of +6 dB varied significantly across talkers from 81.2% to 96.4%. Overall, younger listeners made significantly more errors than older children or adults, and women talkers were more intelligible than other classes of talkers. The relative intelligibility of the 45 talkers was highly consistent across listener groups, suggesting that talker intelligibility is primarily determined by talker-related factors rather than by the interrelation of talker- and listener-related factors. The presence of a precursor sentence providing indexical information did improve word intelligibility for the bottom quartile of listeners in each of the listener groups.