The effect of permutations of time samples in the speech waveform on intelligibility.

This paper describes the effect of two types of temporal permutations of the speech waveform on speech intelligibility. Using an overlap-add procedure with triangular-shaped windows for frame lengths of 1/8 to 2048 ms, the temporal order of the speech samples within each frame was subjected to either of two types of permutations: time-reversal or randomization. For both permutations, speech intelligibility tests expectantly show 100% intelligibility for the very short frame lengths containing only a few speech samples. Intelligibility drops to essentially zero toward longer frame lengths of around 1 ms. Interestingly, only for the reverse condition, intelligibility recovers to essentially 100% for frame lengths in the 4-32 ms range, dropping again to zero for frame lengths exceeding about 100 ms. Tests for the Japanese and the English language show essentially similar results. The data are interpreted along the lines of a previous paper by Kazama and the present authors [J. Acoust. Soc. Am. 127(3), 1432-1439 (2010)]. As in that previous paper, the loss of temporal envelope correlation shows a pattern very similar to that of the intelligibility data, illustrating again the importance of the preservation of narrow-band envelopes for speech intelligibility.

Evaluation of the preliminary auditory profile test battery in an international multi-centre study.

OBJECTIVE: This paper describes the composition and international multi-centre evaluation of a battery of tests termed the preliminary auditory profile. It includes measures of loudness perception, listening effort, speech perception, spectral and temporal resolution, spatial hearing, self-reported disability and handicap, and cognition. Clinical applicability and comparability across different centres are investigated. DESIGN: Headphone tests were conducted in five centres divided over four countries. Effects of test-retest, ear, and centre were investigated. Results for normally-hearing (NH) and hearing-impaired (HI) listeners are presented. STUDY SAMPLE: Thirty NH listeners aged 19-39 years, and 72 HI listeners aged 22-91 years with a broad range of hearing losses were included. RESULTS: Test-retest reliability was generally good and there were very few right/left ear effects. Results of all tests were comparable across centres for NH listeners after baseline correction to account for necessary differences between test materials. For HI listeners, results were comparable across centres for the language-independent tests. CONCLUSIONS: The auditory profile forms a clinical test battery that is applicable in four different languages. Even after baseline correction, differences between test materials have to be taken into account when interpreting results of language-dependent tests in HI listeners.

The binaural intelligibility level difference in hearing-impaired listeners: the role of supra-threshold deficits.

Reduced binaural performance of hearing-impaired listeners may not only be caused by raised hearing thresholds (reduced audibility), but also by supra-threshold coding deficits in signal cues. This question was investigated in the present study using binaural intelligibility level difference (BILD) comparisons: the improvement of speech-reception threshold scores for N(0)S(π) relative to N(0)S(0) presentation conditions. Investigated was what types of supra-threshold deficits play a role in reducing BILDs in hearing-impaired subjects. BILDs were investigated for 25 mild to moderate sensorineural hearing-impaired listeners, under conditions where optimal audibility was assured. All stimuli were bandpass filtered (250-4000 Hz). A distortion-sensitivity approach was used to investigate the sensitivity of subjects BILDs to external stimulus perturbations in the phase, frequency, time, and intensity domains. The underlying assumption of this approach was that an auditory coding deficit occurring in a signal cue in a particular domain will result in a low sensitivity to external perturbations applied in that domain. Compared to reference data for listeners with normal BILDs, distortion-sensitivity data for a subgroup of eight listeners with reduced BILDs suggests that these reductions in BILD were caused by coding deficits in the phase and time domains.

On the significance of phase in the short term Fourier spectrum for speech intelligibility.

This paper investigates the significance of the magnitude or the phase in the short term Fourier spectrum for speech intelligibility as a function of the time-window length. For a wide range of window lengths (1/16-2048 ms), two hybrid signals were obtained by a cross-wise combination of the magnitude and phase spectra of speech and white noise. Speech intelligibility data showed the significance of the phase spectrum for longer windows (>256 ms) and for very short windows (<4 ms), and that of the magnitude spectrum for medium-range window lengths. The hybrid signals used in the intelligibility test were analyzed in terms of the preservation of the original narrow-band speech envelopes. Correlations between the narrow-band envelopes of the original speech and the hybrid signals show a similar pattern as a function of window length. This result illustrates the importance of the preservation of narrow-band envelopes for speech intelligibility. The observed significance of the phase spectrum in recovering the narrow-band envelopes for the long term windows and for the very short term windows is discussed.

User evaluation of a communication system that automatically generates captions to improve telephone communication.

This study examined the subjective benefit obtained from automatically generated captions during telephone-speech comprehension in the presence of babble noise. Short stories were presented by telephone either with or without captions that were generated offline by an automatic speech recognition (ASR) system. To simulate online ASR, the word accuracy (WA) level of the captions was 60% or 70% and the text was presented delayed to the speech. After each test, the hearing impaired participants (n = 20) completed the NASA-Task Load Index and several rating scales evaluating the support from the captions. Participants indicated that using the erroneous text in speech comprehension was difficult and the reported task load did not differ between the audio + text and audio-only conditions. In a follow-up experiment (n = 10), the perceived benefit of presenting captions increased with an increase of WA levels to 80% and 90%, and elimination of the text delay. However, in general, the task load did not decrease when captions were presented. These results suggest that the extra effort required to process the text could have been compensated for by less effort required to comprehend the speech. Future research should aim at reducing the complexity of the task to increase the willingness of hearing impaired persons to use an assistive communication system automatically providing captions. The current results underline the need for obtaining both objective and subjective measures of benefit when evaluating assistive communication systems.

The influence of age, hearing, and working memory on the speech comprehension benefit derived from an automatic speech recognition system.

OBJECTIVE: The aim of the current study was to examine whether partly incorrect subtitles that are automatically generated by an Automatic Speech Recognition (ASR) system, improve speech comprehension by listeners with hearing impairment. In an earlier study (Zekveld et al. 2008), we showed that speech comprehension in noise by young listeners with normal hearing improves when presenting partly incorrect, automatically generated subtitles. The current study focused on the effects of age, hearing loss, visual working memory capacity, and linguistic skills on the benefit obtained from automatically generated subtitles during listening to speech in noise. DESIGN: In order to investigate the effects of age and hearing loss, three groups of participants were included: 22 young persons with normal hearing (YNH, mean age = 21 years), 22 middle-aged adults with normal hearing (MA-NH, mean age = 55 years) and 30 middle-aged adults with hearing impairment (MA-HI, mean age = 57 years). The benefit from automatic subtitling was measured by Speech Reception Threshold (SRT) tests (Plomp & Mimpen, 1979). Both unimodal auditory and bimodal audiovisual SRT tests were performed. In the audiovisual tests, the subtitles were presented simultaneously with the speech, whereas in the auditory test, only speech was presented. The difference between the auditory and audiovisual SRT was defined as the audiovisual benefit. Participants additionally rated the listening effort. We examined the influences of ASR accuracy level and text delay on the audiovisual benefit and the listening effort using a repeated measures General Linear Model analysis. In a correlation analysis, we evaluated the relationships between age, auditory SRT, visual working memory capacity and the audiovisual benefit and listening effort. RESULTS: The automatically generated subtitles improved speech comprehension in noise for all ASR accuracies and delays covered by the current study. Higher ASR accuracy levels resulted in more benefit obtained from the subtitles. Speech comprehension improved even for relatively low ASR accuracy levels; for example, participants obtained about 2 dB SNR audiovisual benefit for ASR accuracies around 74%. Delaying the presentation of the text reduced the benefit and increased the listening effort. Participants with relatively low unimodal speech comprehension obtained greater benefit from the subtitles than participants with better unimodal speech comprehension. We observed an age-related decline in the working-memory capacity of the listeners with normal hearing. A higher age and a lower working memory capacity were associated with increased effort required to use the subtitles to improve speech comprehension. CONCLUSIONS: Participants were able to use partly incorrect and delayed subtitles to increase their comprehension of speech in noise, regardless of age and hearing loss. This supports the further development and evaluation of an assistive listening system that displays automatically recognized speech to aid speech comprehension by listeners with hearing impairment.

Measuring cognitive factors in speech comprehension: the value of using the Text Reception Threshold test as a visual equivalent of the SRT test.

The ability to comprehend speech in noise is influenced by bottom-up auditory and top-down cognitive capacities. Separate examination of these capacities is relevant for various purposes. Speech-Reception-Threshold (SRT) tests measure an individual's ability to comprehend speech. This paper addresses the value of the Text-Reception-Threshold (TRT) test (a visual parallel of the SRT test) to assess the cognitive capacities allocated during speech comprehension. We conducted a secondary data analysis, including 87 normally-hearing adults (aged 18 to 78 years). Correlation coefficients between age, TRT, working memory (Spatial Span) and SRT were examined. The TRT and SRT correlated significantly (r = 0.30), supporting the value of TRT in explaining inter-individual differences in SRTs. The relations between age and TRT and between SSP and TRT were non-significant. The results indicate that the current TRT test does not fully cover the cognitive aspects relevant in speech comprehension. Adaptation of the test is required before clinical implementation can be considered.

The benefit obtained from visually displayed text from an automatic speech recognizer during listening to speech presented in noise.

OBJECTIVES: The aim of this study was to evaluate the benefit that listeners obtain from visually presented output from an automatic speech recognition (ASR) system during listening to speech in noise. DESIGN: Auditory-alone and audiovisual speech reception thresholds (SRTs) were measured. The SRT is defined as the speech-to-noise ratio at which 50% of the test sentences are reproduced correctly. In the auditory-alone SRT tests, the test sentences were presented only auditorily; in the audiovisual SRT test, the ASR output of each test sentence was also presented textually. The ASR system was used in two recognition modes: recognition of spoken words (word output), or recognition of speech sounds or phones (phone output). The benefit obtained from the ASR output was defined as the difference between the auditory-alone and the audiovisual SRT. We also examined the readability of unimodally displayed ASR output (i.e., the percentage of sentences in which ASR errors were identified and accurately corrected). In experiment 1, the readability and benefit obtained from ASR word output (n = 14) was compared with the benefit obtained from ASR phone output (n = 10). In experiment 2, the effect of presenting an indication of the ASR confidence level was examined (n = 14). The effect of delaying the presentation of the text relative to the speech (up to 6 sec) was examined in experiment 3 (n = 24). The ASR accuracy level was varied systematically in each experiment. RESULTS: Mean readability scores ranged from 0 to 46%, depending on ASR accuracy. Speech comprehension improved when the ASR output was displayed. For example, when the ASR output corresponded to readability scores of only about 20% correct, the text improved the SRT by about 3 dB SNR in the audiovisual SRT test. This improvement corresponds to an increase in speech comprehension of about 35% in critical conditions. Equally readable phone and word output provides similar benefit in speech comprehension. For equal ASR accuracies, both the readability and the benefit from the word output generally exceeded the benefits from the phone output. Presenting information about the ASR confidence level did not influence either the readability or the benefit obtained from the word output. Delaying the text relative to the speech moderately decreased the benefit. CONCLUSIONS: The present study indicates that speech comprehension improves considerably by textual ASR output with moderate accuracies. The study shows that this improvement depends on the readability of the ASR output. Word output has better accuracy and readability than phone output. Listeners are therefore better able to use the ASR word output than phone output to improve speech comprehension. The ability of older listeners and listeners with hearing impairments to use ASR output in speech comprehension requires further study.

The concept of signal-to-noise ratio in the modulation domain and speech intelligibility.

A new concept is proposed that relates to intelligibility of speech in noise. The concept combines traditional estimations of signal-to-noise ratios (S/N) with elements from the modulation transfer function model, which results in the definition of the signal-to-noise ratio in the modulation domain: the (SN)(mod). It is argued that this (SN)(mod), quantifying the strength of speech modulations relative to a floor of spurious modulations arising from the speech-noise interaction, is the key factor in relation to speech intelligibility. It is shown that, by using a specific test signal, the strength of these spurious modulations can be measured, allowing an estimation of the (SN)(mod) for various conditions of additive noise, noise suppression, and amplitude compression. By relating these results to intelligibility data for these same conditions, the relevance of the (SN)(mod) as the key factor underlying speech intelligibility is clearly illustrated. For instance, it is shown that the commonly observed limited effect of noise suppression on speech intelligibility is correctly "predicted" by the (SN)(mod), whereas traditional measures such as the speech transmission index, considering only the changes in the speech modulations, fall short in this respect. It is argued that (SN)(mod) may provide a relevant tool in the design of successful noise-suppression systems.

The combined effects of reverberation and nonstationary noise on sentence intelligibility.

Listening conditions in everyday life typically include a combination of reverberation and nonstationary background noise. It is well known that sentence intelligibility is adversely affected by these factors. To assess their combined effects, an approach is introduced which combines two methods of predicting speech intelligibility, the extended speech intelligibility index (ESII) and the speech transmission index. First, the effects of reverberation on nonstationary noise (i.e., reduction of masker modulations) and on speech modulations are evaluated separately. Subsequently, the ESII is applied to predict the speech reception threshold (SRT) in the masker with reduced modulations. To validate this approach, SRTs were measured for ten normal-hearing listeners, in various combinations of nonstationary noise and artificially created reverberation. After taking the characteristics of the speech corpus into account, results show that the approach accurately predicts SRTs in nonstationary noise and reverberation for normal-hearing listeners. Furthermore, it is shown that, when reverberation is present, the benefit from masker fluctuations may be substantially reduced.

On the auditory and cognitive functions that may explain an individual's elevation of the speech reception threshold in noise.

It is generally recognized that poor results of speech-in-noise tests by hearing-impaired persons cannot be fully explained by the elevated pure-tone hearing threshold. Plomp has shown, among others, that an additional factor has to be taken into account, often referred to in general terms as distortion. In an attempt to specify auditory and cognitive functions which may underlie this distortion, various studies on this topic originating from Plomp's research group are reviewed, as well as other relevant studies which provide quantitative data on the correlations between various types of auditory or cognitive tests (the predictor tests) and speech-in-noise tests. The predictor variables considered include, besides the pure-tone audiogram, measures of spectral and temporal resolution, intensity difference limen, age, and some cognitive aspects. The results indicate that, by and large, these variables fall short in fully explaining the variance observed in the speech-in-noise tests. This strongly suggests that the predictor variables considered so far do not cover all sources of variance relevant for speech reception in noise.

Modulation detection interference for asynchronous presentation of masker and target in listeners with normal and impaired hearing.

PURPOSE: The sensitivity to sinusoidal amplitude modulations (SAMs) is reduced when other modulated maskers are presented simultaneously at a distant frequency (also referred to as modulation detection interference [MDI]). This article describes the results of onset differences between masker and target as a parameter. METHOD: Carrier frequencies were 1 kHz (target: 625 ms, 8 Hz SAM) and 2 kHz (masker: 625 ms, 8 Hz SAM; modulation depth = 1) presented at 25 dB SL for listeners with impaired hearing (n = 8) and at 25 dB SL and 50 dB SL for listeners with normal hearing (n = 6). Masker was delayed by 0, 125, 250, 500, 625, or 750 ms relative to the target. RESULTS: Sensitivity to SAMs was reduced in both groups by a modulated masker simultaneous presentation. Reducing the temporal overlap (i.e., increasing the onset delay between masker and target) increased the sensitivity to SAMs in the presence of modulated maskers. CONCLUSION: The gradual reduction in MDI with increasing asynchrony between masker and target suggests that MDI is not solely related to perceptual grouping. Reduced sensitivity to SAMs due to prior stimulation with SAM stimuli (forward masking), and deficits in across-channel integration, are other factors that may play a role.

A detailed study on the effects of noise on speech intelligibility.

A wavelet representation of speech was used to display the instantaneous amplitude and phase within 14 octave frequency bands, representing the envelope and the carrier within each band. Adding stationary noise alters the wavelet pattern, which can be understood as a combination of three simultaneously occurring subeffects: two effects on the wavelet levels (one systematic and one stochastic) and one effect on the wavelet phases. Specific types of signal processing were applied to speech, which allowed each effect to be either included or excluded. The impact of each effect (and of combinations) on speech intelligibility was measured with CVC's. It appeared that the systematic level effect (i.e., the increase of each speech wavelet intensity with the mean noise intensity) has the most degrading effect on speech intelligibility, which is in accordance with measures such as the modulation transfer function and the speech transmission index. However, also the introduction of stochastic level fluctuations and disturbance of the carrier phase seriously contribute to reduced intelligibility in noise. It is argued that these stochastic effects are responsible for the limited success of spectral subtraction as a means to improve speech intelligibility. Results can provide clues for effective noise suppression with respect to intelligibility.

The development of the text reception threshold test: a visual analogue of the speech reception threshold test.

PURPOSE: In this study, the authors aimed to develop a visual analogue of the widely used Speech Reception Threshold (SRT; R. Plomp & A. M. Mimpen, 1979b) test. The Text Reception Threshold (TRT) test, in which visually presented sentences are masked by a bar pattern, enables the quantification of modality-aspecific variance in speech-in-noise comprehension to obtain more insight into interindividual differences in this ability. METHOD: Using an adaptive procedure similar to the SRT test, the TRT test determines the percentage of unmasked text needed to read 50% of sentences correctly. SRTs in stationary noise (SRT(STAT)), modulated noise (SRT(MOD)), and TRTs were determined for 34 participants with normal hearing, aged 19 to 78 years. RESULTS: The results indicate that about 30% of the variance in SRT(STAT) and SRT(MOD) is shared with variance in TRT, which reflects the shared involvement of a modality-aspecific cognitive or linguistic ability in forming meaningful wholes of fragments of sentences. CONCLUSION: The TRT test, a visual analogue of the SRT test, has been developed to measure the variance in speech-in-noise comprehension associated with modality-aspecific cognitive skills. In future research, normative data of the TRT test should be developed. It would also be interesting to measure TRTs of individuals experiencing difficulties understanding speech.

The precedence effect for lateralization for the mild sensory neural hearing impaired.

Using dichotic signals presented by headphone, stimulus onset dominance (the precedence effect for lateralization) was investigated for six sensorineural hearing-impaired subjects. Stimuli were based on 2400-Hz lowpass-filtered 5-ms noise bursts. We used the paradigm, as described by Goverts et al. (2000, Hear. Res. 148, 88-94), in which a single noise burst is divided into leading and lagging parts, with opposite lateralization cues (viz. an interaural time delay of 0.2 ms). The occurrence of onset dominance was investigated by measuring the lateral perception of the stimulus ('left' or 'right') with fixed, equal durations of leading and lagging parts, while decreasing the absolute signal level or adding a filtered white noise. The dominance of the leading part was quantified by measuring the lateral perception of the stimulus as a function of the relative duration of the leading (and thus the lagging) part. This was done at about 40 dB sensation level (SL) in quiet and in filtered white noise, at a signal-to-noise ratio resulting in a SL of about 6 dB. Results are compared to normal hearing reference data at various SLs. Hearing-impaired data show a large variance and overall a decreased precedence effect in terms of both occurrence and quantification, which cannot be explained on basis of reduced audibility. Mean performance of the hearing-impaired subjects at 40 dB in quiet was similar to normal hearing performance in masking noise at a signal-to-noise ratio of 0 dB.

Cognitive abilities relate to self-reported hearing disability.

PURPOSE: In this explorative study, the authors investigated the relationship between auditory and cognitive abilities and self-reported hearing disability. METHOD: Thirty-two adults with mild to moderate hearing loss completed the Amsterdam Inventory for Auditory Disability and Handicap (AIADH; Kramer, Kapteyn, Festen, & Tobi, 1996) and performed the Text Reception Threshold (TRT; Zekveld, George, Kramer, Goverts, & Houtgast, 2007) test as well as tests of spatial working memory (SWM) and visual sustained attention. Regression analyses examined the predictive value of age, hearing thresholds (pure-tone averages [PTAs]), speech perception in noise (speech reception thresholds in noise [SRTNs]), and the cognitive tests for the 5 AIADH factors. RESULTS: Besides the variance explained by age, PTA, and SRTN, cognitive abilities were related to each hearing factor. The reported difficulties with sound detection and speech perception in quiet were less severe for participants with higher age, lower PTAs, and better TRTs. Fewer sound localization and speech perception in noise problems were reported by participants with better SRTNs and smaller SWM. Fewer sound discrimination difficulties were reported by subjects with better SRTNs and TRTs and smaller SWM. CONCLUSIONS: The results suggest a general role of the ability to read partly masked text in subjective hearing. Large working memory was associated with more reported hearing difficulties. This study shows that besides auditory variables and age, cognitive abilities are related to self-reported hearing disability.

On the use of the distortion-sensitivity approach in examining the role of linguistic abilities in speech understanding in noise.

PURPOSE: Researchers have used the distortion-sensitivity approach in the psychoacoustical domain to investigate the role of auditory processing abilities in speech perception in noise (van Schijndel, Houtgast, & Festen, 2001; Goverts & Houtgast, 2010). In this study, the authors examined the potential applicability of the distortion-sensitivity approach for investigating the role of linguistic abilities in speech understanding in noise. METHOD: The authors applied the distortion-sensitivity approach by measuring the processing of visually presented masked text in a condition with manipulated syntactic, lexical, and semantic cues and while using the Text Reception Threshold (George et al., 2007; Kramer, Zekveld, & Houtgast, 2009; Zekveld, George, Kramer, Goverts, & Houtgast, 2007) method. Two groups that differed in linguistic abilities were studied: 13 native and 10 non-native speakers of Dutch, all typically hearing university students. RESULTS: As expected, the non-native subjects showed substantially reduced performance. The results of the distortion-sensitivity approach yielded differentiated results on the use of specific linguistic cues in the 2 groups. CONCLUSION: The results show the potential value of the distortion-sensitivity approach in studying the role of linguistic abilities in speech understanding in noise of individuals with hearing impairment.

Measuring the effects of reverberation and noise on sentence intelligibility for hearing-impaired listeners.

PURPOSE: The Speech Transmission Index (STI; Houtgast, Steeneken, & Plomp, 1980; Steeneken & Houtgast, 1980) is commonly used to quantify the adverse effects of reverberation and stationary noise on speech intelligibility for normal-hearing listeners. Duquesnoy and Plomp (1980) showed that the STI can be applied for presbycusic listeners, relating speech reception thresholds (SRTs) in various reverberant conditions to a fixed, subject-dependent STI value. The current study aims at extending their results to a wider range of hearing-impaired listeners. METHOD: A reverberant analogue of the SRT is presented--the speech reception reverberation threshold (SRRT)--which determines the amount of reverberation that a listener can sustain to understand 50% of the presented sentences. SRTs are performed and evaluated in terms of STI for 5 normal-hearing participants and 36 randomly selected hearing-impaired participants. RESULTS: Results show that differences in STI between reverberant and noisy conditions are only small, equivalent to a change in speech-to-noise ratio < 1.3 dB. CONCLUSION: The STI appears to be a convenient, single number to quantify speech reception of hearing-impaired listeners in noise and/or reverberation, regardless of the nature of the hearing loss. In future research, the SRRT may be applied to further investigate the supposed importance of cognitive processing in reverberant listening conditions.

Audiovisual perception of speech in noise and masked written text.

OBJECTIVE: The aim of this study was to examine the support obtained from degraded visual information in the comprehension of speech in noise. DESIGN: We presented sentences auditorily (speech reception threshold test), visually (text reception threshold test), and audiovisually. Presenting speech in noise and masked written text enabled the quantification and systematic variation of the amount of information presented in both modalities. Eighteen persons with normal hearing (aged 19 to 31 yr) participated. For half of them a bar pattern masked the text and for the other half random dots masked the text. The text was presented simultaneously or delayed relative to the speech. Using an adaptive procedure, the amount of information required for a correct reproduction of 50% of the sentences was determined for both the unimodal and the audiovisual stimuli. Bimodal support was defined as the difference between the observed bimodal performance and that predicted by an independent channels model. Nonparametric tests were used to evaluate the bimodal support and the effect of delaying the text. RESULTS: Masked text substantially supported the comprehension of speech in noise; the bimodal support ranged from 15% to 25% correct. A negative effect of delaying the text was observed in some conditions for the participants who were presented the text masked by the bar pattern. CONCLUSIONS: The ability of participants to reproduce bimodally presented sentences exceeds the performance as predicted by an independent channels model. This indicates that a relatively small amount of visual information can substantially augment speech comprehension in noise, which supports the use of visual information to improve speech comprehension by participants with hearing impairment, even if the visual information is incomplete.