Objective intelligibility measurement of reverberant vocoded speech for normal-hearing listeners: Towards facilitating the development of speech enhancement algorithms for cochlear implants.

Cochlear implant (CI) recipients often struggle to understand speech in reverberant environments. Speech enhancement algorithms could restore speech perception for CI listeners by removing reverberant artifacts from the CI stimulation pattern. Listening studies, either with cochlear-implant recipients or normal-hearing (NH) listeners using a CI acoustic model, provide a benchmark for speech intelligibility improvements conferred by the enhancement algorithm but are costly and time consuming. To reduce the associated costs during algorithm development, speech intelligibility could be estimated offline using objective intelligibility measures. Previous evaluations of objective measures that considered CIs primarily assessed the combined impact of noise and reverberation and employed highly accurate enhancement algorithms. To facilitate the development of enhancement algorithms, we evaluate twelve objective measures in reverberant-only conditions characterized by a gradual reduction of reverberant artifacts, simulating the performance of an enhancement algorithm during development. Measures are validated against the performance of NH listeners using a CI acoustic model. To enhance compatibility with reverberant CI-processed signals, measure performance was assessed after modifying the reference signal and spectral filterbank. Measures leveraging the speech-to-reverberant ratio, cepstral distance and, after modifying the reference or filterbank, envelope correlation are strong predictors of intelligibility for reverberant CI-processed speech.

The effects of reverberant self- and overlap-masking on speech recognition in cochlear implant listeners.

Many cochlear implant (CI) listeners experience decreased speech recognition in reverberant environments [Kokkinakis et al., J. Acoust. Soc. Am. 129(5), 3221-3232 (2011)], which may be caused by a combination of self- and overlap-masking [Bolt and MacDonald, J. Acoust. Soc. Am. 21(6), 577-580 (1949)]. Determining the extent to which these effects decrease speech recognition for CI listeners may influence reverberation mitigation algorithms. This study compared speech recognition with ideal self-masking mitigation, with ideal overlap-masking mitigation, and with no mitigation. Under these conditions, mitigating either self- or overlap-masking resulted in significant improvements in speech recognition for both normal hearing subjects utilizing an acoustic model and for CI listeners using their own devices.

Using channel-specific statistical models to detect reverberation in cochlear implant stimuli.

Reverberation is especially detrimental for cochlear implant listeners; thus, mitigating its effects has the potential to provide significant improvements to cochlear implant communication. Efforts to model and correct for reverberation in acoustic listening scenarios can be quite complex, requiring estimation of the room transfer function and localization of the source and receiver. However, due to the limited resolution associated with cochlear implant stimulation, simpler processing for reverberation detection and mitigation may be possible for cochlear implants. This study models speech stimuli in a cochlear implant on a per-channel basis both in quiet and in reverberation, and assesses the efficacy of these models for detecting the presence of reverberation. This study was able to successfully detect reverberation in cochlear implant pulse trains, and the results appear to be robust to varying room conditions and cochlear implant stimulation parameters. Reverberant signals were detected 100% of the time for a long reverberation time of 1.2 s and 86% of the time for a shorter reverberation time of 0.5 s.

Stream segregation on a single electrode as a function of pulse rate in cochlear implant listeners.

While cochlear implants (CIs) usually provide high levels of speech recognition in quiet, speech recognition in noise remains challenging. To overcome these difficulties, it is important to understand how implanted listeners separate a target signal from interferers. Stream segregation has been studied extensively in both normal and electric hearing, as a function of place of stimulation. However, the effects of pulse rate, independent of place, on the perceptual grouping of sequential sounds in electric hearing have not yet been investigated. A rhythm detection task was used to measure stream segregation. The results of this study suggest that while CI listeners can segregate streams based on differences in pulse rate alone, the amount of stream segregation observed decreases as the base pulse rate increases. Further investigation of the perceptual dimensions encoded by the pulse rate and the effect of sequential presentation of different stimulation rates on perception could be beneficial for the future development of speech processing strategies for CIs.

Parameter tuning of time-frequency masking algorithms for reverberant artifact removal within the cochlear implant stimulus.

Cochlear implant recipients struggle to understand speech in reverberant environments. To restore speech perception, artifacts due to reverberant reflections can be removed from the cochlear implant stimulus by applying a matrix of gain values, a technique referred to as time-frequency masking. In this study, two common time-frequency masking strategies are implemented within cochlear implant processing, either introducing complete retention or deletion of stimulus components using a binary mask or continuous attenuation of stimulus components using a ratio mask. Parameters of each masking strategy control the level of attenuation imposed by the gain values. In this study, we perceptually tune the parameters of the masking strategy to determine a balance between speech retention and artifact removal. We measure the intelligibility of reverberant signals mitigated by each strategy with speech recognition testing in normal-hearing listeners using vocoding as a simulation of cochlear implant perception. For both masking strategies, we find parameterizations that maximize the intelligibility of the mitigated signals. At the best-performing parameterizations, binary-masked reverberant signals yield larger intelligibility improvements than ratio-masked signals. The results provide a perceptually optimized objective for the removal of reverberant artifacts from cochlear implant stimuli, facilitating improved speech recognition performance for cochlear implant recipients in reverberant environments.

Language Model-Guided Classifier Adaptation for Brain-Computer Interfaces for Communication.

Brain-computer interfaces (BCIs), such as the P300 speller, can provide a means of communication for individuals with severe neuromuscular limitations. BCIs interpret electroencephalography (EEG) signals in order to translate embedded information about a user's intent into executable commands to control external devices. However, EEG signals are inherently noisy and nonstationary, posing a challenge to extended BCI use. Conventionally, a BCI classifier is trained via supervised learning in an offline calibration session; once trained, the classifier is deployed for online use and is not updated. As the statistics of a user's EEG data change over time, the performance of a static classifier may decline with extended use. It is therefore desirable to automatically adapt the classifier to current data statistics without requiring offline recalibration. In an existing semi-supervised learning approach, the classifier is trained on labeled EEG data and is then updated using incoming unlabeled EEG data and classifier-predicted labels. To reduce the risk of learning from incorrect predictions, a threshold is imposed to exclude unlabeled data with low-confidence label predictions from the expanded training set when retraining the adaptive classifier. In this work, we propose the use of a language model for spelling error correction and disambiguation to provide information about label correctness during semi-supervised learning. Results from simulations with multi-session P300 speller user EEG data demonstrate that our language-guided semi-supervised approach significantly improves spelling accuracy relative to conventional BCI calibration and threshold-based semi-supervised learning.

Mitigating the Impact of Psychophysical Effects During Adaptive Stimulus Selection in the P300 Speller Brain-Computer Interface.

Stimulus-driven brain-computer interfaces (BCIs), such as the P300 speller, rely on using sensory stimuli to elicit specific neural signal components called event-related potentials (ERPs) to control external devices. However, psychophysical factors, such as refractory effects and adjacency distractions, may negatively impact ERP elicitation and BCI performance. Although conventional BCI stimulus presentation paradigms usually design stimulus presentation schedules in a pseudo-random manner, recent studies have shown that controlling the stimulus selection process can enhance ERP elicitation. In prior work, we developed an algorithm to adaptively select BCI stimuli using an objective criterion that maximizes the amount of information about the user's intent that can be elicited with the presented stimuli given current data conditions. Here, we enhance this adaptive BCI stimulus selection algorithm to mitigate adjacency distractions and refractory effects by modeling temporal dependencies of ERP elicitation in the objective function and imposing spatial restrictions in the stimulus search space. Results from simulations using synthetic data and human data from a BCI study show that the enhanced adaptive stimulus selection algorithm can improve spelling speeds relative to conventional BCI stimulus presentation paradigms.Clinical relevance-Increased communication rates with our enhanced adaptive stimulus selection algorithm can potentially facilitate the translation of BCIs as viable communication alternatives for individuals with severe neuromuscular limitations.

Novel Acoustic Biomarker of Quality of Life in Left Ventricular Assist Device Recipients.

Background Although technological advances to pump design have improved survival, left ventricular assist device (LVAD) recipients experience variable improvements in quality of life. Methods for optimizing LVAD support to improve quality of life are needed. We investigated whether acoustic signatures obtained from digital stethoscopes can predict patient-centered outcomes in LVAD recipients. Methods and Results We followed precordial sounds over 6 months in 24 LVAD recipients (8 HeartWare HVAD™, 16 HeartMate 3 [HM3]). Subjects recorded their precordial sounds with a digital stethoscope and completed a Kansas City Cardiomyopathy Questionnaire weekly. We developed a novel algorithm to filter LVAD sounds from recordings. Unsupervised clustering of LVAD-mitigated sounds revealed distinct groups of acoustic features. Of 16 HM3 recipients, 6 (38%) had a unique acoustic feature that we have termed the pulse synchronized sound based on its temporal association with the artificial pulse of the HM3. HM3 recipients with the pulse synchronized sound had significantly better Kansas City Cardiomyopathy Questionnaire scores at baseline (median, 89.1 [interquartile range, 86.2-90.4] versus 66.1 [interquartile range, 31.1-73.7]; P=0.03) and over the 6-month study period (marginal mean, 77.6 [95% CI, 66.3-88.9] versus 59.9 [95% CI, 47.9-70.0]; P<0.001). Mechanistically, the pulse synchronized sound shares acoustic features with patient-derived intrinsic sounds. Finally, we developed a machine learning algorithm to automatically detect the pulse synchronized sound within precordial sounds (area under the curve, 0.95, leave-one-subject-out cross-validation). Conclusions We have identified a novel acoustic biomarker associated with better quality of life in HM3 LVAD recipients, which may provide a method for assaying optimized LVAD support.

Heart Sound Analysis in Individuals Supported With Left Ventricular Assist Devices.

OBJECTIVE: LVADs are surgically implanted mechanical pumps that improve survival rates of individuals with advanced heart failure. LVAD therapy is associated with high morbidity, which can be partially attributed to challenges with detecting LVAD complications before adverse events occur. Current methods used to monitor for complications with LVAD support require frequent clinical assessments at specialized LVAD centers. Analysis of recorded precordial sounds may enable real-time, remote monitoring of device and cardiac function for early detection of LVAD complications. The dominance of LVAD sounds in the precordium limits the utility of routine cardiac auscultation of LVAD recipients. In this work, we develop a signal processing pipeline to mitigate sounds generated by the LVAD. METHODS: We collected in vivo precordial sounds from 17 LVAD recipients, and contemporaneous echocardiograms from 12 of these individuals, to validate heart valve closure timings. RESULTS: We characterized various acoustic signatures of heart sounds extracted from in vivo recordings, and report preliminary findings linking fundamental heart sound characteristics and level of LVAD support. CONCLUSION: Mitigation of LVAD sounds from precordial sound recordings of LVAD recipients enables analysis of intrinsic heart sounds. SIGNIFICANCE: These findings provide proof-of-concept evidence of the clinical utility of heart sound analysis for bedside and remote monitoring of LVAD recipients.

Investigating the effects of stimulus duration and context on pitch perception by cochlear implant users.

Cochlear implant sound processing strategies that use time-varying pulse rates to transmit fine structure information are one proposed method for improving the spectral representation of a sound with the eventual goal of improving speech recognition in noisy conditions, speech recognition in tonal languages, and music identification and appreciation. However, many of the perceptual phenomena associated with time-varying rates are not well understood. In this study, the effects of stimulus duration on both the place and rate-pitch percepts were investigated via psychophysical experiments. Four Nucleus CI24 cochlear implant users participated in these experiments, which included a short-duration pitch ranking task and three adaptive pulse rate discrimination tasks. When duration was fixed from trial-to-trial and rate was varied adaptively, results suggested that both the place-pitch and rate-pitch percepts may be independent of duration for durations above 10 and 20 ms, respectively. When duration was varied and pulse rates were fixed, performance was highly variable within and across subjects. Implications for multi-rate sound processing strategies are discussed.

Mandarin Chinese tone identification in cochlear implants: predictions from acoustic models.

It has been established that current cochlear implants do not supply adequate spectral information for perception of tonal languages. Comprehension of a tonal language, such as Mandarin Chinese, requires recognition of lexical tones. New strategies of cochlear stimulation such as variable stimulation rate and current steering may provide the means of delivering more spectral information and thus may provide the auditory fine-structure required for tone recognition. Several cochlear implant signal processing strategies are examined in this study, the continuous interleaved sampling (CIS) algorithm, the frequency amplitude modulation encoding (FAME) algorithm, and the multiple carrier frequency algorithm (MCFA). These strategies provide different types and amounts of spectral information. Pattern recognition techniques can be applied to data from Mandarin Chinese tone recognition tasks using acoustic models as a means of testing the abilities of these algorithms to transmit the changes in fundamental frequency indicative of the four lexical tones. The ability of processed Mandarin Chinese tones to be correctly classified may predict trends in the effectiveness of different signal processing algorithms in cochlear implants. The proposed techniques can predict trends in performance of the signal processing techniques in quiet conditions but fail to do so in noise.

Comparison of adaptive psychometric procedures motivated by the theory of optimal experiments: simulated and experimental results.

The wide use of psychometric assessments and the time necessary to conduct comprehensive psychometric tests has motivated significant research into the development of psychometric testing procedures that will provide accurate and efficient estimates of the parameters of interest. One potential framework for developing adaptive psychometric procedures is the Theory of Optimal Experiments. The Theory of Optimal Experiments provides several metrics for determining informative stimulus values based on a model of the psychometric function to be provided by the investigator. In this study, two methods based on a previous implementation of the Theory of Optimal Experiments are presented for comparison to two fixed step size staircase methods and also an existing adaptive method that utilizes a Bayesian framework. The psychometric procedures were used to measure detection thresholds and discrimination limens on two separate psychoacoustic tasks with normal-hearing subjects. Computer simulations were performed based on the outcomes of the experimental psychoacoustic detection task to analyze performance over a large sample size in the case of known truth. Results suggest that the proposed stimulus selection rules motivated by the Theory of Optimal Experiments perform better than previous techniques and also extend estimation to multiple parameters.

Assessing the pitch structure associated with multiple rates and places for cochlear implant users.

Cochlear implant subjects continue to experience difficulty understanding speech in noise and performing pitch-based musical tasks. Acoustic model studies have suggested that transmitting additional fine structure via multiple stimulation rates is a potential mechanism for addressing these issues [Nie et al., IEEE Trans. Biomed. Eng. 52, 64-73 (2005); Throckmorton et al., Hear. Res. 218, 30-42 (2006)]; however, results from preliminary cochlear implant studies have been less compelling. Multirate speech processing algorithms previously assumed a place-dependent pitch structure in that a basal electrode would always elicit a higher pitch percept than an apical electrode, independent of stimulation rate. Some subjective evidence contradicts this assumption [H. J. McDermott and C. M. McKay, J. Acoust. Soc. Am. 101, 1622-1630 (1997); R. V. Shannon, Hear. Res. 11, 157-189 (1983)]. The purpose of this study is to test the hypothesis that the introduction of multiple rates may invalidate the tonotopic pitch structure resulting from place-pitch alone. The SPEAR3 developmental speech processor was used to collect psychophysical data from five cochlear implant users to assess the tonotopic structure for stimuli presented at two rates on all active electrodes. Pitch ranking data indicated many cases where pitch percepts overlapped across electrodes and rates. Thus, the results from this study suggest that pitch-based tuning across rate and electrode may be necessary to optimize performance of a multirate sound processing strategy in cochlear implant subjects.

Application of a Graphical Model to Investigate the Utility of Cross-channel Information for Mitigating Reverberation in Cochlear Implants.

Individuals with cochlear implants (CIs) experience more difficulty understanding speech in reverberant environ-ments than normal hearing listeners. As a result, recent research has targeted mitigating the effects of late reverberant signal reflections in CIs by using a machine learning approach to detect and delete affected segments in the CI stimulus pattern. Previous work has trained electrode-specific classification models to mitigate late reverberant signal reflections based on features extracted from only the acoustic activity within the electrode of interest. Since adjacent CI electrodes tend to be activated concurrently during speech, we hypothesized that incorporating additional information from the other electrode channels, termed cross-channel information, as features could improve classification performance. Cross-channel information extracted in real-world conditions will likely contain errors that will impact classification performance. To simulate extracting cross-channel information in realistic conditions, we developed a graphical model based on the Ising model to systematically introduce errors to specific types of cross-channel information. The Ising-like model allows us to add errors while maintaining the important geometric information contained in cross-channel information, which is due to the spectro-temporal structure of speech. Results suggest the potential utility of leveraging cross-channel information to improve the performance of the reverberation mitigation algorithm from the baseline channel-based features, even when the cross-channel information contains errors.

USING MACHINE LEARNING TO MITIGATE THE EFFECTS OF REVERBERATION AND NOISE IN COCHLEAR IMPLANTS.

In listening environments with room reverberation and background noise, cochlear implant (CI) users experience substantial difficulties in understanding speech. Because everyday environments have different combinations of reverberation and noise, there is a need to develop algorithms that can mitigate both effects to improve speech intelligibility. Desmond et al. (2014) developed a machine learning approach to mitigate the adverse effects of late reverberant reflections of speech signals by using a classifier to detect and remove affected segments in CI pulse trains. This study aimed to investigate the robustness of the reverberation mitigation algorithm in environments with both reverberation and noise. Sentence recognition tests were conducted in normal hearing listeners using vocoded speech with unmitigated and mitigated reverberant-only or noisy reverberant speech signals, across different reverberation times and noise types. Improvements in speech intelligibility were observed in mitigated reverberant-only conditions. However, mixed results were obtained in the mitigated noisy reverberant conditions as a reduction in speech intelligibility was observed for noise types whose spectra were similar to that of anechoic speech. Based on these results, the focus of future work is to develop a context-dependent approach that activates different mitigation strategies for different acoustic environments.

Evaluating Brain-Computer Interface Performance in an ALS Population: Checkerboard and Color Paradigms.

The objective of this study was to investigate the performance of 3 brain-computer interface (BCI) paradigms in an amyotrophic lateral sclerosis (ALS) population (n = 11). Using a repeated-measures design, participants completed 3 BCI conditions: row/column (RCW), checkerboard (CBW), and gray-to-color (CBC). Based on previous studies, it is hypothesized that the CBC and CBW conditions will result in higher accuracy, information transfer rate, waveform amplitude, and user preference over the RCW condition. An offline dynamic stopping simulation will also increase information transfer rate. Higher mean accuracy was observed in the CBC condition (89.7%), followed by the CBW (84.3%) condition, and lowest in the RCW condition (78.7%); however, these differences did not reach statistical significance ( P = .062). Eight of the eleven participants preferred the CBC and the remaining three preferred the CBW conditions. The offline dynamic stopping simulation significantly increased information transfer rate ( P = .005) and decreased accuracy ( P < .000). The findings of this study suggest that color stimuli provide a modest improvement in performance and that participants prefer color stimuli over monochromatic stimuli. Given these findings, BCI paradigms that use color stimuli should be considered for individuals who have ALS.

Predictions of psychophysical measurements for sinusoidal amplitude modulated (SAM) pulse-train stimuli from a stochastic model.

Two approaches have been proposed to reduce the synchrony of the neural response to electrical stimuli in cochlear implants. One approach involves adding noise to the pulse-train stimulus, and the other is based on using a high-rate pulse-train carrier. Hypotheses regarding the efficacy of the two approaches can be tested using computational models of neural responsiveness prior to time-intensive psychophysical studies. In our previous work, we have used such models to examine the effects of noise on several psychophysical measures important to speech recognition. However, to date there has been no parallel analytic solution investigating the neural response to the high-rate pulse-train stimuli and their effect on psychophysical measures. This work investigates the properties of the neural response to high-rate pulse-train stimuli with amplitude modulated envelopes using a stochastic auditory nerve model. The statistics governing the neural response to each pulse are derived using a recursive method. The agreement between the theoretical predictions and model simulations is demonstrated for sinusoidal amplitude modulated (SAM) high rate pulse-train stimuli. With our approach, predicting the neural response in modern implant devices becomes tractable. Psychophysical measurements are also predicted using the stochastic auditory nerve model for SAM high-rate pulse-train stimuli. Changes in dynamic range (DR) and intensity discrimination are compared with that observed for noise-modulated pulse-train stimuli. Modulation frequency discrimination is also studied as a function of stimulus level and pulse rate. Results suggest that high rate carriers may positively impact such psychophysical measures.

Expediting the identification of impaired channels in cochlear implants via analysis of speech-based confusion matrices.

There is significant variability in the benefit provided by cochlear implants to severely deafened individuals. The reasons why some subjects exhibit low speech recognition scores are unknown; however, underlying physiological or psychophysical factors may be involved. Certain phenomena, such as indiscriminable electrodes and nonmonotonic pitch rankings, might hint at limitations in the ability of individual channels in the cochlear implant and/or sensorineural pathway to convey speech information. In this paper, four approaches for analyzing the results of a simple listening test using speech stimuli are investigated for the purpose of targeting channels of concern in order for follow-on psychophysical experiments to correctly identify channels performing in an "impaired" or anomalous manner. Listening tests were first conducted with normal-hearing subjects and acoustic models simulating channel-specific anomalies. Results indicate that these proposed analyses perform significantly better than chance in providing information about the location of anomalous channels. Vowel and consonant confusion matrices from six cochlear implant subjects were also analyzed to test the robustness of the proposed analyses to variability intrinsic to cochlear implant data. The current study suggests that confusion matrix analyses have the potential to expedite the identification of impaired channels by providing preliminary information prior to exhaustive psychophysical testing.

Acoustic model investigation of a multiple carrier frequency algorithm for encoding fine frequency structure: implications for cochlear implants.

Current cochlear implants provide frequency resolution through the number of channels. Improving resolution by increasing channels is limited by factors such as the physiological feasibility of increasing the number of electrodes, the inability to increase the number of channels for those already implanted, and the increased possibility of channel interactions reducing channel efficacy. Recent studies have suggested an alternative method: providing a continuum of pitch percepts for each channel based on the frequency content of that channel. This study seeks to determine the frequency resolution necessary for the highest performance gain, which may give some indication of the feasibility for implementation in implants. A discrete set of carrier frequencies, instead of a continuum, are evaluated using an acoustic model to measure speech recognition. Performance increased as the number of available frequencies increased, and substantive improvement was seen with as few as two frequencies per channel. The effect of variable frequency discrimination was also assessed, and the results suggest that frequency modulation can still provide benefits with poor frequency discrimination on some channels. These results suggest that if two or more discriminable frequencies per channel can be generated for cochlear implant subjects then an improvement in speech recognition may be possible.

Predicting dynamic range and intensity discrimination for electrical pulse-train stimuli using a stochastic auditory nerve model: the effects of stimulus noise.

This work investigates dynamic range and intensity discrimination for electrical pulse-train stimuli that are modulated by noise using a stochastic auditory nerve model. Based on a hypothesized monotonic relationship between loudness and the number of spikes elicited by a stimulus, theoretical prediction of the uncomfortable level has previously been determined by comparing spike counts to a fixed threshold, Nucl. However, no specific rule for determining Nucl has been suggested. Our work determines the uncomfortable level based on the excitation pattern of the neural response in a normal ear. The number of fibers corresponding to the portion of the basilar membrane driven by a stimulus at an uncomfortable level in a normal ear is related to Nucl at an uncomfortable level of the electrical stimulus. Intensity discrimination limens are predicted using signal detection theory via the probability mass function of the neural response and via experimental simulations. The results show that the uncomfortable level for pulse-train stimuli increases slightly as noise level increases. Combining this with our previous threshold predictions, we hypothesize that the dynamic range for noise-modulated pulse-train stimuli should increase with additive noise. However, since our predictions indicate that intensity discrimination under noise degrades, overall intensity coding performance may not improve significantly.