Objective intelligibility measurement of reverberant vocoded speech for normal-hearing listeners: Towards facilitating the development of speech enhancement algorithms for cochlear implants.

Cochlear implant (CI) recipients often struggle to understand speech in reverberant environments. Speech enhancement algorithms could restore speech perception for CI listeners by removing reverberant artifacts from the CI stimulation pattern. Listening studies, either with cochlear-implant recipients or normal-hearing (NH) listeners using a CI acoustic model, provide a benchmark for speech intelligibility improvements conferred by the enhancement algorithm but are costly and time consuming. To reduce the associated costs during algorithm development, speech intelligibility could be estimated offline using objective intelligibility measures. Previous evaluations of objective measures that considered CIs primarily assessed the combined impact of noise and reverberation and employed highly accurate enhancement algorithms. To facilitate the development of enhancement algorithms, we evaluate twelve objective measures in reverberant-only conditions characterized by a gradual reduction of reverberant artifacts, simulating the performance of an enhancement algorithm during development. Measures are validated against the performance of NH listeners using a CI acoustic model. To enhance compatibility with reverberant CI-processed signals, measure performance was assessed after modifying the reference signal and spectral filterbank. Measures leveraging the speech-to-reverberant ratio, cepstral distance and, after modifying the reference or filterbank, envelope correlation are strong predictors of intelligibility for reverberant CI-processed speech.

Parameter tuning of time-frequency masking algorithms for reverberant artifact removal within the cochlear implant stimulus.

Cochlear implant recipients struggle to understand speech in reverberant environments. To restore speech perception, artifacts due to reverberant reflections can be removed from the cochlear implant stimulus by applying a matrix of gain values, a technique referred to as time-frequency masking. In this study, two common time-frequency masking strategies are implemented within cochlear implant processing, either introducing complete retention or deletion of stimulus components using a binary mask or continuous attenuation of stimulus components using a ratio mask. Parameters of each masking strategy control the level of attenuation imposed by the gain values. In this study, we perceptually tune the parameters of the masking strategy to determine a balance between speech retention and artifact removal. We measure the intelligibility of reverberant signals mitigated by each strategy with speech recognition testing in normal-hearing listeners using vocoding as a simulation of cochlear implant perception. For both masking strategies, we find parameterizations that maximize the intelligibility of the mitigated signals. At the best-performing parameterizations, binary-masked reverberant signals yield larger intelligibility improvements than ratio-masked signals. The results provide a perceptually optimized objective for the removal of reverberant artifacts from cochlear implant stimuli, facilitating improved speech recognition performance for cochlear implant recipients in reverberant environments.

Language Model-Guided Classifier Adaptation for Brain-Computer Interfaces for Communication.

Brain-computer interfaces (BCIs), such as the P300 speller, can provide a means of communication for individuals with severe neuromuscular limitations. BCIs interpret electroencephalography (EEG) signals in order to translate embedded information about a user's intent into executable commands to control external devices. However, EEG signals are inherently noisy and nonstationary, posing a challenge to extended BCI use. Conventionally, a BCI classifier is trained via supervised learning in an offline calibration session; once trained, the classifier is deployed for online use and is not updated. As the statistics of a user's EEG data change over time, the performance of a static classifier may decline with extended use. It is therefore desirable to automatically adapt the classifier to current data statistics without requiring offline recalibration. In an existing semi-supervised learning approach, the classifier is trained on labeled EEG data and is then updated using incoming unlabeled EEG data and classifier-predicted labels. To reduce the risk of learning from incorrect predictions, a threshold is imposed to exclude unlabeled data with low-confidence label predictions from the expanded training set when retraining the adaptive classifier. In this work, we propose the use of a language model for spelling error correction and disambiguation to provide information about label correctness during semi-supervised learning. Results from simulations with multi-session P300 speller user EEG data demonstrate that our language-guided semi-supervised approach significantly improves spelling accuracy relative to conventional BCI calibration and threshold-based semi-supervised learning.

Mitigating the Impact of Psychophysical Effects During Adaptive Stimulus Selection in the P300 Speller Brain-Computer Interface.

Stimulus-driven brain-computer interfaces (BCIs), such as the P300 speller, rely on using sensory stimuli to elicit specific neural signal components called event-related potentials (ERPs) to control external devices. However, psychophysical factors, such as refractory effects and adjacency distractions, may negatively impact ERP elicitation and BCI performance. Although conventional BCI stimulus presentation paradigms usually design stimulus presentation schedules in a pseudo-random manner, recent studies have shown that controlling the stimulus selection process can enhance ERP elicitation. In prior work, we developed an algorithm to adaptively select BCI stimuli using an objective criterion that maximizes the amount of information about the user's intent that can be elicited with the presented stimuli given current data conditions. Here, we enhance this adaptive BCI stimulus selection algorithm to mitigate adjacency distractions and refractory effects by modeling temporal dependencies of ERP elicitation in the objective function and imposing spatial restrictions in the stimulus search space. Results from simulations using synthetic data and human data from a BCI study show that the enhanced adaptive stimulus selection algorithm can improve spelling speeds relative to conventional BCI stimulus presentation paradigms.Clinical relevance-Increased communication rates with our enhanced adaptive stimulus selection algorithm can potentially facilitate the translation of BCIs as viable communication alternatives for individuals with severe neuromuscular limitations.

Psychophysiological Markers of Performance and Learning during Simulated Marksmanship in Immersive Virtual Reality.

The fusion of immersive virtual reality, kinematic movement tracking, and EEG offers a powerful test bed for naturalistic neuroscience research. Here, we combined these elements to investigate the neuro-behavioral mechanisms underlying precision visual-motor control as 20 participants completed a three-visit, visual-motor, coincidence-anticipation task, modeled after Olympic Trap Shooting and performed in immersive and interactive virtual reality. Analyses of the kinematic metrics demonstrated learning of more efficient movements with significantly faster hand RTs, earlier trigger response times, and higher spatial precision, leading to an average of 13% improvement in shot scores across the visits. As revealed through spectral and time-locked analyses of the EEG beta band (13-30 Hz), power measured prior to target launch and visual-evoked potential amplitudes measured immediately after the target launch correlated with subsequent reactive kinematic performance in the shooting task. Moreover, both launch-locked and shot/feedback-locked visual-evoked potentials became earlier and more negative with practice, pointing to neural mechanisms that may contribute to the development of visual-motor proficiency. Collectively, these findings illustrate EEG and kinematic biomarkers of precision motor control and changes in the neurophysiological substrates that may underlie motor learning.

Novel Acoustic Biomarker of Quality of Life in Left Ventricular Assist Device Recipients.

Background Although technological advances to pump design have improved survival, left ventricular assist device (LVAD) recipients experience variable improvements in quality of life. Methods for optimizing LVAD support to improve quality of life are needed. We investigated whether acoustic signatures obtained from digital stethoscopes can predict patient-centered outcomes in LVAD recipients. Methods and Results We followed precordial sounds over 6 months in 24 LVAD recipients (8 HeartWare HVAD™, 16 HeartMate 3 [HM3]). Subjects recorded their precordial sounds with a digital stethoscope and completed a Kansas City Cardiomyopathy Questionnaire weekly. We developed a novel algorithm to filter LVAD sounds from recordings. Unsupervised clustering of LVAD-mitigated sounds revealed distinct groups of acoustic features. Of 16 HM3 recipients, 6 (38%) had a unique acoustic feature that we have termed the pulse synchronized sound based on its temporal association with the artificial pulse of the HM3. HM3 recipients with the pulse synchronized sound had significantly better Kansas City Cardiomyopathy Questionnaire scores at baseline (median, 89.1 [interquartile range, 86.2-90.4] versus 66.1 [interquartile range, 31.1-73.7]; P=0.03) and over the 6-month study period (marginal mean, 77.6 [95% CI, 66.3-88.9] versus 59.9 [95% CI, 47.9-70.0]; P<0.001). Mechanistically, the pulse synchronized sound shares acoustic features with patient-derived intrinsic sounds. Finally, we developed a machine learning algorithm to automatically detect the pulse synchronized sound within precordial sounds (area under the curve, 0.95, leave-one-subject-out cross-validation). Conclusions We have identified a novel acoustic biomarker associated with better quality of life in HM3 LVAD recipients, which may provide a method for assaying optimized LVAD support.

Heart Sound Analysis in Individuals Supported With Left Ventricular Assist Devices.

OBJECTIVE: LVADs are surgically implanted mechanical pumps that improve survival rates of individuals with advanced heart failure. LVAD therapy is associated with high morbidity, which can be partially attributed to challenges with detecting LVAD complications before adverse events occur. Current methods used to monitor for complications with LVAD support require frequent clinical assessments at specialized LVAD centers. Analysis of recorded precordial sounds may enable real-time, remote monitoring of device and cardiac function for early detection of LVAD complications. The dominance of LVAD sounds in the precordium limits the utility of routine cardiac auscultation of LVAD recipients. In this work, we develop a signal processing pipeline to mitigate sounds generated by the LVAD. METHODS: We collected in vivo precordial sounds from 17 LVAD recipients, and contemporaneous echocardiograms from 12 of these individuals, to validate heart valve closure timings. RESULTS: We characterized various acoustic signatures of heart sounds extracted from in vivo recordings, and report preliminary findings linking fundamental heart sound characteristics and level of LVAD support. CONCLUSION: Mitigation of LVAD sounds from precordial sound recordings of LVAD recipients enables analysis of intrinsic heart sounds. SIGNIFICANCE: These findings provide proof-of-concept evidence of the clinical utility of heart sound analysis for bedside and remote monitoring of LVAD recipients.

USING MACHINE LEARNING TO MITIGATE THE EFFECTS OF REVERBERATION AND NOISE IN COCHLEAR IMPLANTS.

In listening environments with room reverberation and background noise, cochlear implant (CI) users experience substantial difficulties in understanding speech. Because everyday environments have different combinations of reverberation and noise, there is a need to develop algorithms that can mitigate both effects to improve speech intelligibility. Desmond et al. (2014) developed a machine learning approach to mitigate the adverse effects of late reverberant reflections of speech signals by using a classifier to detect and remove affected segments in CI pulse trains. This study aimed to investigate the robustness of the reverberation mitigation algorithm in environments with both reverberation and noise. Sentence recognition tests were conducted in normal hearing listeners using vocoded speech with unmitigated and mitigated reverberant-only or noisy reverberant speech signals, across different reverberation times and noise types. Improvements in speech intelligibility were observed in mitigated reverberant-only conditions. However, mixed results were obtained in the mitigated noisy reverberant conditions as a reduction in speech intelligibility was observed for noise types whose spectra were similar to that of anechoic speech. Based on these results, the focus of future work is to develop a context-dependent approach that activates different mitigation strategies for different acoustic environments.

Application of a Graphical Model to Investigate the Utility of Cross-channel Information for Mitigating Reverberation in Cochlear Implants.

Individuals with cochlear implants (CIs) experience more difficulty understanding speech in reverberant environ-ments than normal hearing listeners. As a result, recent research has targeted mitigating the effects of late reverberant signal reflections in CIs by using a machine learning approach to detect and delete affected segments in the CI stimulus pattern. Previous work has trained electrode-specific classification models to mitigate late reverberant signal reflections based on features extracted from only the acoustic activity within the electrode of interest. Since adjacent CI electrodes tend to be activated concurrently during speech, we hypothesized that incorporating additional information from the other electrode channels, termed cross-channel information, as features could improve classification performance. Cross-channel information extracted in real-world conditions will likely contain errors that will impact classification performance. To simulate extracting cross-channel information in realistic conditions, we developed a graphical model based on the Ising model to systematically introduce errors to specific types of cross-channel information. The Ising-like model allows us to add errors while maintaining the important geometric information contained in cross-channel information, which is due to the spectro-temporal structure of speech. Results suggest the potential utility of leveraging cross-channel information to improve the performance of the reverberation mitigation algorithm from the baseline channel-based features, even when the cross-channel information contains errors.

Moving Away From Error-Related Potentials to Achieve Spelling Correction in P300 Spellers.

P300 spellers can provide a means of communication for individuals with severe neuromuscular limitations. However, its use as an effective communication tool is reliant on high P300 classification accuracies ( > 70%) to account for error revisions. Error-related potentials (ErrP), which are changes in EEG potentials when a person is aware of or perceives erroneous behavior or feedback, have been proposed as inputs to drive corrective mechanisms that veto erroneous actions by BCI systems. The goal of this study is to demonstrate that training an additional ErrP classifier for a P300 speller is not necessary, as we hypothesize that error information is encoded in the P300 classifier responses used for character selection. We perform offline simulations of P300 spelling to compare ErrP and non-ErrP based corrective algorithms. A simple dictionary correction based on string matching and word frequency significantly improved accuracy (35-185%), in contrast to an ErrP-based method that flagged, deleted and replaced erroneous characters (-47-0%) . Providing additional information about the likelihood of characters to a dictionary-based correction further improves accuracy. Our Bayesian dictionary-based correction algorithm that utilizes P300 classifier confidences performed comparably (44-416%) to an oracle ErrP dictionary-based method that assumed perfect ErrP classification (43-433%).

Utilizing a language model to improve online dynamic data collection in P300 spellers.

P300 spellers provide a means of communication for individuals with severe physical limitations, especially those with locked-in syndrome, such as amyotrophic lateral sclerosis. However, P300 speller use is still limited by relatively low communication rates due to the multiple data measurements that are required to improve the signal-to-noise ratio of event-related potentials for increased accuracy. Therefore, the amount of data collection has competing effects on accuracy and spelling speed. Adaptively varying the amount of data collection prior to character selection has been shown to improve spelling accuracy and speed. The goal of this study was to optimize a previously developed dynamic stopping algorithm that uses a Bayesian approach to control data collection by incorporating a priori knowledge via a language model. Participants ( n = 17) completed online spelling tasks using the dynamic stopping algorithm, with and without a language model. The addition of the language model resulted in improved participant performance from a mean theoretical bit rate of 46.12 bits/min at 88.89% accuracy to 54.42 bits/min ( ) at 90.36% accuracy.