A deep neural-network classifier for photograph-based estimation of hearing protection attenuation and fit.

Occupational and recreational acoustic noise exposure is known to cause permanent hearing damage and reduced quality of life, which indicates the importance of noise controls including hearing protection devices (HPDs) in situations where high noise levels exist. While HPDs can provide adequate protection for many noise exposures, it is often a challenge to properly train HPD users and maintain compliance with usage guidelines. HPD fit-testing systems are commercially available to ensure proper attenuation is achieved, but they often require specific facilities designed for hearing testing (e.g., a quiet room or an audiometric booth) or special equipment (e.g., modified HPDs designed specifically for fit testing). In this study, we explored using visual information from a photograph of an HPD inserted into the ear to estimate hearing protector attenuation. Our dataset consists of 960 unique photographs from four types of hearing protectors across 160 individuals. We achieved 73% classification accuracy in predicting if the fit was greater or less than the median measured attenuation (29 dB at 1 kHz) using a deep neural network. Ultimately, the fit-test technique developed in this research could be used for training as well as for automated compliance monitoring in noisy environments to prevent hearing loss.

Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid.

Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to acoustic streams in the environment. To address the former, we present a system for joint speaker separation and noise suppression, referred to as the Binaural Enhancement via Attention Masking Network (BEAMNET). The BEAMNET system is an end-to-end neural network architecture based on self-attention. Binaural input waveforms are mapped to a joint embedding space via a learned encoder, and separate multiplicative masking mechanisms are included for noise suppression and speaker separation. Pairs of output binaural waveforms are then synthesized using learned decoders, each capturing a separated speaker while maintaining spatial cues. A key contribution of BEAMNET is that the architecture contains a separation path, an enhancement path, and an autoencoder path. This paper proposes a novel loss function which simultaneously trains these paths, so that disabling the masking mechanisms during inference causes BEAMNET to reconstruct the input speech signals. This allows dynamic control of the level of suppression applied by BEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speaker separation. This paper also proposes a perceptually-motivated waveform distance measure. Using objective speech quality metrics, the proposed system is demonstrated to perform well at separating two equal-energy talkers, even in high levels of background noise. Subjective testing shows an improvement in speech intelligibility across a range of noise levels, for signals with artificially added head-related transfer functions and background noise. Finally, when used as part of an auditory attention decoder (AAD) system using existing electroencephalogram (EEG) data, BEAMNET is found to maintain the decoding accuracy achieved with ideal speaker separation, even in severe acoustic conditions. These results suggest that this enhancement system is highly effective at decoding auditory attention in realistic noise environments, and could possibly lead to improved speech perception in a cognitively controlled hearing aid.

Audio, Visual, and Electrodermal Arousal Signals as Predictors of Mental Fatigue Following Sustained Cognitive Work.

Lapses in vigilance and slowed reactions due to mental fatigue can increase risk of accidents and injuries and degrade performance. This paper describes a method for rapid, unobtrusive detection of mental fatigue based on changes in electrodermal arousal (EDA), and changes in neuromotor coordination derived from speaking. Twenty-nine Soldiers completed a 2-hour battery of cognitive tasks intended to induce fatigue. Behavioral markers derived from audio and video during speech were acquired before and after the 2hour cognitive load tasks, as was EDA. Exposure to cognitive load produced detectable increases in neuromotor variability in speech and facial measures after load and even after a recovery period. A Gaussian mixture model classifier with crossvalidation and fusion across speech, video, and EDA produced an accuracy of AUC=0.99 in detecting a change in cognitive fatigue relative to a personalized baseline.

Sensorimotor conflict tests in an immersive virtual environment reveal subclinical impairments in mild traumatic brain injury.

Current clinical tests lack the sensitivity needed for detecting subtle balance impairments associated with mild traumatic brain injury (mTBI). Patient-reported symptoms can be significant and have a huge impact on daily life, but impairments may remain undetected or poorly quantified using clinical measures. Our central hypothesis was that provocative sensorimotor perturbations, delivered in a highly instrumented, immersive virtual environment, would challenge sensory subsystems recruited for balance through conflicting multi-sensory evidence, and therefore reveal that not all subsystems are performing optimally. The results show that, as compared to standard clinical tests, the provocative perturbations illuminate balance impairments in subjects who have had mild traumatic brain injuries. Perturbations delivered while subjects were walking provided greater discriminability (average accuracy ≈ 0.90) than those delivered during standing (average accuracy ≈ 0.65) between mTBI subjects and healthy controls. Of the categories of features extracted to characterize balance, the lower limb accelerometry-based metrics proved to be most informative. Further, in response to perturbations, subjects with an mTBI utilized hip strategies more than ankle strategies to prevent loss of balance and also showed less variability in gait patterns. We have shown that sensorimotor conflicts illuminate otherwise-hidden balance impairments, which can be used to increase the sensitivity of current clinical procedures. This augmentation is vital in order to robustly detect the presence of balance impairments after mTBI and potentially define a phenotype of balance dysfunction that enhances risk of injury.

Analysis of Phonetic Balance in Standard English Passages.

Purpose A common way of eliciting speech from individuals is by using passages of written language that are intended to be read aloud. Read passages afford the opportunity for increased control over the phonetic properties of elicited speech, of which phonetic balance is an often-noted example. No comprehensive analysis of the phonetic balance of read passages has been reported in the literature. The present article provides a quantitative comparison of the phonetic balance of widely used passages in English. Method Assessment of phonetic balance is carried out by comparing the distribution of phonemes in several passages to distributions consistent with typical spoken English. Data regarding the distribution of phonemes in spoken American English are aggregated from the published literature and large speech corpora. Phoneme distributions are compared using Spearman rank order correlation coefficient to quantify similarities of phoneme counts in those sources. Results Correlations between phoneme distributions in read passages and aggregated material representative of spoken American English ranged from .70 to .89. Correlations between phoneme counts from all passages, literature sources, and corpus sources ranged from .55 to .99. All correlations were statistically significant at the Bonferroni-adjusted level. Conclusions Passages considered in the present work provide high, but not ideal, phonetic balance. Space exists for the creation of new passages that more closely match the phoneme distributions observed in spoken American English. The Caterpillar provided the best phonetic balance, but phoneme distributions in all considered materials were highly similar to each other.

Deep Neural Network Model of Hearing-Impaired Speech-in-Noise Perception.

Many individuals struggle to understand speech in listening scenarios that include reverberation and background noise. An individual's ability to understand speech arises from a combination of peripheral auditory function, central auditory function, and general cognitive abilities. The interaction of these factors complicates the prescription of treatment or therapy to improve hearing function. Damage to the auditory periphery can be studied in animals; however, this method alone is not enough to understand the impact of hearing loss on speech perception. Computational auditory models bridge the gap between animal studies and human speech perception. Perturbations to the modeled auditory systems can permit mechanism-based investigations into observed human behavior. In this study, we propose a computational model that accounts for the complex interactions between different hearing damage mechanisms and simulates human speech-in-noise perception. The model performs a digit classification task as a human would, with only acoustic sound pressure as input. Thus, we can use the model's performance as a proxy for human performance. This two-stage model consists of a biophysical cochlear-nerve spike generator followed by a deep neural network (DNN) classifier. We hypothesize that sudden damage to the periphery affects speech perception and that central nervous system adaptation over time may compensate for peripheral hearing damage. Our model achieved human-like performance across signal-to-noise ratios (SNRs) under normal-hearing (NH) cochlear settings, achieving 50% digit recognition accuracy at -20.7 dB SNR. Results were comparable to eight NH participants on the same task who achieved 50% behavioral performance at -22 dB SNR. We also simulated medial olivocochlear reflex (MOCR) and auditory nerve fiber (ANF) loss, which worsened digit-recognition accuracy at lower SNRs compared to higher SNRs. Our simulated performance following ANF loss is consistent with the hypothesis that cochlear synaptopathy impacts communication in background noise more so than in quiet. Following the insult of various cochlear degradations, we implemented extreme and conservative adaptation through the DNN. At the lowest SNRs (<0 dB), both adapted models were unable to fully recover NH performance, even with hundreds of thousands of training samples. This implies a limit on performance recovery following peripheral damage in our human-inspired DNN architecture.

Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods.

Auditory attention decoding (AAD) through a brain-computer interface has had a flowering of developments since it was first introduced by Mesgarani and Chang (2012) using electrocorticograph recordings. AAD has been pursued for its potential application to hearing-aid design in which an attention-guided algorithm selects, from multiple competing acoustic sources, which should be enhanced for the listener and which should be suppressed. Traditionally, researchers have separated the AAD problem into two stages: reconstruction of a representation of the attended audio from neural signals, followed by determining the similarity between the candidate audio streams and the reconstruction. Here, we compare the traditional two-stage approach with a novel neural-network architecture that subsumes the explicit similarity step. We compare this new architecture against linear and non-linear (neural-network) baselines using both wet and dry electroencephalogram (EEG) systems. Our results indicate that the new architecture outperforms the baseline linear stimulus-reconstruction method, improving decoding accuracy from 66% to 81% using wet EEG and from 59% to 87% for dry EEG. Also of note was the finding that the dry EEG system can deliver comparable or even better results than the wet, despite the latter having one third as many EEG channels as the former. The 11-subject, wet-electrode AAD dataset for two competing, co-located talkers, the 11-subject, dry-electrode AAD dataset, and our software are available for further validation, experimentation, and modification.

Current models of speech motor control: A control-theoretic overview of architectures and properties.

This paper reviews the current state of several formal models of speech motor control, with particular focus on the low-level control of the speech articulators. Further development of speech motor control models may be aided by a comparison of model attributes. The review builds an understanding of existing models from first principles, before moving into a discussion of several models, showing how each is constructed out of the same basic domain-general ideas and components-e.g., generalized feedforward, feedback, and model predictive components. This approach allows for direct comparisons to be made in terms of where the models differ, and their points of agreement. Substantial differences among models can be observed in their use of feedforward control, process of estimating system state, and method of incorporating feedback signals into control. However, many commonalities exist among the models in terms of their reliance on higher-level motor planning, use of feedback signals, lack of time-variant adaptation, and focus on kinematic aspects of control and biomechanics. Ongoing research bridging hybrid feedforward/feedback pathways with forward dynamic control, as well as feedback/internal model-based state estimation, is discussed.

Correlating an Ambulatory Voice Measure to Electrodermal Activity in Patients with Vocal Hyperfunction.

We investigate the connection between the autonomic nervous system and the voice in patients with vocal hyperfunction and healthy-control groups. We present a methodology and preliminary results of two multi-modal measurement streams that capture this relationship. Subjects were instrumented for daily, ambulatory collection of their voice and wrist-based electrodermal activity. Measures of vocal function (e.g., fundamental frequency) were computed, as well as measures of autonomic function (e.g., skin conductance response). Spearman correlation coefficients were calculated to measure the relationship between vocal and autonomic function over sliding windows throughout each observation day. We found preliminary evidence that patients with a subtype of vocal hyperfunction (non-phonotraumatic vocal hyperfunction) exhibit a coupling between the autonomic nervous system and the vocal system. Understanding how the autonomic nervous system interacts with the voice may provide new insights into the etiology/pathophysiology of vocal hyperfunction and improve prevention, diagnosis and treatment of these disorders.

Listening Into 2030 Workshop: An Experiment in Envisioning the Future of Hearing and Communication Science.

Here we report the methods and output of a workshop examining possible futures of speech and hearing science out to 2030. Using a design thinking approach, a range of human-centered problems in communication were identified that could provide the motivation for a wide range of research. Nine main research programs were distilled and are summarized: (a) measuring brain and other physiological parameters, (b) auditory and multimodal displays of information, (c) auditory scene analysis, (d) enabling and understanding shared auditory virtual spaces, (e) holistic approaches to health management and hearing impairment, (f) universal access to evolving and individualized technologies, (g) biological intervention for hearing dysfunction, (h) understanding the psychosocial interactions with technology and other humans as mediated by technology, and (i) the impact of changing models of security and privacy. The design thinking approach attempted to link the judged level of importance of different research areas to the "end in mind" through empathy for the real-life problems embodied in the personas created during the workshop.