Robust Three-Microphone Speech Source Localization Using Randomized Singular Value Decomposition.

Direction-of-arrival (DOA) estimation is a fundamental technique in array signal processing due to its wide applications in beamforming, speech enhancement and many other assistive speech processing technologies. In this paper, we devise a novel DOA technique based on randomized singular value decomposition (RSVD) to improve the performance of non-uniform non-linear microphone arrays (NUNLA). The accurate and efficient singular value decomposition of large data matrices is computationally challenging, and randomization provides an effective tool for performing matrix approximation, therefore, the developed DOA estimation utilizes a modified dictionary-based RSVD method for localizing single speech sources under low signal-to-noise ratios (SNR). Unlike previous methods developed for uniform linear microphone arrays, the proposed approach with L-shaped three microphone setup has no 'left-right' ambiguity. We present the performance of our proposed method in comparison to other techniques. The demonstrated experiments shows at-least 20% performance improvement using simulated data and 25% performance improvement using real data when compared with similar DoA estimation techniques for NUNLA. The proposed method exploits frame-based online time delay of arrival (TDOA) measurements which facilitates the proposed algorithm to run on real-time devices. We also show an efficient real-time implementation of the proposed method on a Pixel 3 Android smartphone using its built-in three microphones for hearing aid applications.

Smartphone-based single-channel speech enhancement application for hearing aids.

This work presents a single-channel speech enhancement (SE) framework based on the super-Gaussian extension of the joint maximum a posteriori (SGJMAP) estimation rule. The developed SE algorithm is an open-source research smartphone-based application for hearing improvement studies. In this algorithm, the SGJMAP-based estimation for noisy speech mixture is smoothed along the frequency axis by a Mel filter-bank, resulting in a Mel-warped frequency-domain SGJMAP estimation. The impulse response of this Mel-warped estimation is obtained by applying a Mel-warped inverse discrete cosine transform (Mel-IDCT). This helps in filtering out the background noise and enhancing the speech signal. The proposed application is implemented on an iPhone (Apple, Cupertino, CA) to operate in real time and tested with normal-hearing (NH) and hearing-impaired (HI) listeners with different types of hearing aids through wireless connectivity. The objective speech quality and intelligibility test results are used to compare the performance of the proposed algorithm to existing conventional single-channel SE methods. Additionally, test results from NH and HI listeners show substantial improvement in speech recognition with the developed method in simulated real-world noisy conditions at different signal-to-noise ratio levels.

Real-time dual-channel speech enhancement by VAD assisted MVDR beamformer for hearing aid applications using smartphone.

In this paper, a dual-channel speech enhancement (SE) method is proposed. The proposed method is a combination of minimum variance distortionless response (MVDR) beamformer and a super-Gaussian joint maximum a posteriori (SGJMAP) based SE gain function. The proposed SE method runs on a smartphone in real-time, providing a portable device for hearing aid (HA) applications. Spectral Flux based voice activity detector (VAD) is used to improve the accuracy of the beamformer output. The efficiency of the proposed SE method is evaluated using speech quality and intelligibility measures and compared with that of other SE techniques. The objective and subjective test results show the capability of the proposed SE method in three different noisy conditions at low signal to noise ratios (SNRs) of -5, 0, and +5 dB.

Automated machine learning based speech classification for hearing aid applications and its real-time implementation on smartphone.

Deep neural networks (DNNs) have been useful in solving benchmark problems in various domains including audio. DNNs have been used to improve several speech processing algorithms that improve speech perception for hearing impaired listeners. To make use of DNNs to their full potential and to configure models easily, automated machine learning (AutoML) systems are developed, focusing on model optimization. As an application of AutoML to audio and hearing aids, this work presents an AutoML based voice activity detector (VAD) that is implemented on a smartphone as a real-time application. The developed VAD can be used to elevate the performance of speech processing applications like speech enhancement that are widely used in hearing aid devices. The classification model generated by AutoML is computationally fast and has minimal processing delay, which enables an efficient, real-time operation on a smartphone. The steps involved in real-time implementation are discussed in detail. The key contribution of this work include the utilization of AutoML platform for hearing aid applications and the realization of AutoML model on smartphone. The experimental analysis and results demonstrate the significance and importance of using the AutoML for the current approach. The evaluations also show improvements over the state of art techniques and reflect the practical usability of the developed smartphone app in different noisy environments.

Efficient Real-Time Acoustic Feedback Cancellation using Adaptive Noise Injection Algorithm.

Acoustic feedback cancellation is a challenging problem in the design of sound reinforcement systems, hearing aids, etc. Acoustic feedback is inevitable when the acoustic signal path forms a loop between the microphone and loudspeaker. An efficient short duration noise injection algorithm is proposed in this paper to estimate the impulse response of the acoustic feedback path model. The algorithm does not require any prior information about the acoustic feedback path. It is capable of optimally estimate the acoustic feedback path for cancellation, and avoid the occurrence of any howling episode, in varying acoustic environments. Presented algorithm is efficiently implemented on smartphone device having close proximity of loudspeaker and microphone to emulate the feedback condition. The algorithm being platform-independent can also be implemented for any set-up or system. The experimental results of the proposed method shows satisfying results and its ability to track and cancel the acoustic feedback in changing characteristics of the acoustic path.

Compression Fitting of Hearing Aids and Implementation.

A compressor in hearing aid devices (HADs) is responsible for mapping the dynamic range of input signals to the residual dynamic range of hearing-impaired (HI) patients. Gains and parameters of the compressor are set according to the HI patient's preferences. In different surroundings depending upon noise level, the patient may seek to tune the parameters to improve performance. Traditionally, fitting of the hearing aids is done by an audiologist using hearing aid software and the HI patient's opinion at a clinic. In this paper, we propose a frequency-based multi-band compressor implemented as a smartphone application, which can be used as an alternative to that of the traditional HADs. The proposed solution allows the user to tune the compression parameters for each band along with a choice of compression speed and fitting strategy. Exploiting smartphone processing and hardware capabilities, the application can be used for bilateral hearing loss. The performance of this easy-to-use smartphone-based application is compared with traditional HADs using a hearing aid test system. Objective and subjective evaluations are also carried out to quantify the performance.

Design and Integration of Alert Signal Detector and Separator for Hearing Aid Applications.

Alert signals like sirens and home alarms are important as they warn people of precarious situations. This work presents the detection and separation of these acoustically important alert signals, not to be attenuated as noise, to assist the hearing impaired listeners. The proposed method is based on convolutional neural network (CNN) and convolutional-recurrent neural network (CRNN). The developed method consists of two blocks, the detector block, and the separator block. The entire setup is integrated with speech enhancement (SE) algorithms, and before the compression stage, used in a hearing aid device (HAD) signal processing pipeline. The detector recognizes the presence of alert signal in various noisy environments. The separator block separates the alert signal from the mixture of noisy signals before passing it through SE to ensure minimal or no attenuation of the alert signal. It is implemented on a smartphone as an application that seamlessly works with HADs in real-time. This smartphone assistive setup allows the hearing aid users to know the presence of the alert sounds even when these are out of sight. The algorithm is computationally efficient with a low processing delay. The key contribution of this paper includes the development and integration of alert signal separator block with SE and the realization of the entire setup on a smartphone in real-time. The proposed method is compared with several state-of-the-art techniques through objective measures in various noisy conditions. The experimental analysis demonstrates the effectiveness and practical usefulness of the developed setup in real-world noisy scenarios.

Efficient two-microphone speech enhancement using basic recurrent neural network cell for hearing and hearing aids.

This work presents a two-microphone speech enhancement (SE) framework based on basic recurrent neural network (RNN) cell. The proposed method operates in real-time, improving the speech quality and intelligibility in noisy environments. The RNN model trained using a simple feature set-real and imaginary parts of the short-time Fourier transform (STFT) are computationally efficient with a minimal input-output processing delay. The proposed algorithm can be used in any stand-alone platform such as a smartphone using its two inbuilt microphones. The detailed operation of the real-time implementation on the smartphone is presented. The developed application works as an assistive tool for hearing aid devices (HADs). Speech quality and intelligibility test results are used to compare the proposed algorithm to existing conventional and neural network-based SE methods. Subjective and objective scores show the superior performance of the developed method over several conventional methods in different noise conditions and low signal to noise ratios (SNRs).

CONVOLUTIONAL RECURRENT NEURAL NETWORK BASED DIRECTION OF ARRIVAL ESTIMATION METHOD USING TWO MICROPHONES FOR HEARING STUDIES.

This work proposes a convolutional recurrent neural network (CRNN) based direction of arrival (DOA) angle estimation method, implemented on the Android smartphone for hearing aid applications. The proposed app provides a 'visual' indication of the direction of a talker on the screen of Android smartphones for improving the hearing of people with hearing disorders. We use real and imaginary parts of short-time Fourier transform (STFT) as a feature set for the proposed CRNN architecture for DOA angle estimation. Real smartphone recordings are utilized for assessing performance of the proposed method. The accuracy of the proposed method reaches 87.33% for unseen (untrained) environments. This work also presents real-time inference of the proposed method, which is done on an Android smartphone using only its two built-in microphones and no additional component or external hardware. The real-time implementation also proves the generalization and robustness of the proposed CRNN based model.

Spectral Flux-Based Convolutional Neural Network Architecture for Speech Source Localization and Its Real-Time Implementation.

In this article, we present a real-time convolutional neural network (CNN)-based Speech source localization (SSL) algorithm that is robust to realistic background acoustic conditions (noise and reverberation). We have implemented and tested the proposed method on a prototype (Raspberry Pi) for real-time operation. We have used the combination of the imaginary-real coefficients of the short-time Fourier transform (STFT) and Spectral Flux (SF) with delay-and-sum (DAS) beamforming as the input feature. We have trained the CNN model using noisy speech recordings collected from different rooms and inference on an unseen room. We provide quantitative comparison with five other previously published SSL algorithms under several realistic noisy conditions, and show significant improvements by incorporating the Spectral Flux (SF) with beamforming as an additional feature to learn temporal variation in speech spectra. We perform real-time inferencing of our CNN model on the prototyped platform with low latency (21 milliseconds (ms) per frame with a frame length of 30 ms) and high accuracy (i.e. 89.68% under Babble noise condition at 5dB SNR). Lastly, we provide a detailed explanation of real-time implementation and on-device performance (including peak power consumption metrics) that sets this work apart from previously published works. This work has several notable implications for improving the audio-processing algorithms for portable battery-operated Smart loudspeakers and hearing improvement (HI) devices.

Real-time single-channel deep neural network-based speech enhancement on edge devices.

In this paper, we present a deep neural network architecture comprising of both convolutional neural network (CNN) and recurrent neural network (RNN) layers for real-time single-channel speech enhancement (SE). The proposed neural network model focuses on enhancing the noisy speech magnitude spectrum on a frame-by-frame process. The developed model is implemented on the smartphone (edge device), to demonstrate the real-time usability of the proposed method. Perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) test results are used to compare the proposed algorithm to previously published conventional and deep learning-based SE methods. Subjective ratings show the performance improvement of the proposed model over the other baseline SE methods.

A Real-Time Convolutional Neural Network Based Speech Enhancement for Hearing Impaired Listeners Using Smartphone.

This paper presents a Speech Enhancement (SE) technique based on multi-objective learning convolutional neural network to improve the overall quality of speech perceived by Hearing Aid (HA) users. The proposed method is implemented on a smartphone as an application that performs real-time SE. This arrangement works as an assistive tool to HA. A multi-objective learning architecture including primary and secondary features uses a mapping-based convolutional neural network (CNN) model to remove noise from a noisy speech spectrum. The algorithm is computationally fast and has a low processing delay which enables it to operate seamlessly on a smartphone. The steps and the detailed analysis of real-time implementation are discussed. The proposed method is compared with existing conventional and neural network-based SE techniques through speech quality and intelligibility metrics in various noisy speech conditions. The key contribution of this paper includes the realization of CNN SE model on a smartphone processor that works seamlessly with HA. The experimental results demonstrate significant improvements over the state-of-the-art techniques and reflect the usability of the developed SE application in noisy environments.

Frequency-based multi-band adaptive compression for hearing aid application.

Multi-band Dynamic Range (MBDR) Compression is a key part of the signal processing operation in hearing aid devices (HADs). Operating speed of the MBDR compressor plays an important role in preserving the quality and intelligibility of the output signal. Traditional fast-acting compressor preserves the audible cues in quiet speech but, in presence of surrounding noise, it can degrade the sound quality by introducing pumping and breathing effects. Alternatively, slow-acting compressor maintains the temporal cues and the listening comfort but may provide inadequate gain for soft inputs that come right after loud inputs. HADs may operate in a variable acoustic environment. Therefore, a fixed speed in compression might affect the performance of the hearing aids. In this study, we propose a frequency(FFT) based nine-band adaptive MBDR compression which uses spectral flux as a measure of the intensity change in input level to adapt the speed of the compressor in each band. Gain, threshold and compression ratio of the compressor for nine bands are adjusted based on the audiogram of the hearing impaired patient. The proposed frequency-based adaptive MBDR compression method is implemented on smartphone. The objective and subjective test results demonstrate the performance of proposed method compared to fixed compression approaches.

Direction of arrival estimation using deep neural network for hearing aid applications using smartphone.

Deep neural network (DNN) techniques are gaining popularity due to performance boost in many applications. In this work we propose a DNN-based method for finding the direction of arrival (DOA) of speech source for hearing study improvement and hearing aid applications using popular smartphone with no external components as a cost-effective stand-alone platform. We consider the DOA estimation as a classification problem and use the magnitude and phase of speech signal as a feature set for DNN training stage and obtaining appropriate model. The model is trained and derived using real speech and real noisy speech data recorded on smartphone in different noisy environments under low signal to noise ratios (SNRs). The DNN-based DOA method with the pre-trained model is implemented and run on Android smartphone in real time. The performance of proposed method is evaluated objectively and subjectively in the both training and unseen environments. The test results are presented showing the superior performance of proposed method over conventional methods.

Real-time Convolutional Neural Network based Speech Source Localization on Smartphone.

In this paper, we present a real-time convolutional neural network (CNN) based approach for speech source localization (SSL) using Android-based smartphone and its two built-in microphones under noisy conditions. We propose a new input feature set - using real and imaginary parts of the short-time Fourier transform (STFT) for CNN-based SSL. We use simulated noisy data from popular datasets that was augmented with few hours of real recordings collected on smartphones to train our CNN model. We compare the proposed method to recent CNN-based SSL methods that are trained on our dataset and show that our CNN-based SSL method offers higher accuracy on identical test datasets. Another unique aspect of this work is that we perform real-time inferencing of our CNN model on an Android smartphone with low latency (14 milliseconds(ms) for single frame-based estimation, 180 ms for multi frame-based estimation and frame length is 20 ms for both cases) and high accuracy (i.e. 88.83% at 0dB SNR). We show that our CNN model is rather robust to smartphone hardware mismatch, hence we may not need to retrain the entire model again for use with different smartphones. The proposed application provides a 'visual' indication of the direction of a talker on the screen of Android smartphones for improving the hearing of people with hearing disorders.

Efficient Modeling of Acoustic Feedback Path in Hearing Aids by Voice Activity Detector-Supervised Multiple Noise Injections.

Adaptive Feedback Cancellation (AFC) techniques are widely used to eliminate the undesired acoustic feedback effect arising in the Hearing Aid Devices (HADs) due to the coupling between the speaker and the microphone of the HAD. This paper proposes a method to eliminate the acoustic feedback effect in the HADs in presence of noisy environment. The method involves utilization of a computationally efficient Spectral Flux feature-based voice activity detector (VAD), which controls the process of Noise Injection in the proposed AFC algorithm (SFNIAFC). The proposed algorithm's performance is objectively evaluated using Misalignment (MISA) and Perceptual Evaluation of Speech Quality (PESQ) criteria for realistic noisy conditions. The simulations performed for the proposed method shows faster convergence and reduction in the MISA values with high PESQ values in comparison to the earlier method. Subjective test results support the effectiveness and better performance of the proposed algorithm for the HAD applications over earlier method.

Smartphone based real-time super Gaussian single microphone Speech Enhancement to improve intelligibility for hearing aid users using formant information.

In this paper, we present a Speech Enhancement (SE) technique to improve intelligibility of speech perceived by Hearing Aid users using smartphone as an assistive device. We use the formant frequency information to improve the overall quality and intelligibility of the speech. The proposed SE method is based on new super Gaussian joint maximum a Posteriori (SGJMAP) estimator. Using the priori information of formant frequency locations, the derived gain function has " tradeoff" factors that allows the smartphone user to customize perceptual preference, by controlling the amount of noise suppression and speech distortion in real-time. The formant frequency information helps the hearing aid user to control the gains over the non-formant frequency band, allowing the HA users to attain more noise suppression while maintaining the speech intelligibility using a smartphone application. Objective intelligibility measures and subjective results reflect the usability of the developed SE application in noisy real world acoustic environment.

Influence of MVDR beamformer on a Speech Enhancement based Smartphone application for Hearing Aids.

This paper presents the minimum variance distortionless response (MVDR) beamformer combined with a Speech Enhancement (SE) gain function as a real-time application running on smartphones that work as an assistive device to Hearing Aids. It has been shown that beamforming techniques improve the Signal to Noise Ratio (SNR) in noisy conditions. In the proposed algorithm, MVDR beamformer is used as an SNR booster for the SE method. The proposed SE gain is based on the Log-Spectral Amplitude estimator to improve the speech quality in the presence of different background noises. Objective evaluation and intelligibility measures support the theoretical analysis and show significant improvements of the proposed method in comparison with existing methods. Subjective test results show the effectiveness of the application in real-world noisy conditions at SNR levels of -5 dB, 0 dB, and 5 dB.

Formant Frequency-based Speech Enhancement Technique to improve Intelligibility for hearing aid users with smartphone as an assistive device.

In this paper, we present a Speech Enhancement (SE) method implemented on a smartphone, and this arrangement functions as an assistive device to hearing aids (HA). Many benchmark single channel SE algorithms implemented on HAs provide considerable improvement in speech quality, while speech intelligibility improvement still remains a prime challenge. The proposed SE method based on Log spectral amplitude estimator improves speech intelligibility in the noisy real world acoustic environment using the priori information of formant frequency locations. The formant frequency information avails us to control the amount of speech distortion in these frequency bands, thereby controlling speech distortion. We introduce a 'scaling' parameter for the SE gain function, which controls the gains over the non-formant frequency band, allowing the HA users to customize the playback speech using a smartphone application to their listening preference. Objective intelligibility measures show the effectiveness of the proposed SE method. Subjective results reflect the suitability of the developed Speech Enhancement application in real-world noisy conditions at SNR levels of -5 dB, 0 dB and 5 dB.

Improving quality and intelligibility of speech using single microphone for the broadband fMRI noise at low SNR.

Functional Magnetic Resonance Imaging (fMRI) is used in many diagnostic procedures for neurological related disorders. Strong broadband acoustic noise generated during fMRI scan interferes with the speech communication between the physician and the patient. In this paper, we propose a single microphone Speech Enhancement (SE) technique which is based on the supervised machine learning technique and a statistical model based SE technique. The proposed algorithm is robust and computationally efficient and has capability to run in real-time. Objective and Subjective evaluations show that the proposed SE method outperforms the existing state-of-the-art algorithms in terms of quality and intelligibility of the recovered speech at low Signal to Noise Ratios (SNRs).