Improved tactile speech robustness to background noise with a dual-path recurrent neural network noise-reduction method.

Many people with hearing loss struggle to understand speech in noisy environments, making noise robustness critical for hearing-assistive devices. Recently developed haptic hearing aids, which convert audio to vibration, can improve speech-in-noise performance for cochlear implant (CI) users and assist those unable to access hearing-assistive devices. They are typically body-worn rather than head-mounted, allowing additional space for batteries and microprocessors, and so can deploy more sophisticated noise-reduction techniques. The current study assessed whether a real-time-feasible dual-path recurrent neural network (DPRNN) can improve tactile speech-in-noise performance. Audio was converted to vibration on the wrist using a vocoder method, either with or without noise reduction. Performance was tested for speech in a multi-talker noise (recorded at a party) with a 2.5-dB signal-to-noise ratio. An objective assessment showed the DPRNN improved the scale-invariant signal-to-distortion ratio by 8.6 dB and substantially outperformed traditional noise-reduction (log-MMSE). A behavioural assessment in 16 participants showed the DPRNN improved tactile-only sentence identification in noise by 8.2%. This suggests that advanced techniques like the DPRNN could substantially improve outcomes with haptic hearing aids. Low-cost haptic devices could soon be an important supplement to hearing-assistive devices such as CIs or offer an alternative for people who cannot access CI technology.

Improved tactile speech perception using audio-to-tactile sensory substitution with formant frequency focusing.

Haptic hearing aids, which provide speech information through tactile stimulation, could substantially improve outcomes for both cochlear implant users and for those unable to access cochlear implants. Recent advances in wide-band haptic actuator technology have made new audio-to-tactile conversion strategies viable for wearable devices. One such strategy filters the audio into eight frequency bands, which are evenly distributed across the speech frequency range. The amplitude envelopes from the eight bands modulate the amplitudes of eight low-frequency tones, which are delivered through vibration to a single site on the wrist. This tactile vocoder strategy effectively transfers some phonemic information, but vowels and obstruent consonants are poorly portrayed. In 20 participants with normal touch perception, we tested (1) whether focusing the audio filters of the tactile vocoder more densely around the first and second formant frequencies improved tactile vowel discrimination, and (2) whether focusing filters at mid-to-high frequencies improved obstruent consonant discrimination. The obstruent-focused approach was found to be ineffective. However, the formant-focused approach improved vowel discrimination by 8%, without changing overall consonant discrimination. The formant-focused tactile vocoder strategy, which can readily be implemented in real time on a compact device, could substantially improve speech perception for haptic hearing aid users.

Improving speech perception for hearing-impaired listeners using audio-to-tactile sensory substitution with multiple frequency channels.

Cochlear implants (CIs) have revolutionised treatment of hearing loss, but large populations globally cannot access them either because of disorders that prevent implantation or because they are expensive and require specialist surgery. Recent technology developments mean that haptic aids, which transmit speech through vibration, could offer a viable low-cost, non-invasive alternative. One important development is that compact haptic actuators can now deliver intense stimulation across multiple frequencies. We explored whether these multiple frequency channels can transfer spectral information to improve tactile phoneme discrimination. To convert audio to vibration, the speech amplitude envelope was extracted from one or more audio frequency bands and used to amplitude modulate one or more vibro-tactile tones delivered to a single-site on the wrist. In 26 participants with normal touch sensitivity, tactile-only phoneme discrimination was assessed with one, four, or eight frequency bands. Compared to one frequency band, performance improved by 5.9% with four frequency bands and by 8.4% with eight frequency bands. The multi-band signal-processing approach can be implemented in real-time on a compact device, and the vibro-tactile tones can be reproduced by the latest compact, low-powered actuators. This approach could therefore readily be implemented in a low-cost haptic hearing aid to deliver real-world benefits.

Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?

Cochlear implants (CIs) have been remarkably successful at restoring hearing in severely-to-profoundly hearing-impaired individuals. However, users often struggle to deconstruct complex auditory scenes with multiple simultaneous sounds, which can result in reduced music enjoyment and impaired speech understanding in background noise. Hearing aid users often have similar issues, though these are typically less acute. Several recent studies have shown that haptic stimulation can enhance CI listening by giving access to sound features that are poorly transmitted through the electrical CI signal. This "electro-haptic stimulation" improves melody recognition and pitch discrimination, as well as speech-in-noise performance and sound localization. The success of this approach suggests it could also enhance auditory perception in hearing-aid users and other hearing-impaired listeners. This review focuses on the use of haptic stimulation to enhance music perception in hearing-impaired listeners. Music is prevalent throughout everyday life, being critical to media such as film and video games, and often being central to events such as weddings and funerals. It represents the biggest challenge for signal processing, as it is typically an extremely complex acoustic signal, containing multiple simultaneous harmonic and inharmonic sounds. Signal-processing approaches developed for enhancing music perception could therefore have significant utility for other key issues faced by hearing-impaired listeners, such as understanding speech in noisy environments. This review first discusses the limits of music perception in hearing-impaired listeners and the limits of the tactile system. It then discusses the evidence around integration of audio and haptic stimulation in the brain. Next, the features, suitability, and success of current haptic devices for enhancing music perception are reviewed, as well as the signal-processing approaches that could be deployed in future haptic devices. Finally, the cutting-edge technologies that could be exploited for enhancing music perception with haptics are discussed. These include the latest micro motor and driver technology, low-power wireless technology, machine learning, big data, and cloud computing. New approaches for enhancing music perception in hearing-impaired listeners could substantially improve quality of life. Furthermore, effective haptic techniques for providing complex sound information could offer a non-invasive, affordable means for enhancing listening more broadly in hearing-impaired individuals.

Sensitivity to Haptic Sound-Localization Cues at Different Body Locations.

Cochlear implants (CIs) recover hearing in severely to profoundly hearing-impaired people by electrically stimulating the cochlea. While they are extremely effective, spatial hearing is typically severely limited. Recent studies have shown that haptic stimulation can supplement the electrical CI signal (electro-haptic stimulation) and substantially improve sound localization. In haptic sound-localization studies, the signal is extracted from the audio received by behind-the-ear devices and delivered to each wrist. Localization is achieved using tactile intensity differences (TIDs) across the wrists, which match sound intensity differences across the ears (a key sound localization cue). The current study established sensitivity to across-limb TIDs at three candidate locations for a wearable haptic device, namely: the lower tricep and the palmar and dorsal wrist. At all locations, TID sensitivity was similar to the sensitivity to across-ear intensity differences for normal-hearing listeners. This suggests that greater haptic sound-localization accuracy than previously shown can be achieved. The dynamic range was also measured and far exceeded that available through electrical CI stimulation for all of the locations, suggesting that haptic stimulation could provide additional sound-intensity information. These results indicate that an effective haptic aid could be deployed for any of the candidate locations, and could offer a low-cost, non-invasive means of improving outcomes for hearing-impaired listeners.

Electro-Haptic Stimulation: A New Approach for Improving Cochlear-Implant Listening.

Cochlear implants (CIs) have been remarkably successful at restoring speech perception for severely to profoundly deaf individuals. Despite their success, several limitations remain, particularly in CI users' ability to understand speech in noisy environments, locate sound sources, and enjoy music. A new multimodal approach has been proposed that uses haptic stimulation to provide sound information that is poorly transmitted by the implant. This augmenting of the electrical CI signal with haptic stimulation (electro-haptic stimulation; EHS) has been shown to improve speech-in-noise performance and sound localization in CI users. There is also evidence that it could enhance music perception. We review the evidence of EHS enhancement of CI listening and discuss key areas where further research is required. These include understanding the neural basis of EHS enhancement, understanding the effectiveness of EHS across different clinical populations, and the optimization of signal-processing strategies. We also discuss the significant potential for a new generation of haptic neuroprosthetic devices to aid those who cannot access hearing-assistive technology, either because of biomedical or healthcare-access issues. While significant further research and development is required, we conclude that EHS represents a promising new approach that could, in the near future, offer a non-invasive, inexpensive means of substantially improving clinical outcomes for hearing-impaired individuals.

Sensitivity to haptic sound-localisation cues.

Hearing aid and cochlear implant (CI) users often struggle to locate and segregate sounds. The dominant sound-localisation cues are time and intensity differences across the ears. A recent study showed that CI users locate sounds substantially better when these cues are provided through haptic stimulation on each wrist. However, the sensitivity of the wrists to these cues and the robustness of this sensitivity to aging is unknown. The current study showed that time difference sensitivity is much poorer across the wrists than across the ears and declines with age. In contrast, high sensitivity to across-wrist intensity differences was found that was robust to aging. This high sensitivity was observed across a range of stimulation intensities for both amplitude modulated and unmodulated sinusoids and matched across-ear intensity difference sensitivity for normal-hearing individuals. Furthermore, the usable dynamic range for haptic stimulation on the wrists was found to be around four times larger than for CIs. These findings suggest that high-precision haptic sound-localisation can be achieved, which could aid many hearing-impaired listeners. Furthermore, the finding that high-fidelity across-wrist intensity information can be transferred could be exploited in human-machine interfaces to enhance virtual reality and improve remote control of military, medical, or research robots.

A Compact Two-Loudspeaker Virtual Sound Reproduction System for Clinical Testing of Spatial Hearing With Hearing-Assistive Devices.

Exciting developments in hearing aid and cochlear implant technology for linking signal processing across the ears have improved spatial hearing outcomes. This has resulted in an increased emphasis on clinical assessment of the spatial hearing abilities of hearing-assistive device users. Effective assessment of spatial hearing currently requires a large and costly loudspeaker array system, housed in a heavily acoustically treated testing room. This imposes economic and logistical constraints that limit proliferation of array systems, particularly in developing nations. Despite their size and cost, the ability of current clinical array systems to reproduce realistic spatial sound fields is limited, which substantially reduces the range of realistic acoustic scenes that can be used for diagnostic testing. We propose an alternative low-cost, compact virtual acoustics system with just two loudspeakers. This system uses crosstalk cancelation to reproduce pressure signals at the device microphones that match those for real-world sound sources. Furthermore, in contrast to clinical array systems, the system can adapt to different room acoustics, removing the requirement for a heavily acoustically treated testing environment. We conducted a proof-of-concept study in two stages: in the first, we evaluated the physical performance of the system for a stationary listener in anechoic conditions and in a small audiological testing booth with moderate acoustic treatment. To do this, a head and torso simulator was fitted with specially adapted hearing-assistive devices that allowed direct access to the microphone signals. These microphone signals were compared for real and virtual sound sources at numerous source locations. In the second stage, we quantified the system's robustness to head rotations with and without the system adapting for head position. In the stationary case, the system was found to be highly effective at reproducing signals, such as speech, at all tested source locations. When head rotation was added, it performed well for rotations of up to 2°, even without adapting. However, performance improved markedly for larger rotations when the system adapted. These findings suggest that a compact, low-cost virtual acoustics system can give wider access to advanced and ecologically valid audiological testing, which could substantially improve clinical assessment of hearing-assistive device users.

Using haptic stimulation to enhance auditory perception in hearing-impaired listeners.

INTRODUCTION: Hearing-assistive devices, such as hearing aids and cochlear implants, transform the lives of hearing-impaired people. However, users often struggle to locate and segregate sounds. This leads to impaired threat detection and an inability to understand speech in noisy environments. Recent evidence suggests that segregation and localization can be improved by providing missing sound-information through haptic stimulation. AREAS COVERED: This article reviews the evidence that haptic stimulation can effectively provide sound information. It then discusses the research and development required for this approach to be implemented in a clinically viable device. This includes discussion of what sound information should be provided and how that information can be extracted and delivered. EXPERT OPINION: Although this research area has only recently emerged, it builds on a significant body of work showing that sound information can be effectively transferred through haptic stimulation. Current evidence suggests that haptic stimulation is highly effective at providing missing sound-information to cochlear implant users. However, a great deal of work remains to implement this approach in an effective wearable device. If successful, such a device could offer an inexpensive, noninvasive means of improving educational, work, and social experiences for hearing-impaired individuals, including those without access to hearing-assistive devices.

Haptic sound-localisation for use in cochlear implant and hearing-aid users.

Users of hearing-assistive devices often struggle to locate and segregate sounds, which can make listening in schools, cafes, and busy workplaces extremely challenging. A recent study in unilaterally implanted CI users showed that sound-localisation was improved when the audio received by behind-the-ear devices was converted to haptic stimulation on each wrist. We built on this work, using a new signal-processing approach to improve localisation accuracy and increase generalisability to a wide range of stimuli. We aimed to: (1) improve haptic sound-localisation accuracy using a varied stimulus set and (2) assess whether accuracy improved with prolonged training. Thirty-two adults with normal touch perception were randomly assigned to an experimental or control group. The experimental group completed a 5-h training regime and the control group were not trained. Without training, haptic sound-localisation was substantially better than in previous work on haptic sound-localisation. It was also markedly better than sound-localisation by either unilaterally or bilaterally implanted CI users. After training, accuracy improved, becoming better than for sound-localisation by bilateral hearing-aid users. These findings suggest that a wrist-worn haptic device could be effective for improving spatial hearing for a range of hearing-impaired listeners.

Electro-haptic stimulation enhances speech recognition in spatially separated noise for cochlear implant users.

Hundreds of thousands of profoundly hearing-impaired people perceive sounds through electrical stimulation of the auditory nerve using a cochlear implant (CI). However, CI users are often poor at understanding speech in noisy environments and separating sounds that come from different locations. We provided missing speech and spatial hearing cues through haptic stimulation to augment the electrical CI signal. After just 30 min of training, we found this "electro-haptic" stimulation substantially improved speech recognition in multi-talker noise when the speech and noise came from different locations. Our haptic stimulus was delivered to the wrists at an intensity that can be produced by a compact, low-cost, wearable device. These findings represent a significant step towards the production of a non-invasive neuroprosthetic that can improve CI users' ability to understand speech in realistic noisy environments.

Enhanced Pitch Discrimination for Cochlear Implant Users with a New Haptic Neuroprosthetic.

The cochlear implant (CI) is the most widely used neuroprosthesis, recovering hearing for more than half a million severely-to-profoundly hearing-impaired people. However, CIs still have significant limitations, with users having severely impaired pitch perception. Pitch is critical to speech understanding (particularly in noise), to separating different sounds in complex acoustic environments, and to music enjoyment. In recent decades, researchers have attempted to overcome shortcomings in CIs by improving implant technology and surgical techniques, but with limited success. In the current study, we take a new approach of providing missing pitch information through haptic stimulation on the forearm, using our new mosaicOne_B device. The mosaicOne_B extracts pitch information in real-time and presents it via 12 motors that are arranged in ascending pitch along the forearm, with each motor representing a different pitch. In normal-hearing subjects listening to CI simulated audio, we showed that participants were able to discriminate pitch differences at a similar performance level to that achieved by normal-hearing listeners. Furthermore, the device was shown to be highly robust to background noise. This enhanced pitch discrimination has the potential to significantly improve music perception, speech recognition, and speech prosody perception in CI users.

Electro-Haptic Enhancement of Spatial Hearing in Cochlear Implant Users.

Cochlear implants (CIs) have enabled hundreds of thousands of profoundly hearing-impaired people to perceive sounds by electrically stimulating the auditory nerve. However, CI users are often very poor at locating sounds, which leads to impaired sound segregation and threat detection. We provided missing spatial hearing cues through haptic stimulation to augment the electrical CI signal. We found that this "electro-haptic" stimulation dramatically improved sound localisation. Furthermore, participants were able to effectively integrate spatial information transmitted through these two senses, performing better with combined audio and haptic stimulation than with either alone. Our haptic signal was presented to the wrists and could readily be delivered by a low-cost wearable device. This approach could provide a non-invasive means of improving outcomes for the vast majority of CI users who have only one implant, without the expense and risk of a second implantation.

Electro-haptic enhancement of speech-in-noise performance in cochlear implant users.

Cochlear implant (CI) users receive only limited sound information through their implant, which means that they struggle to understand speech in noisy environments. Recent work has suggested that combining the electrical signal from the CI with a haptic signal that provides crucial missing sound information ("electro-haptic stimulation"; EHS) could improve speech-in-noise performance. The aim of the current study was to test whether EHS could enhance speech-in-noise performance in CI users using: (1) a tactile signal derived using an algorithm that could be applied in real time, (2) a stimulation site appropriate for a real-world application, and (3) a tactile signal that could readily be produced by a compact, portable device. We measured speech intelligibility in multi-talker noise with and without vibro-tactile stimulation of the wrist in CI users, before and after a short training regime. No effect of EHS was found before training, but after training EHS was found to improve the number of words correctly identified by an average of 8.3%-points, with some users improving by more than 20%-points. Our approach could offer an inexpensive and non-invasive means of improving speech-in-noise performance in CI users.

Measurements of ultrasonic deterrents and an acoustically branded hairdryer: Ambiguities in guideline compliance.

Acoustic radiation from three commercial pest deterrents and two hair dryers were measured in an anechoic chamber. The deterrents were chosen because the frequency range at which they emit the most energy is either in the very high-frequency sound band (11.2-17.8 kHz) or the ultrasound band (greater than 17.8 kHz). These are sources that may be heard by a subset of the general population, with the young typically having better high frequency sensitivity. A hairdryer reported to increase the frequency of the motor noise above the audible hearing range was compared with a standard hairdryer. The outputs of the deterrents are compared against six international regulations and guidelines for audible and ultrasound exposure. Multiple ambiguities in the application of these guidelines are discussed. These ambiguities could lead to a device being considered as in compliance despite unconventionally high levels. Even if a device measured here meets a guideline, actual exposures can exceed those taken here and may therefore breach guidelines if the listener is closer to the device or reflections increase the exposure level.

Public exposure to ultrasound and very high-frequency sound in air.

Recent work showing the presence of a new generation of ultrasound (US) sources in public places has reopened the debate about whether there are adverse effects of US on humans, and has identified weaknesses in standards and exposure guidelines. Systems that rely on very high-frequency sound (VHFS) and US include public-address voice-alarm (PAVA) systems (whose operational status is often monitored using tones at ∼20 kHz) and pest deterrents. In this study, sound pressure levels (SPLs) produced by 16 sources that were either publically available or installed in busy public spaces were measured. These sources were identified through a citizen science project, wherein members of the public were asked to provide smartphone recordings of VHFS/US sources. With measurements made in realistic listening positions, pest deterrents were found that produced levels of up to 100 dB SPL at ∼20 kHz, and a hand dryer was found to produce 84 dB SPL at 40 kHz. PAVA systems were found to emit lower levels of up to 76 dB SPL at ∼20 kHz. Pest deterrents measured breach recommended safe listening limits for public exposure for people who are nearby even for relatively short periods.

Effects of very high-frequency sound and ultrasound on humans. Part II: A double-blind randomized provocation study of inaudible 20-kHz ultrasound.

Some people have reported symptoms such as nausea, dizziness, and headaches that they attribute to ultrasound (US) emitted by devices in public places. The primary aim of the present study was to investigate whether inaudible US can provoke adverse symptoms compared to a sham presentation, under double-blind conditions. A second aim was to investigate whether the expectation of US being present could provoke adverse symptoms (a nocebo response). The US stimulus was a 20 kHz tone presented continuously for 20 min set to at least 15 dB below the participants' detection threshold, giving a typical sound pressure level (SPL) of 84 dB. No evidence that US provoked symptoms was found, but there was evidence of small nocebo effects. A case study on an individual with high self-reported sensitivity to US gave similar results. The present study did not reproduce the severe symptoms reported previously by some members of the public; this may be due to the SPL or duration of the stimulus, or strength of the nocebo stimulus. These findings cannot be used to predict outcomes from exposures to sounds that are audible to the individual in question, or to sounds with higher SPLs, longer durations, or different frequency content.

Effects of very high-frequency sound and ultrasound on humans. Part I: Adverse symptoms after exposure to audible very-high frequency sound.

Various adverse symptoms resulting from exposure to very high-frequency sound (VHFS) and ultrasound (US) have previously been reported. This study aimed to establish whether these symptoms are experienced under controlled laboratory conditions and are specific to VHFS/US. To do this, participants were exposed to VHFS/US (at frequencies between 13.5 and 20 kHz and sound pressure levels between 82 and 92 dB) and to a 1 kHz reference stimulus, both at 25 dB above their hearing threshold. The VHFS/US and reference stimuli were presented 4 times, each time for 3 min, during which participants performed a sustained attention task, rated their symptom severity, and had their galvanic skin response (GSR) measured to assess their level of anxiety. Prior to exposure, participants were assigned either to a symptomatic or an asymptomatic group, based on their prior history of symptoms that they attributed to VHFS/US. In both groups, overall discomfort ratings were higher in the VHFS/US condition than the reference condition. In the symptomatic group only, difficulty concentrating and annoyance were also rated higher in the VHFS/US than the reference condition. No difference between the two stimulus conditions was seen in performance on the attention task or on average GSRs for either group.

Vibro-Tactile Enhancement of Speech Intelligibility in Multi-talker Noise for Simulated Cochlear Implant Listening.

Many cochlear implant (CI) users achieve excellent speech understanding in acoustically quiet conditions but most perform poorly in the presence of background noise. An important contributor to this poor speech-in-noise performance is the limited transmission of low-frequency sound information through CIs. Recent work has suggested that tactile presentation of this low-frequency sound information could be used to improve speech-in-noise performance for CI users. Building on this work, we investigated whether vibro-tactile stimulation can improve speech intelligibility in multi-talker noise. The signal used for tactile stimulation was derived from the speech-in-noise using a computationally inexpensive algorithm. Eight normal-hearing participants listened to CI simulated speech-in-noise both with and without concurrent tactile stimulation of their fingertip. Participants' speech recognition performance was assessed before and after a training regime, which took place over 3 consecutive days and totaled around 30 min of exposure to CI-simulated speech-in-noise with concurrent tactile stimulation. Tactile stimulation was found to improve the intelligibility of speech in multi-talker noise, and this improvement was found to increase in size after training. Presentation of such tactile stimulation could be achieved by a compact, portable device and offer an inexpensive and noninvasive means for improving speech-in-noise performance in CI users.

Effect of Contralateral Medial Olivocochlear Feedback on Perceptual Estimates of Cochlear Gain and Compression.

The active cochlear mechanism amplifies responses to low-intensity sounds, compresses the range of input sound intensities to a smaller output range, and increases cochlear frequency selectivity. The gain of the active mechanism can be modulated by the medial olivocochlear (MOC) efferent system, creating the possibility of top-down control at the earliest level of auditory processing. In humans, MOC function has mostly been measured by the suppression of otoacoustic emissions (OAEs), typically as a result of MOC activation by a contralateral elicitor sound. The exact relationship between OAE suppression and cochlear gain reduction, however, remains unclear. Here, we measured the effect of a contralateral MOC elicitor on perceptual estimates of cochlear gain and compression, obtained using the established temporal masking curve (TMC) method. The measurements were taken at a signal frequency of 2 kHz and compared with measurements of click-evoked OAE suppression. The elicitor was a broadband noise, set to a sound pressure level of 54 dB to avoid triggering the middle ear muscle reflex. Despite its low level, the elicitor had a significant effect on the TMCs, consistent with a reduction in cochlear gain. The amount of gain reduction was estimated as 4.4 dB on average, corresponding to around 18 % of the without-elicitor gain. As a result, the compression exponent increased from 0.18 to 0.27.