The neural representation of an auditory spatial cue in the primate cortex.

Humans make use of small differences in the timing of sounds at the two ears-interaural time differences (ITDs)-to locate their sources. Despite extensive investigation, however, the neural representation of ITDs in the human brain is contentious, particularly the range of ITDs explicitly represented by dedicated neural detectors. Here, using magneto- and electro-encephalography (MEG and EEG), we demonstrate evidence of a sparse neural representation of ITDs in the human cortex. The magnitude of cortical activity to sounds presented via insert earphones oscillated as a function of increasing ITD-within and beyond auditory cortical regions-and listeners rated the perceptual quality of these sounds according to the same oscillating pattern. This pattern was accurately described by a population of model neurons with preferred ITDs constrained to the narrow, sound-frequency-dependent range evident in other mammalian species. When scaled for head size, the distribution of ITD detectors in the human cortex is remarkably like that recorded in vivo from the cortex of rhesus monkeys, another large primate that uses ITDs for source localization. The data solve a long-standing issue concerning the neural representation of ITDs in humans and suggest a representation that scales for head size and sound frequency in an optimal manner.

Functional connectivity learning via Siamese-based SPD matrix representation of brain imaging data.

Measurement of brain functional connectivity has become a dominant approach to explore the interaction dynamics between brain regions of subjects under examination. Conventional functional connectivity measures largely originate from deterministic models on empirical analysis, usually demanding application-specific settings (e.g., Pearson's Correlation and Mutual Information). To bridge the technical gap, this study proposes a Siamese-based Symmetric Positive Definite (SPD) Matrix Representation framework (SiameseSPD-MR) to derive the functional connectivity of brain imaging data (BID) such as Electroencephalography (EEG), thus the alternative application-independent measure (in the form of SPD matrix) can be automatically learnt: (1) SiameseSPD-MR first exploits graph convolution to extract the representative features of BID with the adjacency matrix computed considering the anatomical structure; (2) Adaptive Gaussian kernel function then applies to obtain the functional connectivity representations from the deep features followed by SPD matrix transformation to address the intrinsic functional characteristics; and (3) Two-branch (Siamese) networks are combined via an element-wise product followed by a dense layer to derive the similarity between the pairwise inputs. Experimental results on two EEG datasets (autism spectrum disorder, emotion) indicate that (1) SiameseSPD-MR can capture more significant differences in functional connectivity between neural states than the state-of-the-art counterparts do, and these findings properly highlight the typical EEG characteristics of ASD subjects, and (2) the obtained functional connectivity representations conforming to the proposed measure can act as meaningful markers for brain network analysis and ASD discrimination.

Disentangled and Side-Aware Unsupervised Domain Adaptation for Cross-Dataset Subjective Tinnitus Diagnosis.

EEG-based tinnitus classification is a valuable tool for tinnitus diagnosis, research, and treatments. Most current works are limited to a single dataset where data patterns are similar. But EEG signals are highly non-stationary, resulting in model's poor generalization to new users, sessions or datasets. Thus, designing a model that can generalize to new datasets is beneficial and indispensable. To mitigate distribution discrepancy across datasets, we propose to achieve Disentangled and Side-aware Unsupervised Domain Adaptation (DSUDA) for cross-dataset tinnitus diagnosis. A disentangled auto-encoder is developed to decouple class-irrelevant information from the EEG signals to improve the classifying ability. The side-aware unsupervised domain adaptation module adapts the class-irrelevant information as domain variance to a new dataset and excludes the variance to obtain the class-distill features for the new dataset classification. It also aligns signals of left and right ears to overcome inherent EEG pattern difference. We compare DSUDA with state-of-the-art methods, and our model achieves significant improvements over competitors regarding comprehensive evaluation criteria. The results demonstrate our model can successfully generalize to a new dataset and effectively diagnose tinnitus.

Cross-modal functional connectivity supports speech understanding in cochlear implant users.

Sensory deprivation can lead to cross-modal cortical changes, whereby sensory brain regions deprived of input may be recruited to perform atypical function. Enhanced cross-modal responses to visual stimuli observed in auditory cortex of postlingually deaf cochlear implant (CI) users are hypothesized to reflect increased activation of cortical language regions, but it is unclear if this cross-modal activity is "adaptive" or "mal-adaptive" for speech understanding. To determine if increased activation of language regions is correlated with better speech understanding in CI users, we assessed task-related activation and functional connectivity of auditory and visual cortices to auditory and visual speech and non-speech stimuli in CI users (n = 14) and normal-hearing listeners (n = 17) and used functional near-infrared spectroscopy to measure hemodynamic responses. We used visually presented speech and non-speech to investigate neural processes related to linguistic content and observed that CI users show beneficial cross-modal effects. Specifically, an increase in connectivity between the left auditory and visual cortices-presumed primary sites of cortical language processing-was positively correlated with CI users' abilities to understand speech in background noise. Cross-modal activity in auditory cortex of postlingually deaf CI users may reflect adaptive activity of a distributed, multimodal speech network, recruited to enhance speech understanding.

Side-Aware Meta-Learning for Cross-Dataset Listener Diagnosis With Subjective Tinnitus.

With the development of digital technology, machine learning has paved the way for the next generation of tinnitus diagnoses. Although machine learning has been widely applied in EEG-based tinnitus analysis, most current models are dataset-specific. Each dataset may be limited to a specific range of symptoms, overall disease severity, and demographic attributes; further, dataset formats may differ, impacting model performance. This paper proposes a side-aware meta-learning for cross-dataset tinnitus diagnosis, which can effectively classify tinnitus in subjects of divergent ages and genders from different data collection processes. Owing to the superiority of meta-learning, our method does not rely on large-scale datasets like conventional deep learning models. Moreover, we design a subject-specific training process to assist the model in fitting the data pattern of different patients or healthy people. Our method achieves a high accuracy of 73.8% in the cross-dataset classification. We conduct an extensive analysis to show the effectiveness of side information of ears in enhancing model performance and side-aware meta-learning in improving the quality of the learned features.

Deep reinforcement learning guided graph neural networks for brain network analysis.

Modern neuroimaging techniques enable us to construct human brains as brain networks or connectomes. Capturing brain networks' structural information and hierarchical patterns is essential for understanding brain functions and disease states. Recently, the promising network representation learning capability of graph neural networks (GNNs) has prompted related methods for brain network analysis to be proposed. Specifically, these methods apply feature aggregation and global pooling to convert brain network instances into vector representations encoding brain structure induction for downstream brain network analysis tasks. However, existing GNN-based methods often neglect that brain networks of different subjects may require various aggregation iterations and use GNN with a fixed number of layers to learn all brain networks. Therefore, how to fully release the potential of GNNs to promote brain network analysis is still non-trivial. In our work, a novel brain network representation framework, BN-GNN, is proposed to solve this difficulty, which searches for the optimal GNN architecture for each brain network. Concretely, BN-GNN employs deep reinforcement learning (DRL) to automatically predict the optimal number of feature propagations (reflected in the number of GNN layers) required for a given brain network. Furthermore, BN-GNN improves the upper bound of traditional GNNs' performance in eight brain network disease analysis tasks.

Modulation Depth Discrimination by Cochlear Implant Users.

Cochlear implants (CIs) convey the amplitude envelope of speech by modulating high-rate pulse trains. However, not all of the envelope may be necessary to perceive amplitude modulations (AMs); the effective envelope depth may be limited by forward and backward masking from the envelope peaks. Three experiments used modulated pulse trains to measure which portions of the envelope can be effectively processed by CI users as a function of AM frequency. Experiment 1 used a three-interval forced-choice task to test the ability of CI users to discriminate less-modulated pulse trains from a fully modulated standard, without controlling for loudness. The stimuli in experiment 2 were identical, but a two-interval task was used in which participants were required to choose the less-modulated interval, ignoring loudness. Catch trials, in which judgements based on level or modulation depth would give opposing answers, were included. Experiment 3 employed novel stimuli whose modulation envelope could be modified below a variable point in the dynamic range, without changing the loudness of the stimulus. Overall, results showed that substantial portions of the envelope are not accurately encoded by CI users. In experiment 1, where loudness cues were available, participants on average were insensitive to changes in the bottom 30% of their dynamic range. In experiment 2, where loudness was controlled, participants appeared insensitive to changes in the bottom 50% of the dynamic range. In experiment 3, participants were insensitive to changes in the bottom 80% of the dynamic range. We discuss potential reasons for this insensitivity and implications for CI speech-processing strategies.

Understanding degraded speech leads to perceptual gating of a brainstem reflex in human listeners.

The ability to navigate "cocktail party" situations by focusing on sounds of interest over irrelevant, background sounds is often considered in terms of cortical mechanisms. However, subcortical circuits such as the pathway underlying the medial olivocochlear (MOC) reflex modulate the activity of the inner ear itself, supporting the extraction of salient features from auditory scene prior to any cortical processing. To understand the contribution of auditory subcortical nuclei and the cochlea in complex listening tasks, we made physiological recordings along the auditory pathway while listeners engaged in detecting non(sense) words in lists of words. Both naturally spoken and intrinsically noisy, vocoded speech-filtering that mimics processing by a cochlear implant (CI)-significantly activated the MOC reflex, but this was not the case for speech in background noise, which more engaged midbrain and cortical resources. A model of the initial stages of auditory processing reproduced specific effects of each form of speech degradation, providing a rationale for goal-directed gating of the MOC reflex based on enhancing the representation of the energy envelope of the acoustic waveform. Our data reveal the coexistence of 2 strategies in the auditory system that may facilitate speech understanding in situations where the signal is either intrinsically degraded or masked by extrinsic acoustic energy. Whereas intrinsically degraded streams recruit the MOC reflex to improve representation of speech cues peripherally, extrinsically masked streams rely more on higher auditory centres to denoise signals.

Generalizable Sample-Efficient Siamese Autoencoder for Tinnitus Diagnosis in Listeners With Subjective Tinnitus.

Electroencephalogram (EEG)-based neurofeedback has been widely studied for tinnitus therapy in recent years. Most existing research relies on experts' cognitive prediction, and studies based on machine learning and deep learning are either data-hungry or not well generalizable to new subjects. In this paper, we propose a robust, data-efficient model for distinguishing tinnitus from the healthy state based on EEG-based tinnitus neurofeedback. We propose trend descriptor, a feature extractor with lower fineness, to reduce the effect of electrode noises on EEG signals, and a siamese encoder-decoder network boosted in a supervised manner to learn accurate alignment and to acquire high-quality transferable mappings across subjects and EEG signal channels. Our experiments show the proposed method significantly outperforms state-of-the-art algorithms when analyzing subjects' EEG neurofeedback to 90dB and 100dB sound, achieving an accuracy of 91.67%-94.44% in predicting tinnitus and control subjects in a subject-independent setting. Our ablation studies on mixed subjects and parameters show the method's stability in performance.

Hidden Hearing Loss Impacts the Neural Representation of Speech in Background Noise.

Many individuals with seemingly normal hearing abilities struggle to understand speech in noisy backgrounds. To understand why this might be the case, we investigated the neural representation of speech in the auditory midbrain of gerbils with "hidden hearing loss" through noise exposure that increased hearing thresholds only temporarily. In noise-exposed animals, we observed significantly increased neural responses to speech stimuli, with a more pronounced increase at moderate than at high sound intensities. Noise exposure reduced discriminability of neural responses to speech in background noise at high sound intensities, with impairment most severe for tokens with relatively greater spectral energy in the noise-exposure frequency range (2-4 kHz). At moderate sound intensities, discriminability was surprisingly improved, which was unrelated to spectral content. A model combining damage to high-threshold auditory nerve fibers with increased response gain of central auditory neurons reproduced these effects, demonstrating that a specific combination of peripheral damage and central compensation could explain listening difficulties despite normal hearing thresholds.

Tolerable delay for speech production and perception: effects of hearing ability and experience with hearing aids.

OBJECTIVE: Processing delay is one of the important factors that limit the development of novel algorithms for hearing devices. In this study, both normal-hearing listeners and listeners with hearing loss were tested for their tolerance of processing delay up to 50 ms using a real-time setup for own-voice and external-voice conditions based on linear processing to avoid confounding effects of time-dependent gain. DESIGN: Participants rated their perceived subjective annoyance for each condition on a 7-point Likert scale. STUDY SAMPLE: Twenty normal-hearing participants and twenty participants with a range of mild to moderate hearing losses. RESULTS: Delay tolerance was significantly greater for the participants with hearing loss in two out of three voice conditions. The average slopes of annoyance ratings were negatively correlated with the degree of hearing loss across participants. A small trend of higher tolerance of delay by experienced users of hearing aids in comparison to new users was not significant. CONCLUSION: The increased tolerance of processing delay for speech production and perception with hearing loss and reduced sensitivity to changes in delay with stronger hearing loss may be beneficial for novel algorithms for hearing devices but the setup used in this study differed from commercial hearing aids.

Influence of envelope waveform on ITD sensitivity of neurons in the auditory midbrain.

Interaural time differences (ITDs) conveyed by the modulated envelopes of high-frequency sounds can serve as a cue for localizing a sound source. Klein-Hennig et al. (J Acoust Soc Am 129: 3856, 2011) demonstrated the envelope attack (the rate at which stimulus energy in the envelope increases) and the duration of the pause (the interval between successive envelope pulses) as important factors affecting sensitivity to envelope ITDs in human listeners. Modulated sounds with rapid attacks and long pauses produce the lowest ITD discrimination thresholds. The duration of the envelope's sustained component (sustain) and the rate at which stimulus energy falls at the offset of the envelope (decay) are only minor factors. We assessed the responses of 71 single neurons, recorded from the midbrains of 15 urethane-anesthetized tri-colored guinea pigs, to envelope shapes in which the four envelope components, i.e., attack, sustain, decay, and pause, were systematically varied. We confirmed the importance of the attack and pause components in generating ITD-sensitive responses. Analysis of neural firing rates demonstrated more neurons (49/71) show ITD sensitivity in response to "damped" stimuli (fast attack and slow decay) compared with "ramped" stimuli (slow attack and fast decay) (14/71). Furthermore, the lowest threshold for the damped stimulus (91 µs) was lower by a factor of 4 than that for the temporally reversed ramped envelope shape (407 µs). The data confirm the importance of fast attacks and optimal pause durations in generating sensitivity to ITDs conveyed in the modulated envelopes of high-frequency sounds and are incompatible with models of ITD processing based on the integration of sound energy over time.NEW & NOTEWORTHY Using single-neuron electrophysiology, we show that the precise shape of a sound's "energy envelope" is a critical factor in determining how well midbrain neurons are able to convey information about auditory spatial cues. Consistent with human behavioral performance, sounds with rapidly rising energy and relatively long intervals between energy bursts are best at conveying spatial information. The data suggest specific sound energy patterns that might best be applied to hearing devices to aid spatial listening.

Auditory inspired machine learning techniques can improve speech intelligibility and quality for hearing-impaired listeners.

Machine-learning based approaches to speech enhancement have recently shown great promise for improving speech intelligibility for hearing-impaired listeners. Here, the performance of three machine-learning algorithms and one classical algorithm, Wiener filtering, was compared. Two algorithms based on neural networks were examined, one using a previously reported feature set and one using a feature set derived from an auditory model. The third machine-learning approach was a dictionary-based sparse-coding algorithm. Speech intelligibility and quality scores were obtained for participants with mild-to-moderate hearing impairments listening to sentences in speech-shaped noise and multi-talker babble following processing with the algorithms. Intelligibility and quality scores were significantly improved by each of the three machine-learning approaches, but not by the classical approach. The largest improvements for both speech intelligibility and quality were found by implementing a neural network using the feature set based on auditory modeling. Furthermore, neural network based techniques appeared more promising than dictionary-based, sparse coding in terms of performance and ease of implementation.

Speech enhancement based on neural networks improves speech intelligibility in noise for cochlear implant users.

Speech understanding in noisy environments is still one of the major challenges for cochlear implant (CI) users in everyday life. We evaluated a speech enhancement algorithm based on neural networks (NNSE) for improving speech intelligibility in noise for CI users. The algorithm decomposes the noisy speech signal into time-frequency units, extracts a set of auditory-inspired features and feeds them to the neural network to produce an estimation of which frequency channels contain more perceptually important information (higher signal-to-noise ratio, SNR). This estimate is used to attenuate noise-dominated and retain speech-dominated CI channels for electrical stimulation, as in traditional n-of-m CI coding strategies. The proposed algorithm was evaluated by measuring the speech-in-noise performance of 14 CI users using three types of background noise. Two NNSE algorithms were compared: a speaker-dependent algorithm, that was trained on the target speaker used for testing, and a speaker-independent algorithm, that was trained on different speakers. Significant improvements in the intelligibility of speech in stationary and fluctuating noises were found relative to the unprocessed condition for the speaker-dependent algorithm in all noise types and for the speaker-independent algorithm in 2 out of 3 noise types. The NNSE algorithms used noise-specific neural networks that generalized to novel segments of the same noise type and worked over a range of SNRs. The proposed algorithm has the potential to improve the intelligibility of speech in noise for CI users while meeting the requirements of low computational complexity and processing delay for application in CI devices.

A method to enhance the use of interaural time differences for cochlear implants in reverberant environments.

The ability of normal-hearing (NH) listeners to exploit interaural time difference (ITD) cues conveyed in the modulated envelopes of high-frequency sounds is poor compared to ITD cues transmitted in the temporal fine structure at low frequencies. Sensitivity to envelope ITDs is further degraded when envelopes become less steep, when modulation depth is reduced, and when envelopes become less similar between the ears, common factors when listening in reverberant environments. The vulnerability of envelope ITDs is particularly problematic for cochlear implant (CI) users, as they rely on information conveyed by slowly varying amplitude envelopes. Here, an approach to improve access to envelope ITDs for CIs is described in which, rather than attempting to reduce reverberation, the perceptual saliency of cues relating to the source is increased by selectively sharpening peaks in the amplitude envelope judged to contain reliable ITDs. Performance of the algorithm with room reverberation was assessed through simulating listening with bilateral CIs in headphone experiments with NH listeners. Relative to simulated standard CI processing, stimuli processed with the algorithm generated lower ITD discrimination thresholds and increased extents of laterality. Depending on parameterization, intelligibility was unchanged or somewhat reduced. The algorithm has the potential to improve spatial listening with CIs.

Sensitivity to Envelope Interaural Time Differences at High Modulation Rates.

Sensitivity to interaural time differences (ITDs) conveyed in the temporal fine structure of low-frequency tones and the modulated envelopes of high-frequency sounds are considered comparable, particularly for envelopes shaped to transmit similar fidelity of temporal information normally present for low-frequency sounds. Nevertheless, discrimination performance for envelope modulation rates above a few hundred Hertz is reported to be poor-to the point of discrimination thresholds being unattainable-compared with the much higher (>1,000 Hz) limit for low-frequency ITD sensitivity, suggesting the presence of a low-pass filter in the envelope domain. Further, performance for identical modulation rates appears to decline with increasing carrier frequency, supporting the view that the low-pass characteristics observed for envelope ITD processing is carrier-frequency dependent. Here, we assessed listeners' sensitivity to ITDs conveyed in pure tones and in the modulated envelopes of high-frequency tones. ITD discrimination for the modulated high-frequency tones was measured as a function of both modulation rate and carrier frequency. Some well-trained listeners appear able to discriminate ITDs extremely well, even at modulation rates well beyond 500 Hz, for 4-kHz carriers. For one listener, thresholds were even obtained for a modulation rate of 800 Hz. The highest modulation rate for which thresholds could be obtained declined with increasing carrier frequency for all listeners. At 10 kHz, the highest modulation rate at which thresholds could be obtained was 600 Hz. The upper limit of sensitivity to ITDs conveyed in the envelope of high-frequency modulated sounds appears to be higher than previously considered.

Factors affecting the use of envelope interaural time differences in reverberation.

At high frequencies, interaural time differences (ITDs) are conveyed by the sound envelope. Sensitivity to envelope ITDs depends crucially on the envelope shape. Reverberation degrades the envelope shape, reducing the modulation depth of the envelope and the slope of its flanks. Reverberation also reduces the envelope interaural coherence (i.e., the similarity of the envelopes at two ears). The current study investigates the extent to which these changes affect sensitivity to envelope ITDs. The first experiment measured ITD discrimination thresholds at low and high frequencies in a simulated room. The stimulus was either a low-frequency narrowband noise or the same noise transposed to a higher frequency. The results suggest that the effect of reverberation on ITD thresholds was multiplicative. Given that the threshold without reverberation was larger for the transposed than for the low-frequency stimulus, this meant that, in absolute terms, the thresholds for the transposed stimulus showed a much greater increase due to reverberation than those for the low-frequency stimulus. Three further experiments indicated that the effect of reverberation on the envelope ITD thresholds was due to the combined effect of the reduction in the envelope modulation depth and slopes, as well as the decrease in the envelope interaural coherence.

Development of social vocalizations in mice.

Adult mice are highly vocal animals, with both males and females vocalizing in same sex and cross sex social encounters. Mouse pups are also highly vocal, producing isolation vocalizations when they are cold or removed from the nest. This study examined patterns in the development of pup isolation vocalizations, and compared these to adult vocalizations. In three litters of CBA/CaJ mice, we recorded isolation vocalizations at ages postnatal day 5 (p5), p7, p9, p11, and p13. Adult vocalizations were obtained in a variety of social situations. Altogether, 28,384 discrete vocal signals were recorded using high-frequency-sensitive equipment and analyzed for syllable type, spectral and temporal features, and the temporal sequencing within bouts. We found that pups produced all but one of the 11 syllable types recorded from adults. The proportions of syllable types changed developmentally, but even the youngest pups produced complex syllables with frequency-time variations. When all syllable types were pooled together for analysis, changes in the peak frequency or the duration of syllables were small, although significant, from p5 through p13. However, individual syllable types showed different, large patterns of change over development, requiring analysis of each syllable type separately. Most adult syllables were substantially lower in frequency and shorter in duration. As pups aged, the complexity of vocal bouts increased, with a greater tendency to switch between syllable types. Vocal bouts from older animals, p13 and adult, had significantly more sequential structure than those from younger mice. Overall, these results demonstrate substantial changes in social vocalizations with age. Future studies are required to identify whether these changes result from developmental processes affecting the vocal tract or control of vocalization, or from vocal learning. To provide a tool for further research, we developed a MATLAB program that generates bouts of vocalizations that correspond to mice of different ages.

A statistical, formant-pattern model for segregating vowel type and vocal-tract length in developmental formant data.

This paper investigates the theoretical basis for estimating vocal-tract length (VTL) from the formant frequencies of vowel sounds. A statistical inference model was developed to characterize the relationship between vowel type and VTL, on the one hand, and formant frequency and vocal cavity size, on the other. The model was applied to two well known developmental studies of formant frequency. The results show that VTL is the major source of variability after vowel type and that the contribution due to other factors like developmental changes in oral-pharyngeal ratio is small relative to the residual measurement noise. The results suggest that speakers adjust the shape of the vocal tract as they grow to maintain a specific pattern of formant frequencies for individual vowels. This formant-pattern hypothesis motivates development of a statistical-inference model for estimating VTL from formant-frequency data. The technique is illustrated using a third developmental study of formant frequencies. The VTLs of the speakers are estimated and used to provide a more accurate description of the complicated relationship between VTL and glottal pulse rate as children mature into adults.