Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception.

Hearing-impaired people often struggle to follow the speech stream of an individual talker in noisy environments. Recent studies show that the brain tracks attended speech and that the attended talker can be decoded from neural data on a single-trial level. This raises the possibility of "neuro-steered" hearing devices in which the brain-decoded intention of a hearing-impaired listener is used to enhance the voice of the attended speaker from a speech separation front-end. So far, methods that use this paradigm have focused on optimizing the brain decoding and the acoustic speech separation independently. In this work, we propose a novel framework called brain-informed speech separation (BISS)1 in which the information about the attended speech, as decoded from the subject's brain, is directly used to perform speech separation in the front-end. We present a deep learning model that uses neural data to extract the clean audio signal that a listener is attending to from a multi-talker speech mixture. We show that the framework can be applied successfully to the decoded output from either invasive intracranial electroencephalography (iEEG) or non-invasive electroencephalography (EEG) recordings from hearing-impaired subjects. It also results in improved speech separation, even in scenes with background noise. The generalization capability of the system renders it a perfect candidate for neuro-steered hearing-assistive devices.

Multiway canonical correlation analysis of brain data.

Brain data recorded with electroencephalography (EEG), magnetoencephalography (MEG) and related techniques often have poor signal-to-noise ratios due to the presence of multiple competing sources and artifacts. A common remedy is to average responses over repeats of the same stimulus, but this is not applicable for temporally extended stimuli that are presented only once (speech, music, movies, natural sound). An alternative is to average responses over multiple subjects that were presented with identical stimuli, but differences in geometry of brain sources and sensors reduce the effectiveness of this solution. Multiway canonical correlation analysis (MCCA) brings a solution to this problem by allowing data from multiple subjects to be fused in such a way as to extract components common to all. This paper reviews the method, offers application examples that illustrate its effectiveness, and outlines the caveats and risks entailed by the method.

Decoding the auditory brain with canonical component analysis.

The relation between a stimulus and the evoked brain response can shed light on perceptual processes within the brain. Signals derived from this relation can also be harnessed to control external devices for Brain Computer Interface (BCI) applications. While the classic event-related potential (ERP) is appropriate for isolated stimuli, more sophisticated "decoding" strategies are needed to address continuous stimuli such as speech, music or environmental sounds. Here we describe an approach based on Canonical Correlation Analysis (CCA) that finds the optimal transform to apply to both the stimulus and the response to reveal correlations between the two. Compared to prior methods based on forward or backward models for stimulus-response mapping, CCA finds significantly higher correlation scores, thus providing increased sensitivity to relatively small effects, and supports classifier schemes that yield higher classification scores. CCA strips the brain response of variance unrelated to the stimulus, and the stimulus representation of variance that does not affect the response, and thus improves observations of the relation between stimulus and response.

Cortical hemispheric asymmetries are present at young ages and further develop into adolescence.

Specialization of the auditory cortices for pure tone listening may develop with age. In adults, the right hemisphere dominates when listening to pure tones and music; we thus hypothesized that (a) asymmetric function between auditory cortices increases with age and (b) this development is specific to tonal rather than broadband/non-tonal stimuli. Cortical responses to tone-bursts and broadband click-trains were recorded by multichannel electroencephalography in young children (5.1 ± 0.8 years old) and adolescents (15.2 ± 1.7 years old) with normal hearing. Peak dipole moments indicating activity strength in right and left auditory cortices were calculated using the Time Restricted, Artefact and Coherence source Suppression (TRACS) beamformer. Monaural click-trains and tone-bursts in young children evoked a dominant response in the contralateral right cortex by left ear stimulation and, similarly, a contralateral left cortex response to click-trains in the right ear. Responses to tone-bursts in the right ear were more bilateral. In adolescents, peak activity dominated in the right cortex in most conditions (tone-bursts from either ear and to clicks from the left ear). Bilateral activity was evoked by right ear click stimulation. Thus, right hemispheric specialization for monaural tonal stimuli begins in children as young as 5 years of age and becomes more prominent by adolescence. These changes were marked by consistent dipole moments in the right auditory cortex with age in contrast to decreases in dipole activity in all other stimulus conditions. Together, the findings reveal increasingly asymmetric function for the two auditory cortices, potentially to support greater cortical specialization with development into adolescence.

Bilateral input protects the cortex from unilaterally-driven reorganization in children who are deaf.

Unilateral hearing in childhood restricts input along the bilateral auditory pathways, possibly causing permanent reorganization. In this study we asked: (i) do the auditory pathways develop abnormally in children who are bilaterally deaf and hear with a unilateral cochlear implant? and (ii) can such differences be reversed by restoring input to the deprived ear? We measured multichannel electroencephalography in 34 children using cochlear implants and seven normal hearing peers. Dipole moments of activity became abnormally high in the auditory cortex contralateral to the first implant as unilateral cochlear implant use exceeded 1.5 years. This resulted in increased lateralization of activity to the auditory cortex contralateral to the stimulated ear and a decline in normal contralateral activity in response to stimulation from the newly implanted ear, corresponding to poorer speech perception. These results reflect an abnormal strengthening of pathways from the stimulated ear in consequence to the loss of contralateral activity including inhibitory processes normally involved in bilateral hearing. Although this reorganization occurred within a fairly short period (∼1.5 years of unilateral hearing), it was not reversed by long-term (3-4 years) bilateral cochlear implant stimulation. In bilateral listeners, effects of side of stimulation were assessed; children with long periods of unilateral cochlear implant use prior to bilateral implantation showed a reduction in normal dominance of contralateral input in the auditory cortex ipsilateral to the stimulated ear, further confirming an abnormal strengthening of pathways from the stimulated ear. By contrast, cortical activity in children using bilateral cochlear implants after limited or no unilateral cochlear implant exposure normally lateralized to the hemisphere contralateral to side of stimulation and retained normal contralateral dominance of auditory input in both hemispheres. Results demonstrate that the immature human auditory cortex reorganizes, potentially permanently, with unilateral stimulation and that bilateral auditory input provided with limited delay can protect the brain from such changes. These results indicate for the first time that there is a sensitive period for bilateral auditory input in human development with implications for functional hearing.

Lateralization of interimplant timing and level differences in children who use bilateral cochlear implants.

OBJECTIVES: Interaural level differences (ILD) and interaural timing differences (ITD) are important cues for locating sounds in space. Adult bilateral cochlear implant (CI) users use ILDs more effectively than ITDs. Few studies investigated the ability of children who use bilateral CIs to make use of these binaural cues. Our working hypothesis was that children using bilateral CIs are able to perceive changes in ITDs and ILDs similar to their normal-hearing (NH) peers. DESIGN: Participants were two groups of children; 19 bilateral implant users (CI) and nine NH children. The children in the CI group had received a second CI after 4.9 +/- 2.8 yrs of unilateral use. Children performed a four alternative forced-choice lateralization task in which they were asked to describe stimuli as coming from the left side, right side, middle of the head, or from both sides simultaneously. Stimuli were 500 msec trains of electrical pulses delivered to apical electrode no. 18 (CI group) or clicks (NH group) presented 11 times per second with either ITDs (0, 400, 1000, or 2000 microsec delay between sides) or level differences (0, 10, or 20 Current Units (CI group) or 0, 10, or 20 dB (NH group) difference between sides). ITDs were presented using current levels that were balanced using left and right electrically evoked brain stem responses. Stimulus levels evoking response amplitudes that were most similar were used. RESULTS: Responses from children in the CI group changed significantly with changes in ILD of bilateral stimuli, but not with changes in ITD. Responses from children in the CI group were significantly different from those in the NH group in three ways. Children in the CI group perceived bilaterally presented electrical pulses: (1) to come from the second implanted side more often than the first, (2) to rarely come from the middle, and (3) to come from both sides of the head simultaneously. Perceived changes in lateralization with ILD changes were correlated with differences in amplitudes of electrically evoked brain stem responses by the left versus right CI. CONCLUSIONS: The results of this study illustrate that children who use bilateral CIs can lateralize stimuli on the basis of level cues, but have difficulty interpreting interimplant timing differences. Perceived lateralization of bilaterally presented stimuli to the second implanted side in many of the stimulus conditions may relate to the use of different device generations between sides. Further differences from normal lateralization responses could be due to abnormal binaural processing, possibly resulting from a period of unilateral hearing before the provision of a second implant or due to insufficiently matched interimplant stimuli. It may be possible to use objective measures such as electrically evoked auditory brain stem responses wave eV amplitudes to provide balanced levels of bilateral stimulation in children who have had no binaural hearing experience.

A Comparison of Regularization Methods in Forward and Backward Models for Auditory Attention Decoding.

The decoding of selective auditory attention from noninvasive electroencephalogram (EEG) data is of interest in brain computer interface and auditory perception research. The current state-of-the-art approaches for decoding the attentional selection of listeners are based on linear mappings between features of sound streams and EEG responses (forward model), or vice versa (backward model). It has been shown that when the envelope of attended speech and EEG responses are used to derive such mapping functions, the model estimates can be used to discriminate between attended and unattended talkers. However, the predictive/reconstructive performance of the models is dependent on how the model parameters are estimated. There exist a number of model estimation methods that have been published, along with a variety of datasets. It is currently unclear if any of these methods perform better than others, as they have not yet been compared side by side on a single standardized dataset in a controlled fashion. Here, we present a comparative study of the ability of different estimation methods to classify attended speakers from multi-channel EEG data. The performance of the model estimation methods is evaluated using different performance metrics on a set of labeled EEG data from 18 subjects listening to mixtures of two speech streams. We find that when forward models predict the EEG from the attended audio, regularized models do not improve regression or classification accuracies. When backward models decode the attended speech from the EEG, regularization provides higher regression and classification accuracies.

Characterization of retentive capacity of the subpericranial pocket in cochlear implants with and without a pedestal.

OBJECTIVES/HYPOTHESIS: To quantify the retentive capacity (RC) of the subpericranial pocket (SpP) in children undergoing cochlear implantation (CI) and measure improvements in RC with the addition of a pedestal to the device base. Retention of a CI in an SpP relies on the integrity of surrounding tissues to determine device position and resist movement from external forces. We hypothesize that device position can be controlled and resistance to movement can be improved with placement of a small pedestal on the base of the CI receiver stimulator. STUDY DESIGN: Analysis of prospectively assembled data. METHODS: Ninety-seven patients (145 devices) underwent CI (48 bilateral, 49 unilateral). Intraoperatively, a force gauge measured the displacement force on a template Nucleus 5 (Cochlear Corporation, Sydney, Australia) implant placed in an SpP prior to routine suture fixation of a standard device. In 47 patients (64 devices), displacement forces were also measured for a custom template Nucleus 5 implant with pedestal. RESULTS: Average RC of the SpP for the standard device was 5.59 N ± 2.73 standard deviation (SD), which increased to 9.401 N ± 4.6267 SD with a pedestaled device. Resistance to displacement decreased significantly across trials in both groups (P <.0001). Retentive capacity of the SpP increased significantly with the addition of a pedestaled device (P < .0001). The interaction between device and trial was also found to be significant (P = .05). CONCLUSIONS: The RC of the SpP in children and the ability to resist device migration in the absence of fixation may improve with the addition of a pedestal attached to the device. LEVEL OF EVIDENCE: 2b. Laryngoscope, 126:1175-1179, 2016.

MEG/EEG source reconstruction, statistical evaluation, and visualization with NUTMEG.

NUTMEG is a source analysis toolbox geared towards cognitive neuroscience researchers using MEG and EEG, including intracranial recordings. Evoked and unaveraged data can be imported to the toolbox for source analysis in either the time or time-frequency domains. NUTMEG offers several variants of adaptive beamformers, probabilistic reconstruction algorithms, as well as minimum-norm techniques to generate functional maps of spatiotemporal neural source activity. Lead fields can be calculated from single and overlapping sphere head models or imported from other software. Group averages and statistics can be calculated as well. In addition to data analysis tools, NUTMEG provides a unique and intuitive graphical interface for visualization of results. Source analyses can be superimposed onto a structural MRI or headshape to provide a convenient visual correspondence to anatomy. These results can also be navigated interactively, with the spatial maps and source time series or spectrogram linked accordingly. Animations can be generated to view the evolution of neural activity over time. NUTMEG can also display brain renderings and perform spatial normalization of functional maps using SPM's engine. As a MATLAB package, the end user may easily link with other toolboxes or add customized functions.

Cortical function in children receiving bilateral cochlear implants simultaneously or after a period of interimplant delay.

HYPOTHESIS: Children using bilateral cochlear implants (CIs) develop normal patterns of cortical activity when interimplant delays are minimized. BACKGROUND: It is not clear whether bilateral CIs can promote normally functioning bilateral auditory pathways in children. METHODS: Cortical responses were recorded from 64 cephalic sites in 2 normal hearing participants and 8 children with 3 to 4 years of bilateral CI experience (age at first CI, 0.9-4.1 yr; age at second CI, 1.1-9.7 yr; interimplant delay, 0-5.8 yr). RESULTS: Beamformer analyses on the dominant positive peak in CI users and P1 in normal hearers revealed that stimuli delivered from the left side evoked responses lateralized to right auditory cortex in the 2 participants with normal hearing and the 3 children receiving bilateral CIs with minimal interimplant delay at young ages. These 5 participants showed a shift in cortical lateralization away from the right cortical hemisphere when stimuli were moved to the right. In contrast, 4 of 5 children receiving bilateral CIs after longer delays and at older ages showed abnormal ipsilateral parietal activity in response to left stimuli and lateralization to the left cortical hemisphere in response to both right and left stimuli. The fifth child in this group showed abnormal lateralization to the ipsilateral cortex in response to both right and left stimuli. CONCLUSION: The data suggest that, after 3 to 4 years of bilateral CI use, normal-like patterns of bilateral cortical activity are promoted in children receiving bilateral CI with minimal interimplant delays and young ages but are not present in older children who had longer interimplant delays.

Beamformer suppression of cochlear implant artifacts in an electroencephalography dataset.

Localization of cortical auditory evoked potentials in cochlear implant (CI) users is confounded by the presence of a stimulus artifact produced by the implant. Linearly constrained minimum variance (LCMV) beamformers are a class of adaptive spatial filters that localize sources of interest by minimizing the contributions of other uncorrelated sources. We have developed an artifact suppression method that enables an LCMV beamformer to reconstruct cortical activity with minimal artifact interference. This is accomplished by formulating the beamformer to enforce zero-gain on sources that generate the artifact lead potential. The artifact suppression method is first applied to normal hearing subject data to verify that it does not significantly distort the measured cortical responses. The effectiveness of the method is then demonstrated using simulated data and electroencephalography data from a CI user.