The role of attention in immersion: The two-competitor model.

Currently, we face an exponentially increasing interest in immersion, especially sensory-driven immersion, mainly due to the rapid development of ideas and business models centered around a digital virtual universe as well as the increasing availability of affordable immersive technologies for education, communication, and entertainment. However, a clear definition of 'immersion', in terms of established neurocognitive concepts and measurable properties, remains elusive, slowing research on the human side of immersive interfaces. To address this problem, we propose a conceptual, taxonomic model of attention in immersion. We argue (a) modeling immersion theoretically as well as studying immersion experimentally requires a detailed characterization of the role of attention in immersion, even though (b) attention, while necessary, cannot be a sufficient condition for defining immersion. Our broader goal is to characterize immersion in terms that will be compatible with established psychophysiolgical measures that could then in principle be used for the assessment and eventually the optimization of an immersive experience. We start from the perspective that immersion requires the projection of attention to an induced reality, and build on accepted taxonomies of different modes of attention for the development of our two-competitor model. The two-competitor model allows for a quantitative implementation and has an easy graphical interpretation. It helps to highlight the important link between different modes of attention and affect in studying immersion.

Assessment of Vestigial Auriculomotor Activity to Acoustic Stimuli Using Electrodes In and Around the Ear.

Recently, it has been demonstrated that electromyographic (EMG) activity of auricular muscles in humans, especially the postauricular muscle (PAM), depends on the spatial location of auditory stimuli. This observation has only been shown using wet electrodes placed directly on auricular muscles. To move towards a more applied, out-of-the-laboratory setting, this study aims to investigate if similar results can be obtained using electrodes placed in custom-fitted earpieces. Furthermore, with the exception of the ground electrode, only dry-contact electrodes were used to record EMG signals, which require little to no skin preparation and can therefore be applied extremely fast. In two experiments, auditory stimuli were presented to ten participants from different spatial directions. In experiment 1, stimuli were rapid onset naturalistic stimuli presented in silence, and in experiment 2, the corresponding participant's first name, presented in a "cocktail party" environment. In both experiments, ipsilateral responses were significantly larger than contralateral responses. Furthermore, machine learning models objectively decoded the direction of stimuli significantly above chance level on a single trial basis (PAM: ≈ 80%, in-ear: ≈ 69%). There were no significant differences when participants repeated the experiments after several weeks. This study provides evidence that auricular muscle responses can be recorded reliably using an almost entirely dry-contact in-ear electrode system. The location of the PAM, and the fact that in-ear electrodes can record comparable signals, would make hearing aids interesting devices to record these auricular EMG signals and potentially utilize them as control signals in the future.

Speech Tracking in Complex Auditory Scenes with Differentiated In- and Out-Field-Of-View Processing in Hearing Aids.

In naturalistic auditory scenes, relevant information is rarely concentrated at a single location, but rather unpredictably scattered in- and out-field-of-view (in-/out-FOV). Although the parsing of a complex auditory scene is a fairly simple job for a healthy human auditory system, the uncertainty represents a major issue in the development of effective hearing aid (HA) processing strategies. Whereas traditional omnidirectional microphones (OM) amplify the complete auditory scene without enhancing signal-to-noise-ratio (SNR) between in- and out-FOV streams, directional microphones (DM) may greatly increase SNR at the cost of preventing HA users to perceive out-FOV information. The present study compares the conventional OM and DM HA settings to a split processing (SP) scheme differentiating between in- and out-FOV processing. We recorded electroencephalographic data of ten young, normal-hearing listeners who solved a cocktail-party-scenario-paradigm with continuous auditory streams and analyzed neural tracking of speech with a stimulus reconstruction (SR) approach. While results for all settings exhibited significantly higher SR accuracies for attended in-FOV than unattended out-FOV streams, there were distinct differences between settings. In-FOV SR performance was dominated by DM and SP and out-FOV SR accuracies were significantly higher for SP compared to OM and DM. Our results demonstrate the potential of a SP approach to combine the advantages of traditional OM and DM settings without introduction of significant compromises.

Speech induced binaural beats: Electrophysiological assessment of binaural interaction.

This paper introduces and evaluates a speech signal manipulation scheme that generates transient speech induced binaural beats (SBBs). These SBBs can only be perceived when different signals are presented dichotically (to both ears). Event-related potentials were recorded in 22 normal-hearing subjects. Dichotic stimulus presentation reliably evoked auditory late responses (ALRs) in all subjects using such manipulated signals. As control measurements, diotic stimulation modalities were presented to confirm that the ALRs were not evoked by the speech signal itself or that the signal manipulation scheme created audible artifacts. Since diotic measurements evoked no ALRs, responses from dichotic stimulation are a pure correlate of binaural interaction. While there are several auditory stimuli (mostly modulated sinusoids or noise) that share this characteristic, none of them are based on running speech. Because SBBs can be added to any arbitrary speech signal, they could easily be combined with psychoacoustic tests, for example speech reception thresholds, adding an objective measure of binaural interaction.

Vestigial auriculomotor activity indicates the direction of auditory attention in humans.

Unlike dogs and cats, people do not point their ears as they focus attention on novel, salient, or task-relevant stimuli. Our species may nevertheless have retained a vestigial pinna-orienting system that has persisted as a 'neural fossil' within in the brain for about 25 million years. Consistent with this hypothesis, we demonstrate that the direction of auditory attention is reflected in sustained electrical activity of muscles within the vestigial auriculomotor system. Surface electromyograms (EMGs) were taken from muscles that either move the pinna or alter its shape. To assess reflexive, stimulus-driven attention we presented novel sounds from speakers at four different lateral locations while the participants silently read a boring text in front of them. To test voluntary, goal-directed attention we instructed participants to listen to a short story coming from one of these speakers, while ignoring a competing story from the corresponding speaker on the opposite side. In both experiments, EMG recordings showed larger activity at the ear on the side of the attended stimulus, but with slightly different patterns. Upward movement (perking) differed according to the lateral focus of attention only during voluntary orienting; rearward folding of the pinna's upper-lateral edge exhibited such differences only during reflexive orienting. The existence of a pinna-orienting system in humans, one that is experimentally accessible, offers opportunities for basic as well as applied science.

Dogs, cats, monkeys and other animals perk their ears in the direction of sounds they are interested in. Humans and their closest ape relatives, however, appear to have lost this ability. Some humans are able to wiggle their ears, suggesting that some of the brain circuits and muscles that allow automatic ear movements towards sounds are still present. This may be a 'vestigial feature', an ability that is maintained even though it no longer serves its original purpose. Now, Strauss et al. show that vestigial movements of muscles around the ear indicate the direction of sounds a person is paying attention to. In the experiments, human volunteers tried to read a boring text while surprising sounds like a traffic jam, a baby crying, or footsteps played. During this exercise, Strauss et al. recorded the electrical activity in the muscles of their ears to see if they moved in response to the direction the sound came from. In a second set of experiments, the same electrical recordings were made as participants listened to a podcast while a second podcast was playing from a different direction. The individuals' ears were also recorded using high resolution video. Both sets of experiments revealed tiny involuntary movements in muscles surrounding the ear closest to the direction of a sound the person is listening to. When the participants tried to listen to one podcast and tune out another, they also made ear 'perking' movements in the direction of their preferred podcast. The results suggest that movements of the vestigial muscles in the human ear indicate the direction of sounds a person is paying attention to. These tiny movements could be used to develop better hearing aids that sense the electrical activity in the ear muscles and amplify sounds the person is trying to focus on, while minimizing other sounds.

Implementation and Long-Term Evaluation of a Hearing Aid Supported Tinnitus Treatment Using Notched Environmental Sounds.

Recent work has shown that sharp spectral edges in acoustic stimuli might have advantageous effects in the treatment of tonal tinnitus. In the course of this paper, we evaluate the long-term effects of spectrally notched hearing aids on the subjective tinnitus distress. By merging recent experimental work with a computational tinnitus model, we modified the commercially available behind-the-ear hearing aids so that a frequency band of 0.5 octaves, centered on the patient's individual tinnitus frequency, was blocked out. Those hearing aids employ a steep notch filter that filters environmental sounds to suppress the tinnitus-related changes in neural firing by lateral inhibition. The computational model reveals a renormalization of pathologically increased neural response reliability and synchrony in response to spectrally modified input. The target group, fitted with spectrally notched hearing aids, was matched with a comparable control group, fitted with standard hearing aids of the same type but without a notch filter. We analyze the subjective self-assessment by tinnitus questionnaires, and we monitor the objective distress correlates in auditory evoked response phase data. Both, subjective and objective results show a noticeable trend of a larger therapeutic benefit for notched hearing correction.

Reducing the Effect of Spurious Phase Variations in Neural Oscillatory Signals.

The phase-reset model of oscillatory EEG activity has received a lot of attention in the last decades for decoding different cognitive processes. Based on this model, the ERPs are assumed to be generated as a result of phase reorganization in ongoing EEG. Alignment of the phase of neuronal activities can be observed within or between different assemblies of neurons across the brain. Phase synchronization has been used to explore and understand perception, attentional binding and considering it in the domain of neuronal correlates of consciousness. The importance of the topic and its vast exploration in different domains of the neuroscience presses the need for appropriate tools and methods for measuring the level of phase synchronization of neuronal activities. Measuring the level of instantaneous phase (IP) synchronization has been used extensively in numerous studies of ERPs as well as oscillatory activity for a better understanding of the underlying cognitive binding with regard to different set of stimulations such as auditory and visual. However, the reliability of results can be challenged as a result of noise artifact in IP. Phase distortion due to environmental noise artifacts as well as different pre-processing steps on signals can lead to generation of artificial phase jumps. One of such effects presented recently is the effect of low envelope on the IP of signal. It has been shown that as the instantaneous envelope of the analytic signal approaches zero, the variations in the phase increase, effectively leading to abrupt transitions in the phase. These abrupt transitions can distort the phase synchronization results as they are not related to any neurophysiological effect. These transitions are called spurious phase variation. In this study, we present a model to remove generated artificial phase variations due to the effect of low envelope. The proposed method is based on a simplified form of a Kalman smoother, that is able to model the IP behavior in narrow-bandpassed oscillatory signals. In this work we first explain the details of the proposed Kalman smoother for modeling the dynamics of the phase variations in narrow-bandpassed signals and then evaluate it on a set of synthetic signals. Finally, we apply the model on ongoing-EEG signals to assess the removal of spurious phase variations.

Free-field evoked auditory brainstem responses in cochlear implant users.

The importance of binaural cues in auditory stream formation and sound source segregation is widely accepted. When treating one ear with a cochlear implant (CI) the peripheral auditory system gets partially replaced and processing delays get added potentially, thus important interaural time differences get altered. However, these effects are not fully understood, leaving a lack of systematic binaural fitting strategies with respect to an optimal binaural fusion. To get new insights into such alterations, we suggest a novel method of free-field auditory brainstem evoked responses (ABRs) analysis in CI users. This method does not bypass the technically induced intrinsic delays of the sound processor while leaving the whole electrode array active, thus the most natural way of stimulation is provided. We compared the ABRs collected of 12 CI users and 12 normal hearing listeners using two different stimuli (chirp, click) at four different intensities each. We analyzed the ABRs using the average of 2000 trials as well as a single trial analysis and found consistent results in the ABRs' amplitudes and latencies, as well as in single trial relationships between both groups. This method provides a new perspective into the natural CI users' ABRs and can be useful in future research regarding binaural interaction and fusion.

Neurodynamic evaluation of hearing aid features using EEG correlates of listening effort.

In this study, we propose a novel estimate of listening effort using electroencephalographic data. This method is a translation of our past findings, gained from the evoked electroencephalographic activity, to the oscillatory EEG activity. To test this technique, electroencephalographic data from experienced hearing aid users with moderate hearing loss were recorded, wearing hearing aids. The investigated hearing aid settings were: a directional microphone combined with a noise reduction algorithm in a medium and a strong setting, the noise reduction setting turned off, and a setting using omnidirectional microphones without any noise reduction. The results suggest that the electroencephalographic estimate of listening effort seems to be a useful tool to map the exerted effort of the participants. In addition, the results indicate that a directional processing mode can reduce the listening effort in multitalker listening situations.

Robust extraction of the N1-effect in dichotic listening using hardy space mappings of auditory late single trials.

It is known that signs of (early) auditory selective attention are reflected in the N1-wave of auditory late potentials. In recent years, we have analyzed this N1-effect using multiscale neurofunctional modeling and (instantaneous) phase synchronization measures of Hardy space mapped data. To acquire the electroencephalographic data in dichotic listening, we repeated the seminal experiment of Hillyard et al. which first described the N1-effect. Here we apply for the first time phase synchronization measures to these dichotic data and compare the results to the conventional time-domain analysis. Our results suggest that a single trial synchronization analysis of the instantaneous phase at the α/θ-border shows the N1-attention effect even more clear than the original averaged time-domain representation. It is concluded that the presented method is a robust tool to analyse (early) signs of selective attention in dichotic listening tasks.

Objective assessment of listening effort: effects of an increased task demand.

In demanding listening situations the individual has to exert an increased listening effort to process interesting auditory signals correctly. Especially people with hearing loss are particularly affected and require more effort to identify sounds compared to those with normal hearing. So far, a suitable objective estimate of listening effort is still not available. In previous studies, we presented an objective estimate of listening effort (OLEosc), which is based on the instantaneous phase distribution of the ongoing EEG activity. In the current study, the proposed measure was analyzed in detail. The task performance was taken into account, which indicates if the participants can solve the auditory task and exert effort or if they ceased solving the task. The EEG data was acquired from people with moderate hearing loss solving a listening task. During the experiment, hearing aid settings with different microphone configurations were tested. The results indicate that the OLEosc reflects the real exerted effort of the participants. Besides the reflection of different task demands related to the hearing aid settings, it was possible to show a decline of focused attention to the auditory stimuli related to an excessive task demand. Furthermore, the data indicate that the OLEosc is not directly correlated to the speech intelligibility presented in the word score data and is not subjectively biased like the results of the presented rating scale.

Impact of monaural frequency compression on binaural fusion at the brainstem level.

A classical objective measure for binaural fusion at the brainstem level is the so-called ß-wave of the binaural interaction component (BIC) in the auditory brainstem response (ABR). However, in some cases it appeared that a reliable detection of this component still remains a challenge. In this study, we investigate the wavelet phase synchronization stability (WPSS) of ABR data for the analysis of binaural fusion and compare it to the BIC. In particular, we examine the impact of monaural nonlinear frequency compression on binaural fusion. As the auditory system is tonotopically organized, an interaural frequency mismatch caused by monaural frequency compression could negatively effect binaural fusion. In this study, only few subjects showed a detectable ß-wave and in most cases only for low ITDs. However, we present a novel objective measure for binaural fusion that outperforms the current state-of-the-art technique (BIC): the WPSS analysis showed a significant difference between the phase stability of the sum of the monaurally evoked responses and the phase stability of the binaurally evoked ABR. This difference could be an indicator for binaural fusion in the brainstem. Furthermore, we observed that monaural frequency compression could indeed effect binaural fusion, as the WPSS results for this condition vary strongly from the results obtained without frequency compression.

Dysfunctional long term habituation to exogeneous tinnitus-matched sounds in patients with high tinnitus distress.

During the last years, the demand of accurate diagnostic tools for individualized tinnitus treatment gradually increased. Today several different psychometric instruments for the estimation of the patients degree of decompensation with clinical relevance have emerged. All of these tools are questionnaires for a subjective self-assessment and have deficits in comparability due to severe differences in their factor structure in the anamnesis. Those questionnaires thus they are only of limited value in the design of an individualized therapeutic approach. Objective diagnostic tools for the categorization of the patients' distress level are lacking in clinical routine. Scientific approaches yet demonstrated the feasibility of individual distress assessment by objective markers in the EEG. In this article we present the preliminary results of our study of a use of habituation correlates as objective indicator for the decompensation degree in high-distress tinnitus patients.

Evaluation of an objective listening effort measure in a selective, multi-speaker listening task using different hearing aid settings.

Speaker recognition in a multi-speaker environment is a complex listening task that requires effort to be solved. Especially people with hearing loss show an increased listening effort in demanding listening situations compared to normal hearing people. However, a standardized method to quantify listening effort does not exist yet. Recently we have shown a possible way to determine listening effort objectively. The aim of this study was to validate the proposed objective measure in a challenging, true-to-life listening situation, and to get an insight on the influence of different hearing aid (HA) settings on the listening effort using the proposed measure. To achieve this we investigated the influence of four different HA settings and two different listening task difficulties (LTD) on the listening effort of people with hearing loss in a selective, real-speech listening task. HA setting A, B and C all had an adaptive compression with static characteristic, but differed in the gain and compression settings (more and less gain and more and less linear). Setting D had an adaptive compression whose characteristic was situation-dependent. To quantify the listening effort the ongoing oscillatory EEG activity was recorded as the basis to calculate the objective measure (OLEosc). By way of comparison a subjective listening effort score was determined on an individual basis (SLEscr). The results show that the OLEosc maps the SLEscr well in every of the tested conditions. Furthermore, the results also suggest that OLEosc might be more sensitive to small variances in listening effort than the employed subjective rating scale.

Towards the assessment of listening effort in real life situations: Mobile EEG recordings in a multimodal driving situation.

The EEG estimation of listening effort has been proven to efficiently map subjectively perceived effort on an objective scale. However, it has mostly been performed in controlled audiometric laboratory environments. In this study, we investigated the feasibility of the EEG listening effort assessment in a multisensory demanding environment. For this purpose, two mobile EEG acquisition systems were used to extract the objective listening effort of the ongoing oscillatory activity (OLEosc) while driving. As an auditory paradigm was presented to trigger listening effort, the simulated driving scenario served as a competing task responding to different modalities. The OLEosc was compared with a subjective rating of the expended listening effort and the task performance. Furthermore, we considered time resolved listening effort profiles over the duration of the paradigm. The results showed that the practical assessment of listening effort in a multimodal real life setting is viable. The OLEosc could be extracted and followed the subjective listening effort. Moreover, the analysis of the dynamic listening effort profiles unveiled further information such as "surrender effects", when the subjects ceased to solve the auditory task due to the intense multimodal load.

Objective assessment of listening effort in the oscillatory EEG: comparison of different hearing aid configurations.

An objective estimate of listening effort could support the hearing aid fitting procedure. Most of the digital hearing aids have already hearing aid settings which are supposed to reduce the listening effort, but the effects of these settings on the individual's listening effort remain unclear. In this study, we propose an objective estimate of listening effort using electroencephalographic data. The new method is based on the phase distribution of the ongoing oscillatory EEG activity. We hypothesize that for a non-effortful listening environment the phase is rather uniformly distributed on the unit circle than for a demanding condition. To prove if the phase is uniformly distributed around the unit circle, the Rayleigh Test was applied to the phase of the EEG. This method was tested in 14 hearing impaired subjects (moderate hearing loss, 65.64 ±7.93 yrs, 7 female). The tested hearing aid settings were a directional microphone combined with a noise reduction algorithm in a medium and a strong setting, the noise reduction setting turned off as well as a setting using omnidirectional microphones. Noise embedded sentences (Oldenburg Sentence Test, OlSa) were used as test materials. The task of the subject was to repeat each sentence. The results indicate that the objective estimate of listening effort maps the subjectively rated effort and for a listening situation like the presented one, the strong setting of the directional microphone requires the smallest effort.

Exploiting the self-similarity in ERP images by nonlocal means for single-trial denoising.

Event related potentials (ERPs) represent a noninvasive and widely available means to analyze neural correlates of sensory and cognitive processing. Recent developments in neural and cognitive engineering proposed completely new application fields of this well-established measurement technique when using an advanced single-trial processing. We have recently shown that 2-D diffusion filtering methods from image processing can be used for the denoising of ERP single-trials in matrix representations, also called ERP images. In contrast to conventional 1-D transient ERP denoising techniques, the 2-D restoration of ERP images allows for an integration of regularities over multiple stimulations into the denoising process. Advanced anisotropic image restoration methods may require directional information for the ERP denoising process. This is especially true if there is a lack of a priori knowledge about possible traces in ERP images. However due to the use of event related experimental paradigms, ERP images are characterized by a high degree of self-similarity over the individual trials. In this paper, we propose the simple and easy to apply nonlocal means method for ERP image denoising in order to exploit this self-similarity rather than focusing on the edge-based extraction of directional information. Using measured and simulated ERP data, we compare our method to conventional approaches in ERP denoising. It is concluded that the self-similarity in ERP images can be exploited for single-trial ERP denoising by the proposed approach. This method might be promising for a variety of evoked and event-related potential applications, including nonstationary paradigms such as changing exogeneous stimulus characteristics or endogenous states during the experiment. As presented, the proposed approach is for the a posteriori denoising of single-trial sequences.

Neural correlates of listening effort related factors: influence of age and hearing impairment.

In the last years, there has been a rising interest to find an objective method to estimate listening effort. Previously, we have shown that the wavelet phase synchronization stability (WPSS), gained from the instantaneous phase of auditory late responses (ALRs), could serve as a feasible measure for listening effort related factors. In the current study, we examined if the WPSS reflects the listening effort in young as well as middle-aged subjects and in persons with a different degree of hearing loss. To evoke ALR sequences, we generated syllabic paradigms with a different level of difficulty to evoke ALR sequences. We expected, due to the varying task demand, that the subjects require a measurable difference in the amount of effort to solve the paradigms. Additionally, a possible age and/or hearing loss related effect on the neural correlates of listening effort was investigated. The results indicate, that WPSS reflects the listening effort related factors needed to solve an auditory task. A further finding was that the reaction time data and the N1 wave amplitude information hardly yield any correlate of the invested listening effort. In addition, we noticed an age as well as hearing sensitivity related effect on the listening effort.

Nonlocal two dimensional denoising of frequency specific chirp evoked ABR single trials.

Recently, we have shown that denoising evoked potential (EP) images is possible using two dimensional diffusion filtering methods. This restoration allows for an integration of regularities over multiple stimulations into the denoising process. In the present work we propose the nonlocal means (NLM) method for EP image denoising. The EP images were constructed using auditory brainstem responses (ABR) collected in young healthy subjects using frequency specific and broadband chirp stimulations. It is concluded that the NLM method is more efficient than conventional approaches in EP imaging denoising, specially in the case of ABRs, where the relevant information can be easily masked by the ongoing EEG activity, i.e., signals suffer from rather low signal-to-noise ratio SNR. The proposed approach is for the a posteriori denoising of single trials after the experiment and not for real time applications.

Quantification of listening effort correlates in the oscillatory EEG activity: a feasibility study.

So far, a generally accepted objective measure for the listening effort estimation in clinical settings is not existent. Such a measure could support the hearing aid fitting in order to reduce the listening effort in hearing impaired patients by an adequate adaption of their personal hearing aids. In the current study, we propose a new measure for the quantification of large-scale listening effort correlates. This measure takes the phase information of the ongoing oscillatory EEG activity into account. The phase was gained from the 32 channel EEG. Then, the entropy of the extracted phase was calculated. We assume that this angular entropy reflects phase synchronization effects of the ongoing activities due to an increased attention on the relevant (speech) signal. Thus, we expect that smaller values of the angular entropy reflect a more "ordered" process of the phase distribution. The new method was tested in 13 young normal hearing subjects using different auditory tasks consisting of differently adapted sentences to create different listening conditions. The results indicate that the angular entropy can be applied to reveal significantly differences between the solving and the relaxing part of the paradigm, i.e. between a more effortful and a more relaxing listening situation. It is concluded, that the further research includes the development of more effortful listening tasks in order to reveal also differences between the auditory paradigms.