The role of attention in immersion: The two-competitor model.

Currently, we face an exponentially increasing interest in immersion, especially sensory-driven immersion, mainly due to the rapid development of ideas and business models centered around a digital virtual universe as well as the increasing availability of affordable immersive technologies for education, communication, and entertainment. However, a clear definition of 'immersion', in terms of established neurocognitive concepts and measurable properties, remains elusive, slowing research on the human side of immersive interfaces. To address this problem, we propose a conceptual, taxonomic model of attention in immersion. We argue (a) modeling immersion theoretically as well as studying immersion experimentally requires a detailed characterization of the role of attention in immersion, even though (b) attention, while necessary, cannot be a sufficient condition for defining immersion. Our broader goal is to characterize immersion in terms that will be compatible with established psychophysiolgical measures that could then in principle be used for the assessment and eventually the optimization of an immersive experience. We start from the perspective that immersion requires the projection of attention to an induced reality, and build on accepted taxonomies of different modes of attention for the development of our two-competitor model. The two-competitor model allows for a quantitative implementation and has an easy graphical interpretation. It helps to highlight the important link between different modes of attention and affect in studying immersion.

Assessment of Vestigial Auriculomotor Activity to Acoustic Stimuli Using Electrodes In and Around the Ear.

Recently, it has been demonstrated that electromyographic (EMG) activity of auricular muscles in humans, especially the postauricular muscle (PAM), depends on the spatial location of auditory stimuli. This observation has only been shown using wet electrodes placed directly on auricular muscles. To move towards a more applied, out-of-the-laboratory setting, this study aims to investigate if similar results can be obtained using electrodes placed in custom-fitted earpieces. Furthermore, with the exception of the ground electrode, only dry-contact electrodes were used to record EMG signals, which require little to no skin preparation and can therefore be applied extremely fast. In two experiments, auditory stimuli were presented to ten participants from different spatial directions. In experiment 1, stimuli were rapid onset naturalistic stimuli presented in silence, and in experiment 2, the corresponding participant's first name, presented in a "cocktail party" environment. In both experiments, ipsilateral responses were significantly larger than contralateral responses. Furthermore, machine learning models objectively decoded the direction of stimuli significantly above chance level on a single trial basis (PAM: ≈ 80%, in-ear: ≈ 69%). There were no significant differences when participants repeated the experiments after several weeks. This study provides evidence that auricular muscle responses can be recorded reliably using an almost entirely dry-contact in-ear electrode system. The location of the PAM, and the fact that in-ear electrodes can record comparable signals, would make hearing aids interesting devices to record these auricular EMG signals and potentially utilize them as control signals in the future.

[New perspectives in orthopedics : Developments through neurotechnology and metaverse]. / Neue Perspektiven in der Orthopädie : Entwicklungen durch Neurotechnologie und Metaversum.

The combination of neurotechnology and metaverse holds high potentials for orthopedics, as it offers a broad spectrum of possibilities to overcome the limits of traditional medical care. The vision of a medical metaverse providing the infrastructure as a link for innovative technologies opens up new opportunities for therapy, medical collaborations and practical, personalized training for aspiring physicians. However, risks and challenges, such as security and privacy, health-related issues, acceptance by patients and doctors, as well as technical hurdles and access to the technologies, remain. Hence, future research and development is paramount. Nonetheless, due to technological progress, the exploration of new research areas, and the improved availability of the technologies paired with cost reduction, the prospects for neurotechnology and metaverse in orthopedics are promising.

Speech Tracking in Complex Auditory Scenes with Differentiated In- and Out-Field-Of-View Processing in Hearing Aids.

In naturalistic auditory scenes, relevant information is rarely concentrated at a single location, but rather unpredictably scattered in- and out-field-of-view (in-/out-FOV). Although the parsing of a complex auditory scene is a fairly simple job for a healthy human auditory system, the uncertainty represents a major issue in the development of effective hearing aid (HA) processing strategies. Whereas traditional omnidirectional microphones (OM) amplify the complete auditory scene without enhancing signal-to-noise-ratio (SNR) between in- and out-FOV streams, directional microphones (DM) may greatly increase SNR at the cost of preventing HA users to perceive out-FOV information. The present study compares the conventional OM and DM HA settings to a split processing (SP) scheme differentiating between in- and out-FOV processing. We recorded electroencephalographic data of ten young, normal-hearing listeners who solved a cocktail-party-scenario-paradigm with continuous auditory streams and analyzed neural tracking of speech with a stimulus reconstruction (SR) approach. While results for all settings exhibited significantly higher SR accuracies for attended in-FOV than unattended out-FOV streams, there were distinct differences between settings. In-FOV SR performance was dominated by DM and SP and out-FOV SR accuracies were significantly higher for SP compared to OM and DM. Our results demonstrate the potential of a SP approach to combine the advantages of traditional OM and DM settings without introduction of significant compromises.

Motion Sickness related Route Profiling for Evaluation of the Sensory Conflict in Real-Driving Studies.

The risk for passengers of an automated vehicle to suffer from motion sickness symptoms increases while performing non-driving tasks. Motion sickness, whether at sea, in the air, in a car or in virtual reality, has been studied for years, but the specific motion patterns of different vehicles and the individual physiology of passengers complicate the definition of general applicable models. Technical progress in vehicles, e.g. the development of the chassis or general digitalization, is constantly changing the influences and marginal effects of motion sickness. In recent years, increasing number of investigations concentrated on the influencing factors on motion sickness. However, the relation between emesis and vehicle dynamics itself is predominantly inadequately presented. Therefore, the results can poorly be incorporate in mathematical models of the sensory conflict theory established as leading theory in the research community. In our research, we suggest a method to prepare and present route and driving information to increase the transparency of real-world driving experiments. We used determined position-based spectrograms to simplify the understanding of the provoked acceleration as well as frequency, known as important motion sickness trigger. Standardized use of this method would support review articles about driving experiments and thus support research regarding motion sickness prediction and occurrence in vehicles.

Lagrangian Motion Magnification with Landmark-Prior and Sparse PCA for Facial Microexpressions and Micromovements.

Video motion magnification methods are motion visualization techniques that aim to magnify subtle and imper-ceptibly small motions in videos. They fall into two main groups where Eulerian methods work on the pixel grid with implicit motion information and Lagrangian methods use explicitly estimated motion and modify point trajectories. The motion in high framerate videos of faces can contain a wide variety of information that ranges from microexpressions over pulse or respiratory rate to cues on speech and affective state. In his work, we propose a novel strategy for Lagrangian motion magnification that integrates landmark information from the face as well as an approach to decompose facial motions in an unsupervised manner using sparse PCA. We decompose the estimated displacements into different movement components that are subsequently amplified selectively. We propose two approaches: A landmark-based decomposition into global and local movements and a decomposition into multiple coherent motion components based on sparse PCA. Optical flow estimation is performed using a state-of-the-art deep learning-based method that we retrain on a microexpression database. Clinical relevance- This method could be applied to the annotation and analysis of micromovements for neurocognitive assessment and even novel, medical applications where micro-motions of the face might play a role.

Software for non-parametric image registration of 2-photon imaging data.

Functional 2-photon microscopy is a key technology for imaging neuronal activity. The recorded image sequences, however, can contain non-rigid movement artifacts which requires high-accuracy movement correction. Variational optical flow (OF) estimation is a group of methods for motion analysis with established performance in many computer vision areas. However, it has yet to be adapted to the statistics of 2-photon neuroimaging data. In this work, we present the motion compensation method Flow-Registration that outperforms previous alignment tools and allows to align and reconstruct even low signal-to-noise ratio 2-photon imaging data and is able to compensate high-divergence displacements during local drug injections. The method is based on statistics of such data and integrates previous advances in variational OF estimation. Our method is available as an easy-to-use ImageJ/FIJI plugin as well as a MATLAB toolbox with modular, object oriented file IO, native multi-channel support and compatibility with existing 2-photon imaging suites.

Multimodal data acquisition at SARS-CoV-2 drive through screening centers: Setup description and experiences in Saarland, Germany.

SARS-CoV-2 drive through screening centers (DTSC) have been implemented worldwide as a fast and secure way of mass screening. We use DTSCs as a platform for the acquisition of multimodal datasets that are needed for the development of remote screening methods. Our acquisition setup consists of an array of thermal, infrared and RGB cameras as well as microphones and we apply methods from computer vision and computer audition for the contactless estimation of physiological parameters. We have recorded a multimodal dataset of DTSC participants in Germany for the development of remote screening methods and symptom identification. Acquisition in the early stages of a pandemic and in regions with high infection rates can facilitate and speed up the identification of infection specific symptoms and large-scale data acquisition at DTSC is possible without disturbing the flow of operation.

Speech induced binaural beats: Electrophysiological assessment of binaural interaction.

This paper introduces and evaluates a speech signal manipulation scheme that generates transient speech induced binaural beats (SBBs). These SBBs can only be perceived when different signals are presented dichotically (to both ears). Event-related potentials were recorded in 22 normal-hearing subjects. Dichotic stimulus presentation reliably evoked auditory late responses (ALRs) in all subjects using such manipulated signals. As control measurements, diotic stimulation modalities were presented to confirm that the ALRs were not evoked by the speech signal itself or that the signal manipulation scheme created audible artifacts. Since diotic measurements evoked no ALRs, responses from dichotic stimulation are a pure correlate of binaural interaction. While there are several auditory stimuli (mostly modulated sinusoids or noise) that share this characteristic, none of them are based on running speech. Because SBBs can be added to any arbitrary speech signal, they could easily be combined with psychoacoustic tests, for example speech reception thresholds, adding an objective measure of binaural interaction.

Fast variational alignment of non-flat 1D displacements for applications in neuroimaging.

BACKGROUND: In the context of signal analysis and pattern matching, alignment of 1D signals for the comparison of signal morphologies is an important problem. For image processing and computer vision, 2D optical flow (OF) methods find wide application for motion analysis and image registration and variational OF methods have been continuously improved over the past decades. NEW METHOD: We propose a variational method for the alignment and displacement estimation of 1D signals. We pose the estimation of non-flat displacements as an optimization problem with a similarity and smoothness term similar to variational OF estimation. To this end, we can make use of efficient optimization strategies that allow real-time applications on consumer grade hardware. RESULTS: We apply our method to two applications from functional neuroimaging: The alignment of 2-photon imaging line scan recordings and the denoising of evoked and event-related potentials in single trial matrices. We can report state of the art results in terms of alignment quality and computing speeds. EXISTING METHODS: Existing methods for 1D alignment target mostly constant displacements, do not allow native subsample precision or precise control over regularization or are slower than the proposed method. CONCLUSIONS: Our method is implemented as a MATLAB toolbox and is online available. It is suitable for 1D alignment problems, where high accuracy and high speed is needed and non-constant displacements occur.

Towards Contactless Estimation of Electrodermal Activity Correlates.

This paper presents a proof-of-concept for contactless and nonintrusive estimation of electrodermal activity (EDA) correlates using a camera. RGB video of the palm under three different lighting conditions showed that for a suitably chosen illumination strategy the data from the camera is sufficient to estimate EDA correlates which agree with the measurements done using laboratory grade physiological sensors. The effects we see in the recorded video can be attributed to sweat gland activity, which inturn is known to be correlated with EDA. These effects are so pronounced that simple pixel statistics can be used to quantify them. Such a method benefits from advances in computer vision and graphics research and has the potential to be used in affective computing and psychophysiology research where contact based sensors may not be suitable.

Time-Multiplexed Illumination for simultaneous Structural and Functional Voltage Sensitive Dye Recordings with a single Photo Sensor.

The in-vivo optical imaging of the cortical surface provides the ability to record different types of biophysiological signals, e.g., structural information, intrinsic signals, like blood oxygenation coupled reflection changes as well as extrinsic properties of voltage sensitive probes, like fluorescent voltage-sensitive dyes. The recorded data sets have very high temporal and spatial resolutions on a meso- to macroscopic scale, which surpass conventional multi-electrode recordings. Both, intrinsic and functional data sets, each provide unique information about temporal and spatial dynamics of cortical functioning, yet have individual drawbacks. To optimize the informational value it would thus be opportune to combine different types of optical imaging in a near simultaneous recording.Due to the low signal-to-noise ratio of voltage-sensitive dyes it is necessary to reduce stray light pollution below the level of the camera's dark noise. It is thus impossible to record full-spectrum optical data sets. We address this problem by a time-multiplexed illumination, bespoke to the utilized voltage sensitive dye, to record an alternating series of intrinsic and extrinsic frames by a high-frequency CMOS sensor. These near simultaneous data series can be used to compare the mutual influence of intrinsic and extrinsic dynamics (with regards to extracorporeal functional imaging) as well as for motion compensation and thus for minimizing frame averaging, which in turn results in increased spatial precision of functional data and in a reduction of necessary experimental data sets (3R principle).

Vestigial auriculomotor activity indicates the direction of auditory attention in humans.

Unlike dogs and cats, people do not point their ears as they focus attention on novel, salient, or task-relevant stimuli. Our species may nevertheless have retained a vestigial pinna-orienting system that has persisted as a 'neural fossil' within in the brain for about 25 million years. Consistent with this hypothesis, we demonstrate that the direction of auditory attention is reflected in sustained electrical activity of muscles within the vestigial auriculomotor system. Surface electromyograms (EMGs) were taken from muscles that either move the pinna or alter its shape. To assess reflexive, stimulus-driven attention we presented novel sounds from speakers at four different lateral locations while the participants silently read a boring text in front of them. To test voluntary, goal-directed attention we instructed participants to listen to a short story coming from one of these speakers, while ignoring a competing story from the corresponding speaker on the opposite side. In both experiments, EMG recordings showed larger activity at the ear on the side of the attended stimulus, but with slightly different patterns. Upward movement (perking) differed according to the lateral focus of attention only during voluntary orienting; rearward folding of the pinna's upper-lateral edge exhibited such differences only during reflexive orienting. The existence of a pinna-orienting system in humans, one that is experimentally accessible, offers opportunities for basic as well as applied science.

Dogs, cats, monkeys and other animals perk their ears in the direction of sounds they are interested in. Humans and their closest ape relatives, however, appear to have lost this ability. Some humans are able to wiggle their ears, suggesting that some of the brain circuits and muscles that allow automatic ear movements towards sounds are still present. This may be a 'vestigial feature', an ability that is maintained even though it no longer serves its original purpose. Now, Strauss et al. show that vestigial movements of muscles around the ear indicate the direction of sounds a person is paying attention to. In the experiments, human volunteers tried to read a boring text while surprising sounds like a traffic jam, a baby crying, or footsteps played. During this exercise, Strauss et al. recorded the electrical activity in the muscles of their ears to see if they moved in response to the direction the sound came from. In a second set of experiments, the same electrical recordings were made as participants listened to a podcast while a second podcast was playing from a different direction. The individuals' ears were also recorded using high resolution video. Both sets of experiments revealed tiny involuntary movements in muscles surrounding the ear closest to the direction of a sound the person is listening to. When the participants tried to listen to one podcast and tune out another, they also made ear 'perking' movements in the direction of their preferred podcast. The results suggest that movements of the vestigial muscles in the human ear indicate the direction of sounds a person is paying attention to. These tiny movements could be used to develop better hearing aids that sense the electrical activity in the ear muscles and amplify sounds the person is trying to focus on, while minimizing other sounds.

Implementation and Long-Term Evaluation of a Hearing Aid Supported Tinnitus Treatment Using Notched Environmental Sounds.

Recent work has shown that sharp spectral edges in acoustic stimuli might have advantageous effects in the treatment of tonal tinnitus. In the course of this paper, we evaluate the long-term effects of spectrally notched hearing aids on the subjective tinnitus distress. By merging recent experimental work with a computational tinnitus model, we modified the commercially available behind-the-ear hearing aids so that a frequency band of 0.5 octaves, centered on the patient's individual tinnitus frequency, was blocked out. Those hearing aids employ a steep notch filter that filters environmental sounds to suppress the tinnitus-related changes in neural firing by lateral inhibition. The computational model reveals a renormalization of pathologically increased neural response reliability and synchrony in response to spectrally modified input. The target group, fitted with spectrally notched hearing aids, was matched with a comparable control group, fitted with standard hearing aids of the same type but without a notch filter. We analyze the subjective self-assessment by tinnitus questionnaires, and we monitor the objective distress correlates in auditory evoked response phase data. Both, subjective and objective results show a noticeable trend of a larger therapeutic benefit for notched hearing correction.

Fast acquisition of full-range auditory event-related potentials using an interleaved deconvolution approach.

This work relates to recent advances in the field of auditory event-related potentials (ERP), specifically deconvolution-based ERP acquisition and single-trial processing. An efficient stimulus sequence optimization method for ERP deconvolution is proposed, achieving consistent noise attenuation within a broad designated frequency range. Furthermore, a stimulus presentation paradigm for the fast, interleaved acquisition of auditory brainstem, middle-latency and late responses featuring alternating periods of high-rate deconvolution sequences, and subsequent low-rate stimulation is investigated in 20 normal hearing subjects. Deconvolved sequence responses containing early and middle-latency ERP components are fused with subsequent late responses using a time-frequency resolved weighted averaging method based on cross-trial regularity, yielding a uniform signal-to-noise ratio of the full-range auditory ERP across investigated timescales. Obtained average ERP waveforms exhibit morphologies consistent with both literature values and reference recordings acquired in 15 normal hearing subjects using a prior art approach to full-range auditory ERP acquisition, with all prominent waves being visible in the grand average waveforms. Results suggest the proposed interleaved stimulus presentation and associated ERP processing methodology to be suitable for the fast, reliable extraction of full-range auditory processing correlates in future ERP studies.

Friction in Passive Tactile Perception Induces Phase Coherency in Late Somatosensory Single Trial Sequences.

Event related potentials represent a noninvasive means for studying sensory and cognitive processes that occur in response to particular stimuli. Here, we report on a phase measure for estimating single trial interaction of late somatosensory potentials (LSPs) following a tribological well defined mechanical stimulation of the human fingertip. Stimuli are presented via a programmable Braille-display with actively switchable pins which was slid along the apex of the passive fingertip, i.e., the fingertip rested stationarily in a finger holding system with circular opening at the bottom. The event was the raising and the lowering of either one, three, or five lines of pins. Differences were identified by measures based on instantaneous phase synchronization to the stimuli across trials, in particular the wavelet phase synchronization stability (WPSS) measure for single trial sequences of LSPs. In particular, we show that the higher the friction the stronger and more localized the induced phase coherency is. We concluded that the WPSS analysis of single sequences of LSPs represents a reliable method which allows for the quantification of brain responses upon distinct tactile stimuli.

Lagrangian Motion Magnification revisited: Continuous, Magnitude Driven Motion Scaling for Psychophysiological Experiments.

Video motion magnification forms a relatively novel family of visualization techniques, that aim to magnify imperceivably small motions in videos. The most prominent techniques are based on Eulerian video processing and local phase shifting, which modify pixel time courses, rather than relying on explicit motion estimation.In this work, we show that under ideal conditions in the context of psychophysiological experiments, a Lagrangian motion magnification approach based on dense optical flow estimation, can be superior to Eulerian motion magnification strategies. We present a novel, continuous and motion magnitude driven forward warping scheme of small motions, which implements motion compensation and magnification into a single motion estimation step. Our approach does not rely on temporal filtering and works in the presence of large motion. It does not require the explicit identification of fast moving objects and more generally no segmentation and or matting in the image domain is necessary. We apply our method to the visualization of blinking related modulations in micro-saccadic eye movements ((i.a.. iridodonesis), pupil dilation (hippus) and micro-expression analysis.

Reducing the Effect of Spurious Phase Variations in Neural Oscillatory Signals.

The phase-reset model of oscillatory EEG activity has received a lot of attention in the last decades for decoding different cognitive processes. Based on this model, the ERPs are assumed to be generated as a result of phase reorganization in ongoing EEG. Alignment of the phase of neuronal activities can be observed within or between different assemblies of neurons across the brain. Phase synchronization has been used to explore and understand perception, attentional binding and considering it in the domain of neuronal correlates of consciousness. The importance of the topic and its vast exploration in different domains of the neuroscience presses the need for appropriate tools and methods for measuring the level of phase synchronization of neuronal activities. Measuring the level of instantaneous phase (IP) synchronization has been used extensively in numerous studies of ERPs as well as oscillatory activity for a better understanding of the underlying cognitive binding with regard to different set of stimulations such as auditory and visual. However, the reliability of results can be challenged as a result of noise artifact in IP. Phase distortion due to environmental noise artifacts as well as different pre-processing steps on signals can lead to generation of artificial phase jumps. One of such effects presented recently is the effect of low envelope on the IP of signal. It has been shown that as the instantaneous envelope of the analytic signal approaches zero, the variations in the phase increase, effectively leading to abrupt transitions in the phase. These abrupt transitions can distort the phase synchronization results as they are not related to any neurophysiological effect. These transitions are called spurious phase variation. In this study, we present a model to remove generated artificial phase variations due to the effect of low envelope. The proposed method is based on a simplified form of a Kalman smoother, that is able to model the IP behavior in narrow-bandpassed oscillatory signals. In this work we first explain the details of the proposed Kalman smoother for modeling the dynamics of the phase variations in narrow-bandpassed signals and then evaluate it on a set of synthetic signals. Finally, we apply the model on ongoing-EEG signals to assess the removal of spurious phase variations.

Somatosensory Evoked Responses Elicited by Haptic Sensations in Midair.

The design of human-machine interfaces (HMIs) by virtual haptics is an emerging field of research. So far, the perception of virtual haptic feedback, e.g., generated by focused ultrasound in midair has not been objectively evaluated. This paper demonstrates the feasibility of eliciting somatosensory evoked potentials (SEPs) with ultrasonic stimuli in midair for the first time. The palm was stimulated by short ultrasonic focal points generated by an ultrasound board. The results are compared with a no stimulation condition as well as with the results of a vibro tactile stimulation. The SEPs are analyzed with the wavelet phase synchronization stability (WPSS) and the M-consecutive averaged WPSS. The results indicate a clear SEP waveform elicited by ultrasound. It can be significantly differentiated from a no stimulation condition by the M-consecutive averaged WPSS. These results could enable the possibility of developing an objective evaluation method for virtual haptic feedback in HMIs.

Free-field evoked auditory brainstem responses in cochlear implant users.

The importance of binaural cues in auditory stream formation and sound source segregation is widely accepted. When treating one ear with a cochlear implant (CI) the peripheral auditory system gets partially replaced and processing delays get added potentially, thus important interaural time differences get altered. However, these effects are not fully understood, leaving a lack of systematic binaural fitting strategies with respect to an optimal binaural fusion. To get new insights into such alterations, we suggest a novel method of free-field auditory brainstem evoked responses (ABRs) analysis in CI users. This method does not bypass the technically induced intrinsic delays of the sound processor while leaving the whole electrode array active, thus the most natural way of stimulation is provided. We compared the ABRs collected of 12 CI users and 12 normal hearing listeners using two different stimuli (chirp, click) at four different intensities each. We analyzed the ABRs using the average of 2000 trials as well as a single trial analysis and found consistent results in the ABRs' amplitudes and latencies, as well as in single trial relationships between both groups. This method provides a new perspective into the natural CI users' ABRs and can be useful in future research regarding binaural interaction and fusion.