Lagrangian Motion Magnification with Landmark-Prior and Sparse PCA for Facial Microexpressions and Micromovements.

Video motion magnification methods are motion visualization techniques that aim to magnify subtle and imper-ceptibly small motions in videos. They fall into two main groups where Eulerian methods work on the pixel grid with implicit motion information and Lagrangian methods use explicitly estimated motion and modify point trajectories. The motion in high framerate videos of faces can contain a wide variety of information that ranges from microexpressions over pulse or respiratory rate to cues on speech and affective state. In his work, we propose a novel strategy for Lagrangian motion magnification that integrates landmark information from the face as well as an approach to decompose facial motions in an unsupervised manner using sparse PCA. We decompose the estimated displacements into different movement components that are subsequently amplified selectively. We propose two approaches: A landmark-based decomposition into global and local movements and a decomposition into multiple coherent motion components based on sparse PCA. Optical flow estimation is performed using a state-of-the-art deep learning-based method that we retrain on a microexpression database. Clinical relevance- This method could be applied to the annotation and analysis of micromovements for neurocognitive assessment and even novel, medical applications where micro-motions of the face might play a role.

Software for non-parametric image registration of 2-photon imaging data.

Functional 2-photon microscopy is a key technology for imaging neuronal activity. The recorded image sequences, however, can contain non-rigid movement artifacts which requires high-accuracy movement correction. Variational optical flow (OF) estimation is a group of methods for motion analysis with established performance in many computer vision areas. However, it has yet to be adapted to the statistics of 2-photon neuroimaging data. In this work, we present the motion compensation method Flow-Registration that outperforms previous alignment tools and allows to align and reconstruct even low signal-to-noise ratio 2-photon imaging data and is able to compensate high-divergence displacements during local drug injections. The method is based on statistics of such data and integrates previous advances in variational OF estimation. Our method is available as an easy-to-use ImageJ/FIJI plugin as well as a MATLAB toolbox with modular, object oriented file IO, native multi-channel support and compatibility with existing 2-photon imaging suites.

Multimodal data acquisition at SARS-CoV-2 drive through screening centers: Setup description and experiences in Saarland, Germany.

SARS-CoV-2 drive through screening centers (DTSC) have been implemented worldwide as a fast and secure way of mass screening. We use DTSCs as a platform for the acquisition of multimodal datasets that are needed for the development of remote screening methods. Our acquisition setup consists of an array of thermal, infrared and RGB cameras as well as microphones and we apply methods from computer vision and computer audition for the contactless estimation of physiological parameters. We have recorded a multimodal dataset of DTSC participants in Germany for the development of remote screening methods and symptom identification. Acquisition in the early stages of a pandemic and in regions with high infection rates can facilitate and speed up the identification of infection specific symptoms and large-scale data acquisition at DTSC is possible without disturbing the flow of operation.

Fast variational alignment of non-flat 1D displacements for applications in neuroimaging.

BACKGROUND: In the context of signal analysis and pattern matching, alignment of 1D signals for the comparison of signal morphologies is an important problem. For image processing and computer vision, 2D optical flow (OF) methods find wide application for motion analysis and image registration and variational OF methods have been continuously improved over the past decades. NEW METHOD: We propose a variational method for the alignment and displacement estimation of 1D signals. We pose the estimation of non-flat displacements as an optimization problem with a similarity and smoothness term similar to variational OF estimation. To this end, we can make use of efficient optimization strategies that allow real-time applications on consumer grade hardware. RESULTS: We apply our method to two applications from functional neuroimaging: The alignment of 2-photon imaging line scan recordings and the denoising of evoked and event-related potentials in single trial matrices. We can report state of the art results in terms of alignment quality and computing speeds. EXISTING METHODS: Existing methods for 1D alignment target mostly constant displacements, do not allow native subsample precision or precise control over regularization or are slower than the proposed method. CONCLUSIONS: Our method is implemented as a MATLAB toolbox and is online available. It is suitable for 1D alignment problems, where high accuracy and high speed is needed and non-constant displacements occur.

Towards Contactless Estimation of Electrodermal Activity Correlates.

This paper presents a proof-of-concept for contactless and nonintrusive estimation of electrodermal activity (EDA) correlates using a camera. RGB video of the palm under three different lighting conditions showed that for a suitably chosen illumination strategy the data from the camera is sufficient to estimate EDA correlates which agree with the measurements done using laboratory grade physiological sensors. The effects we see in the recorded video can be attributed to sweat gland activity, which inturn is known to be correlated with EDA. These effects are so pronounced that simple pixel statistics can be used to quantify them. Such a method benefits from advances in computer vision and graphics research and has the potential to be used in affective computing and psychophysiology research where contact based sensors may not be suitable.

Time-Multiplexed Illumination for simultaneous Structural and Functional Voltage Sensitive Dye Recordings with a single Photo Sensor.

The in-vivo optical imaging of the cortical surface provides the ability to record different types of biophysiological signals, e.g., structural information, intrinsic signals, like blood oxygenation coupled reflection changes as well as extrinsic properties of voltage sensitive probes, like fluorescent voltage-sensitive dyes. The recorded data sets have very high temporal and spatial resolutions on a meso- to macroscopic scale, which surpass conventional multi-electrode recordings. Both, intrinsic and functional data sets, each provide unique information about temporal and spatial dynamics of cortical functioning, yet have individual drawbacks. To optimize the informational value it would thus be opportune to combine different types of optical imaging in a near simultaneous recording.Due to the low signal-to-noise ratio of voltage-sensitive dyes it is necessary to reduce stray light pollution below the level of the camera's dark noise. It is thus impossible to record full-spectrum optical data sets. We address this problem by a time-multiplexed illumination, bespoke to the utilized voltage sensitive dye, to record an alternating series of intrinsic and extrinsic frames by a high-frequency CMOS sensor. These near simultaneous data series can be used to compare the mutual influence of intrinsic and extrinsic dynamics (with regards to extracorporeal functional imaging) as well as for motion compensation and thus for minimizing frame averaging, which in turn results in increased spatial precision of functional data and in a reduction of necessary experimental data sets (3R principle).

Vestigial auriculomotor activity indicates the direction of auditory attention in humans.

Unlike dogs and cats, people do not point their ears as they focus attention on novel, salient, or task-relevant stimuli. Our species may nevertheless have retained a vestigial pinna-orienting system that has persisted as a 'neural fossil' within in the brain for about 25 million years. Consistent with this hypothesis, we demonstrate that the direction of auditory attention is reflected in sustained electrical activity of muscles within the vestigial auriculomotor system. Surface electromyograms (EMGs) were taken from muscles that either move the pinna or alter its shape. To assess reflexive, stimulus-driven attention we presented novel sounds from speakers at four different lateral locations while the participants silently read a boring text in front of them. To test voluntary, goal-directed attention we instructed participants to listen to a short story coming from one of these speakers, while ignoring a competing story from the corresponding speaker on the opposite side. In both experiments, EMG recordings showed larger activity at the ear on the side of the attended stimulus, but with slightly different patterns. Upward movement (perking) differed according to the lateral focus of attention only during voluntary orienting; rearward folding of the pinna's upper-lateral edge exhibited such differences only during reflexive orienting. The existence of a pinna-orienting system in humans, one that is experimentally accessible, offers opportunities for basic as well as applied science.

Dogs, cats, monkeys and other animals perk their ears in the direction of sounds they are interested in. Humans and their closest ape relatives, however, appear to have lost this ability. Some humans are able to wiggle their ears, suggesting that some of the brain circuits and muscles that allow automatic ear movements towards sounds are still present. This may be a 'vestigial feature', an ability that is maintained even though it no longer serves its original purpose. Now, Strauss et al. show that vestigial movements of muscles around the ear indicate the direction of sounds a person is paying attention to. In the experiments, human volunteers tried to read a boring text while surprising sounds like a traffic jam, a baby crying, or footsteps played. During this exercise, Strauss et al. recorded the electrical activity in the muscles of their ears to see if they moved in response to the direction the sound came from. In a second set of experiments, the same electrical recordings were made as participants listened to a podcast while a second podcast was playing from a different direction. The individuals' ears were also recorded using high resolution video. Both sets of experiments revealed tiny involuntary movements in muscles surrounding the ear closest to the direction of a sound the person is listening to. When the participants tried to listen to one podcast and tune out another, they also made ear 'perking' movements in the direction of their preferred podcast. The results suggest that movements of the vestigial muscles in the human ear indicate the direction of sounds a person is paying attention to. These tiny movements could be used to develop better hearing aids that sense the electrical activity in the ear muscles and amplify sounds the person is trying to focus on, while minimizing other sounds.

Lagrangian Motion Magnification revisited: Continuous, Magnitude Driven Motion Scaling for Psychophysiological Experiments.

Video motion magnification forms a relatively novel family of visualization techniques, that aim to magnify imperceivably small motions in videos. The most prominent techniques are based on Eulerian video processing and local phase shifting, which modify pixel time courses, rather than relying on explicit motion estimation.In this work, we show that under ideal conditions in the context of psychophysiological experiments, a Lagrangian motion magnification approach based on dense optical flow estimation, can be superior to Eulerian motion magnification strategies. We present a novel, continuous and motion magnitude driven forward warping scheme of small motions, which implements motion compensation and magnification into a single motion estimation step. Our approach does not rely on temporal filtering and works in the presence of large motion. It does not require the explicit identification of fast moving objects and more generally no segmentation and or matting in the image domain is necessary. We apply our method to the visualization of blinking related modulations in micro-saccadic eye movements ((i.a.. iridodonesis), pupil dilation (hippus) and micro-expression analysis.

Pyramid approach for the reduction of parallax-related artefacts in optical recordings of moving translucent volumes.

Functional optical imaging (OI) of intrinsic signals (like blood oxygenation coupled reflection changes) and of extrinsic properties of voltage sensitive probes (like voltage-sensitive dyes (VSD)) forms a group of invasive neuroimaging techniques, that possess up to date the highest temporal and spatial resolution on a meso- to macroscopic scale. There are different sources that contribute to the OI signal of which many are noise. In our previous works, we have used dense optical flow for the reduction of movement artefacts. The translucent surface of the cortex allows contributions from multiple depths. Due to the depth offield (DOF) effect, we get an implicit relation of depth and 2D frequency components. In this work, we introduce registration on the levels of a Laplacian pyramid to remove movement artefacts which have different motion components in different spatial frequency bands. This aims to resolve artefacts that remain after normal registration and are caused e.g. by parallax motion, dead pixels or dust on the sensor and other high frequent, moving particles on the cortex surface without the compromise of using high smoothness weights.

Motion invariant contrast enhancement of optical imaging data in the gradient domain.

Functional optical imaging (OI) of intrinsic signals (like blood oxygenation coupled reflection changes) and of extrinsic properties of voltage sensitive probes (like voltage-sensitive dyes (VSD)) forms a group of neuroimaging techniques that possess up to date highest temporal and spatial resolution on a meso-to macroscopic scale. An inherent problem of OI is a very low signal to noise ratio (SNR), which restricts the recordings to be completely motionless and requires detailed knowledge of the properties of the different noise sources. In our experiments we performed a durectomy and did not use an imaging chamber to allow us future joint electroencephalography-optical imaging (EEG-OI) measures, which resulted in movement artifacts. With the goal of motion compensation in OI recordings and magnification of signal changes, we present a novel processing pipeline, which is based on optic flow guided denoising and gradient domain tone mapping for spatiotemporal contrast enhancement.

Compensation of pulsation artifacts during optical imaging with and without cranial chamber.

Functional Optical Imaging (OI) through the opened skull forms a group of Neuroimaging techniques characterized by a high temporal and spatial resolution on a meso-to macroscopic scale. State of the art OI experiments are generally difficult to execute, with a very timely surgical preparation preceding the experiment, that requires a skilled surgeon to mount a sealed imaging chamber onto the skull. The chamber reduces brain pulsation artifacts and swelling of the brain through movement restriction. In this work, we present preliminary results of a novel approach that does not rely on the usage of an imaging chamber with the goal to facilitate heavily the surgical animal preparation and to allow straightforward joint Electroencephalography - Optical Imaging recordings in the future. We carried out experiments to compare the movement restricting properties of the imaging chamber with the movement in a recording of an unconstrained and periodically irrigated brain. We used high-level image processing techniques to reduce brain pulsation artifacts and did a quantitative movement analysis of the recordings. Our results suggest that while recordings with imaging chamber show less sagittal movement, both with and without imaging chamber comprise the same lateral movements.