Realness of face images can be decoded from non-linear modulation of EEG responses.

Artificially created human faces play an increasingly important role in our digital world. However, the so-called uncanny valley effect may cause people to perceive highly, yet not perfectly human-like faces as eerie, bringing challenges to the interaction with virtual agents. At the same time, the neurocognitive underpinnings of the uncanny valley effect remain elusive. Here, we utilized an electroencephalography (EEG) dataset of steady-state visual evoked potentials (SSVEP) in which participants were presented with human face images of different stylization levels ranging from simplistic cartoons to actual photographs. Assessing neuronal responses both in frequency and time domain, we found a non-linear relationship between SSVEP amplitudes and stylization level, that is, the most stylized cartoon images and the real photographs evoked stronger responses than images with medium stylization. Moreover, realness of even highly similar stylization levels could be decoded from the EEG data with task-related component analysis (TRCA). Importantly, we also account for confounding factors, such as the size of the stimulus face's eyes, which previously have not been adequately addressed. Together, this study provides a basis for future research and neuronal benchmarking of real-time detection of face realness regarding three aspects: SSVEP-based neural markers, efficient classification methods, and low-level stimulus confounders.

Data Fusion for Cross-Domain Real-Time Object Detection on the Edge.

We investigate an edge-computing scenario for robot control, where two similar neural networks are running on one computational node. We test the feasibility of using a single object-detection model (YOLOv5) with the benefit of reduced computational resources against the potentially more accurate independent and specialized models. Our results show that using one single convolutional neural network (for object detection and hand-gesture classification) instead of two separate ones can reduce resource usage by almost 50%. For many classes, we observed an increase in accuracy when using the model trained with more labels. For small datasets (a few hundred instances per label), we found that it is advisable to add labels with many instances from another dataset to increase detection accuracy.

Assessing the Value of Multimodal Interfaces: A Study on Human-Machine Interaction in Weld Inspection Workstations.

Multimodal user interfaces promise natural and intuitive human-machine interactions. However, is the extra effort for the development of a complex multisensor system justified, or can users also be satisfied with only one input modality? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and the use of multiple input modalities is valuable and that it is difficult to predict the usability of individual input modalities for complex systems.

EEGSourceSim: A framework for realistic simulation of EEG scalp data using MRI-based forward models and biologically plausible signals and noise.

BACKGROUND: Electroencephalography (EEG) is widely used to investigate human brain function. Simulation studies are essential for assessing the validity of EEG analysis methods and the interpretability of results. NEW METHOD: Here we present a simulation environment for generating EEG data by embedding biologically plausible signal and noise into MRI-based forward models that incorporate individual-subject variability in structure and function. RESULTS: The package includes pipelines for the evaluation and validation of EEG analysis tools for source estimation, functional connectivity, and spatial filtering. EEG dynamics can be simulated using realistic noise and signal models with user specifiable signal-to-noise ratio (SNR). We also provide a set of quantitative metrics tailored to source estimation, connectivity and spatial filtering applications. COMPARISON WITH EXISTING METHOD(S): We provide a larger set of forward solutions for individual MRI-based head models than has been available previously. These head models are surface-based and include two sets of regions-of-interest (ROIs) that have been brought into registration with the brain of each individual using surface-based alignment - one from a whole brain and the other from a visual cortex atlas. We derive a realistic model of noise by fitting different model components to measured resting state EEG. We also provide a set of quantitative metrics for evaluating source-localization, functional connectivity and spatial filtering methods. CONCLUSIONS: The inclusion of a larger number of individual head-models, combined with surface-atlas based labeling of ROIs and plausible models of signal and noise, allows for simulation of EEG data with greater realism than previous packages.

Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment.

We present a deep neural network-based approach to image quality assessment (IQA). The network is trained end-to-end and comprises ten convolutional layers and five pooling layers for feature extraction, and two fully connected layers for regression, which makes it significantly deeper than related IQA models. Unique features of the proposed architecture are that: 1) with slight adaptations it can be used in a no-reference (NR) as well as in a full-reference (FR) IQA setting and 2) it allows for joint learning of local quality and local weights, i.e., relative importance of local quality to the global quality estimate, in an unified framework. Our approach is purely data-driven and does not rely on hand-crafted features or other types of prior domain knowledge about the human visual system or image statistics. We evaluate the proposed approach on the LIVE, CISQ, and TID2013 databases as well as the LIVE In the wild image quality challenge database and show superior performance to state-of-the-art NR and FR IQA methods. Finally, cross-database evaluation shows a high ability to generalize between different databases, indicating a high robustness of the learned features.

Quality assessment of 3D visualizations with vertical disparity: An ERP approach.

In an objective approach for the assessment of quality of experience the neural correlates of EEG data are studied when stereoscopic images are presented in three different conditions containing vertical disparity. These conditions are compared to a similar image in 2D both on the channel level by studying the ERP components and on the source level by the localization of the corresponding ERP component. Our findings posit that P1 component in the occipital cortex has significantly increased in amplitude for 3D condition without vertical disparity compared to the 2D condition. According to previous studies, this component increases when depth information are added to the stimulus which is in line with our findings. However the amplitude of this component has significantly decreased for 3D condition with maximum vertical disparity compared to the 3D condition without vertical disparity. We have concluded that the perception of stereoscopic depth by subjects have decreased in this case due to the distortion introduced by vertical disparity. The underlying sources corresponding to P1 component are localized. Except for the power differences, the source locations do not differ for different conditions.

Objective quality assessment of stereoscopic images with vertical disparity using EEG.

OBJECTIVE: Neurophysiological correlates of vertical disparity in 3D images are studied in an objective approach using EEG technique. These disparities are known to negatively affect the quality of experience and to cause visual discomfort in stereoscopic visualizations. APPROACH: We have presented four conditions to subjects: one in 2D and three conditions in 3D, one without vertical disparity and two with different vertical disparity levels. Event related potentials (ERPs) are measured for each condition and the differences between ERP components are studied. Analysis is also performed on the induced potentials in the time frequency domain. MAIN RESULTS: Results show that there is a significant increase in the amplitude of P1 components in 3D conditions in comparison to 2D. These results are consistent with previous studies which have shown that P1 amplitude increases due to the depth perception in 3D compared to 2D. However the amplitude is significantly smaller for maximum vertical disparity (3D-3) in comparison to 3D with no vertical disparity. Our results therefore suggest that the vertical disparity in 3D-3 condition decreases the perception of depth compared to other 3D conditions and the amplitude of P1 component can be used as a discriminative feature. SIGNIFICANCE: The results show that the P1 component increases in amplitude due to the depth perception in the 3D stimuli compared to the 2D stimulus. On the other hand the vertical disparity in the stereoscopic images is studied here. We suggest that the amplitude of P1 component is modulated with this parameter and decreases due to the decrease in the perception of depth.

EEG-based classification of video quality perception using steady state visual evoked potentials (SSVEPs).

OBJECTIVE: Recent studies exploit the neural signal recorded via electroencephalography (EEG) to get a more objective measurement of perceived video quality. Most of these studies capitalize on the event-related potential component P3. We follow an alternative approach to the measurement problem investigating steady state visual evoked potentials (SSVEPs) as EEG correlates of quality changes. Unlike the P3, SSVEPs are directly linked to the sensory processing of the stimuli and do not require long experimental sessions to get a sufficient signal-to-noise ratio. Furthermore, we investigate the correlation of the EEG-based measures with the outcome of the standard behavioral assessment. APPROACH: As stimulus material, we used six gray-level natural images in six levels of degradation that were created by coding the images with the HM10.0 test model of the high efficiency video coding (H.265/MPEG-HEVC) using six different compression rates. The degraded images were presented in rapid alternation with the original images. In this setting, the presence of SSVEPs is a neural marker that objectively indicates the neural processing of the quality changes that are induced by the video coding. We tested two different machine learning methods to classify such potentials based on the modulation of the brain rhythm and on time-locked components, respectively. MAIN RESULTS: Results show high accuracies in classification of the neural signal over the threshold of the perception of the quality changes. Accuracies significantly correlate with the mean opinion scores given by the participants in the standardized degradation category rating quality assessment of the same group of images. SIGNIFICANCE: The results show that neural assessment of video quality based on SSVEPs is a viable complement of the behavioral one and a significantly fast alternative to methods based on the P3 component.

3D high-efficiency video coding for multi-view video and depth data.

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

Toward a direct measure of video quality perception using EEG.

An approach to the direct measurement of perception of video quality change using electroencephalography (EEG) is presented. Subjects viewed 8-s video clips while their brain activity was registered using EEG. The video signal was either uncompressed at full length or changed from uncompressed to a lower quality level at a random time point. The distortions were introduced by a hybrid video codec. Subjects had to indicate whether they had perceived a quality change. In response to a quality change, a positive voltage change in EEG (the so-called P3 component) was observed at latency of about 400-600 ms for all subjects. The voltage change positively correlated with the magnitude of the video quality change, substantiating the P3 component as a graded neural index of the perception of video quality change within the presented paradigm. By applying machine learning techniques, we could classify on a single-trial basis whether a subject perceived a quality change. Interestingly, some video clips wherein changes were missed (i.e., not reported) by the subject were classified as quality changes, suggesting that the brain detected a change, although the subject did not press a button. In conclusion, abrupt changes of video quality give rise to specific components in the EEG that can be detected on a single-trial basis. Potentially, a neurotechnological approach to video assessment could lead to a more objective quantification of quality change detection, overcoming the limitations of subjective approaches (such as subjective bias and the requirement of an overt response). Furthermore, it allows for real-time applications wherein the brain response to a video clip is monitored while it is being viewed.