Task-Dependent Visual Behavior in Immersive Environments: A Comparative Study of Free Exploration, Memory and Visual Search.

Visual behavior depends on both bottom-up mechanisms, where gaze is driven by the visual conspicuity of the stimuli, and top-down mechanisms, guiding attention towards relevant areas based on the task or goal of the viewer. While this is well-known, visual attention models often focus on bottom-up mechanisms. Existing works have analyzed the effect of high-level cognitive tasks like memory or visual search on visual behavior; however, they have often done so with different stimuli, methodology, metrics and participants, which makes drawing conclusions and comparisons between tasks particularly difficult. In this work we present a systematic study of how different cognitive tasks affect visual behavior in a novel within-subjects design scheme. Participants performed free exploration, memory and visual search tasks in three different scenes while their eye and head movements were being recorded. We found significant, consistent differences between tasks in the distributions of fixations, saccades and head movements. Our findings can provide insights for practitioners and content creators designing task-oriented immersive applications.

D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos.

Understanding human visual behavior within virtual reality environments is crucial to fully leverage their potential. While previous research has provided rich visual data from human observers, existing gaze datasets often suffer from the absence of multimodal stimuli. Moreover, no dataset has yet gathered eye gaze trajectories (i.e., scanpaths) for dynamic content with directional ambisonic sound, which is a critical aspect of sound perception by humans. To address this gap, we introduce D-SAV360, a dataset of 4,609 head and eye scanpaths for 360° videos with first-order ambisonics. This dataset enables a more comprehensive study of multimodal interaction on visual behavior in virtual reality environments. We analyze our collected scanpaths from a total of 87 participants viewing 85 different videos and show that various factors such as viewing mode, content type, and gender significantly impact eye movement statistics. We demonstrate the potential of D-SAV360 as a benchmarking resource for state-of-the-art attention prediction models and discuss its possible applications in further research. By providing a comprehensive dataset of eye movement data for dynamic, multimodal virtual environments, our work can facilitate future investigations of visual behavior and attention in virtual reality.

A Study of Change Blindness in Immersive Environments.

Human performance is poor at detecting certain changes in a scene, a phenomenon known as change blindness. Although the exact reasons of this effect are not yet completely understood, there is a consensus that it is due to our constrained attention and memory capacity: We create our own mental, structured representation of what surrounds us, but such representation is limited and imprecise. Previous efforts investigating this effect have focused on 2D images; however, there are significant differences regarding attention and memory between 2D images and the viewing conditions of daily life. In this work, we present a systematic study of change blindness using immersive 3D environments, which offer more natural viewing conditions closer to our daily visual experience. We devise two experiments; first, we focus on analyzing how different change properties (namely type, distance, complexity, and field of view) may affect change blindness. We then further explore its relation with the capacity of our visual working memory and conduct a second experiment analyzing the influence of the number of changes. Besides gaining a deeper understanding of the change blindness effect, our results may be leveraged in several VR applications such as redirected walking, games, or even studies on saliency or attention prediction.

Development of oculomotor control throughout childhood: A multicenter and multiethnic study.

Although steady fixation is a key aspect of a proper visual function, it is only subjectively assessed in young and uncooperative children. In the present study, we characterize the development of fixational behavior throughout childhood in a large group of healthy children 5 months of age and up, recruited in five geographically diverse sites. In order to do it, we examined 802 healthy children from April 2019 to February 2020. Their oculomotor behavior was analyzed by means of an automated digital system, based on eye-tracking technology. Oculomotor outcomes were gaze stability, fixation stability and duration of fixations (for both long and short fixational tasks), and saccadic reaction time. Ninety-nine percent of all recruited children were successfully examined. Fixational and saccadic performance improved with age throughout childhood, with more pronounced changes during the first 2 years of life. Gaze and fixation tended to be more stable with age (p < 0.001 for most the outcomes), and saccades tended to be faster. In a multivariate analysis, including age and ethnicity as independent variables and adjusting by data quality, age was related with most fixational outcomes. Our automated digital system and eye-tracking data allow us to quantitatively describe the development of oculomotor control during childhood, assess visual fixation and saccadic performance in children 5 months of age and up, and provide a normative reference of fixational outcomes for clinical practice.

Larger visual changes compress time: The inverted effect of asemantic visual features on interval time perception.

Time perception is fluid and affected by manipulations to visual inputs. Previous literature shows that changes to low-level visual properties alter time judgments at the millisecond-level. At longer intervals, in the span of seconds and minutes, high-level cognitive effects (e.g., emotions, memories) elicited by visual inputs affect time perception, but these effects are confounded with semantic information in these inputs, and are therefore challenging to measure and control. In this work, we investigate the effect of asemantic visual properties (pure visual features devoid of emotional or semantic value) on interval time perception. Our experiments were conducted with binary and production tasks in both conventional and head-mounted displays, testing the effects of four different visual features (spatial luminance contrast, temporal frequency, field of view, and visual complexity). Our results reveal a consistent pattern: larger visual changes all shorten perceived time in intervals of up to 3min, remarkably contrary to their effect on millisecond-level perception. Our findings may help alter participants' time perception, which can have broad real-world implications.

ScanGAN360: A Generative Model of Realistic Scanpaths for 360° Images.

Understanding and modeling the dynamics of human gaze behavior in 360° environments is crucial for creating, improving, and developing emerging virtual reality applications. However, recruiting human observers and acquiring enough data to analyze their behavior when exploring virtual environments requires complex hardware and software setups, and can be time-consuming. Being able to generate virtual observers can help overcome this limitation, and thus stands as an open problem in this medium. Particularly, generative adversarial approaches could alleviate this challenge by generating a large number of scanpaths that reproduce human behavior when observing new scenes, essentially mimicking virtual observers. However, existing methods for scanpath generation do not adequately predict realistic scanpaths for 360° images. We present ScanGAN360, a new generative adversarial approach to address this problem. We propose a novel loss function based on dynamic time warping and tailor our network to the specifics of 360° images. The quality of our generated scanpaths outperforms competing approaches by a large margin, and is almost on par with the human baseline. ScanGAN360 allows fast simulation of large numbers of virtual observers, whose behavior mimics real users, enabling a better understanding of gaze behavior, facilitating experimentation, and aiding novel applications in virtual reality and beyond.

Perception of material appearance: A comparison between painted and rendered images.

Painters are masters in replicating the visual appearance of materials. While the perception of material appearance is not yet fully understood, painters seem to have acquired an implicit understanding of the key visual cues that we need to accurately perceive material properties. In this study, we directly compare the perception of material properties in paintings and in renderings by collecting professional realistic paintings of rendered materials. From both type of images, we collect human judgments of material properties and compute a variety of image features that are known to reflect material properties. Our study reveals that, despite important visual differences between the two types of depiction, material properties in paintings and renderings are perceived very similarly and are linked to the same image features. This suggests that we use similar visual cues independently of the medium and that the presence of such cues is sufficient to provide a good appearance perception of the materials.

Influence of Directional Sound Cues on Users' Exploration Across 360° Movie Cuts.

Virtual reality (VR) is a powerful medium for $360^{\circ }$360∘ storytelling, yet content creators are still in the process of developing cinematographic rules for effectively communicating stories in VR. Traditional cinematography has relied for over a century on well-established techniques for editing, and one of the most recurrent resources for this are cinematic cuts that allow content creators to seamlessly transition between scenes. One fundamental assumption of these techniques is that the content creator can control the camera; however, this assumption breaks in VR: Users are free to explore $360^{\circ }$360∘ around them. Recent works have studied the effectiveness of different cuts in $360^{\circ }$360∘ content, but the effect of directional sound cues while experiencing these cuts has been less explored. In this work, we provide the first systematic analysis of the influence of directional sound cues in users' behavior across $360^{\circ }$360∘ movie cuts, providing insights that can have an impact on deriving conventions for VR storytelling.

The joint role of geometry and illumination on material recognition.

Observing and recognizing materials is a fundamental part of our daily life. Under typical viewing conditions, we are capable of effortlessly identifying the objects that surround us and recognizing the materials they are made of. Nevertheless, understanding the underlying perceptual processes that take place to accurately discern the visual properties of an object is a long-standing problem. In this work, we perform a comprehensive and systematic analysis of how the interplay of geometry, illumination, and their spatial frequencies affects human performance on material recognition tasks. We carry out large-scale behavioral experiments where participants are asked to recognize different reference materials among a pool of candidate samples. In the different experiments, we carefully sample the information in the frequency domain of the stimuli. From our analysis, we find significant first-order interactions between the geometry and the illumination, of both the reference and the candidates. In addition, we observe that simple image statistics and higher-order image histograms do not correlate with human performance. Therefore, we perform a high-level comparison of highly nonlinear statistics by training a deep neural network on material recognition tasks. Our results show that such models can accurately classify materials, which suggests that they are capable of defining a meaningful representation of material appearance from labeled proximal image data. Last, we find preliminary evidence that these highly nonlinear models and humans may use similar high-level factors for material recognition tasks.

Evaluation of Fixational Behavior throughout Life.

BACKGROUND: To quantify development of gaze stability throughout life during short and long fixational tasks using eye tracking technology. METHODS: Two hundred and fifty-nine participants aged between 5 months and 77 years were recruited along the study. All participants underwent a complete ophthalmological assessment. Fixational behavior during long and short fixational tasks was analyzed using a DIVE (Device for an Integral Visual Examination), a digital test assisted with eye tracking technology. The participants were divided into ten groups according to their age. Group 1, 0-2 years; group 2, 2-5 years; group 3, 5-10 years; group 4, 10-20 years; group 5, 20-30 years; group 6, 30-40 years; group 7, 40-50 years; group 8, 50-60 years; group 9, 60-70 years; and group 10, over 70 years. RESULTS: Gaze stability, assessed by logBCEA (log-transformed bivariate contour ellipse area), improved with age from 5 months to 30 years (1.27 vs. 0.57 deg2 for long fixational task, 0.73 vs. -0.04 deg2 for short fixational task), while fixations tend to be longer (1.95 vs. 2.80 msec for long fixational tasks and 0.80 vs. 1.71 msec for short fixational tasks). All fixational outcomes worsened progressively from the fifth decade of life. Log-transformed bivariate contour ellipse area (0.79, 0.83, 0.91, 1.42 deg2 for long fixational task and 0.01, 0.18, 0.28, 0.44 deg2 for short fixational task, for group 7, 8, 9, and 10 respectively). Stimuli features may influence oculomotor performance, with smaller stimuli providing prolonged fixations. CONCLUSIONS: Fixational behavior can be accurately assessed from 5 months of age using a DIVE. We report normative data of gaze stability and duration of fixations for every age group. Currently available technology may increase the accuracy of our visual assessments at any age.

Auditory stimuli degrade visual performance in virtual reality.

We report an auditory effect of visual performance degradation in a virtual reality (VR) setting, where the viewing conditions are significantly different from previous studies. With the presentation of temporally congruent but spatially incongruent sound, we can degrade visual performance significantly at detection and recognition levels. We further show that this effect is robust to different types and locations of both auditory and visual stimuli. We also analyze participants behavior with an eye tracker to study the underlying cause of the degradation effect. We find that the performance degradation occurs even in the absence of saccades towards the sound source, during normal gaze behavior. This suggests that this effect is not caused by oculomotor phenomena, but rather by neural interactions or attentional shifts.

Development of a system based on artificial intelligence to identify visual problems in children: study protocol of the TrackAI project.

INTRODUCTION: Around 70% to 80% of the 19 million visually disabled children in the world are due to a preventable or curable disease, if detected early enough. Vision screening in childhood is an evidence-based and cost-effective way to detect visual disorders. However, current screening programmes face several limitations: training required to perform them efficiently, lack of accurate screening tools and poor collaboration from young children.Some of these limitations can be overcome by new digital tools. Implementing a system based on artificial intelligence systems avoid the challenge of interpreting visual outcomes.The objective of the TrackAI Project is to develop a system to identify children with visual disorders. The system will have two main components: a novel visual test implemented in a digital device, DIVE (Device for an Integral Visual Examination); and artificial intelligence algorithms that will run on a smartphone to analyse automatically the visual data gathered by DIVE. METHODS AND ANALYSIS: This is a multicentre study, with at least five centres located in five geographically diverse study sites participating in the recruitment, covering Europe, USA and Asia.The study will include children aged between 6 months and 14 years, both with normal or abnormal visual development.The project will be divided in two consecutive phases: design and training of an artificial intelligence (AI) algorithm to identify visual problems, and system development and validation. The study protocol will consist of a comprehensive ophthalmological examination, performed by an experienced paediatric ophthalmologist, and an exam of the visual function using a DIVE.For the first part of the study, diagnostic labels will be given to each DIVE exam to train the neural network. For the validation, diagnosis provided by ophthalmologists will be compared with AI system outcomes. ETHICS AND DISSEMINATION: The study will be conducted in accordance with the principles of Good Clinical Practice. This protocol was approved by the Clinical Research Ethics Committee of Aragón, CEICA, on January 2019 (Code PI18/346).Results will be published in peer-reviewed journals and disseminated in scientific meetings. TRIAL REGISTRATION NUMBER: ISRCTN17316993.

Oculomotor deficits in children adopted from Eastern Europe.

AIM: We aim to assess oculomotor behaviour in children adopted from Eastern Europe, who are at high risk of maternal alcohol consumption. METHODS: This cross-sectional study included 29 adoptees and 29 age-matched controls. All of them underwent a complete ophthalmological examination. Oculomotor control, including fixation and saccadic performance, was assessed using a DIVE device, with eye tracking technology. Anthropometric and facial measurements were obtained from all the adopted children, to identify features of foetal alcohol spectrum disorders (FASD). Fixational and saccadic outcomes were compared between groups, and the effect of adoption and FASD features quantified. RESULTS: Oculomotor performance was poorer in adopted children. They presented shorter (0.53 vs 1.43 milliseconds in the long task and 0.43 vs 0.82 in the short task) and more unstable fixations (with a bivariate contour ellipse area of 27.9 vs 11.6 degree2 during the long task and 6.9 vs 1.3 degree2 during the short task) and slower saccadic reactions (278 vs 197 milliseconds). Children with sentinel finding for FASD showed the worst oculomotor outcomes. CONCLUSION: Children adopted from Eastern Europe present oculomotor deficits, affecting both fixation and saccadic skills. We highlight prenatal exposure to alcohol as the main cause for these deficits.

Motion parallax for 360° RGBD video.

We present a method for adding parallax and real-time playback of 360° videos in Virtual Reality headsets. In current video players, the playback does not respond to translational head movement, which reduces the feeling of immersion, and causes motion sickness for some viewers. Given a 360° video and its corresponding depth (provided by current stereo 360° stitching algorithms), a naive image-based rendering approach would use the depth to generate a 3D mesh around the viewer, then translate it appropriately as the viewer moves their head. However, this approach breaks at depth discontinuities, showing visible distortions, whereas cutting the mesh at such discontinuities leads to ragged silhouettes and holes at disocclusions. We address these issues by improving the given initial depth map to yield cleaner, more natural silhouettes. We rely on a three-layer scene representation, made up of a foreground layer and two static background layers, to handle disocclusions by propagating information from multiple frames for the first background layer, and then inpainting for the second one. Our system works with input from many of today's most popular 360° stereo capture devices (e.g., Yi Halo or GoPro Odyssey), and works well even if the original video does not provide depth information. Our user studies confirm that our method provides a more compelling viewing experience than without parallax, increasing immersion while reducing discomfort and nausea.

Computational Imaging and Displays: Capturing and displaying richer representations of the world.

Computational imaging techniques allow capturing richer, more complete representations of a scene through the introduction of novel computational algorithms, overcoming the limitations imposed by hardware and optics. The areas of application range from medical imaging to security, to areas in engineering, to name a few. Computational displays combine optics, hardware and computation to faithfully reproduce the world as seen by our eyes, something that current displays still cannot do. This article looks at a few examples of both, aiming to convey the power of the joint co-design of hardware and computational algorithms, also taking into account visual perception.

Saliency in VR: How Do People Explore Virtual Environments?

Understanding how people explore immersive virtual environments is crucial for many applications, such as designing virtual reality (VR) content, developing new compression algorithms, or learning computational models of saliency or visual attention. Whereas a body of recent work has focused on modeling saliency in desktop viewing conditions, VR is very different from these conditions in that viewing behavior is governed by stereoscopic vision and by the complex interaction of head orientation, gaze, and other kinematic constraints. To further our understanding of viewing behavior and saliency in VR, we capture and analyze gaze and head orientation data of 169 users exploring stereoscopic, static omni-directional panoramas, for a total of 1980 head and gaze trajectories for three different viewing conditions. We provide a thorough analysis of our data, which leads to several important insights, such as the existence of a particular fixation bias, which we then use to adapt existing saliency predictors to immersive VR conditions. In addition, we explore other applications of our data and analysis, including automatic alignment of VR video cuts, panorama thumbnails, panorama video synopsis, and saliency-basedcompression.