Using XR (Extended Reality) for Behavioral, Clinical, and Learning Sciences Requires Updates in Infrastructure and Funding.

Extended reality (XR, including augmented and virtual reality) creates a powerful intersection between information technology and cognitive, clinical, and education sciences. XR technology has long captured the public imagination, and its development is the focus of major technology companies. This article demonstrates the potential of XR to (1) deliver behavioral insights, (2) transform clinical treatments, and (3) improve learning and education. However, without appropriate policy, funding, and infrastructural investment, many research institutions will struggle to keep pace with the advances and opportunities of XR. To realize the full potential of XR for basic and translational research, funding should incentivize (1) appropriate training, (2) open software solutions, and (3) collaborations between complementary academic and industry partners. Bolstering the XR research infrastructure with the right investments and incentives is vital for delivering on the potential for transformative discoveries, innovations, and applications.

Reading the room: Autistic traits, gaze behaviour, and the ability to infer social relationships.

Individuals high in autistic traits can have difficulty understanding verbal and non-verbal cues, and may display atypical gaze behaviour during social interactions. The aim of this study was to examine differences among neurotypical individuals with high and low levels of autistic traits with regard to their gaze behaviour and their ability to assess peers' social status accurately. Fifty-four university students who completed the 10-item Autism Quotient (AQ-10) were eye-tracked as they watched six 20-second video clips of people ("targets") involved in a group decision-making task. Simulating natural, everyday social interactions, the video clips included moments of debate, humour, interruptions, and cross talk. Results showed that high-scorers on the AQ-10 (i.e., those with more autistic traits) did not differ from the low-scorers in either gaze behaviour or assessing the targets' relative social status. The results based on this neurotypical group of participants suggest that the ability of individuals high in autistic traits to read social cues may be preserved in certain tasks crucial to navigating day-to-day social relationships. These findings are discussed in terms of their implications for theory of mind, weak central coherence, and social motivation theories of autism.

Eye and head movements while encoding and recognizing panoramic scenes in virtual reality.

One approach to studying the recognition of scenes and objects relies on the comparison of eye movement patterns during encoding and recognition. Past studies typically analyzed the perception of flat stimuli of limited extent presented on a computer monitor that did not require head movements. In contrast, participants in the present study saw omnidirectional panoramic scenes through an immersive 3D virtual reality viewer, and they could move their head freely to inspect different parts of the visual scenes. This allowed us to examine how unconstrained observers use their head and eyes to encode and recognize visual scenes. By studying head and eye movement within a fully immersive environment, and applying cross-recurrence analysis, we found that eye movements are strongly influenced by the content of the visual environment, as are head movements-though to a much lesser degree. Moreover, we found that the head and eyes are linked, with the head supporting, and by and large mirroring the movements of the eyes, consistent with the notion that the head operates to support the acquisition of visual information by the eyes.

Eye Tracking in Virtual Reality.

This chapter explores the current state of the art in eye tracking within 3D virtual environments. It begins with the motivation for eye tracking in Virtual Reality (VR) in psychological research, followed by descriptions of the hardware and software used for presenting virtual environments as well as for tracking eye and head movements in VR. This is followed by a detailed description of an example project on eye and head tracking while observers look at 360° panoramic scenes. The example is illustrated with descriptions of the user interface and program excerpts to show the measurement of eye and head movements in VR. The chapter continues with fundamentals of data analysis, in particular methods for the determination of fixations and saccades when viewing spherical displays. We then extend these methodological considerations to determining the spatial and temporal coordination of the eyes and head in VR perception. The chapter concludes with a discussion of outstanding problems and future directions for conducting eye- and head-tracking research in VR. We hope that this chapter will serve as a primer for those intending to implement VR eye tracking in their own research.

Eye spy: Gaze communication and deception during hide-and-seek.

Gaze behaviour is an important component of successful social interactions. Existing research on social gaze and attention has largely focused on gaze detection and following, rather than the two-way communicative component of gaze that operates between individuals. The present study sought to address this in two experiments. First, "hiders" were eye-tracked while they selected hiding places among a grid of boxes on a computer screen; these boxes were either homogeneous or contained a visually unique pop-out item. Importantly, sometimes hiders believed that their gaze would be seen by hypothetical "seekers" who they might wish to deceive or communicate truthful information to; and sometime hiders believed that their gaze would be concealed. In a second experiment, seekers were asked to select the hiders' locations after viewing the hiders' gaze behaviour, including the eye movements that hiders had been (falsely) told would be concealed. Results indicate that seekers are most accurate when hiders use their gaze to truthfully communicate their selected locations and least accurate when hiders aim to deceive. Notably, both communication and interpretation strategies were affected by the visual display type (e.g., hiders looked to and preferentially selected pop-out items when communicating truthfully while seekers interpreted gaze differently when allocated to these pop-out items), indicating that the visual context can be integrated with gaze to facilitate mis/communication. Our study illuminates how the gaze of an individual acquires and signals information, and that individuals will spontaneously adjust the balance between these two functions based on their current goal and visual environment.

Scene meaningfulness guides eye movements even during mind-wandering.

During scene viewing, semantic information in the scene has been shown to play a dominant role in guiding fixations compared to visual salience (e.g., Henderson & Hayes, 2017). However, scene viewing is sometimes disrupted by cognitive processes unrelated to the scene. For example, viewers sometimes engage in mind-wandering, or having thoughts unrelated to the current task. How do meaning and visual salience account for fixation allocation when the viewer is mind-wandering, and does it differ from when the viewer is on-task? We asked participants to study a series of real-world scenes in preparation for a later memory test. Thought probes occasionally occurred after a subset of scenes to assess whether participants were on-task or mind-wandering. We used salience maps (Graph-Based Visual Saliency; Harel, Koch, & Perona, 2007) and meaning maps (Henderson & Hayes, 2017) to represent the distribution of visual salience and semantic richness in the scene, respectively. Because visual salience and meaning were represented similarly, we could directly compare how well they predicted fixation allocation. Our results indicate that fixations prioritized meaningful over visually salient regions in the scene during mind-wandering just as during attentive viewing. These results held across the entire viewing time. A re-analysis of an independent study (Krasich, Huffman, Faber, & Brockmole Journal of Vision, 20(9), 10, 2020) showed similar results. Therefore, viewers appear to prioritize meaningful regions over visually salient regions in real-world scenes even during mind-wandering.

Social modulation of on-screen looking behaviour.

While passive social information (e.g. pictures of people) routinely draws one's eyes, our willingness to look at live others is more nuanced. People tend not to stare at strangers and will modify their gaze behaviour to avoid sending undesirable social signals; yet they often continue to monitor others covertly "out of the corner of their eyes." What this means for looks that are being made near to live others is unknown. Will the eyes be drawn towards the other person, or pushed away? We evaluate changes in two elements of gaze control: image-independent principles guiding how people look (e.g. biases to make eye movements along the cardinal directions) and image-dependent principles guiding what people look at (e.g. a preference for meaningful content within a scene). Participants were asked to freely view semantically unstructured (fractals) and semantically structured (rotated landscape) images, half of which were located in the space near to a live other. We found that eye movements were horizontally displaced away from a visible other starting at 1032 ms after stimulus onset when fractals but not landscapes were viewed. We suggest that the avoidance of looking towards live others extends to the near space around them, at least in the absence of semantically meaningful gaze targets.

Refixation patterns of mind-wandering during real-world scene perception.

Recent studies have shown that mind-wandering (MW) is associated with changes in eye movement parameters, but have not explored how MW affects the sequential pattern of eye movements involved in making sense of complex visual information. Eye movements naturally unfold over time and this process may reveal novel information about cognitive processing during MW. The current study used Recurrence Quantification Analysis (Anderson, Bischof, Laidlaw, Risko, & Kingstone, 2013) to describe the pattern of refixations (fixations directed to previously inspected regions) during MW. Participants completed a real-world scene encoding task and responded to thought probes assessing intentional and unintentional MW. Both types of MW were associated with worse memory of the scenes. More important, RQA showed that scanpaths during unintentional MW were more repetitive than during on-task episodes, as indicated by a higher recurrence rate and more stereotypical fixation sequences. This increased repetitiveness suggests an adaptive response to processing failures through reexamining previous locations. Moreover, this increased repetitiveness contributed to fixations focusing on a smaller spatial scale of the stimuli. Finally, we were also able to validate several traditional measures: both intentional and unintentional MW were associated with fewer and longer fixations; eye-blinking increased numerically during both types of MW but the difference was only significant for unintentional MW. Overall, the results advanced our understanding of how visual processing is affected during MW by highlighting the sequential aspect of eye movements. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Visual exploration of omnidirectional panoramic scenes.

How do we explore the visual environment around us, and how are head and eye movements coordinated during our exploration? To investigate this question, we had observers look at omnidirectional panoramic scenes, composed of both landscape and fractal images, using a virtual reality viewer while their eye and head movements were tracked. We analyzed the spatial distribution of eye fixations and the distribution of saccade directions and the spatial distribution of head positions and the distribution of head shifts, as well as the relation between eye and head movements. The results show that, for landscape scenes, eye and head behavior best fit the allocentric frame defined by the scene horizon, especially when head tilt (i.e., head rotation around the view axis) is considered. For fractal scenes, which have an isotropic texture, eye and head movements were executed primarily along the cardinal directions in world coordinates. The results also show that eye and head movements are closely linked in space and time in a complementary way, with stimulus-driven eye movements predominantly leading the head movements. Our study is the first to systematically examine eye and head movements in a panoramic virtual reality environment, and the results demonstrate that a virtual reality environment constitutes a powerful and informative research alternative to traditional methods for investigating looking behavior.

Physically attractive faces attract us physically.

When interacting with other humans, facial expressions provide valuable information for approach or avoid decisions. Here, we consider facial attractiveness as another important dimension upon which approach-avoidance behaviours may be based. In Experiments 1-3, we measured participants' responses to attractive and unattractive women's faces in an approach-avoidance paradigm in which there was no explicit instruction to evaluate facial attractiveness or any other stimulus attribute. Attractive faces were selected more often, a bias that may be sensitive to response outcomes and was reduced when the faces were inverted. Experiment 4 explored an entirely implicit measure of approach, with participants passively viewing single faces while standing on a force platform. We found greater lean towards attractive faces, with this pattern being most obvious in male participants. Taken together, these results demonstrate that attractiveness activates approach-avoidance tendencies, even in the absence of any task demand.

Turning the (virtual) world around: Patterns in saccade direction vary with picture orientation and shape in virtual reality.

Research investigating gaze in natural scenes has identified a number of spatial biases in where people look, but it is unclear whether these are partly due to constrained testing environments (e.g., a participant with their head restrained and looking at a landscape image framed within a computer monitor). We examined the extent to which image shape (square vs. circle), image rotation, and image content (landscapes vs. fractal images) influence eye and head movements in virtual reality (VR). Both the eyes and head were tracked while observers looked at natural scenes in a virtual environment. In line with previous work, we found a bias for saccade directions parallel to the image horizon, regardless of image shape or content. We found that, when allowed to do so, observers move both their eyes and head to explore images. Head rotation, however, was idiosyncratic; some observers rotated a lot, whereas others did not. Interestingly, the head rotated in line with the rotation of landscape but not fractal images. That head rotation and gaze direction respond differently to image content suggests that they may be under different control systems. We discuss our findings in relation to current theories on head and eye movement control and how insights from VR might inform more traditional eye-tracking studies.

Eye and head movements while looking at rotated scenes in VR. Session "Beyond the screen's edge" at the 20th European Conference on Eye Movement Research (ECEM) in Alicante, 19.8.2019.

We examined the extent to which image shape (square vs. circle), image rotation, and image content (landscapes vs. fractal images) influenced eye and head movements. Both the eyes and head were tracked while observers looked at natural scenes in a virtual reality (VR) environment. In line with previous work, we found a horizontal bias in saccade directions, but this was affected by both the image shape and its content. Interestingly, when viewing landscapes (but not fractals), observers rotated their head in line with the image rotation, presumably to make saccades in cardinal, rather than oblique, directions. We discuss our findings in relation to current theories on eye movement control, and how insights from VR might inform traditional eyetracking studies. - Part 2: Observers looked at panoramic, 360 degree scenes using VR goggles while eye and head movements were tracked. Fixations were determined using IDT (Salvucci & Goldberg, 2000) adapted to a spherical coordinate system. We then analyzed a) the spatial distribution of fixations and the distribution of saccade directions, b) the spatial distribution of head positions and the distribution of head movements, and c) the relation between gaze and head movements. We found that, for landscape scenes, gaze and head best fit the allocentric frame defined by the scene horizon, especially when taking head tilt (i.e., head rotation around the view axis) into account. For fractal scenes, which are isotropic on average, the bias toward a body-centric frame gaze is weak for gaze and strong for the head. Furthermore, our data show that eye and head movements are closely linked in space and time in stereotypical ways, with volitional eye movements predominantly leading the head. We discuss our results in terms of models of visual exploratory behavior in panoramic scenes, both in virtual and real environments. Video stream: https://vimeo.com/356859979 Production and publication of the video stream was sponsored by SCIANS Ltd http://www.scians.ch/.

Looking at paintings in the Vincent Van Gogh Museum: Eye movement patterns of children and adults.

In the present study, we examined the eye movement behaviour of children and adults looking at five Van Gogh paintings in the Van Gogh Museum, Amsterdam. The goal of the study was to determine the role of top-down and bottom-up attentional processes in the first stages of participants' aesthetic experience. Bottom-up processes were quantified by determining a salience map for each painting. Top-down processing was manipulated by first allowing participants to view the paintings freely, then providing background information about each painting, and then allowing them to view the paintings a second time. The salience analysis showed differences between the eye movement behaviour of children and adults, and differences between the two phases. In the children, the first five fixations during the free viewing phase were strongly related to visually salient features of the paintings-indicating a strong role for bottom-up factors. In the second phase, after children had received background information, top-down factors played a more prominent role. By contrast, adults' observed patterns were similar in both phases, indicating that bottom-up processes did not play a major role when they viewed the paintings. In the second phase, children and adults both spent more time looking at regions that were mentioned in the background information. This effect was greater for adults than for children, confirming the notion that adults, when viewing paintings, rely much more on top-down processing than children.

Salient object changes influence overt attentional prioritization and object-based targeting in natural scenes.

A change to an object in natural scenes attracts attention when it occurs during a fixation. However, when a change occurs during a saccade, and is masked by saccadic suppression, it typically does not capture the gaze in a bottom-up manner. In the present work, we investigated how the type and direction of salient changes to objects affect the prioritization and targeting of objects in natural scenes. We asked observers to look around a scene in preparation for a later memory test. After a period of time, an object in the scene was increased or decreased in salience either during a fixation (with a transient signal) or during a saccade (without transient signal), or it was not changed at all. Changes that were made during a fixation attracted the eyes both when the change involved an increase and a decrease in salience. However, changes that were made during a saccade only captured the eyes when the change was an increase in salience, relative to the baseline no-change condition. These results suggest that the prioritization of object changes can be influenced by the underlying salience of the changed object. In addition, object changes that occurred with a transient signal (which is itself a salient signal) resulted in more central object targeting. Taken together, our results suggest that salient signals in a natural scene are an important component in both object prioritization and targeting in natural scene viewing, insofar as they align with object locations.

The influence of a scene preview on eye movement behavior in natural scenes.

Rich contextual and semantic information can be extracted from only a brief presentation of a natural scene. This is presumed to be activated quickly enough to guide initial eye movements into a scene. However, early, short-latency eye movements in natural scenes have been shown to be dependent on the salience distribution across the image (Anderson, Ort, Kruijne, Meeter, & Donk, 2015). In the present work, we manipulated the salience distribution across a natural scene by changing the global contrast. We showed participants a brief real or nonsense preview of the scene and examined the time-course of eye movement guidance. A real preview decreased the latency and increased the amplitude of initial saccades into the image, suggesting that the preview allowed observers to obtain additional contextual information that would otherwise not be available. However, the preview did not completely override the initial tendency for short-latency saccades to be guided by the underlying salience distribution of the image. We discuss these findings in the context of oculomotor selection based on the integration of contextual information and low-level features in a natural scene.

Motion influences gaze direction discrimination and disambiguates contradictory luminance cues.

In two experiments, we investigated the role of apparent motion in discriminating left/right gaze deviation judgments. We demonstrated that discrimination accuracy and response confidence was significantly higher when the eyes were moved to the left or right, compared to when the eyes were presented in their final shifted position (static images). To dissociate the role of motion signals from luminance signals, gaze stimuli were also presented in reverse contrast. Replicating past studies polarity reversal had a profound and detrimental effect on gaze discrimination in static images, although, intriguingly, while response confidence remained low, participant performance improved as gaze angle increased. In striking contrast to these data, polarity reversal had no negative effect on performance when the eyes were moved. We discuss these findings in the context of a multiple-cue account of gaze perception.

It depends on when you look at it: Salience influences eye movements in natural scene viewing and search early in time.

It is generally accepted that salience affects eye movements in simple artificially created search displays. However, no such consensus exists for eye movements in natural scenes, with several reports arguing that it is mostly high-level cognitive factors that control oculomotor behavior in natural scenes. Here, we manipulate the salience distribution across images by decreasing or increasing the contrast in a gradient across the image. We recorded eye movements in an encoding task (Experiment 1) and a visual search task (Experiment 2) and analyzed the relationship between the latency of fixations and subsequent saccade targeting throughout scene viewing. We find that short-latency first saccades are more likely to land on a region of the image with high salience than long-latency and subsequent saccades in both the encoding and visual search tasks. This implies that salience indeed influences oculomotor behavior in natural scenes, albeit on a different timescale than previously reported. We discuss our findings in relation to current theories of saccade control in natural scenes.

A comparison of scanpath comparison methods.

Interest has flourished in studying both the spatial and temporal aspects of eye movement behavior. This has sparked the development of a large number of new methods to compare scanpaths. In the present work, we present a detailed overview of common scanpath comparison measures. Each of these measures was developed to solve a specific problem, but quantifies different aspects of scanpath behavior and requires different data-processing techniques. To understand these differences, we applied each scanpath comparison method to data from an encoding and recognition experiment and compared their ability to reveal scanpath similarities within and between individuals looking at natural scenes. Results are discussed in terms of the unique aspects of scanpath behavior that the different methods quantify. We conclude by making recommendations for choosing an appropriate scanpath comparison measure.

Temporal dynamics of eye movements are related to differences in scene complexity and clutter.

Recent research has begun to explore not just the spatial distribution of eye fixations but also the temporal dynamics of how we look at the world. In this investigation, we assess how scene characteristics contribute to these fixation dynamics. In a free-viewing task, participants viewed three scene types: fractal, landscape, and social scenes. We used a relatively new method, recurrence quantification analysis (RQA), to quantify eye movement dynamics. RQA revealed that eye movement dynamics were dependent on the scene type viewed. To understand the underlying cause for these differences we applied a technique known as fractal analysis and discovered that complexity and clutter are two scene characteristics that affect fixation dynamics, but only in scenes with meaningful content. Critically, scene primitives-revealed by saliency analysis-had no impact on performance. In addition, we explored how RQA differs from the first half of the trial to the second half, as well as the potential to investigate the precision of fixation targeting by changing RQA radius values. Collectively, our results suggest that eye movement dynamics result from top-down viewing strategies that vary according to the meaning of a scene and its associated visual complexity and clutter.

Recurrence quantification analysis of eye movements.

Recurrence quantification analysis (RQA) has been successfully used for describing dynamic systems that are too complex to be characterized adequately by standard methods in time series analysis. More recently, RQA has been used for analyzing the coordination of gaze patterns between cooperating individuals. Here, we extend RQA to the characterization of fixation sequences, and we show that the global and local temporal characteristics of fixation sequences can be captured by a small number of RQA measures that have a clear interpretation in this context. We applied RQA to the analysis of a study in which observers looked at different scenes under natural or gaze-contingent viewing conditions, and we found large differences in the RQA measures between the viewing conditions, indicating that RQA is a powerful new tool for the analysis of the temporal patterns of eye movement behavior.