Eyes meet, hands greet: The art of timing in social interactions.

Shaking hands is a fundamental form of social interaction. The current study used high-definition cameras during a university graduation ceremony to examine the temporal sequencing of eye contact and shaking hands. Analyses revealed that mutual gaze always preceded shaking hands. A follow up investigation manipulated gaze when shaking hands, and found that participants take significantly longer to accept a handshake when an outstretched hand precedes eye contact. These findings demonstrate that the timing between a person's gaze and their offer to shake hands is critical to how their action is interpreted.

A tutorial: Analyzing eye and head movements in virtual reality.

This tutorial provides instruction on how to use the eye tracking technology built into virtual reality (VR) headsets, emphasizing the analysis of head and eye movement data when an observer is situated in the center of an omnidirectional environment. We begin with a brief description of how VR eye movement research differs from previous forms of eye movement research, as well as identifying some outstanding gaps in the current literature. We then introduce the basic methodology used to collect VR eye movement data both in general and with regard to the specific data that we collected to illustrate different analytical approaches. We continue with an introduction of the foundational ideas regarding data analysis in VR, including frames of reference, how to map eye and head position, and event detection. In the next part, we introduce core head and eye data analyses focusing on determining where the head and eyes are directed. We then expand on what has been presented, introducing several novel spatial, spatio-temporal, and temporal head-eye data analysis techniques. We conclude with a reflection on what has been presented, and how the techniques introduced in this tutorial provide the scaffolding for extensions to more complex and dynamic VR environments.

Visual exploration of omnidirectional panoramic scenes.

How do we explore the visual environment around us, and how are head and eye movements coordinated during our exploration? To investigate this question, we had observers look at omnidirectional panoramic scenes, composed of both landscape and fractal images, using a virtual reality viewer while their eye and head movements were tracked. We analyzed the spatial distribution of eye fixations and the distribution of saccade directions and the spatial distribution of head positions and the distribution of head shifts, as well as the relation between eye and head movements. The results show that, for landscape scenes, eye and head behavior best fit the allocentric frame defined by the scene horizon, especially when head tilt (i.e., head rotation around the view axis) is considered. For fractal scenes, which have an isotropic texture, eye and head movements were executed primarily along the cardinal directions in world coordinates. The results also show that eye and head movements are closely linked in space and time in a complementary way, with stimulus-driven eye movements predominantly leading the head movements. Our study is the first to systematically examine eye and head movements in a panoramic virtual reality environment, and the results demonstrate that a virtual reality environment constitutes a powerful and informative research alternative to traditional methods for investigating looking behavior.

Turning the (virtual) world around: Patterns in saccade direction vary with picture orientation and shape in virtual reality.

Research investigating gaze in natural scenes has identified a number of spatial biases in where people look, but it is unclear whether these are partly due to constrained testing environments (e.g., a participant with their head restrained and looking at a landscape image framed within a computer monitor). We examined the extent to which image shape (square vs. circle), image rotation, and image content (landscapes vs. fractal images) influence eye and head movements in virtual reality (VR). Both the eyes and head were tracked while observers looked at natural scenes in a virtual environment. In line with previous work, we found a bias for saccade directions parallel to the image horizon, regardless of image shape or content. We found that, when allowed to do so, observers move both their eyes and head to explore images. Head rotation, however, was idiosyncratic; some observers rotated a lot, whereas others did not. Interestingly, the head rotated in line with the rotation of landscape but not fractal images. That head rotation and gaze direction respond differently to image content suggests that they may be under different control systems. We discuss our findings in relation to current theories on head and eye movement control and how insights from VR might inform more traditional eye-tracking studies.

Recurrence quantification analysis of eye movements during mental imagery.

Several studies demonstrated similarities of eye fixations during mental imagery and visual perception but-to our knowledge-the temporal characteristics of eye movements during imagery have not yet been considered in detail. To fill this gap, the same data is analyzed with conventional spatial techniques such as analysis of areas of interest (AOI), ScanMatch, and MultiMatch and with recurrence quantification analysis (RQA), a new way of analyzing gaze data by tracking re-fixations and their temporal dynamics. Participants viewed and afterwards imagined three different kinds of pictures (art, faces, and landscapes) while their eye movements were recorded. While fixation locations during imagery were related to those during perception, participants returned more often to areas they had previously looked at during imagery and their scan paths were more clustered and more repetitive when compared to visual perception. Furthermore, refixations of the same area occurred sooner after initial fixation during mental imagery. The results highlight not only content-driven spatial similarities between imagery and perception but also shed light on the processes of mental imagery maintenance and interindividual differences in these processes.

A new method for tongue rehabilitation with computer games: Pilot study.

BACKGROUND: Lingual exercises are commonly used in clinical practice for swallowing rehabilitation. Associating lingual exercises with computer games increases motivation, which influences tongue motor performance. OBJECTIVE: To investigate the effects of tongue movement direction; resistance force level; repetition number; sustained tongue contraction duration; age and gender on tongue motor performance in healthy adults using computer games. METHODS: An observational pilot study was carried out at a university laboratory. Nine healthy adults, aged 22 to 38 years, used an intra-oral joystick controlled by the tongue to play four computer games. The participants had to reach 12 targets that appeared on the computer screen using the intra-oral joystick. Motor performance was measured by the number of attempts to score and the time during which the target force was maintained. Tongue motor performance was compared among tongue movement direction, resistance force level, game round number, and continuous force application time on the target, age and gender. RESULTS: The number of attempts depended significantly on the direction, continuous force application time on the target and age. The time during which the target force was maintained depended significantly on the direction, continuous force application time on the target and game round number. There were no significant differences in the comparisons by gender or by resistance force level. CONCLUSIONS: It was seen that young adults had their best performance in the downward direction, on the third round, holding the force for a shorter time. The performance deteriorated as age increased.

A comparison of scanpath comparison methods.

Interest has flourished in studying both the spatial and temporal aspects of eye movement behavior. This has sparked the development of a large number of new methods to compare scanpaths. In the present work, we present a detailed overview of common scanpath comparison measures. Each of these measures was developed to solve a specific problem, but quantifies different aspects of scanpath behavior and requires different data-processing techniques. To understand these differences, we applied each scanpath comparison method to data from an encoding and recognition experiment and compared their ability to reveal scanpath similarities within and between individuals looking at natural scenes. Results are discussed in terms of the unique aspects of scanpath behavior that the different methods quantify. We conclude by making recommendations for choosing an appropriate scanpath comparison measure.

Temporal dynamics of eye movements are related to differences in scene complexity and clutter.

Recent research has begun to explore not just the spatial distribution of eye fixations but also the temporal dynamics of how we look at the world. In this investigation, we assess how scene characteristics contribute to these fixation dynamics. In a free-viewing task, participants viewed three scene types: fractal, landscape, and social scenes. We used a relatively new method, recurrence quantification analysis (RQA), to quantify eye movement dynamics. RQA revealed that eye movement dynamics were dependent on the scene type viewed. To understand the underlying cause for these differences we applied a technique known as fractal analysis and discovered that complexity and clutter are two scene characteristics that affect fixation dynamics, but only in scenes with meaningful content. Critically, scene primitives-revealed by saliency analysis-had no impact on performance. In addition, we explored how RQA differs from the first half of the trial to the second half, as well as the potential to investigate the precision of fixation targeting by changing RQA radius values. Collectively, our results suggest that eye movement dynamics result from top-down viewing strategies that vary according to the meaning of a scene and its associated visual complexity and clutter.

Eyeing the eyes in social scenes: Evidence for top-down control of stimulus selection in simultanagnosia.

Simultanagnosia is a disorder of visual attention resulting from bilateral parieto-occipital lesions. Healthy individuals look at eyes to infer people's attentional states, but simultanagnosics allocate abnormally few fixations to eyes in scenes. It is unclear why simultanagnosics fail to fixate eyes, but it might reflect that they are (a) unable to locate and fixate them, or (b) do not prioritize attentional states. We compared eye movements of simultanagnosic G.B. to those of healthy subjects viewing scenes normally or through a restricted window of vision. They described scenes and explicitly inferred attentional states of people in scenes. G.B. and subjects viewing scenes through a restricted window made few fixations on eyes when describing scenes, yet increased fixations on eyes when inferring attention. Thus G.B. understands that eyes are important for inferring attentional states and can exert top-down control to seek out and process the gaze of others when attentional states are of interest.

Recurrence quantification analysis of eye movements.

Recurrence quantification analysis (RQA) has been successfully used for describing dynamic systems that are too complex to be characterized adequately by standard methods in time series analysis. More recently, RQA has been used for analyzing the coordination of gaze patterns between cooperating individuals. Here, we extend RQA to the characterization of fixation sequences, and we show that the global and local temporal characteristics of fixation sequences can be captured by a small number of RQA measures that have a clear interpretation in this context. We applied RQA to the analysis of a study in which observers looked at different scenes under natural or gaze-contingent viewing conditions, and we found large differences in the RQA measures between the viewing conditions, indicating that RQA is a powerful new tool for the analysis of the temporal patterns of eye movement behavior.

Gaze Restriction and Reactivation of Place-bound Content Drive Eye Movements During Mental Imagery.

When we imagine a picture, we move our eyes even though the picture is physically not present. These eye movements provide information about the ongoing process of mental imagery. Eye movements unfold over time, and previous research has shown that the temporal gaze dynamics of eye movements in mental imagery have unique properties, which are unrelated to those in perception. In mental imagery, refixations of previously fixated locations happen more often and in a more systematic manner than in perception. The origin of these unique properties remains unclear. We tested how the temporal structure of eye movements is influenced by the complexity of the mental image. Participants briefly saw and then maintained a pattern stimulus, consisting of one (easy condition) to four black segments (most difficult condition). When maintaining a simple pattern in imagery, participants restricted their gaze to a narrow area, and for more complex stimuli, eye movements were more spread out to distant areas. At the same time, fewer refixations were made in imagery when the stimuli were complex. The results show that refixations depend on the imagined content. While fixations of stimulus-related areas reflect the so-called 'looking at nothing' effect, gaze restriction emphasizes differences between mental imagery and perception.

Eye and head movements while encoding and recognizing panoramic scenes in virtual reality.

One approach to studying the recognition of scenes and objects relies on the comparison of eye movement patterns during encoding and recognition. Past studies typically analyzed the perception of flat stimuli of limited extent presented on a computer monitor that did not require head movements. In contrast, participants in the present study saw omnidirectional panoramic scenes through an immersive 3D virtual reality viewer, and they could move their head freely to inspect different parts of the visual scenes. This allowed us to examine how unconstrained observers use their head and eyes to encode and recognize visual scenes. By studying head and eye movement within a fully immersive environment, and applying cross-recurrence analysis, we found that eye movements are strongly influenced by the content of the visual environment, as are head movements-though to a much lesser degree. Moreover, we found that the head and eyes are linked, with the head supporting, and by and large mirroring the movements of the eyes, consistent with the notion that the head operates to support the acquisition of visual information by the eyes.

Eye Tracking in Virtual Reality.

This chapter explores the current state of the art in eye tracking within 3D virtual environments. It begins with the motivation for eye tracking in Virtual Reality (VR) in psychological research, followed by descriptions of the hardware and software used for presenting virtual environments as well as for tracking eye and head movements in VR. This is followed by a detailed description of an example project on eye and head tracking while observers look at 360° panoramic scenes. The example is illustrated with descriptions of the user interface and program excerpts to show the measurement of eye and head movements in VR. The chapter continues with fundamentals of data analysis, in particular methods for the determination of fixations and saccades when viewing spherical displays. We then extend these methodological considerations to determining the spatial and temporal coordination of the eyes and head in VR perception. The chapter concludes with a discussion of outstanding problems and future directions for conducting eye- and head-tracking research in VR. We hope that this chapter will serve as a primer for those intending to implement VR eye tracking in their own research.

Simulating simultanagnosia: spatially constricted vision mimics local capture and the global processing deficit.

Patients with simultanagnosia, which is a component of Bálint syndrome, have a restricted spatial window of visual attention and cannot see more than one object at a time. As a result, these patients see the world in a piecemeal fashion, seeing the local components of objects or scenes at the expense of the global picture. To directly test the relationship between the restriction of the attentional window in simultanagnosia and patients' difficulty with global-level processing, we used a gaze-contingent display to create a literal restriction of vision for healthy participants while they performed a global/local identification task. Participants in this viewing condition were instructed to identify the global and local aspects of hierarchical letter stimuli of different sizes and densities. They performed well at the local identification task, and their patterns of inaccuracies for the global level task were highly similar to the pattern of inaccuracies typically seen with simultanagnosic patients. This suggests that a restricted spatial area of visual processing, combined with normal limits to visual processing, can lead to difficulties with global-level perception.

Hands, tables and groups make rehabilitation awesome!

Technology has helped improve rehabilitation programs by providing patients with engaging alternatives to otherwise monotonous and repetitive exercises. In recent years, therapists have looked towards multi-touch technologies to further enhance patient rehabilitation programs. So far, the focus has mainly been on single-user interaction, largely ignoring many of the benefits patients receive from socially interacting with therapists, caregivers and their peers. To make use of these valuable interactions, we have developed a suite of multi-touch activities for motor and cognitive rehabilitation. These applications can easily be adjusted to meet the needs of individual patients and enable therapists to quantitatively measure patient behavior and performance. We also reflect on design-related discussions we had with practicing occupational therapists and provide a set of design considerations to guide future rehabilitation activities.

Lean on Wii: physical rehabilitation with virtual reality Wii peripherals.

In recent years, a growing number of occupational therapists have integrated video game technologies, such as the Nintendo Wii, into rehabilitation programs. 'Wiihabilitation', or the use of the Wii in rehabilitation, has been successful in increasing patients' motivation and encouraging full body movement. The non-rehabilitative focus of Wii applications, however, presents a number of problems: games are too difficult for patients, they mainly target upper-body gross motor functions, and they lack support for task customization, grading, and quantitative measurements. To overcome these problems, we have designed a low-cost, virtual-reality based system. Our system, Virtual Wiihab, records performance and behavioral measurements, allows for activity customization, and uses auditory, visual, and haptic elements to provide extrinsic feedback and motivation to patients.

The impact of classroom seating location and computer use on student academic performance.

A student's ability to learn effectively in a classroom setting is subject to many factors. While some factors are difficult to regulate, this study explores two factors that a student, or instructor, has full control over, namely 1) seating position, and 2) computer usage. Both factors have been studied considerably with regard to their effects on student performance, and the results indicate that sitting further from the instructor, or using a computer in the classroom, are related to a decline in grade performance. However, it is unclear if the choice of where to sit and whether or not to use a computer in class are mediated by the same cognitive process. If they are the same, then we would expect to see an interaction between the factors, such that, for example, computer usage would most negatively impact the grades of students who sit near the back of a class. This study aims to answer this question by looking at the individual and combined effects of seating position and computer usage on classroom performance. We sampled 1364 students, collecting nearly 3000 total responses across 5 different introductory psychology courses with 4 different instructors on 3 separate occasions. In agreement with previous research, we found that sitting further from the instructor negatively impacted students' grades (0.75 percentage points/row), and using a computer in class negatively impacted grades (by 3.88 percentage points). Our novel finding is that these deleterious effects combined in an additive manner, such that using a computer had the same harmful effect on grade performance regardless of whether the student sat at the front or back of the classroom.

The SNaP Framework: A VR Tool for Assessing Spatial Navigation.

Recent work in psychology has leveraged the power of Virtual Reality (VR) to study the deterioration of navigation abilities in the elderly. Much of this research has focused on determining the behavioral measurements and paradigms appropriate for such diagnoses. We present a system, the Spatial Navigation Paradigm (SNaP) framework, which can be used to implement a battery of spatial navigation paradigms. This framework integrates a popular VR environment development platform with an extensible representation medium to allow for the precise control of paradigms, the switching between input and output devices, and the recording of accurate behavioral measurements. A preliminary study of the framework indicates that novice and expert VR users are able to quickly and easily specify and deploy experiments and that expert VR users can easily modify and extend existing paradigm implementations.

Eye and head movements while looking at rotated scenes in VR. Session "Beyond the screen's edge" at the 20th European Conference on Eye Movement Research (ECEM) in Alicante, 19.8.2019.

We examined the extent to which image shape (square vs. circle), image rotation, and image content (landscapes vs. fractal images) influenced eye and head movements. Both the eyes and head were tracked while observers looked at natural scenes in a virtual reality (VR) environment. In line with previous work, we found a horizontal bias in saccade directions, but this was affected by both the image shape and its content. Interestingly, when viewing landscapes (but not fractals), observers rotated their head in line with the image rotation, presumably to make saccades in cardinal, rather than oblique, directions. We discuss our findings in relation to current theories on eye movement control, and how insights from VR might inform traditional eyetracking studies. - Part 2: Observers looked at panoramic, 360 degree scenes using VR goggles while eye and head movements were tracked. Fixations were determined using IDT (Salvucci & Goldberg, 2000) adapted to a spherical coordinate system. We then analyzed a) the spatial distribution of fixations and the distribution of saccade directions, b) the spatial distribution of head positions and the distribution of head movements, and c) the relation between gaze and head movements. We found that, for landscape scenes, gaze and head best fit the allocentric frame defined by the scene horizon, especially when taking head tilt (i.e., head rotation around the view axis) into account. For fractal scenes, which are isotropic on average, the bias toward a body-centric frame gaze is weak for gaze and strong for the head. Furthermore, our data show that eye and head movements are closely linked in space and time in stereotypical ways, with volitional eye movements predominantly leading the head. We discuss our results in terms of models of visual exploratory behavior in panoramic scenes, both in virtual and real environments. Video stream: https://vimeo.com/356859979 Production and publication of the video stream was sponsored by SCIANS Ltd http://www.scians.ch/.

Orienting in virtual environments: How are surface features and environmental geometry weighted in an orientation task?

We investigated how human adults orient in enclosed virtual environments, when discrete landmark information is not available and participants have to rely on geometric and featural information on the environmental surfaces. In contrast to earlier studies, where, for women, the featural information from discrete landmarks overshadowed the encoding of the geometric information, Experiment 1 showed that when featural information is conjoined with the environmental surfaces, men and women encoded both types of information. Experiment 2 showed that, although both types of information are encoded, performance in locating a goal position is better if it is close to a geometrically or featurally distinct location. Furthermore, although features are relied upon more strongly than geometry, initial experience with an environment influences the relative weighting of featural and geometric cues. Taken together, these results show that human adults use a flexible strategy for encoding spatial information.