Exploring the impact of nonverbal social behavior on learning outcomes in instructional video design.

Online education has become increasingly popular in recent years, and video lectures have emerged as a common instructional format. While the importance of instructors' nonverbal social cues such as gaze, facial expression, and gestures for learning progress in face-to-face teaching is well-established, their impact on instructional videos is not fully understood. Most studies on nonverbal social cues in instructional videos focus on isolated cues rather than considering multimodal nonverbal behavior patterns and their effects on the learning progress. This study examines the role of instructors' nonverbal immediacy (a construct capturing multimodal nonverbal behaviors that reduce psychological distance) in video lectures with respect to learners' cognitive, affective, and motivational outcomes. We carried out an eye-tracking experiment with 87 participants (Mage = 24.11, SD = 4.80). Results of multilevel path analyses indicate that high nonverbal immediacy substantially increases learners' state motivation and enjoyment, but does not affect cognitive learning. Analyses of learners' eye movements show that learners allocate more attention to the instructor than to the learning material with increasing levels of nonverbal immediacy displayed by the instructor. The study highlights the importance of considering the role of multimodal nonverbal behavior patterns in online education and provides insights for effective video lecture design.

Sustained bias of spatial attention in a 3 T MRI scanner.

When lying inside a MRI scanner and even in the absence of any motion, the static magnetic field of MRI scanners induces a magneto-hydrodynamic stimulation of subjects' vestibular organ (MVS). MVS thereby not only causes a horizontal vestibular nystagmus but also induces a horizontal bias in spatial attention. In this study, we aimed to determine the time course of MVS-induced biases in both VOR and spatial attention inside a 3 T MRI-scanner as well as their respective aftereffects after participants left the scanner. Eye movements and overt spatial attention in a visual search task were assessed in healthy volunteers before, during, and after a one-hour MVS period. All participants exhibited a VOR inside the scanner, which declined over time but never vanished completely. Importantly, there was also an MVS-induced horizontal bias in spatial attention and exploration, which persisted throughout the entire hour within the scanner. Upon exiting the scanner, we observed aftereffects in the opposite direction manifested in both the VOR and in spatial attention, which were statistically no longer detectable after 7 min. Sustained MVS effects on spatial attention have important implications for the design and interpretation of fMRI-studies and for the development of therapeutic interventions counteracting spatial neglect.

The role of prediction and visual tracking strategies during manual interception: An exploration of individual differences.

The interception (or avoidance) of moving objects is a common component of various daily living tasks; however, it remains unclear whether precise alignment of foveal vision with a target is important for motor performance. Furthermore, there has also been little examination of individual differences in visual tracking strategy and the use of anticipatory gaze adjustments. We examined the importance of in-flight tracking and predictive visual behaviors using a virtual reality environment that required participants (n = 41) to intercept tennis balls projected from one of two possible locations. Here, we explored whether different tracking strategies spontaneously arose during the task, and which were most effective. Although indices of closer in-flight tracking (pursuit gain, tracking coherence, tracking lag, and saccades) were predictive of better interception performance, these relationships were rather weak. Anticipatory gaze shifts toward the correct release location of the ball provided no benefit for subsequent interception. Nonetheless, two interceptive strategies were evident: 1) early anticipation of the ball's onset location followed by attempts to closely track the ball in flight (i.e., predictive strategy); or 2) positioning gaze between possible onset locations and then using peripheral vision to locate the moving ball (i.e., a visual pivot strategy). Despite showing much poorer in-flight foveal tracking of the ball, participants adopting a visual pivot strategy performed slightly better in the task. Overall, these results indicate that precise alignment of the fovea with the target may not be critical for interception tasks, but that observers can adopt quite varied visual guidance approaches.

Visual search for real-world scenes in patients with Alzheimer's disease and amnestic mild cognitive impairment.

BACKGROUND: Visual attention-related processes that underlie visual search behavior are impaired in both the early stages of Alzheimer's disease (AD) and amnestic mild cognitive impairment (aMCI), which is considered a risk factor for AD. Although traditional computer-based array tasks have been used to investigate visual search, information on the visual search patterns of AD and MCI patients in real-world environments is limited. AIM: The objective of this study was to evaluate the differences in visual search behaviors among individuals with AD, aMCI, and healthy controls (HCs) in real-world scenes. MATERIALS AND METHODS: A total of 92 participants were enrolled, including 28 with AD, 32 with aMCI, and 32 HCs. During the visual search task, participants were instructed to look at a single target object amid distractors, and their eye movements were recorded. RESULTS: The results indicate that patients with AD made more fixations on distractors and fewer fixations on the target, compared to patients with aMCI and HC groups. Additionally, AD patients had longer fixation durations on distractors and spent less time looking at the target than both patients with aMCI and HCs. DISCUSSION: These findings suggest that visual search behavior is impaired in patients with AD and can be distinguished from aMCI and healthy individuals. For future studies, it is important to longitudinally monitor visual search behavior in the progression from aMCI to AD. CONCLUSION: Our study holds significance in elucidating the interplay between impairments in attention, visual processes, and other underlying cognitive processes, which contribute to the functional decline observed in individuals with AD and aMCI.

A Multimodal Assistive-Robotic-Arm Control System to Increase Independence After Tetraplegia.

Following tetraplegia, independence for completing essential daily tasks, such as opening doors and eating, significantly declines. Assistive robotic manipulators (ARMs) could restore independence, but typically input devices for these manipulators require functional use of the hands. We created and validated a hands-free multimodal input system for controlling an ARM in virtual reality using combinations of a gyroscope, eye-tracking, and heterologous surface electromyography (sEMG). These input modalities are mapped to ARM functions based on the user's preferences and to maximize the utility of their residual volitional capabilities following tetraplegia. The two participants in this study with tetraplegia preferred to use the control mapping with sEMG button functions and disliked winking commands. Non-disabled participants were more varied in their preferences and performance, further suggesting that customizability is an advantageous component of the control system. Replacing buttons from a traditional handheld controller with sEMG did not substantively reduce performance. The system provided adequate control to all participants to complete functional tasks in virtual reality such as opening door handles, turning stove dials, eating, and drinking, all of which enable independence and improved quality of life for these individuals.

Classification of short and long term mild traumatic brain injury using computerized eye tracking.

Accurate, and objective diagnosis of brain injury remains challenging. This study evaluated useability and reliability of computerized eye-tracker assessments (CEAs) designed to assess oculomotor function, visual attention/processing, and selective attention in recent mild traumatic brain injury (mTBI), persistent post-concussion syndrome (PPCS), and controls. Tests included egocentric localisation, fixation-stability, smooth-pursuit, saccades, Stroop, and the vestibulo-ocular reflex (VOR). Thirty-five healthy adults performed the CEA battery twice to assess useability and test-retest reliability. In separate experiments, CEA data from 55 healthy, 20 mTBI, and 40 PPCS adults were used to train a machine learning model to categorize participants into control, mTBI, or PPCS classes. Intraclass correlation coefficients demonstrated moderate (ICC > .50) to excellent (ICC > .98) reliability (p < .05) and satisfactory CEA compliance. Machine learning modelling categorizing participants into groups of control, mTBI, and PPCS performed reasonably (balanced accuracy control: 0.83, mTBI: 0.66, and PPCS: 0.76, AUC-ROC: 0.82). Key outcomes were the VOR (gaze stability), fixation (vertical error), and pursuit (total error, vertical gain, and number of saccades). The CEA battery was reliable and able to differentiate healthy, mTBI, and PPCS patients reasonably well. While promising, the diagnostic model accuracy should be improved with a larger training dataset before use in clinical environments.

Column setting and text justification influence return-sweep eye movement behavior during Chinese multi-line reading.

People regularly read multi-line texts in different formats and publishers, internationally, must decide how to present text to make reading most effective and efficient. Relatively few studies have examined multi-line reading, and fewer still Chinese multi-line reading. Here, we examined whether texts presented in single or double columns, and either left-justified or fully-justified affect Chinese reading. Text format had minimal influence on overall reading time; however, it significantly impacted return-sweeps (large saccades moving the eyes from the end of one line of text to the beginning of the next). Return-sweeps were launched and landed further away from margins and involved more corrective saccades in single- than double-column format. For left- compared to fully-justified format, return-sweeps were launched and landed closer to margins. More corrective saccades also occurred. Our results showed more efficient return-sweep behavior for fully- than left-justified text. Moreover, there were clear trade-off effects such that formats requiring increased numbers of shorter return-sweeps produced more accurate targeting and reduced numbers of corrective fixations, whereas formats requiring reduced numbers of longer return-sweeps caused less accurate targeting and an increased rate of corrective fixations. Overall, our results demonstrate that text formats substantially affect return-sweep eye movement behavior during Chinese reading without affecting efficiency and effectiveness, that is, the overall time it takes to read and understand the text.

Eye movement intervention facilitates concurrent perception and memory processing.

A widely used psychotherapeutic treatment for post-traumatic stress disorder (PTSD) involves performing bilateral eye movement (EM) during trauma memory retrieval. However, how this treatment-described as eye movement desensitization and reprocessing (EMDR)-alleviates trauma-related symptoms is unclear. While conventional theories suggest that bilateral EM interferes with concurrently retrieved trauma memories by taxing the limited working memory resources, here, we propose that bilateral EM actually facilitates information processing. In two EEG experiments, we replicated the bilateral EM procedure of EMDR, having participants engaging in continuous bilateral EM or receiving bilateral sensory stimulation (BS) as a control while retrieving short- or long-term memory. During EM or BS, we presented bystander images or memory cues to probe neural representations of perceptual and memory information. Multivariate pattern analysis of the EEG signals revealed that bilateral EM enhanced neural representations of simultaneously processed perceptual and memory information. This enhancement was accompanied by heightened visual responses and increased neural excitability in the occipital region. Furthermore, bilateral EM increased information transmission from the occipital to the frontoparietal region, indicating facilitated information transition from low-level perceptual representation to high-level memory representation. These findings argue for theories that emphasize information facilitation rather than disruption in the EMDR treatment.

Study on the relationship between fixation characteristics and hit rate in psychological procedure training of free throw.

BACKGROUND: Free throw is an important means of scoring in basketball games. With the improvement of basketball competition level and the enhancement of confrontation degree, the number of free throws in the game gradually increases, so the score of free throw will have an important impact on the result of the game. The purpose of this study is to explore the relationship between visual attention characteristics and hit rate of basketball players in free throw psychological procedure training, so as to provide scientific basis for basketball teaching and training. METHODS: Forty players with similar free throw abilities were randomly assigned to the experimental group (10 males, 10 females) and control group (10 males, 10 females). The experimental group was free throw psychological procedure training, while the control group was trained with routine training, Eye movement indices (number of fixations, fixation duration, and pupil dilation) and the free throw hit rate and analyzed before and after the experiment. Group differences were examined using t-tests, while paired sample t-tests were conducted to compare pre- and post-test results within each group. The training time and training times of the two groups were the same. RESULTS: There were significant differences in fixation duration, number of fixations, pupil diameter and free throw hit rate between pre-test and post-test in the experimental group (P < 0.05). Post-test, there were significant differences in number of fixations, fixation duration, pupil diameter and free throw hit rate between the two groups (P < 0.05). There was a significant positive correlation between number of fixations and free throw hit rate in top (P < 0.01), and there was a significant positive correlation between fixation duration and hit rate in front (P < 0.01). CONCLUSIONS: The psychological procedure training can improve the visual information search strategy and information processing ability of free throw, and significantly improve the free throw hit rate. There was a positive correlation between the front fixation time and the free throw hit rate, and there was a positive correlation between the top number of fixations and the free throw hit rate.

Visual Tracking in Amblyopia: A Continuous Psychophysical Approach.

Purpose: This study aimed to explore the underlying mechanisms of the observed visuomotor deficit in amblyopia. Methods: Twenty-four amblyopic (25.8 ± 3.8 years; 15 males) and 22 normal participants (25.8 ± 2.1 years; 8 males) took part in the study. The participants were instructed to continuously track a randomly moving Gaussian target on a computer screen using a mouse. In experiment 1, the participants performed the tracking task at six different target sizes. In experiments 2 and 3, they were asked to track a target with the contrast adjusted to individual's threshold. The tracking performance was represented by the kernel function calculated as the cross-correlation between the target and mouse displacements. The peak, latency, and width of the kernel were extracted and compared between the two groups. Results: In experiment 1, target size had a significant effect on the kernel peak (F(1.649, 46.170) = 200.958, P = 4.420 × 10-22). At the smallest target size, the peak in the amblyopic group was significantly lower than that in the normal group (0.089 ± 0.023 vs. 0.107 ± 0.020, t(28) = -2.390, P = 0.024) and correlated with the contrast sensitivity function (r = 0.739, P = 0.002) in the amblyopic eyes. In experiments 2 and 3, with equally visible stimuli, there were still differences in the kernel between the two groups (all Ps < 0.05). Conclusions: When stimulus visibility was compensated, amblyopic participants still showed significantly poorer tracking performance.

Oculomotor routines for perceptual judgments.

In everyday life we frequently make simple visual judgments about object properties, for example, how big or wide is a certain object? Our goal is to test whether there are also task-specific oculomotor routines that support perceptual judgments, similar to the well-established exploratory routines for haptic perception. In a first study, observers saw different scenes with two objects presented in a photorealistic virtual reality environment. Observers were asked to judge which of two objects was taller or wider while gaze was tracked. All tasks were performed with the same set of virtual objects in the same scenes, so that we can compare spatial characteristics of exploratory gaze behavior to quantify oculomotor routines for each task. Width judgments showed fixations around the center of the objects with larger horizontal spread. In contrast, for height judgments, gaze was shifted toward the top of the objects with larger vertical spread. These results suggest specific strategies in gaze behavior that presumably are used for perceptual judgments. To test the causal link between oculomotor behavior and perception, in a second study, observers could freely gaze at the object or we introduced a gaze-contingent setup forcing observers to fixate specific positions on the object. Discrimination performance was similar between free-gaze and the gaze-contingent conditions for width and height judgments. These results suggest that although gaze is adapted for different tasks, performance seems to be based on a perceptual strategy, independent of potential cues that can be provided by the oculomotor system.

Eye movements track prioritized auditory features in selective attention to natural speech.

Over the last decades, cognitive neuroscience has identified a distributed set of brain regions that are critical for attention. Strong anatomical overlap with brain regions critical for oculomotor processes suggests a joint network for attention and eye movements. However, the role of this shared network in complex, naturalistic environments remains understudied. Here, we investigated eye movements in relation to (un)attended sentences of natural speech. Combining simultaneously recorded eye tracking and magnetoencephalographic data with temporal response functions, we show that gaze tracks attended speech, a phenomenon we termed ocular speech tracking. Ocular speech tracking even differentiates a target from a distractor in a multi-speaker context and is further related to intelligibility. Moreover, we provide evidence for its contribution to neural differences in speech processing, emphasizing the necessity to consider oculomotor activity in future research and in the interpretation of neural differences in auditory cognition.

A behavioral advantage for the face pareidolia illusion in peripheral vision.

Investigation of visual illusions helps us understand how we process visual information. For example, face pareidolia, the misperception of illusory faces in objects, could be used to understand how we process real faces. However, it remains unclear whether this illusion emerges from errors in face detection or from slower, cognitive processes. Here, our logic is straightforward; if examples of face pareidolia activate the mechanisms that rapidly detect faces in visual environments, then participants will look at objects more quickly when the objects also contain illusory faces. To test this hypothesis, we sampled continuous eye movements during a fast saccadic choice task-participants were required to select either faces or food items. During this task, pairs of stimuli were positioned close to the initial fixation point or further away, in the periphery. As expected, the participants were faster to look at face targets than food targets. Importantly, we also discovered an advantage for food items with illusory faces but, this advantage was limited to the peripheral condition. These findings are among the first to demonstrate that the face pareidolia illusion persists in the periphery and, thus, it is likely to be a consequence of erroneous face detection.

Insights into the relationship between eye movements and personality traits in restricted visual fields.

Previous studies have suggested behavioral patterns, such as visual attention and eye movements, relate to individual personality traits. However, these studies mainly focused on free visual tasks, and the impact of visual field restriction remains inadequately understood. The primary objective of this study is to elucidate the patterns of conscious eye movements induced by visual field restriction and to examine how these patterns relate to individual personality traits. Building on previous research, we aim to gain new insights through two behavioral experiments, unraveling the intricate relationship between visual behaviors and individual personality traits. As a result, both Experiment 1 and Experiment 2 revealed differences in eye movements during free observation and visual field restriction. Particularly, simulation results based on the analyzed data showed clear distinctions in eye movements between free observation and visual field restriction conditions. This suggests that eye movements during free observation involve a mixture of conscious and unconscious eye movements. Furthermore, we observed significant correlations between conscious eye movements and personality traits, with more pronounced effects in the visual field restriction condition used in Experiment 2 compared to Experiment 1. These analytical findings provide a novel perspective on human cognitive processes through visual perception.

Spiking activity in the visual thalamus is coupled to pupil dynamics across temporal scales.

The processing of sensory information, even at early stages, is influenced by the internal state of the animal. Internal states, such as arousal, are often characterized by relating neural activity to a single "level" of arousal, defined by a behavioral indicator such as pupil size. In this study, we expand the understanding of arousal-related modulations in sensory systems by uncovering multiple timescales of pupil dynamics and their relationship to neural activity. Specifically, we observed a robust coupling between spiking activity in the mouse dorsolateral geniculate nucleus (dLGN) of the thalamus and pupil dynamics across timescales spanning a few seconds to several minutes. Throughout all these timescales, 2 distinct spiking modes-individual tonic spikes and tightly clustered bursts of spikes-preferred opposite phases of pupil dynamics. This multi-scale coupling reveals modulations distinct from those captured by pupil size per se, locomotion, and eye movements. Furthermore, coupling persisted even during viewing of a naturalistic movie, where it contributed to differences in the encoding of visual information. We conclude that dLGN spiking activity is under the simultaneous influence of multiple arousal-related processes associated with pupil dynamics occurring over a broad range of timescales.

Eye tracking evidence for the reinstatement of emotionally negative and neutral memories.

Recent eye tracking studies have linked gaze reinstatement-when eye movements from encoding are reinstated during retrieval-with memory performance. In this study, we investigated whether gaze reinstatement is influenced by the affective salience of information stored in memory, using an adaptation of the emotion-induced memory trade-off paradigm. Participants learned word-scene pairs, where scenes were composed of negative or neutral objects located on the left or right side of neutral backgrounds. This allowed us to measure gaze reinstatement during scene memory tests based on whether people looked at the side of the screen where the object had been located. Across two experiments, we behaviorally replicated the emotion-induced memory trade-off effect, in that negative object memory was better than neutral object memory at the expense of background memory. Furthermore, we found evidence that gaze reinstatement was related to recognition memory for the object and background scene components. This effect was generally comparable for negative and neutral memories, although the effects of valence varied somewhat between the two experiments. Together, these findings suggest that gaze reinstatement occurs independently of the processes contributing to the emotion-induced memory trade-off effect.

Oculomotor behavior can be adjusted on the basis of artificial feedback signals indicating externally caused errors.

Whether a saccade is accurate and has reached the target cannot be evaluated during its execution, but relies on post-saccadic feedback. If the eye has missed the target object, a secondary corrective saccade has to be made to align the fovea with the target. If a systematic post-saccadic error occurs, adaptive changes to the oculomotor behavior are made, such as shortening or lengthening the saccade amplitude. Systematic post-saccadic errors are typically attributed internally to erroneous motor commands. The corresponding adaptive changes to the motor command reduce the error and the need for secondary corrective saccades, and, in doing so, restore accuracy and efficiency. However, adaptive changes to the oculomotor behavior also occur if a change in saccade amplitude is beneficial for task performance, or if it is rewarded. Oculomotor learning thus is more complex than reducing a post-saccadic position error. In the current study, we used a novel oculomotor learning paradigm and investigated whether human participants are able to adapt their oculomotor behavior to improve task performance even when they attribute the error externally. The task was to indicate the intended target object among several objects to a simulated human-machine interface by making eye movements. The participants were informed that the system itself could make errors. The decoding process depended on a distorted landing point of the saccade, resulting in decoding errors. Two different types of visual feedback were added to the post-saccadic scene and we compared how participants used the different feedback types to adjust their oculomotor behavior to avoid errors. We found that task performance improved over time, regardless of the type of feedback. Thus, error feedback from the simulated human-machine interface was used for post-saccadic error evaluation. This indicates that 1) artificial visual feedback signals and 2) externally caused errors might drive adaptive changes to oculomotor behavior.

Instant interaction driven adaptive gaze control interface.

Gaze estimation is long been recognised as having potential as the basis for human-computer interaction (HCI) systems, but usability and robustness of performance remain challenging . This work focuses on systems in which there is a live video stream showing enough of the subjects face to track eye movements and some means to infer gaze location from detected eye features. Currently, systems generally require some form of calibration or set-up procedure at the start of each user session. Here we explore some simple strategies for enabling gaze based HCI to operate immediately and robustly without any explicit set-up tasks. We explore different choices of coordinate origin for combining extracted features from multiple subjects and the replacement of subject specific calibration by system initiation based on prior models. Results show that referencing all extracted features to local coordinate origins determined by subject start position enables robust immediate operation. Combining this approach with an adaptive gaze estimation model using an interactive user interface enables continuous operation with the 75th percentile gaze errors of 0.7 ∘ , and maximum gaze errors of 1.7 ∘ during prospective testing. There constitute state-of-the-art results and have the potential to enable a new generation of reliable gaze based HCI systems.

Temporal dynamics of ocular torsion and vertical vergence during visual, vestibular, and visuovestibular rotations.

Ocular torsion and vertical divergence reflect the brain's sensorimotor integration of motion through the vestibulo-ocular reflex (VOR) and the optokinetic reflex (OKR) to roll rotations. Torsion and vergence however express different response patterns depending on several motion variables, but research on their temporal dynamics remains limited. This study investigated the onset times of ocular torsion (OT) and vertical vergence (VV) during visual, vestibular, and visuovestibular motion, as well as their relative decay rates following prolonged optokinetic stimulations. Temporal characteristics were retrieved from three separate investigations where the level of visual clutter and acceleration were controlled. Video eye-tracking was used to retrieve the eye-movement parameters from a total of 41 healthy participants across all trials. Ocular torsion consistently initiated earlier than vertical vergence, particularly evident under intensified visual information density, and higher clutter levels were associated with more balanced decay rates. Additionally, stimulation modality and accelerations affected the onsets of both eye movements, with visuovestibular motion triggering earlier responses compared to vestibular motion, and increased accelerations leading to earlier onsets for both movements. The present study showed that joint visuovestibular responses produced more rapid onsets, indicating a synergetic sensorimotor process. It also showed that visual content acted as a fusional force during the decay period, and imposed greater influence over the torsional onset compared to vergence. Acceleration, by contrast, did not affect the temporal relationship between the two eye movements. Altogether, these findings provide insights into the sensorimotor integration of the vestibulo-ocular and optokinetic reflex arcs.

Tablet-Based Cognitive and Eye Movement Measures as Accessible Tools for Schizophrenia Assessment: Multisite Usability Study.

BACKGROUND: Schizophrenia is a complex mental disorder characterized by significant cognitive and neurobiological alterations. Impairments in cognitive function and eye movement have been known to be promising biomarkers for schizophrenia. However, cognitive assessment methods require specialized expertise. To date, data on simplified measurement tools for assessing both cognitive function and eye movement in patients with schizophrenia are lacking. OBJECTIVE: This study aims to assess the efficacy of a novel tablet-based platform combining cognitive and eye movement measures for classifying schizophrenia. METHODS: Forty-four patients with schizophrenia, 67 healthy controls, and 41 patients with other psychiatric diagnoses participated in this study from 10 sites across Japan. A free-viewing eye movement task and 2 cognitive assessment tools (Codebreaker task from the THINC-integrated tool and the CognitiveFunctionTest app) were used for conducting assessments in a 12.9-inch iPad Pro. We performed comparative group and logistic regression analyses for evaluating the diagnostic efficacy of the 3 measures of interest. RESULTS: Cognitive and eye movement measures differed significantly between patients with schizophrenia and healthy controls (all 3 measures; P<.001). The Codebreaker task showed the highest classification effectiveness in distinguishing schizophrenia with an area under the receiver operating characteristic curve of 0.90. Combining cognitive and eye movement measures further improved accuracy with a maximum area under the receiver operating characteristic curve of 0.94. Cognitive measures were more effective in differentiating patients with schizophrenia from healthy controls, whereas eye movement measures better differentiated schizophrenia from other psychiatric conditions. CONCLUSIONS: This multisite study demonstrates the feasibility and effectiveness of a tablet-based app for assessing cognitive functioning and eye movements in patients with schizophrenia. Our results suggest the potential of tablet-based assessments of cognitive function and eye movement as simple and accessible evaluation tools, which may be useful for future clinical implementation.