A Real-Time Eye Tracking System for the Detection of Eye Blinks.

INTRODUCTION: This paper proposes an eye blink detection system that automatically detects eye blinks, which can be an indicator of fatigue or cognitive load, among others. As a key feature, the real-time capability of the system is being required to use it, for example, as a monitoring system for people in potentially critical situations (e.g., drivers or operators of heavy machinery). METHODS: The system uses the Viola-Jones algorithm for face detection and the median flow tracker to track the face in video sequences. Eye detection is implemented using face proportions, and template matching is used for blink detection. RESULTS: The resulting system processes 40-47 frames per second on default consumer hardware and achieves an accuracy of 80.33% and a precision of 85.22% in the evaluation. DISCUSSION: The proposed system shows promising results under ideal viewing conditions but has difficulty maintaining high precision during head movements. The proposed system could be integrated with various health-related assistance systems to monitor the individual's well-being in real time, as long as their head is observed from the front if possible.

Biomechanical Analysis of Golf Swing Motion Using Hilbert-Huang Transform.

In golf swing analysis, high-speed cameras and Trackman devices are traditionally used to collect data about the club, ball, and putt. However, these tools are costly and often inaccessible to golfers. This research proposes an alternative solution, employing an affordable inertial motion capture system to record golf swing movements accurately. The focus is discerning the differences between motions producing straight and slice trajectories. Commonly, the opening motion of the body's left half and the head-up motion are associated with a slice trajectory. We employ the Hilbert-Huang transform (HHT) to examine these motions in detail to conduct a biomechanical analysis. The gathered data are then processed through HHT, calculating their instantaneous frequency and amplitude. The research found discernible differences between straight and slice trajectories in the golf swing's moment of impact within the instantaneous frequency domain. An average golfer, a single handicapper, and three beginner golfers were selected as the subjects in this study and analyzed using the proposed method, respectively. For the average golfer, the head and the left leg amplitudes of the swing motions increase at the moment of impact of the swings, resulting in the slice trajectory. These results indicate that an opening of the legs and head-up movements have been detected and extracted as non-linear frequency components, reviewing the biomechanical meaning in slice trajectory motion. For the single handicapper, the hip and left arm joints could be the target joints to detect the biomechanical motion that triggered the slice trajectory. For the beginners, since their golf swing forms were not finalized, the biomechanical motions regarding slice trajectory were different from each swing, indicating that beginner golfers need more practice to fix their golf swing form first. These results revealed that our proposed framework applied to different golf levels and could help golfers to improve their golf swing skills to achieve straight trajectories.

Comparing Synchronicity in Body Movement among Jazz Musicians with Their Emotions.

This paper presents novel preliminary research that investigates the relationship between the flow of a group of jazz musicians, quantified through multi-person pose synchronization, and their collective emotions. We have developed a real-time software to calculate the physical synchronicity of team members by tracking the difference in arm, leg, and head movements using Lightweight OpenPose. We employ facial expression recognition to evaluate the musicians' collective emotions. Through correlation and regression analysis, we establish that higher levels of synchronized body and head movements correspond to lower levels of disgust, anger, sadness, and higher levels of joy among the musicians. Furthermore, we utilize 1-D CNNs to predict the collective emotions of the musicians. The model leverages 17 body synchrony keypoint vectors as features, resulting in a training accuracy of 61.47% and a test accuracy of 66.17%.

OperatorEYEVP: Operator Dataset for Fatigue Detection Based on Eye Movements, Heart Rate Data, and Video Information.

Detection of fatigue is extremely important in the development of different kinds of preventive systems (such as driver monitoring or operator monitoring for accident prevention). The presence of fatigue for this task should be determined with physiological and objective behavioral indicators. To develop an effective model of fatigue detection, it is important to record a dataset with people in a state of fatigue as well as in a normal state. We carried out data collection using an eye tracker, a video camera, a stage camera, and a heart rate monitor to record a different kind of signal to analyze them. In our proposed dataset, 10 participants took part in the experiment and recorded data 3 times a day for 8 days. They performed different types of activity (choice reaction time, reading, correction test Landolt rings, playing Tetris), imitating everyday tasks. Our dataset is useful for studying fatigue and finding indicators of its manifestation. We have analyzed datasets that have public access to find the best for this task. Each of them contains data of eye movements and other types of data. We evaluated each of them to determine their suitability for fatigue studies, but none of them fully fit the fatigue detection task. We evaluated the recorded dataset by calculating the correspondences between eye-tracking data and CRT (choice reaction time) that show the presence of fatigue.

Head movement direction in football - a field study on visual scanning activity during the UEFA-U17 and -U21 European Championship 2019.

Visual exploration (scanning) of one's environment is a key aspect in team sports. Based on Gibson's (1979) ecological approach of visual perception, this study aims to advance the understanding of scanning by focusing on the direction of head movements in football and its implications for subsequent on-ball actions. The video-based data analysis consisted of nine selected matches and 162 players of the Union of European Football Associations (UEFA) U17 and U21 European Championship 2019. The results indicate that the direction of the last scan prior to receiving the ball is related to the foot used for the first ball contact. This relationship was further analysed in view of the game context (direction of play and opponent pressure) and with information about the player's dominant foot. The findings reveal a relationship between the direction of the last scan before receiving the ball and the direction the game is proceeded in. Further, when a player performs the last scan to the side of their dominant foot, the probability increases that their dominant foot is used for the subsequent first ball contact. Depending on the direction of the last scan, opponent pressure had various effects on the foot used for the first contact.

Importance of head movements in gaze tracking during table tennis forehand stroke.

The purpose of this study was to clarify the properties of gaze and head movements during forehand stroke in table tennis. Collegiate table tennis players (n = 12) conducted forehand strokes toward a ball launched by a skilled experimenter. A total of ten trials were conducted for the experimental task. Horizontal and vertical movements of the ball, gaze, head and eye were analyzed from the image recorded by an eye tracking device. The results showed that participants did not always keep their gaze and head position on the ball throughout the entire ball path. Our results indicate that table tennis players tend to gaze at the ball in the initial ball-tracking phase. Furthermore, there was a significant negative correlation between eye and head position especially in the vertical direction. This result suggests that horizontal VOR is suppressed more than vertical VOR in ball-tracking during table tennis forehand stroke. Finally, multiple regression analysis showed that the effect of head position to gaze position was significantly higher than that of eye position. This result indicates that gaze position during forehand stroke could be associated with head position rather than eye position. Taken together, head movements may play an important role in maintaining the ball in a constant egocentric direction in table tennis forehand stroke.

Gaze Point Tracking Based on a Robotic Body-Head-Eye Coordination Method.

When the magnitude of a gaze is too large, human beings change the orientation of their head or body to assist their eyes in tracking targets because saccade alone is insufficient to keep a target at the center region of the retina. To make a robot gaze at targets rapidly and stably (as a human does), it is necessary to design a body-head-eye coordinated motion control strategy. A robot system equipped with eyes and a head is designed in this paper. Gaze point tracking problems are divided into two sub-problems: in situ gaze point tracking and approaching gaze point tracking. In the in situ gaze tracking state, the desired positions of the eye, head and body are calculated on the basis of minimizing resource consumption and maximizing stability. In the approaching gaze point tracking state, the robot is expected to approach the object at a zero angle. In the process of tracking, the three-dimensional (3D) coordinates of the object are obtained by the bionic eye and then converted to the head coordinate system and the mobile robot coordinate system. The desired positions of the head, eyes and body are obtained according to the object's 3D coordinates. Then, using sophisticated motor control methods, the head, eyes and body are controlled to the desired position. This method avoids the complex process of adjusting control parameters and does not require the design of complex control algorithms. Based on this strategy, in situ gaze point tracking and approaching gaze point tracking experiments are performed by the robot. The experimental results show that body-head-eye coordination gaze point tracking based on the 3D coordinates of an object is feasible. This paper provides a new method that differs from the traditional two-dimensional image-based method for robotic body-head-eye gaze point tracking.

Serial dependencies in visual stability during self-motion.

Every time we move our head, the brain must decide whether the displacement of the visual scene is the result of external or self-produced motion. Gaze shifts generate the biggest and most frequent disturbance of vision. Visual stability during gaze shifts is necessary for both, dissociating self-produced from external motion and retaining bodily balance. Here, we asked participants to perform an eye-head gaze shift to a target that was briefly presented in a head-mounted display. We manipulated the velocity of the scene displacement across trials such that the background moved either too fast or too slow in relation to the head movement speed. Participants were required to report whether they perceived the gaze-contingent visual motion as faster or slower than what they would expect from their head movement velocity. We found that the point of visual stability was attracted to the velocity presented in the previous trial. Our data reveal that serial dependencies in visual stability calibrate the mapping between motor-related signals coding head movement velocity and visual motion velocity. This process is likely to aid in visual stability as the accuracy of this mapping is crucial to maintain visual stability during self-motion.NEW & NOTEWORTHY We report that visual stability during self-motion is maintained by serial dependencies between the current and the previous gaze-contingent visual velocity that was experienced during a head movement. The gaze-contingent scene displacement velocity that appears normal to us thus depends on what we have registered in the recent history of gaze shifts. Serial dependencies provide an efficient means to maintain visual stability during self-motion.

Relation between the kinematic synergy controlling swing foot and visual exploration during obstacle crossing.

To step over obstacles of varying heights, two distinct ongoing streams of activities-visual exploration of the environment and gait adjustment- were required to occur concurrently without interfering each other. Yet, it remains unclear whether and how the manner of embodied behavior of visual exploration is related to the synergistic control of foot trajectory to negotiate with the irregular terrain. Thus, we aimed to explore that how the synergistic control of the vertical trajectory of the swing foot (i.e., obstacle clearance) crossing an obstacle is related to the manner of visual exploration of the environment during approach. Twenty healthy young adults crossed an obstacle (depth: 1 cm, width: 60 cm, height: 8 cm) during their comfortable-speed walking. The visual exploration was evaluated as the amount of time spent in fixating the vicinity of the obstacle on the floor during the period from two to four steps prior to crossing the obstacle, and the strengths of kinematic synergy to control obstacle clearance were estimated using the uncontrolled manifold approach. We found that the participants with relatively weak synergy spent more time fixating at the vicinity of the obstacle from two to four steps prior to crossing the obstacle, and those participants exhibited greater amount of head flexion movement compared to those with stronger kinematic synergy. Taking advantage of this complex relationship between exploratory activities (e.g. looking movement) and performative activities (e.g. adjustment of ground clearance) would be crucial to adapt walking in a complex environment.

Simultaneous multislice EPI prospective motion correction by real-time receiver phase correction and coil sensitivity map interpolation.

PURPOSE: To improve the image reconstruction for prospective motion correction (PMC) of simultaneous multislice (SMS) EPI of the brain, an update of receiver phase and resampling of coil sensitivities are proposed and evaluated. METHODS: A camera-based system was used to track head motion (3 translations and 3 rotations) and dynamically update the scan position and orientation. We derived the change in receiver phase associated with a shifted field of view (FOV) and applied it in real-time to each k-space line of the EPI readout trains. Second, for the SMS reconstruction, we adapted resampled coil sensitivity profiles reflecting the movement of slices. Single-shot gradient-echo SMS-EPI scans were performed in phantoms and human subjects for validation. RESULTS: Brain SMS-EPI scans in the presence of motion with PMC and no phase correction for scan plane shift showed noticeable artifacts. These artifacts were visually and quantitatively attenuated when corrections were enabled. Correcting misaligned coil sensitivity maps improved the temporal SNR (tSNR) of time series by 24% (p = 0.0007) for scans with large movements (up to ˜35 mm and 30°). Correcting the receiver phase improved the tSNR of a scan with minimal head movement by 50% from 50 to 75 for a United Kingdom biobank protocol. CONCLUSION: Reconstruction-induced motion artifacts in single-shot SMS-EPI scans acquired with PMC can be removed by dynamically adjusting the receiver phase of each line across EPI readout trains and updating coil sensitivity profiles during reconstruction. The method may be a valuable tool for SMS-EPI scans in the presence of subject motion.

Two feedback mechanisms involved in the control of leaf fragment size in leaf-cutting ants.

Polymorphic leaf-cutting ants harvest leaf fragments that correlate in size with the workers' body size. When cutting, workers anchor their hind legs on the leaf edge and rotate, removing approximately semicircular fragments. Workers show behavioural plasticity and modify their leg extension while holding onto the leaf edge depending on, for instance, leaf toughness, cutting smaller fragments out of tough leaves. What sensory information workers use to control the cutting trajectory remains unknown. We investigated whether sensory information from both the leg contact with the leaf edge and from head movements underlies fragment size determination. In the laboratory, we recorded Atta sexdens workers cutting standardised ®Parafilm pseudoleaves of different thickness, and quantified cutting behaviour and body reach, i.e. the distance between the mandible and the anchored hind leg tarsus. Experimentally preventing contact with the leaf edge resulted in smaller fragments, evincing that workers control the cutting trajectory using information from the contact of the hind legs with the leaf edge. However, ants were able to cut fragments even when contact of all six legs with the edge was prevented, indicating the use of additional sensory information. Ablation of mechanosensory hairs at the neck joint alone did not influence fragment size determination, yet simultaneously preventing sensory feedback from both mechanosensory hairs and edge contact led to a loss of control over the cutting trajectory. Leaf-cutting ants, therefore, control their cutting trajectory using sensory information from both the leg contact with the leaf edge and the lateral bending of the head.

Head and body cues guide eye movements and facilitate target search in real-world videos.

Static gaze cues presented in central vision result in observer shifts of covert attention and eye movements, and benefits in perceptual performance in the detection of simple targets. Less is known about how dynamic gazer behaviors with head and body motion influence search eye movements and performance in perceptual tasks in real-world scenes. Participants searched for a target person (yes/no task, 50% presence), whereas watching videos of one to three gazers looking at a designated person (50% valid gaze cue, looking at the target). To assess the contributions of different body parts, we digitally erase parts of the gazers in the videos to create three different body parts/whole conditions for gazers: floating heads (only head movements), headless bodies (only lower body movements), and the baseline condition with intact head and body. We show that valid dynamic gaze cues guided participants' eye movements (up to 3 fixations) closer to the target, speeded the time to foveate the target, reduced fixations to the gazers, and improved target detection. The effect of gaze cues in guiding eye movements to the search target was the smallest when the gazer's head was removed from the videos. To assess the inherent information about gaze goal location for each body parts/whole condition, we collected perceptual judgments estimating gaze goals by a separate group of observers with unlimited time. Observers' perceptual judgments showed larger estimate errors when the gazer's head was removed. This suggests that the reduced eye movement guidance from lower body cueing is related to observers' difficulty extracting gaze information without the presence of the head. Together, the study extends previous work by evaluating the impact of dynamic gazer behaviors on search with videos of real-world cluttered scenes.

Development of integrated jaw-neck motor function in children at 6, 10 and 13 years of age compared to adults: A kinematic longitudinal study.

BACKGROUND: The functional integration of the jaw and neck motor systems, of great importance to everyday oral activities, is established in early childhood. Detailed characterisation of this developmental progress is largely unknown. OBJECTIVE: To establish developmental changes in jaw-neck motor function in children over the ages 6-13 years compared to adults. METHODS: Jaw and head movement kinematics during jaw opening-closing and chewing were longitudinally recorded in 20 Swedish children (8 girls) at 6 (6.3 ± 0.4), 10 (10.3 ± 0.3) and 13 (13.5 ± 0.7) years of age and 20 adults (9 women, 28.2 ± 6.7). Movement amplitudes, jaw movement cycle time (CT), coefficient of variation (CV) and head/jaw ratio for amplitudes were analysed. Linear mixed effect analysis and Welch's t-test were used. RESULTS: Children showed pronounced movement variability and longer CT at 6 and 10 years old during opening and chewing (p < .001). Compared to adults, 6-year-olds showed higher head/jaw ratios (p < .02) and longer CT (p < .001) during opening and chewing, and higher CV-head (p < .001) during chewing. Whereas 10-year-olds showed larger jaw and head amplitudes (p < .02) and longer CT (p < .001) during opening, and longer CT (p < .001) and higher CV-head (p < .001) during chewing. For 13-year-olds, longer CT (p < .001) during chewing was found. CONCLUSION: Children showed pronounced movement variability and longer movement cycle time at 6-10 years and developmental progress in jaw-neck integration from 6 to 13 years, with 13-year-olds displaying adult-like movements. These results add new detailed understanding to the typical development of integrated jaw-neck motor function.

Hindbrain modules differentially transform activity of single collicular neurons to coordinate movements.

Seemingly simple behaviors such as swatting a mosquito or glancing at a signpost involve the precise coordination of multiple body parts. Neural control of coordinated movements is widely thought to entail transforming a desired overall displacement into displacements for each body part. Here we reveal a different logic implemented in the mouse gaze system. Stimulating superior colliculus (SC) elicits head movements with stereotyped displacements but eye movements with stereotyped endpoints. This is achieved by individual SC neurons whose branched axons innervate modules in medulla and pons that drive head movements with stereotyped displacements and eye movements with stereotyped endpoints, respectively. Thus, single neurons specify a mixture of endpoints and displacements for different body parts, not overall displacement, with displacements for different body parts computed at distinct anatomical stages. Our study establishes an approach for unraveling motor hierarchies and identifies a logic for coordinating movements and the resulting pose.

Measuring real-time cognitive engagement in remote audiences.

Responses to arts and entertainment media offer a valuable window into human behaviour. Many individuals worldwide spend the vast majority of their leisure time engaging with video content at home. However, there are few ways to study engagement and attention in this natural home viewing context. We used motion-tracking of the head via a web-camera to measure real-time cognitive engagement in 132 individuals while they watched 30 min of streamed theatre content at home. Head movement was negatively associated with engagement across a constellation of measures. Individuals who moved less reported feeling more engaged and immersed, evaluated the performance as more engaging, and were more likely to express interest in watching further. Our results demonstrate the value of in-home remote motion tracking as a low-cost, scalable metric of cognitive engagement, which can be used to collect audience behaviour data in a natural setting.

Quantifying MR head motion in the Rhineland Study - A robust method for population cohorts.

Head motion during MR acquisition reduces image quality and has been shown to bias neuromorphometric analysis. The quantification of head motion, therefore, has both neuroscientific as well as clinical applications, for example, to control for motion in statistical analyses of brain morphology, or as a variable of interest in neurological studies. The accuracy of markerless optical head tracking, however, is largely unexplored. Furthermore, no quantitative analysis of head motion in a general, mostly healthy population cohort exists thus far. In this work, we present a robust registration method for the alignment of depth camera data that sensitively estimates even small head movements of compliant participants. Our method outperforms the vendor-supplied method in three validation experiments: 1. similarity to fMRI motion traces as a low-frequency reference, 2. recovery of the independently acquired breathing signal as a high-frequency reference, and 3. correlation with image-based quality metrics in structural T1-weighted MRI. In addition to the core algorithm, we establish an analysis pipeline that computes average motion scores per time interval or per sequence for inclusion in downstream analyses. We apply the pipeline in the Rhineland Study, a large population cohort study, where we replicate age and body mass index (BMI) as motion correlates and show that head motion significantly increases over the duration of the scan session. We observe weak, yet significant interactions between this within-session increase and age, BMI, and sex. High correlations between fMRI and camera-based motion scores of proceeding sequences further suggest that fMRI motion estimates can be used as a surrogate score in the absence of better measures to control for motion in statistical analyses.

Reduced Vestibulo-Ocular Reflex During Fast Head Rotation in Complete Darkness.

The human vestibulo-ocular reflex (VOR) leads to maintenance of the acuity of an image on the retina and contributes to the perception of orientation during high acceleration head movements. Our objective was to determine whether vision affects the horizontal VOR by assessing and comparing the performance at the boundaries of contribution of: (a) unrestricted visual information and (b) no visual information. Understanding how the VOR performs under both lighted and unlighted conditions is of paramount importance to avoiding falls, perhaps particularly among the elderly. We tested 23 participants (M age = 35.3 years, standard error of mean (SEM) = 2.0 years). The participants were tested with the video Head Impulse Test (vHIT), EyeSeeCam from Interacoustics™, which assesses whether VOR is of the expected angular velocity compared to head movement angular velocity. The vHIT tests were performed under two conditions: (a) in a well-lit room and (b) in complete darkness. The VOR was analyzed by evaluating the gain (quotient between eye and head angular velocity) at 40, 60 and 80 ms time stamps after the start of head movement. Additionally, we calculated the approximate linear gain between 0-100 ms through regression. The gain decreased significantly faster across time stamps in complete darkness (p < .001), by 10% in darkness compared with a 2% decrease in light. In complete darkness, the VOR gain gradually declined, reaching a marked reduction at 80 ms by 10% (p < .001), at which the head velocities were 150°/second or faster. The approximate linear gain value was not significantly different in complete darkness and in light. These findings suggest that information from the visual system can modulate the high velocity VOR. Subsequently, fast head turns might cause postural imbalance and momentary disorientation in poor light in people with reduced sensory discrimination or motor control, like the elderly.

Brief mock-scan training reduces head motion during real scanning for children: A growth curve study.

Pediatric neuroimaging datasets are rapidly increasing in scales. Despite strict protocols in data collection and preprocessing focused on improving data quality, the presence of head motion still impedes our understanding of neurodevelopmental mechanisms. Large head motion can lead to severe noise and artifacts in magnetic resonance imaging (MRI) studies, inflating correlations between adjacent brain areas and decreasing correlations between spatial distant territories, especially in children and adolescents. Here, by leveraging mock-scans of 123 Chinese children and adolescents, we demonstrated the presence of increased head motion in younger participants. Critically, a 5.5-minute training session in an MRI mock scanner was found to effectively suppress the head motion in the children and adolescents. Therefore, we suggest that mock scanner training should be part of the quality assurance routine prior to formal MRI data collection, particularly in large-scale population-level neuroimaging initiatives for pediatrics.

Cognition Guided Human-Object Relationship Detection.

Human-object relationship detection reveals the fine-grained relationship between humans and objects, helping the comprehensive understanding of videos. Previous human-object relationship detection approaches are mainly developed with object features and relation features without exploring the specific information of humans. In this paper, we propose a novel Relation-Pose Transformer (RPT) for human-object relationship detection. Inspired by the coordination of eye-head-body movements in cognitive science, we employ the head pose to find those crucial objects that humans focus on and use the body pose with skeleton information to represent multiple actions. Then, we utilize the spatial encoder to capture spatial contextualized information of the relation pair, which integrates the relation features and pose features. Next, the temporal decoder aims to model the temporal dependency of the relationship. Finally, we adopt multiple classifiers to predict different types of relationships. Extensive experiments on the benchmark Action Genome validate the effectiveness of our proposed method and show the state-of-the-art performance compared with related methods.

Improved Motion Artifact Correction in fNIRS Data by Combining Wavelet and Correlation-Based Signal Improvement.

Functional near-infrared spectroscopy (fNIRS) is an optical non-invasive neuroimaging technique that allows participants to move relatively freely. However, head movements frequently cause optode movements relative to the head, leading to motion artifacts (MA) in the measured signal. Here, we propose an improved algorithmic approach for MA correction that combines wavelet and correlation-based signal improvement (WCBSI). We compare its MA correction accuracy to multiple established correction approaches (spline interpolation, spline-Savitzky-Golay filter, principal component analysis, targeted principal component analysis, robust locally weighted regression smoothing filter, wavelet filter, and correlation-based signal improvement) on real data. Therefore, we measured brain activity in 20 participants performing a hand-tapping task and simultaneously moving their head to produce MAs at different levels of severity. In order to obtain a "ground truth" brain activation, we added a condition in which only the tapping task was performed. We compared the MA correction performance among the algorithms on four predefined metrics (R, RMSE, MAPE, and ΔAUC) and ranked the performances. The suggested WCBSI algorithm was the only one exceeding average performance (p < 0.001), and it had the highest probability to be the best ranked algorithm (78.8% probability). Together, our results indicate that among all algorithms tested, our suggested WCBSI approach performed consistently favorably across all measures.