Depth from motion parallax: Deictic consistency, eye contact, and a serious problem with Zoom.

The dynamics of head and eye gaze between two or more individuals displayed during verbal and nonverbal face-to-face communication contains a wealth of information and is used for both volitionary and unconscious signaling. Current video communication systems convey visual signals about gaze behavior and other directional cues, but the information they carry is often spurious and potentially misleading. I discuss the consequences of this situation, identify the source of the problem as a more general lack of deictic consistency, and demonstrate that using display technologies that simulate motion parallax are both necessary and sufficient to alleviate it. I then devise an avatar-based remote communication solution that achieves deictic consistency and provides natural, dynamic eye contact for computer-mediated audiovisual communication.

Zoom disrupts eye contact behaviour: problems and solutions.

Natural, dynamic eye contact behaviour is critical to social interaction but is dysfunctional in video conferencing. In analysing the problem, I introduce the concept of directionality and emphasize the critical role of motion parallax. I then sketch approaches towards re-establishing directionality and enabling natural, dynamic eye contact in video conferences.

Life Detection From Biological Motion.

Life motion, the active movements of people and other animals, contains a wealth of information that is potentially accessible to the visual system of an observer. Biological-motion point-light displays have been widely used to study both the information contained in life motion stimuli and the visual mechanisms that make use of it. Biological motion conveys motion-mediated dynamic shape, which in turn can be used for identification and recognition of the agent, but it also contains local visual invariants that humans and other animals use as a general detection system that signals the presence of other agents in the visual environment. Here, we review recent research on behavioral, neurophysiological, and genetic aspects of this life-detection system and discuss its functional significance in the light of earlier hypotheses.

Does belief in free will influence biological motion perception?

Previous research suggests that belief in free will correlates with intentionality attribution. However, whether belief in free will is also related to more basic social processes is unknown. Based on evidence that biological motion contains intentionality cues that observers spontaneously extract, we investigate whether people who believe more in free will, or in related constructs, such as dualism and determinism, would be better at picking up such cues and therefore at detecting biological agents hidden in noise, or would be more inclined to detect intentionality cues and therefore to detect biological agents even when there are none. Signal detection theory was used to measure participants' ability to detect biological motion from scrambled background noise (d') and their response bias (c) in doing so. In two experiments, we found that belief in determinism and belief in dualism, but not belief in free will, were associated with biological motion perception. However, no causal effect was found when experimentally manipulating free will-related beliefs. In sum, our results show that biological motion perception, a low-level social process, is related to high-level beliefs about dualism and determinism.

Human (but not animal) motion can be recognized at first sight - After treatment for congenital blindness.

The long-standing nativist vs. empiricist debate asks a foundational question in epistemology - does our knowledge arise through experience or is it available innately? Studies that probe the sensitivity of newborns and patients recovering from congenital blindness are central in informing this dialogue. One of the most robust sensitivities our visual system possesses is to 'biological motion' - the movement patterns of humans and other vertebrates. Various biological motion perception skills (such as distinguishing between movement of human and non-human animals, or between upright and inverted human movement) become evident within the first months of life. The mechanisms of acquiring these capabilities, and specifically the contribution of visual experience to their development, are still under debate. We had the opportunity to directly examine the role of visual experience in biological motion perception, by testing what level of sensitivity is present immediately upon onset of sight following years of congenital visual deprivation. Two congenitally blind patients who underwent sight-restorative cataract-removal surgery late in life (at the ages of 7 and 20 years) were tested before and after sight restoration. The patients were shown displays of walking humans, pigeons, and cats, and asked to describe what they saw. Visual recognition of movement patterns emerged immediately upon eye-opening following surgery, when the patients spontaneously began to identify human, but not animal, biological motion. This recognition ability was evident contemporaneously for upright and inverted human displays. These findings suggest that visual recognition of human motion patterns may not critically depend on visual experience, as it was evident upon first exposure to un-obstructed sight in patients with very limited prior visual exposure, and furthermore, was not limited to the typical (upright) orientation of humans in real-life settings.

Movement markers of schizophrenia: a detailed analysis of patients' gait patterns.

Motor abnormalities occur in the majority of persons with schizophrenia but are generally neglected in clinical care. Psychiatric diagnostics fail to include quantifiable motor variables and few assessment tools examine full-body movement. We assessed full-body movement during gait of 20 patients and 20 controls with motion capture technology, symptom load (PANSS, BPRS) and Neurological Soft Signs (NSS). In a data-driven analysis, participants' motion patterns were quantified and compared between groups. Resulting movement markers (MM) were correlated with the clinical assessment. We identified 16 quantifiable MM of schizophrenia. While walking, patients and controls display significant differences in movement patterns related to posture, velocity, regularity of gait as well as sway, flexibility and integration of body parts. Specifically, the adjustment of body sides, limbs and movement direction were affected. The MM remain significant when controlling for medication load. They are systematically related to NSS. Results add assessment tools, analysis methods as well as theory-independent MM to the growing body of research on motor abnormalities in schizophrenia.

Electrophysiological and behavioral indicators of musical knowledge about unfamiliar music.

Most listeners possess sophisticated knowledge about the music around them without being aware of it or its intricacies. Previous research shows that we develop such knowledge through exposure. This knowledge can then be assessed using behavioral and neurophysiological measures. It remains unknown however, which neurophysiological measures accompany the development of musical long-term knowledge. In this series of experiments, we first identified a potential ERP marker of musical long-term knowledge by comparing EEG activity following musically unexpected and expected tones within the context of known music (n = 30). We then validated the marker by showing that it does not differentiate between such tones within the context of unknown music (n = 34). In a third experiment, we exposed participants to unknown music (n = 40) and compared EEG data before and after exposure to explore effects of time. Although listeners' behavior indicated musical long-term knowledge, we did not find any effects of time on the ERP marker. Instead, the relationship between behavioral and EEG data suggests musical long-term knowledge may have formed before we could confirm its presence through behavioral measures. Listeners are thus not only knowledgeable about music but seem to also be incredibly fast music learners.

The role of binocular disparity and active motion parallax in cybersickness.

Cybersickness is an enduring problem for users of virtual environments. While it is generally assumed that cybersickness is caused by discrepancies in perceived self-motion between the visual and vestibular systems, little is known about the relative contribution of active motion parallax and binocular disparity to the occurrence of cybersickness. We investigated the role of these two depth cues in cybersickness by simulating a roller-coaster ride using a head-mounted display. Participants could see the tracks via a virtual frame placed at the front of the roller-coaster cart. We manipulated the state of the frame, so it behaved like: (1) a window into the virtual scene, (2) a 2D screen, (3) and (4) a window for one of the two depth cues, and a 2D screen for the other. Participants completed the Simulator Sickness Questionnaire before and after the experiment, and verbally reported their level of discomfort at repeated intervals during the ride. Additionally, participants' electrodermal activity (EDA) was recorded. The results of the questionnaire and the continuous ratings revealed the largest increase in cybersickness when the frame behaved like a window, and least increase when the frame behaved like a 2D screen. Cybersickness scores were at an intermediate level for the conditions where the frame simulated only one depth cue. This suggests that neither active motion parallax nor binocular disparity had a more prominent effect on the severity of cybersickness. The EDA responses increased at about the same rate in all conditions, suggesting that EDA is not necessarily coupled with subjectively experienced cybersickness.

MoVi: A large multi-purpose human motion and video dataset.

Large high-quality datasets of human body shape and kinematics lay the foundation for modelling and simulation approaches in computer vision, computer graphics, and biomechanics. Creating datasets that combine naturalistic recordings with high-accuracy data about ground truth body shape and pose is challenging because different motion recording systems are either optimized for one or the other. We address this issue in our dataset by using different hardware systems to record partially overlapping information and synchronized data that lend themselves to transfer learning. This multimodal dataset contains 9 hours of optical motion capture data, 17 hours of video data from 4 different points of view recorded by stationary and hand-held cameras, and 6.6 hours of inertial measurement units data recorded from 60 female and 30 male actors performing a collection of 21 everyday actions and sports movements. The processed motion capture data is also available as realistic 3D human meshes. We anticipate use of this dataset for research on human pose estimation, action recognition, motion modelling, gait analysis, and body shape reconstruction.

Spatiotemporal dynamics of responses to biological motion in the human brain.

We sought to understand the spatiotemporal characteristics of biological motion perception. We presented observers with biological motion walkers that differed in terms of form coherence or kinematics (i.e., the presence or absence of natural acceleration). Participants were asked to discriminate the facing direction of the stimuli while their magnetoencephalographic responses were concurrently imaged. We found that two univariate response components can be observed around ~200 msec and ~650 msec post-stimulus onset, each engaging lateral-occipital and parietal cortex prior to temporal and frontal cortex. Moreover, while univariate responses show biological motion form-specificity only after 300 msec, multivariate patterns specific to form can be well discriminated from those for local cues as early as 100 msec after stimulus onset. By finally examining the representational similarity of fMRI and MEG patterned responses, we show that early responses to biological motion are most likely sourced to occipital cortex while later responses likely originate from extrastriate body areas.

Does anxiety induced by social interaction influence the perception of bistable biological motion?

When observing point light walkers orthographically projected onto a frontoparallel plane, the direction in which they are walking is ambiguous. Nevertheless, observers more often perceive them as facing towards than as facing away from them. This phenomenon is known as the "facing-the-viewer bias" (FTV). Two interpretations of the facing-the-viewer bias exist in the literature: a top-down and a bottom-up interpretation. Support for the top-down interpretation comes from evidence that social anxiety correlates with the FTV bias. However, the direction of the relationship between the FTV bias and social anxiety is inconsistent across studies and evidence for a correlation has mostly been obtained with relatively small samples. Therefore, the first aim of the current study was to provide a strong test of the hypothesized relationship between social anxiety and the facing-the-viewer bias in a large sample of 200 participants recruited online. In addition, a second aim was to further extend top-down accounts by investigating if the FTV bias is also related to autistic traits. Our results replicate the FTV bias, showing that people indeed tend to perceive orthographically projected point light walkers as facing towards them. However, no correlation between the FTV bias and social interaction anxiety (tau = -0.01, p = .86, BF = 0.18) or autistic traits (tau = -0.0039, p = .45, BF = 0.18) was found. As such, our data cannot confirm the top-down interpretation of the facing-the-viewer bias.

Does co-presence affect the way we perceive and respond to emotional interactions?

This study compared how two virtual display conditions of human body expressions influenced explicit and implicit dimensions of emotion perception and response behavior in women and men. Two avatars displayed emotional interactions (angry, sad, affectionate, happy) in a "pictorial" condition depicting the emotional interactive partners on a screen within a virtual environment and a "visual" condition allowing participants to share space with the avatars, thereby enhancing co-presence and agency. Subsequently to stimulus presentation, explicit valence perception and response tendency (i.e. the explicit tendency to avoid or approach the situation) were assessed on rating scales. Implicit responses, i.e. postural and autonomic responses towards the observed interactions were measured by means of postural displacement and changes in skin conductance. Results showed that self-reported presence differed between pictorial and visual conditions, however, it was not correlated with skin conductance responses. Valence perception was only marginally influenced by the virtual condition and not at all by explicit response behavior. There were gender-mediated effects on postural response tendencies as well as gender differences in explicit response behavior but not in valence perception. Exploratory analyses revealed a link between valence perception and preferred behavioral response in women but not in men. We conclude that the display condition seems to influence automatic motivational tendencies but not higher level cognitive evaluations. Moreover, intragroup differences in explicit and implicit response behavior highlight the importance of individual factors beyond gender.

Oxytocin modulates human chemosensory decoding of sex in a dose-dependent manner.

There has been accumulating evidence of human social chemo-signaling, but the underlying mechanisms remain poorly understood. Considering the evolutionarily conserved roles of oxytocin and vasopressin in reproductive and social behaviors, we examined whether the two neuropeptides are involved in the subconscious processing of androsta-4,16,-dien-3-one and estra-1,3,5 (10),16-tetraen-3-ol, two human chemosignals that convey masculinity and femininity to the targeted recipients, respectively. Psychophysical data collected from 216 heterosexual and homosexual men across five experiments totaling 1056 testing sessions consistently showed that such chemosensory communications of masculinity and femininity were blocked by a competitive antagonist of both oxytocin and vasopressin receptors called atosiban, administered nasally. On the other hand, intranasal oxytocin, but not vasopressin, modulated the decoding of androstadienone and estratetraenol in manners that were dose-dependent, nonmonotonic, and contingent upon the recipients' social proficiency. Taken together, these findings establish a causal link between neuroendocrine factors and subconscious chemosensory communications of sex-specific information in humans.

Biological Action Identification Does Not Require Early Visual Input for Development.

Visual input during the first years of life is vital for the development of numerous visual functions. While normal development of global motion perception seems to require visual input during an early sensitive period, the detection of biological motion (BM) does not seem to do so. A more complex form of BM processing is the identification of human actions. Here, we tested whether identification rather than detection of BM is experience dependent. A group of human participants who had been treated for congenital cataracts (CC; of up to 18 years in duration, CC group) had to identify ten actions performed by human line figures. In addition, they performed a coherent motion (CM) detection task, which required identifying the direction of CM amid the movement of random dots. As controls, developmental cataract (DC) reversal individuals (DC group) who had undergone the same surgical treatment as CC group were included. Moreover, normally sighted controls were tested both with vision blurred to match the visual acuity (VA) of CC individuals [vision matched (VM) group] and with full sight [sighted control (SC) group]. The CC group identified biological actions with an extraordinary high accuracy (on average ∼85% correct) and was indistinguishable from the VM control group. By contrast, CM processing impairments of the CC group persisted even after controlling for VA. These results in the same individuals demonstrate an impressive resilience of BM processing to aberrant early visual experience and at the same time a sensitive period for the development of CM processing.

bmlTUX: Design and Control of Experiments in Virtual Reality and Beyond.

Advances in virtual reality technology have made it a valuable new tool for vision and perception researchers. Coding virtual reality experiments from scratch can be difficult and time-consuming, so researchers rely on software such as Unity game engine to create and edit virtual scenes. However, Unity lacks built-in tools for controlling experiments. Existing third-party add-ins requires complicated scripts to define experiments. This can be difficult and requires advanced coding knowledge, especially for multifactorial experimental designs. In this article, we describe a new free and open-source tool called the BiomotionLab Toolkit for Unity Experiments (bmlTUX) that provides a simple interface for controlling experiments in Unity. In contrast to existing tools, bmlTUX provides a graphical interface to automatically handle combinatorics, counterbalancing, randomization, mixed designs, and blocking of trial order. The toolbox works out-of-the-box since simple experiments can be created with almost no coding. Furthermore, multiple design configurations can be swapped with a drag-and-drop interface allowing researchers to test new configurations iteratively while maintaining the ability to easily revert to previous configurations. Despite its simplicity, bmlTUX remains highly flexible and customizable, catering to coding novices and experts alike.

Classifying Elite From Novice Athletes Using Simulated Wearable Sensor Data.

Movement screens are frequently used to identify differences in movement patterns such as pathological abnormalities or skill related differences in sport; however, abnormalities are often visually detected by a human assessor resulting in poor reliability. Therefore, our previous research has focused on the development of an objective movement assessment tool to classify elite and novice athletes' kinematic data using machine learning algorithms. Classifying elite and novice athletes can be beneficial to objectively detect differences in movement patterns between the athletes, which can then be used to provide higher quality feedback to athletes and their coaches. Currently, the method requires optical motion capture, which is expensive and time-consuming to use, creating a barrier for adoption within industry. Therefore, the purpose of this study was to assess whether machine learning could classify athletes as elite or novice using data that can be collected easily and inexpensively in the field using inertial measurement units (IMUs). A secondary purpose of this study was to refine the architecture of the tool to optimize classification rates. Motion capture data from 542 athletes performing seven dynamic screening movements were analyzed. A principal component analysis (PCA)-based pattern recognition technique and machine learning algorithms with the Euclidean norm of the segment linear accelerations and angular velocities as inputs were used to classify athletes based on skill level. Depending on the movement, using metrics achievable with IMUs and a linear discriminant analysis (LDA), 75.1-84.7% of athletes were accurately classified as elite or novice. We have provided evidence that suggests our objective, data-driven method can detect meaningful differences during a movement screening battery when using data that can be collected using IMUs, thus providing a large methodological advance as these can be collected in the field using sensors. This method offers an objective, inexpensive tool that can be easily implemented in the field to potentially enhance screening, assessment, and rehabilitation in sport and clinical settings.

Spatial coding for memory-guided reaching in visual and pictorial spaces.

An essential difference between pictorial space displayed as paintings, photographs, or computer screens, and the visual space experienced in the real world is that the observer has a defined location, and thus valid information about distance and direction of objects, in the latter but not in the former. Thus egocentric information should be more reliable in visual space, whereas allocentric information should be more reliable in pictorial space. The majority of studies relied on pictorial representations (images on a computer screen), leaving it unclear whether the same coding mechanisms apply in visual space. Using a memory-guided reaching task in virtual reality, we investigated allocentric coding in both visual space (on a table in virtual reality) and pictorial space (on a monitor that is on the table in virtual reality). Our results suggest that the brain uses allocentric information to represent objects in both pictorial and visual space. Contrary to our hypothesis, the influence of allocentric cues was stronger in visual space than in pictorial space, also after controlling for retinal stimulus size, confounding allocentric cues, and differences in presentation depth. We discuss possible reasons for stronger allocentric coding in visual than in pictorial space.

Prediction of action outcome: Effects of available information about body structure.

Correctly perceiving the movements of opponents is essential in everyday life as well as in many sports. Several studies have shown a better prediction performance for detailed stimuli compared to point-light displays (PLDs). However, it remains unclear whether differences in prediction performance result from explicit information about articulation or from information about body shape. We therefore presented three different types of stimuli (PLDs, stick figures, and skinned avatars) with different amounts of available information of soccer players' run-ups. Stimulus presentation was faded out at ball contact. Participants had to react to the perceived shot direction with a full-body movement. Results showed no differences for time to virtual ball contact between presentation modes. However, prediction performance was significantly better for avatars and stick figures compared to PLDs, but did not differ between avatars and stick figures, suggesting that explicit information about the articulation of the major joints is mainly relevant for better prediction performance, and plays a larger role than detailed information about body shape. We also tracked eye movements and found that gaze behavior for avatars differed from those for PLDs and stick figures, with no significant differences between PLDs and stick figures. This effect was due to more and longer fixations on the head when avatars were presented.

Cognition modulates action-to-perception transfer in ambiguous perception.

Can cognition penetrate action-to-perception transfer? Participants observed a structure-from-motion cylinder of ambiguous rotation direction. Beforehand, they experienced one of two mechanical models: An unambiguous cylinder was connected to a rod by either a belt (cylinder and rod rotating in the same direction) or by gears (both rotating in opposite directions). During ambiguous cylinder presentation, mechanics and rod were invisible, making both conditions visually identical. Observers inferred the rod's direction from their moment-by-moment subjective perceptual interpretation of the ambiguous cylinder. They reported the (hidden) rod's direction by rotating a manipulandum in either the same or the opposite direction. With respect to their effect on perceptual stability, the resulting match/nonmatch between perceived cylinder rotation and manipulandum rotation showed a significant interaction with the cognitive model they had previously been biased with. For the "belt" model, congruency between cylinder perception and manual action is induced by same-direction report. Here, we found that same-direction movement stabilized the perceived motion direction, replicating a known congruency effect. For the "gear" model, congruency between perception and action is-in contrast-induced by opposite-direction report. Here, no effect of perception-action congruency was found: Perceptual congruency and cognitive model nullified each other. Hence, an observer's internal model of a machine's operation guides action-to-perception transfer.