A methodological framework to assess the accuracy of virtual reality hand-tracking systems: A case study with the Meta Quest 2.

Optical markerless hand-tracking systems incorporated into virtual reality (VR) headsets are transforming the ability to assess fine motor skills in VR. This promises to have far-reaching implications for the increased applicability of VR across scientific, industrial, and clinical settings. However, so far, there are little data regarding the accuracy, delay, and overall performance of these types of hand-tracking systems. Here we present a novel methodological framework based on a fixed grid of targets, which can be easily applied to measure these systems' absolute positional error and delay. We also demonstrate a method to assess finger joint-angle accuracy. We used this framework to evaluate the Meta Quest 2 hand-tracking system. Our results showed an average fingertip positional error of 1.1cm, an average finger joint angle error of 9.6∘ and an average temporal delay of 45.0 ms. This methodological framework provides a powerful tool to ensure the reliability and validity of data originating from VR-based, markerless hand-tracking systems.

Skin properties and afferent density in the deterioration of tactile spatial acuity with age.

Tactile sensitivity is affected by age, as shown by the deterioration of spatial acuity assessed with the two-point discrimination task. This is assumed to be partly a result of age-related changes of the peripheral somatosensory system. In particular, in the elderly, the density of mechanoreceptive afferents decreases with age and the skin tends to become drier, less elastic and less stiff. To assess to what degree mechanoreceptor density, skin hydration, elasticity and stiffness can account for the deterioration of tactile spatial sensitivity observed in the elderly, several approaches were combined, including psychophysics, measurements of finger properties, modelling and simulation of the response of first-order tactile neurons. Psychophysics confirmed that the Elderly group has lower tactile acuity than the Young group. Correlation and commonality analysis showed that age was the most important factor in explaining decreases in behavioural performance. Biological elasticity, hydration and finger pad area were also involved. These results were consistent with the outcome of simulations showing that lower afferent density and lower Young's modulus (i.e. lower stiffness) negatively affected the tactile encoding of stimulus information. Simulations revealed that these changes resulted in a lower build-up of task-relevant stimulus information. Importantly, the reduction in discrimination performance with age in the simulation was less than that observed in the psychophysical testing, indicating that there are additional peripheral as well as central factors responsible for age-related changes in tactile discrimination. KEY POINTS: Ageing effects on tactile perception involve the deterioration of spatial sensitivity, although the contribution of central and peripheral factors is not clear. We combined psychophysics, measurements of finger properties, modelling and simulation of the response of first-order tactile neurons to investigate to what extent skin elasticity, stiffness, hydration, finger pad area and afferent density can account for the lower spatial sensitivity observed in the elderly. Correlation and commonality analysis revealed that age was the most important factor to predict behavioural performance. Skin biological elasticity, hydration and finger pad area contributed to a lesser extent. The simulation of first-order tactile neuron responses indicated that reduction in afferent density plays a major role in the deterioration of tactile spatial acuity. Simulations also showed that lower skin stiffness and lower afferent density affect the build-up of stimulus information and the response of SA1 (i.e. type 1 slowly adapting fibres) and RA1 (i.e. type 1 rapidly adapting fibres) afferent fibres.

Perceived time and temporal structure: Neural entrainment to isochronous stimulation increases duration estimates.

Distortions of perceived duration can give crucial insights into the mechanisms that underlie the processing and representation of stimulus timing. One factor that affects duration estimates is the temporal structure of stimuli that fill an interval. For example, regular filling (isochronous interval) leads to an overestimation of perceived duration as compared to irregular filling (anisochronous interval). In the present article, we use electroencephalography (EEG) to investigate the neural basis of this subjective lengthening of perceived duration with isochrony. In a two-interval forced choice task, participants judged which of two intervals lasts longer - one always being isochronous, the other one anisochronous. Response proportions confirm the subjective overestimation of isochronous intervals. At the neural level, isochronous sequences are associated with enhanced pairwise phase consistency (PPC) at the stimulation frequency, reflecting the brain's entrainment to the regular stimulation. The PPC over the entrainment channels is further enhanced for isochronous intervals that are reported to be longer, and the magnitude of this PCC effect correlates with the amount of perceptual bias. Neural entrainment has been proposed as a mechanism of attentional selection, enabling increased neural responsiveness toward stimuli that arrive at an expected point in time. The present results support the proposed relationship between neural response magnitudes and temporal estimates: An increase in neural responsiveness leads to a more pronounced representation of the individual stimuli filling the interval and in turn to a subjective increase in duration.

Look but don't touch: Visual cues to surface structure drive somatosensory cortex.

When planning interactions with nearby objects, our brain uses visual information to estimate shape, material composition, and surface structure before we come into contact with them. Here we analyse brain activations elicited by different types of visual appearance, measuring fMRI responses to objects that are glossy, matte, rough, or textured. In addition to activation in visual areas, we found that fMRI responses are evoked in the secondary somatosensory area (S2) when looking at glossy and rough surfaces. This activity could be reliably discriminated on the basis of tactile-related visual properties (gloss, rough, and matte), but importantly, other visual properties (i.e., coloured texture) did not substantially change fMRI activity. The activity could not be solely due to tactile imagination, as asking explicitly to imagine such surface properties did not lead to the same results. These findings suggest that visual cues to an object's surface properties evoke activity in neural circuits associated with tactile stimulation. This activation may reflect the a-priori probability of the physics of the interaction (i.e., the expectation of upcoming friction) that can be used to plan finger placement and grasp force.

Differential processing of binocular and monocular gloss cues in human visual cortex.

The visual impression of an object's surface reflectance ("gloss") relies on a range of visual cues, both monocular and binocular. Whereas previous imaging work has identified processing within ventral visual areas as important for monocular cues, little is known about cortical areas involved in processing binocular cues. Here, we used human functional MRI (fMRI) to test for brain areas selectively involved in the processing of binocular cues. We manipulated stereoscopic information to create four conditions that differed in their disparity structure and in the impression of surface gloss that they evoked. We performed multivoxel pattern analysis to find areas whose fMRI responses allow classes of stimuli to be distinguished based on their depth structure vs. material appearance. We show that higher dorsal areas play a role in processing binocular gloss information, in addition to known ventral areas involved in material processing, with ventral area lateral occipital responding to both object shape and surface material properties. Moreover, we tested for similarities between the representation of gloss from binocular cues and monocular cues. Specifically, we tested for transfer in the decoding performance of an algorithm trained on glossy vs. matte objects defined by either binocular or by monocular cues. We found transfer effects from monocular to binocular cues in dorsal visual area V3B/kinetic occipital (KO), suggesting a shared representation of the two cues in this area. These results indicate the involvement of mid- to high-level visual circuitry in the estimation of surface material properties, with V3B/KO potentially playing a role in integrating monocular and binocular cues.

Erratum: Touching with the eyes: Oculomotor self-touch induces illusory body ownership.

[This corrects the article DOI: 10.1016/j.isci.2023.106180.].

Touching with the eyes: Oculomotor self-touch induces illusory body ownership.

Self-touch plays a central role in the construction and plasticity of the bodily self. But which mechanisms support this role? Previous accounts emphasize the convergence of proprioceptive and tactile signals from the touching and the touched body parts. Here, we hypothesise that proprioceptive information is not necessary for self-touch modulation of body-ownership. Because eye movements do not rely on proprioceptive signals as limb movements do, we developed a novel oculomotor self-touch paradigm where voluntary eye movements generated corresponding tactile sensations. We then compared the effectiveness of eye versus hand self-touch movements in generating an illusion of owning a rubber hand. Voluntary oculomotor self-touch was as effective as hand-driven self-touch, suggesting that proprioception does not contribute to body ownership during self-touch. Self-touch may contribute to a unified sense of bodily self by binding voluntary actions toward our own body with their tactile consequences.

Multisensory simultaneity recalibration: storage of the aftereffect in the absence of counterevidence.

Recent studies show that repeated exposure to an asynchrony between auditory and visual stimuli shifts the point of subjective simultaneity. Usually, the measurement stimuli used to assess this aftereffect are interleaved with short re-exposures to the asynchrony. In a first experiment, we show that the aftereffect declines during measurement in spite of the use of re-exposures. In a second experiment, we investigate whether the observed decline is either due to a dissipation of the aftereffect with the passage of time, or the result of using measurement stimuli with a distribution of asynchronies different from the exposure stimulus. To this end, we introduced a delay before measuring the aftereffects and we compared the magnitude of the aftereffect with and without delay. We find that the aftereffect does not dissipate during the delay but instead is stored until new sensory information in the form of measurement stimuli is presented as counterevidence (i.e., stimuli with an asynchrony that differs from the one used during exposure).

Skin and Mechanoreceptor Contribution to Tactile Input for Perception: A Review of Simulation Models.

We review four current computational models that simulate the response of mechanoreceptors in the glabrous skin to tactile stimulation. The aim is to inform researchers in psychology, sensorimotor science and robotics who may want to implement this type of quantitative model in their research. This approach proves relevant to understanding of the interaction between skin response and neural activity as it avoids some of the limitations of traditional measurement methods of tribology, for the skin, and neurophysiology, for tactile neurons. The main advantage is to afford new ways of looking at the combined effects of skin properties on the activity of a population of tactile neurons, and to examine different forms of coding by tactile neurons. Here, we provide an overview of selected models from stimulus application to neuronal spiking response, including their evaluation in terms of existing data, and their applicability in relation to human tactile perception.

Within- and cross-modal distance information disambiguate visual size-change perception.

Perception is fundamentally underconstrained because different combinations of object properties can generate the same sensory information. To disambiguate sensory information into estimates of scene properties, our brains incorporate prior knowledge and additional "auxiliary" (i.e., not directly relevant to desired scene property) sensory information to constrain perceptual interpretations. For example, knowing the distance to an object helps in perceiving its size. The literature contains few demonstrations of the use of prior knowledge and auxiliary information in combined visual and haptic disambiguation and almost no examination of haptic disambiguation of vision beyond "bistable" stimuli. Previous studies have reported humans integrate multiple unambiguous sensations to perceive single, continuous object properties, like size or position. Here we test whether humans use visual and haptic information, individually and jointly, to disambiguate size from distance. We presented participants with a ball moving in depth with a changing diameter. Because no unambiguous distance information is available under monocular viewing, participants rely on prior assumptions about the ball's distance to disambiguate their -size percept. Presenting auxiliary binocular and/or haptic distance information augments participants' prior distance assumptions and improves their size judgment accuracy-though binocular cues were trusted more than haptic. Our results suggest both visual and haptic distance information disambiguate size perception, and we interpret these results in the context of probabilistic perceptual reasoning.

Causality shifts the perceived temporal order of audiovisual events.

Causality poses explicit constraints to the timing of sensory signals produced by events, as sound travels slower than light, making auditory stimulation to lag visual stimulation. Previous studies show that implied causality between unrelated events can change the tolerance of simultaneity judgments for audiovisual asynchronies. Here, we tested whether apparent causality between audiovisual events may also affect their perceived temporal order. To this aim, we used a disambiguated stream-bounce display, with stimuli either bouncing or streaming upon each other. These two possibilities were accompanied by a sound played around the time of contact between the objects, which could be perceived as causally related to the visual event according to the condition. Participants reported whether the visual contact occurred before or after the sound. Our results show that when the audiovisual stimuli are consistent with a causal interpretation (i.e., the bounce caused the sound), their perceived temporal order is systematically biased. Namely, a stimulus dynamic consistent with a causal relation induces a perceptual delay in the audio component, even if the sound was presented first. We thus conclude that causality can systematically bias the perceived temporal order of events, possibly due to expectations based on the dynamics of events in the real world. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Contact forces in roughness discrimination.

Roughness perception through fingertip contact with a textured surface can involve spatial and temporal cues from skin indentation and vibration respectively. Both types of cue may be affected by contact forces when feeling a surface and we ask whether, on a given trial, discrimination performance relates to contact forces. We examine roughness discrimination performance in a standard psychophysical method (2-interval forced choice, in which the participant identifies which of two spatial textures formed by parallel grooves feels rougher) while continuously measuring the normal and tangential forces applied by the index finger. Fourteen participants discriminated spatial gratings in fine (spatial period of 320-580 micron) and coarse (1520-1920 micron) ranges using static pressing or sliding contact of the index finger. Normal contact force (mean and variability) during pressing or sliding had relatively little impact on accuracy of roughness judgments except when pressing on surfaces in the coarse range. Discrimination was better for sliding than pressing in the fine but not the coarse range. In contrast, tangential force fluctuations during sliding were strongly related to roughness judgment accuracy.

Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency.

After exposure to asynchronous sound and light stimuli, perceived audio-visual synchrony changes to compensate for the asynchrony. Here we investigate to what extent this audio-visual recalibration effect transfers to visual-tactile and audio-tactile simultaneity perception in order to infer the mechanisms responsible for temporal recalibration. Results indicate that audio-visual recalibration of simultaneity can transfer to audio-tactile and visual-tactile stimuli depending on the way in which the multisensory stimuli are presented. With presentation of co-located multisensory stimuli, we found a change in the perceptual latency of the visual stimuli. Presenting auditory stimuli through headphones, on the other hand, induced a change in the perceptual latency of the auditory stimuli. We argue that the difference in transfer depends on the relative trust in the auditory and visual estimates. Interestingly, these findings were confirmed by showing that audio-visual recalibration influences simple reaction time to visual and auditory stimuli. Presenting co-located stimuli during asynchronous exposure induced a change in reaction time to visual stimuli, while with headphones the change in reaction time occurred for the auditory stimuli. These results indicate that the perceptual latency is altered with repeated exposure to asynchronous audio-visual stimuli in order to compensate (at least in part) for the presented asynchrony.

Musical Scales in Tone Sequences Improve Temporal Accuracy.

Predicting the time of stimulus onset is a key component in perception. Previous investigations of perceived timing have focused on the effect of stimulus properties such as rhythm and temporal irregularity, but the influence of non-temporal properties and their role in predicting stimulus timing has not been exhaustively considered. The present study aims to understand how a non-temporal pattern in a sequence of regularly timed stimuli could improve or bias the detection of temporal deviations. We presented interspersed sequences of 3, 4, 5, and 6 auditory tones where only the timing of the last stimulus could slightly deviate from isochrony. Participants reported whether the last tone was 'earlier' or 'later' relative to the expected regular timing. In two conditions, the tones composing the sequence were either organized into musical scales or they were random tones. In one experiment, all sequences ended with the same tone; in the other experiment, each sequence ended with a different tone. Results indicate higher discriminability of anisochrony with musical scales and with longer sequences, irrespective of the knowledge of the final tone. Such an outcome suggests that the predictability of non-temporal properties, as enabled by the musical scale pattern, can be a factor in determining the sensitivity of time judgments.

Modality-specific temporal constraints for state-dependent interval timing.

The ability to discriminate temporal intervals in the milliseconds-to-seconds range has been accounted for by proposing that duration is encoded in the dynamic change of a neuronal network state. A critical limitation of such networks is that their activity cannot immediately return to the initial state, a restriction that could hinder the processing of intervals presented in rapid succession. Empirical evidence in the literature consistently shows impaired duration discrimination performance for 100 ms intervals demarked by short auditory stimuli immediately preceded by a similar interval. Here we tested whether a similar interference is present with longer intervals (300 ms) demarked either by auditory or by visual stimuli. Our results show that while temporal estimates of auditory stimuli in this range are not affected by the interval between them, duration discrimination with this duration is significantly impaired with visual intervals presented in rapid succession. The difference in performance between modalities is overall consistent with state-dependent temporal computations, as it suggests that the limits due to slow neuronal dynamics greatly depends on the sensory modality with which the intervals are demarked, in line with the idea of intrinsic, modality-specific neural mechanisms for interval timing.

An experimental setup to test dual-joystick directional responses to vibrotactile stimuli.

In this paper we investigate the influence of the location of vibrotactile stimulation in triggering the response made using two handheld joysticks. In particular, we compare performance with stimuli delivered either using tactors placed on the palm or on the back of the hand and with attractive (move toward the vibration) or repulsive prompts (move away from the vibration). The experimental set-up comprised two joysticks and two gloves, each equipped with four pager motors along the cardinal directions. In different blocks, fifty-three volunteers were asked to move the joysticks as fast as possible either towards or away with respect to the direction specified by a set of vibrating motors. Results indicate that participants performed better with attractive prompts (i.e. responses were faster and with fewer errors in conditions where participants were asked to move the joysticks in the direction of the felt vibration) and that the stimulation delivered on the back of the hand from the gloves gives better results than the stimulation on the palm delivered by the joysticks. Finally, we analyse the laterality, the relation between correct responses and reaction times, the direction patterns for wrong responses and we perform an analysis on the Stimulus-Response Compatibility and on the training effect.

Peri-personal space as a prior in coupling visual and proprioceptive signals.

It has been suggested that the integration of multiple body-related sources of information within the peri-personal space (PPS) scaffolds body ownership. However, a normative computational framework detailing the functional role of PPS is still missing. Here we cast PPS as a visuo-proprioceptive Bayesian inference problem whereby objects we see in our environment are more likely to engender sensations as they come near to the body. We propose that PPS is the reflection of such an increased a priori probability of visuo-proprioceptive coupling that surrounds the body. To test this prediction, we immersed participants in a highly realistic virtual reality (VR) simulation of their right arm and surrounding environment. We asked participants to perform target-directed reaches toward visual, proprioceptive, and visuo-proprioceptive targets while visually displaying their reaching arm (body visible condition) or not (body invisible condition). Reach end-points are analyzed in light of the coupling prior framework, where the extension of PPS is taken to be represented by the spatial dispersion of the coupling prior between visual and proprioceptive estimates of arm location. Results demonstrate that if the body is not visible, the spatial dispersion of the visuo-proprioceptive coupling relaxes, whereas the strength of coupling remains stable. By demonstrating a distance-dependent alteration in visual and proprioceptive localization attractive pull toward one another (stronger pull at small spatial discrepancies) when the body is rendered invisible - an effect that is well accounted for by the visuo-proprioceptive coupling prior - the results suggest that the visible body grounds visuo-proprioceptive coupling preferentially in the near vs. far space.

The relation between disparity and velocity signals of rigidly moving objects constrains depth order perception.

In two experiments, observers were asked to judge the relative depth of a probe and one or two flanker dots. In Experiment 1, we found that such judgments were influenced by the properties of adjacent image regions, that is, by the amount of angular rotation of a surrounding cloud of dots. In Experiment 2, we found that the properties of the adjacent image regions affected the precision of the observers' judgments. With only the probe and the flanker dots presented in isolation, the precision of observers' judgments was much lower than when probe and the flanker dots were surrounded by a rigidly-connected cloud of dots. Conversely, a non-rigid rotation of the surrounding dots was detrimental to the precision of visual performance. These data can be accounted for by the Intrinsic Constraint model [Domini, F., Caudek, C., & Tassinari, H. (2006). Stereo and motion information are not independently processed by the visual system. Vision Research, 46, 1707-1723], which incorporates the mutual constraints relating disparity and motion signals. The present investigation does not show that the rigidity constraint affects the visual interpretation of motion information alone. Rather, our results show that perceptual performance is affected by the linear relation between disparity and velocity signals, when both depth-cues are present and the distal object is, in fact, rigid.

Temporal Regularity of the Environment Drives Time Perception.

It's reasonable to assume that a regularly paced sequence should be perceived as regular, but here we show that perceived regularity depends on the context in which the sequence is embedded. We presented one group of participants with perceptually regularly paced sequences, and another group of participants with mostly irregularly paced sequences (75% irregular, 25% regular). The timing of the final stimulus in each sequence could be varied. In one experiment, we asked whether the last stimulus was regular or not. We found that participants exposed to an irregular environment frequently reported perfectly regularly paced stimuli to be irregular. In a second experiment, we asked participants to judge whether the final stimulus was presented before or after a flash. In this way, we were able to determine distortions in temporal perception as changes in the timing necessary for the sound and the flash to be perceived synchronous. We found that within a regular context, the perceived timing of deviant last stimuli changed so that the relative anisochrony appeared to be perceptually decreased. In the irregular context, the perceived timing of irregular stimuli following a regular sequence was not affected. These observations suggest that humans use temporal expectations to evaluate the regularity of sequences and that expectations are combined with sensory stimuli to adapt perceived timing to follow the statistics of the environment. Expectations can be seen as a-priori probabilities on which perceived timing of stimuli depend.

Optimal Perceived Timing: Integrating Sensory Information with Dynamically Updated Expectations.

The environment has a temporal structure, and knowing when a stimulus will appear translates into increased perceptual performance. Here we investigated how the human brain exploits temporal regularity in stimulus sequences for perception. We find that the timing of stimuli that occasionally deviate from a regularly paced sequence is perceptually distorted. Stimuli presented earlier than expected are perceptually delayed, whereas stimuli presented on time and later than expected are perceptually accelerated. This result suggests that the brain regularizes slightly deviant stimuli with an asymmetry that leads to the perceptual acceleration of expected stimuli. We present a Bayesian model for the combination of dynamically-updated expectations, in the form of a priori probability of encountering future stimuli, with incoming sensory information. The asymmetries in the results are accounted for by the asymmetries in the distributions involved in the computational process.