Effects of self-avatar cast shadow and foot vibration on telepresence, virtual walking experience, and cybersickness from omnidirectional movie.

Human locomotion is most naturally achieved through walking, which is good for both mental and physical health. To provide a virtual walking experience to seated users, a system utilizing foot vibrations and simulated optical flow was developed. The current study sought to augment this system and examine the effect of an avatar's cast shadow and foot vibrations on the virtual walking experience and cybersickness. The omnidirectional movie and the avatar's walking animation were synchronized, with the cast shadow reflecting the avatar's movement on the ground. Twenty participants were exposed to the virtual walking in six conditions (with/without foot vibrations and no/short/long shadow) and were asked to rate their sense of telepresence, walking experience, and occurrences of cybersickness. Our findings indicate that the synchronized foot vibrations enhanced telepresence as well as self-motion, walking, and leg-action sensations, while also reducing instances of nausea and disorientation sickness. The avatar's cast shadow was found to improve telepresence and leg-action sensation, but had no impact on self-motion and walking sensation. These results suggest that observation of the self-body cast shadow does not directly improve walking sensation, but is effective in enhancing telepresence and leg-action sensation, while foot vibrations are effective in improving telepresence and walking experience and reducing instances of cybersickness.

Temporal visuomotor synchrony induces embodiment towards an avatar with biomechanically impossible arm movements.

Visuomotor synchrony in time and space induces a sense of embodiment towards virtual bodies experienced in first-person view using Virtual Reality (VR). Here, we investigated whether temporal visuomotor synchrony affects avatar embodiment even when the movements of the virtual arms are spatially altered from those of the user in a non-human-like manner. In a within-subjects design VR experiment, participants performed a reaching task controlling an avatar whose lower arms bent in inversed and biomechanically impossible directions from the elbow joints. They performed the reaching task using this "unnatural avatar" as well as a "natural avatar," whose arm movements and positions spatially matched the user. The reaching tasks were performed with and without a one second delay between the real and virtual movements. While the senses of body ownership and agency towards the unnatural avatar were significantly lower compared to those towards the natural avatar, temporal visuomotor synchrony did significantly increase the sense of embodiment towards the unnatural avatar as well as the natural avatar. These results suggest that temporal visuomotor synchrony is crucial for inducing embodiment even when the spatial match between the real and virtual limbs is disrupted with movements outside the pre-existing cognitive representations of the human body.

Empathic embarrassment towards non-human agents in virtual environments.

Humans feel empathic embarrassment by witnessing others go through embarrassing situations. We examined whether we feel such empathic embarrassment even with robot avatars. Participants observed a human avatar and a robot avatar face a series of embarrassing and non-embarrassing scenarios. We collected data for their empathic embarrassment and the cognitive empathy on a 7-point Likert scale. Both empathic embarrassment and cognitive empathy were significantly higher in the embarrassed condition compared to the non-embarrassed condition with both avatars, and the cognitive empathy was significantly higher with the human avatar. There was a tendency of participants showing a higher level of skin conductance while watching the human avatar go through embarrassing situations compared to the robot avatar. A following experiment showed that the average plausibility of the embarrassed condition was significantly higher with the human avatar compared to the robot avatar. However, plausibility scores for emotion were not significantly different among the conditions. These results suggest that humans can feel empathic embarrassment as well as cognitive empathy for robot avatars while cognitive empathy for robot avatars is comparatively lower, and that part of the empathic difference between human and robot avatars might be due to the difference of their plausibility.

Sensory Attenuation with a Virtual Robotic Arm Controlled Using Facial Movements.

When humans generate stimuli voluntarily, they perceive the stimuli more weakly than those produced by others, which is called sensory attenuation (SA). SA has been investigated in various body parts, but it is unclear whether an extended body induces SA. This study investigated the SA of audio stimuli generated by an extended body. SA was assessed using a sound comparison task in a virtual environment. We prepared robotic arms as extended bodies, and the robotic arms were controlled by facial movements. To evaluate the SA of robotic arms, we conducted two experiments. Experiment 1 investigated the SA of the robotic arms under four conditions. The results showed that robotic arms manipulated by voluntary actions attenuated audio stimuli. Experiment 2 investigated the SA of the robotic arm and innate body under five conditions. The results indicated that the innate body and robotic arm induced SA, while there were differences in the sense of agency between the innate body and robotic arm. Analysis of the results indicated three findings regarding the SA of the extended body. First, controlling the robotic arm with voluntary actions in a virtual environment attenuates the audio stimuli. Second, there were differences in the sense of agency related to SA between extended and innate bodies. Third, the SA of the robotic arm was correlated with the sense of body ownership.

Effect of connection induced upper body movements on embodiment towards a limb controlled by another during virtual co-embodiment.

Even if we cannot control them, or when we receive no tactile or proprioceptive feedback from them, limbs attached to our bodies can still provide indirect proprioceptive and haptic stimulations to the body parts they are attached to simply due to the physical connections. In this study we investigated whether such indirect movement and haptic feedbacks from a limb contribute to a feeling of embodiment towards it. To investigate this issue, we developed a 'Joint Avatar' setup in which two individuals were given full control over the limbs in different sides (left and right) of an avatar during a reaching task. The backs of the two individuals were connected with a pair of solid braces through which they could exchange forces and match the upper body postures with one another. Coupled with the first-person view, this simulated an experience of the upper body being synchronously dragged by the partner-controlled virtual arm when it moved. We observed that this passive synchronized upper-body movement significantly reduced the feeling of the partner-controlled limb being owned or controlled by another. In summary, our results suggest that even in total absence of control, connection induced upper body movements synchronized with the visible limb movements can positively affect the sense of embodiment towards partner-controlled or autonomous limbs.

Knowing the intention behind limb movements of a partner increases embodiment towards the limb of joint avatar.

We explored a concept called "virtual co-embodiment", which enables users to share their virtual avatars with others. Co-embodiment of avatars and robots can be applied for collaboratively performing complicated tasks, skill training, rehabilitation, and aiding disabled users. We conducted an experiment where two users could co-embody one "joint avatar" in first person view and control different arms to collaboratively perform three types of reaching tasks. We measured their senses of agency and ownership towards the two arms of the avatar and changes in skin conductance levels in response to visual stimuli threatening the two virtual arms. We found that sense of agency, ownership, and skin conductance were significantly higher towards the virtual arm with control compared to the arm controlled by the partner. Furthermore, the senses of agency and ownership towards the arm controlled by the partner were significantly higher when the participant dyads shared a common intention or when they were allowed to see their partner's target, compared to when the partner's target was invisible. These results show that while embodiment towards partner-controlled limbs is lower compared to limbs with control, visual information necessary for predicting the partner's intentions can significantly enhance embodiment towards partner-controlled limbs during virtual co-embodiment.

Embodiment of supernumerary robotic limbs in virtual reality.

The supernumerary robotic limb system expands the motor function of human users by adding extra artificially designed limbs. It is important for us to embody the system as if it is a part of one's own body and to maintain cognitive transparency in which the cognitive load is suppressed. Embodiment studies have been conducted with an expansion of bodily functions through a "substitution" and "extension". However, there have been few studies on the "addition" of supernumerary body parts. In this study, we developed a supernumerary robotic limb system that operates in a virtual environment, and then evaluated whether the extra limb can be regarded as a part of one's own body using a questionnaire and whether the perception of peripersonal space changes with a visuotactile crossmodal congruency task. We found that the participants can embody the extra-limbs after using the supernumerary robotic limb system. We also found a positive correlation between the perceptual change in the crossmodal congruency task and the subjective feeling that the number of one's arms had increased (supernumerary limb sensation). These results suggest that the addition of an extra body part may cause the participants to feel that they had acquired a new body part that differs from their original body part through a functional expansion.

The effects of body direction and posture on taking the perspective of a humanoid avatar in a virtual environment.

Visual perspective taking is inferring how the world looks to another person. To clarify this process, we investigated whether employing a humanoid avatar as the viewpoint would facilitate an imagined perspective shift in a virtual environment, and which factor of the avatar is effective for the facilitation effect. We used a task that involved reporting how an object looks by a simple direction judgment, either from the avatar's position or from the position of an empty chair. We found that the humanoid avatar's presence improved task performance. Furthermore, the avatar's facilitation effect was observed only when the avatar was facing the visual stimulus to be judged; performance was worse when it faced backwards than when there was only an empty chair facing forwards. This suggests that the avatar does not simply attract spatial attention, but the posture of the avatar is crucial for the facilitation effect. In addition, when the directions of the head and the torso were opposite (i.e., an impossible posture), the avatar's facilitation effect disappeared. Thus, visual perspective taking might not be facilitated by the avatar when its posture is biomechanically impossible because we cannot embody it. Finally, even when the avatar's head of the possible posture was covered with a bucket, the facilitation effect was found with the forward-facing avatar rather than the backward-facing avatar. That is, the head/gaze direction cue, or presumably the belief that the visual stimulus to be judged can be seen by the avatar, was not required. These results suggest that explicit perspective taking is facilitated by embodiment towards humanoid avatars.

Great apes' understanding of biomechanics: eye-tracking experiments using three-dimensional computer-generated animations.

Visual processing of the body movements of other animals is important for adaptive animal behaviors. It is widely known that animals can distinguish articulated animal movements even when they are just represented by points of light such that only information about biological motion is retained. However, the extent to which nonhuman great apes comprehend the underlying structural and physiological constraints affecting each moving body part, i.e., biomechanics, is still unclear. To address this, we examined the understanding of biomechanics in bonobos (Pan paniscus) and chimpanzees (Pan troglodytes), following a previous study on humans (Homo sapiens). Apes underwent eye tracking while viewing three-dimensional computer-generated (CG) animations of biomechanically possible or impossible elbow movements performed by a human, robot, or nonhuman ape. Overall, apes did not differentiate their gaze between possible and impossible movements of elbows. However, some apes looked at elbows for longer when viewing impossible vs. possible robot movements, which indicates that they may have had knowledge of biomechanics and that this knowledge could be extended to a novel agent. These mixed results make it difficult to draw a firm conclusion regarding the extent to which apes understand biomechanics. We discuss some methodological features that may be responsible for the results, as well as implications for future nonhuman animal studies involving the presentation of CG animations or measurement of gaze behaviors.

Peripersonal space in the front, rear, left and right directions for audio-tactile multisensory integration.

Peripersonal space (PPS) is important for humans to perform body-environment interactions. However, many previous studies only focused on the specific direction of the PPS, such as the front space, despite suggesting that there were PPSs in all directions. We aimed to measure and compare the peri-trunk PPS in four directions (front, rear, left, and right). To measure the PPS, we used a tactile and an audio stimulus because auditory information is available at any time in all directions. We used the approaching and receding task-irrelevant sounds in the experiment. Observers were asked to respond as quickly as possible when a tactile stimulus was applied to a vibrator on their chest. We found that peri-trunk PPS representations exist with an approaching sound, irrespective of the direction.

Individuals Prioritize the Reach Straightness and Hand Jerk of a Shared Avatar over Their Own.

Cyber space enables us to "share" bodies whose movements are a consequence of movements by several individuals. But whether and how our motor behavior is affected during body sharing remains unclear. Here we examined this issue in arm reaching performed by a shared avatar, whose movement was generated by averaging the movements of two participants. We observed that participants exhibited improved reaction times with a shared avatar than alone. Moreover, the reach trajectory of the shared avatar was straighter than that of either participant and correlated with their subjective embodiment of the avatar. Finally, the jerk of the avatar's hand was less than either participant's own hand, both when they reached alone and in the shared body. Movement straightness and hand jerk are well known characteristics of human reach behavior, and our results suggest that during body sharing, humans prioritize these movement characteristics of the shared body over their own.

Scrambled body differentiates body part ownership from the full body illusion.

Illusory body ownership can be induced in a body part or a full body by visual-motor synchronisation. A previous study indicated that an invisible full body illusion can be induced by the synchronous movement of only the hands and feet. The difference between body part ownership and the full body illusion has not been explained in detail because there is no method for separating these two illusions. To develop a method to do so, we scrambled or randomised the positions of the hands and feet and compared it with the normal layout stimulus by manipulating visual-motor synchronisation. In Experiment 1, participants observed the stimuli from a third-person perspective, and the questionnaire results showed that the scrambled body stimulus induced only body part ownership, while the normal layout stimulus induced both body part ownership and full body ownership when the stimuli were synchronous with participants' actions. In Experiment 2, we found similar results as with the first-person perspective stimuli in a questionnaire. We did not find significant skin conductance response difference between any conditions in either Experiment 2 or 3. These results suggest that a spatial relationship is necessary for the full body illusion, but not for body part ownership.

Pseudo-Sensation of Walking Generated by Passive Whole-Body Motions in Heave and Yaw Directions.

Walking is an innate human behavior that propels the body forward. Recent studies have investigated the creation of a walking sensation wherein, the body neither moves nor is forced to move. However, it is unclear which whole-body motions effectively induce the sensation of walking. Here, we show that passive whole-body motions, such as heave and/or yaw motions, produced by a motorized chair induced a sensation of walking for seated participants in virtual environments as if the participant were walking while viewing a virtual reality scene through a head-mounted display. Our findings suggest that the passive whole-body motions in the gravitational axis-and to a lesser extent in the yaw axis-provide a clear perception of pseudo-walking, but only with limited motion amplitudes, namely one-fourth or less than those of actual walking. In addition, we found a negative correlation between the scores of walking sensation and motion sickness.

Re-association of Body Parts: Illusory Ownership of a Virtual Arm Associated With the Contralateral Real Finger by Visuo-Motor Synchrony.

Illusory ownership can be induced in a virtual body by visuo-motor synchrony. Our aim was to test the possibility of a re-association of the right thumb with a virtual left arm and express the illusory body ownership of the re-associated arm through a synchronous or asynchronous movement of the body parts through action and vision. Participants felt that their right thumb was the virtual left arm more strongly in the synchronous condition than in the asynchronous one, and the feeling of ownership of the virtual arm was also stronger in the synchronous condition. We did not find a significant difference in the startle responses to a sudden knife appearance to the virtual arm between the two synchrony conditions, as there was no proprioceptive drift of the thumb. These results suggest that a re-association of the right thumb with the virtual left arm could be induced by visuo-motor synchronization; however, it may be weaker than the natural association.

Virtual Walking Sensation by Prerecorded Oscillating Optic Flow and Synchronous Foot Vibration.

This article reports the first psychological evidence that the combination of oscillating optic flow and synchronous foot vibration evokes a walking sensation. In this study, we first captured a walker's first-person-view scenes with footstep timings. Participants observed the naturally oscillating scenes on a head-mounted display with vibrations on their feet and rated walking-related sensations using a Visual Analogue Scale. They perceived stronger sensations of self-motion, walking, leg action, and telepresence from the oscillating visual flow with foot vibrations than with randomized-timing vibrations or without vibrations. The artificial delay of foot vibrations with respect to the scenes diminished the walking-related sensations. These results suggest that the oscillating visual scenes and synchronous foot vibrations are effective for creating virtual walking sensations.

Remapping Peripersonal Space by Using Foot-Sole Vibrations Without Any Body Movement.

The limited space immediately surrounding our body, known as peripersonal space (PPS), has been investigated by focusing on changes in the multisensory processing of audio-tactile stimuli occurring within or outside the PPS. Some studies have reported that the PPS representation is extended by body actions such as walking. However, it is unclear whether the PPS changes when a walking-like sensation is induced but the body neither moves nor is forced to move. Here, we show that a rhythmic pattern consisting of walking-sound vibrations applied to the soles of the feet, but not the forearms, boosted tactile processing when looming sounds were located near the body. The findings suggest that an extension of the PPS representation can be triggered by stimulating the soles in the absence of body action, which may automatically drive a motor program for walking, leading to a change in spatial cognition around the body.

Social contingency modulates the perceived distance between self and other.

Although social contingency, namely contingent reactions of other to one's own actions, critically affects attachment formation, can it also modulate the perceived distance between self and other? Previous studies have suggested a positive answer. However, these studies are criticized for not showing true top-down effects on perception because of pitfalls such as task demands. We show that social contingency reduced the perceived distance between self and other while avoiding pitfalls. According to Emmert's law, the perceived size of an afterimage increases with perceived distance. Thus, if social contingency modulates the perceived distance, the perceived size of afterimage should inevitably reflect it. The results showed that the size of the afterimages of a face that contingently responded to participants' actions was perceived as smaller than those of non-contingent and unresponsive faces. This effect was more salient with increasing viewing distances. Thus, prior knowledge of interaction with environment modulates online perceptual processing in size constancy, probably through its influence on perceived distance.

Illusory body ownership of an invisible body interpolated between virtual hands and feet via visual-motor synchronicity.

Body ownership can be modulated through illusory visual-tactile integration or visual-motor synchronicity/contingency. Recently, it has been reported that illusory ownership of an invisible body can be induced by illusory visual-tactile integration from a first-person view. We aimed to test whether a similar illusory ownership of the invisible body could be induced by the active method of visual-motor synchronicity and if the illusory invisible body could be experienced in front of and facing away from the observer. Participants observed left and right white gloves and socks in front of them, at a distance of 2 m, in a virtual room through a head-mounted display. The white gloves and socks were synchronized with the observers' actions. In the experiments, we tested the effect of synchronization, and compared this to a whole-body avatar, measuring self-localization drift. We observed that visual hands and feet were sufficient to induce illusory body ownership, and this effect was as strong as using a whole-body avatar.

Social information affects adults' evaluation of fairness in distributions: An ERP approach.

The sense of fairness has been observed in early infancy. Because many studies of fairness in adults have used economic games such as the Ultimatum Game, it has been difficult to compare fairness between adults and infants. Further, recent studies have suggested that social information about actors who behave fairly or unfairly may influence the judgement of fairness in infants. Therefore, to compare the sense of fairness between infants and adults, the study using paradigm in infant research is required. We examined how social information about two characters, either prosocial or antisocial, affects the event-related potential response (ERP) to fair or unfair resource distributions in adults. In the habituation phase, participants were informed about characters' social information through their actions. One character then distributed resources fairly or unfairly, and ERP was measured at the end of the distribution. Data from eighteen adult participants were analysed. A significant interaction of social information and fairness was found for late positive potential (LPP), but a post-hoc t test revealed a significant difference between fair and unfair conditions only for actions of the antisocial character. We found that LPP can reflect the sense of fairness affected by social information. Comparison with infant studies suggests that the sense of fairness may change during development.

Measuring empathy for human and robot hand pain using electroencephalography.

This study provides the first physiological evidence of humans' ability to empathize with robot pain and highlights the difference in empathy for humans and robots. We performed electroencephalography in 15 healthy adults who observed either human- or robot-hand pictures in painful or non-painful situations such as a finger cut by a knife. We found that the descending phase of the P3 component was larger for the painful stimuli than the non-painful stimuli, regardless of whether the hand belonged to a human or robot. In contrast, the ascending phase of the P3 component at the frontal-central electrodes was increased by painful human stimuli but not painful robot stimuli, though the interaction of ANOVA was not significant, but marginal. These results suggest that we empathize with humanoid robots in late top-down processing similarly to human others. However, the beginning of the top-down process of empathy is weaker for robots than for humans.