The left-right reversed visual feedback of the hand affects multisensory interaction within peripersonal space.

The interaction between vision and touch, known as the crossmodal congruency effect, has been extensively investigated in several research studies. Recent studies have revealed that the crossmodal congruency effect involves body representations. However, it is unclear how bodily information (e.g., location, posture, motion) is linked to visual and tactile inputs. Three experiments were conducted to investigate this issue. In Experiment 1, participants performed a crossmodal congruency task in which both their hand appearance and the motor trajectories were left-right reversed. The results showed that the crossmodal congruency effect was not observed in the reversal condition, whereas participants showed significant crossmodal congruency in the control condition, in which there was no visual manipulation of the hand. In Experiments 2 and 3, where either the hand appearance or motor trajectory was left-right reversed individually, a significant crossmodal congruency effect was observed. This study demonstrated that visual manipulation of hand appearance and motor trajectories both affected the crossmodal congruency effect, although neither showed a dominant effect that solely altered the crossmodal congruency effect. The present results provide insights into the relationship between visual-tactile interactions and bodily information.

Reduction of Forgetting by Contextual Variation During Encoding Using 360-Degree Video-Based Immersive Virtual Environments.

Recall impairment in a different environmental context from learning is called context-dependent forgetting. Two learning methods have been proposed to prevent context-dependent forgetting: reinstatement and decontextualization. Reinstatement matches the environmental context between learning and retrieval, whereas decontextualization involves repeated learning in various environmental contexts and eliminates the context dependency of memory. Conventionally, these methods have been validated by switching between physical rooms. However, in this study, we use immersive virtual environments (IVEs) as the environmental context assisted by virtual reality (VR), which is known for its low cost and high reproducibility compared to traditional manipulation. Whereas most existing studies using VR have failed to reveal the reinstatement effect, we test its occurrence using a 360-degree video-based IVE with improved familiarity and realism instead of a computer graphics-based IVE. Furthermore, we are the first to address decontextualization using VR. Our experiment showed that repeated learning in the same constant IVE as retrieval did not significantly reduce forgetting compared to repeated learning in different constant IVEs. Conversely, repeated learning in various IVEs significantly reduced forgetting than repeated learning in constant IVEs. These findings contribute to the design of IVEs for VR-based applications, particularly in educational settings.

Fear in action: Fear conditioning and alleviation through body movements.

Fear memories enhance survival especially when the memories guide defensive movements to minimize harm. Accordingly, fear memories and body movements have tight relationships in animals: Fear memory acquisition results in adapting reactive defense movements, while training active defense movements reduces fear memory. However, evidence in humans is scarce because their movements are typically suppressed in experiments. Here, we tracked adult participants' body motions while they underwent ecologically valid fear conditioning in a 3D virtual space. First, with body motion tracking, we revealed that distinct spatiotemporal body movement patterns emerge through fear conditioning. Second, subsequent training to actively avoid threats with naturalistic defensive actions led to a long-term (24 h) reduction of physiological and embodied conditioned responses, while extinction or vicarious training only transiently reduced the responses. Together, our results highlight the role of body movements in human fear memory and its intervention.

Effects of the Visual Fidelity of Virtual Environments on Presence, Context-dependent Forgetting, and Source-monitoring Error.

Advances in virtual reality technology have enabled the creation of virtual environments (VEs) with significantly high visual fidelity when compared to real environments (REs). In this study, we use a high-fidelity VE to examine two effects caused by alternating VE and RE experiences: "context-dependent forgetting" and "source-monitoring errors." The former effect is that memories learned in VEs are more easily recalled in VEs than in REs, whereas memories learned in REs are more easily recalled in REs than in VEs. The source-monitoring error is that memories learned in VEs are easily confused with those learned in REs, making discriminating the source of the memory difficult. We hypothesized that the visual fidelity of VEs is responsible for these effects and conducted an experiment using two types of VEs: a high-fidelity VE created using photogrammetry techniques and low-fidelity VE created with primitive shapes and materials. The results show that the high-fidelity VE significantly improved the sense of presence. However, the level of the visual fidelity of the VEs did not show any effect on context-dependent forgetting and source-monitoring errors. Notably, the null results of the context-dependent forgetting between the VE and RE were strongly supported by Bayesian analysis. Thus, we indicate that context-dependent forgetting does not necessarily occur, which will be helpful for VR-based education and training.

Effects of bone-conducted vibration stimulation of various frequencies on the vertical vection.

Illusory self-motion ("vection") has been used to present a sense of movement in virtual reality (VR) and other similar applications. It is crucial in vection research to present a stronger sense of movement. Bone-conducted vibration (BCV) is a small and generally acceptable method for enhancing the sense of movement in VR. However, its effects on vection have not been extensively studied. Here, we conducted two experiments to investigate the effect of BCV on the vection, which generates an upward sensation under the hypothesis that BCV stimulation to the mastoid processes causes noise in the vestibular system and enhances visually-induced self-motion perception. The experiments focused on the effects of BCV stimuli of different frequencies on the vection experience. The results suggested that 500 Hz BCV was more effective as noise to the vestibular system than other frequency BCVs and improved self-motion sensation. This study examines the effects of BCV with different frequencies on the vection experience and designs a theory for using BCV in VR.

Leveraging Tendon Vibration to Enhance Pseudo-Haptic Perceptions in VR.

Pseudo-haptic techniques are used to modify haptic perception by appropriately changing visual feedback to body movements. Based on the knowledge that tendon vibration can affect our somatosensory perception, this paper proposes a method for leveraging tendon vibration to enhance pseudo-haptics during free arm motion. Three experiments were performed to examine the impact of tendon vibration on the range and resolution of pseudo-haptics. The first experiment investigated the effect of tendon vibration on the detection threshold of the discrepancy between visual and physical motion. The results indicated that vibrations applied to the inner tendons of the wrist and elbow increased the threshold, suggesting that tendon vibration can augment the applicable visual motion gain by approximately 13% without users detecting the visual/physical discrepancy. Furthermore, the results demonstrate that tendon vibration acts as noise on haptic motion cues. The second experiment assessed the impact of tendon vibration on the resolution of pseudo-haptics by determining the just noticeable difference in pseudo-weight perception. The results suggested that the tendon vibration does not largely compromise the resolution of pseudo-haptics. The third experiment evaluated the equivalence between the weight perception triggered by tendon vibration and that by visual motion gain, that is, the point of subjective equality. The results revealed that vibration amplifies the weight perception and its effect was equivalent to that obtained using a gain of 0.64 without vibration, implying that the tendon vibration also functions as an additional haptic cue. Our results provide design guidelines and future work for enhancing pseudo-haptics with tendon vibration.

Presenting Morphing Shape Illusion: Enhanced Sense of Morphing Virtual Object With Weight Shifting VR Controller by Computational Perception Model.

The haptic sensation is crucial for virtual reality, as it gives the presence of objects in a virtual world and thus gives a greater sense of immersion. To provide a sense of the shape of handheld objects, a haptic device that changes weight distribution is proposed. It is known that the visual feedback enhances the haptic sensation of shape, and it is also known that it does for morphing shape as well. Our previous publication presented a perception model for the static shape of a virtual object. In this article, we extend the model to produce a plausible sense of the morphing shape of handheld objects. Our stochastic model predicts the proper weight actuation for the weight-shifting haptic device, which users can plausibly feel while reducing hardware effort. We evaluated our perception model and resulted in the model accuracy average of 8.1% error. Using this perception model, the amount of weight actuation at 75% probability of plausibility is reduced up to 37%.

Effects of Collaborative Training Using Virtual Co-embodiment on Motor Skill Learning.

Virtual reality (VR) is a promising tool for motor skill learning. Previous studies have indicated that observing and following a teacher's movements from a first-person perspective using VR facilitates motor skill learning. Conversely, it has also been pointed out that this learning method makes the learner so strongly aware of the need to follow that it weakens their sense of agency (SoA) for motor skills and prevents them from updating the body schema, thereby preventing long-term retention of motor skills. To address this problem, we propose applying "virtual co-embodiment" to motor skill learning. Virtual co-embodiment is a system in which a virtual avatar is controlled based on the weighted average of the movements of multiple entities. Because users in virtual co-embodiment overestimate their SoA, we hypothesized that learning using virtual co-embodiment with a teacher would improve motor skill retention. In this study, we focused on learning a dual task to evaluate the automation of movement, which is considered an essential element of motor skills. As a result, learning in virtual co-embodiment with the teacher improves motor skill learning efficiency compared with sharing the teacher's first-person perspective or learning alone.

Revisiting Walking-in-Place by Introducing Step-Height Control, Elastic Input, and Pseudo-Haptic Feedback.

Walking-in-place (WIP) is a locomotion technique that enables users to "walk infinitely" through vast virtual environments using walking-like gestures within a limited physical space. This paper investigates alternative interaction schemes for WIP, addressing successively the control, input, and output of WIP. First, we introduce a novel height-based control to increase advanced speed. Second, we introduce a novel input system for WIP based on elastic and passive strips. Third, we introduce the use of pseudo-haptic feedback as a novel output for WIP meant to alter walking sensations. The results of a series of user studies show that height and frequency based control of WIP can facilitate higher virtual speed with greater efficacy and ease than in frequency-based WIP. Second, using an upward elastic input system can result in a stable virtual speed control, although excessively strong elastic forces may impact the usability and user experience. Finally, using a pseudo-haptic approach can improve the perceived realism of virtual slopes. Taken together, our results suggest that, for future VR applications, there is value in further research into the use of alternative interaction schemes for walking-in-place.

Odor-Induced Taste Enhancement Is Specific to Naturally Occurring Temporal Order and the Respiration Phase.

Interaction between odor and taste information creates flavor perception. There are many possible determinants of the interaction between odor and taste, one of which may be the somatic sensations associated with breathing. We assumed that a smell stimulus accompanied by inhaling or exhaling enhances taste intensity if the order is congruent with natural drinking. To present an olfactory stimulus from the identical location during inhalation and exhalation, we blocked the gap between the tube presenting the olfactory stimulus and the nostril. Participants breathed and ingested the solution according to the instructions on the screen and evaluated the solution's taste intensity. Vanilla odor enhanced the sweet taste in both retronasal and orthonasal conditions when the order of stimuli was congruent with natural drinking, but it did not do so in either condition when they were incongruent. The results suggest that breathing is a determinant of odor-taste interaction. The methods of presenting olfactory stimuli used in this study were compared and discussed in relation to those used in previous studies. Odor-induced taste enhancement depends on the time order of smell with breathing and taste congruency in natural drinking. Taste enhancement was induced by odor in both conditions by minimizing differences in odor presentation between them.

Effect of Avatar Appearance on Detection Thresholds for Remapped Hand Movements.

Hand interaction techniques in virtual reality often exploit visual dominance over proprioception to remap physical hand movements onto different virtual movements. However, when the offset between virtual and physical hands increases, the remapped virtual hand movements are hardly self-attributed, and the users become aware of the remapping. Interestingly, the sense of self-attribution of a body is called the sense of body ownership (SoBO) in the field of psychology, and the realistic the avatar, the stronger is the SoBO. Hence, we hypothesized that realistic avatars (i.e., human hands) can foster self-attribution of the remapped movements better than abstract avatars (i.e., spherical pointers), thus making the remapping less noticeable. In this article, we present an experiment in which participants repeatedly executed reaching movements with their right hand while different amounts of horizontal shifts were applied. We measured the remapping detection thresholds for each combination of shift directions (left or right) and avatar appearances (realistic or abstract). The results show that realistic avatars increased the detection threshold (i.e., lowered sensitivity) by 31.3 percent than the abstract avatars when the leftward shift was applied (i.e., when the hand moved in the direction away from the body-midline). In addition, the proprioceptive drift (i.e., the displacement of self-localization toward an avatar) was larger with realistic avatars for leftward shifts, indicating that visual information was given greater preference during visuo-proprioceptive integration in realistic avatars. Our findings quantifiably show that realistic avatars can make remapping less noticeable for larger mismatches between virtual and physical movements and can potentially improve a wide variety of hand-remapping techniques without changing the mapping itself.

Redirected Walking using Continuous Curvature Manipulation.

In this paper, we propose a novel redirected walking (RDW) technique that applies dynamic bending and curvature gains so that users perceive less discomfort than existing techniques that apply constant gains. Humans are less likely to notice continuous changes than those that are sudden. Therefore, instead of applying constant bending or curvature gains to users, we propose a dynamic method that continuously changes the gains. We conduct experiments to investigate the effect of dynamic gains in bending and curvature manipulation with regards to discomfort. The experimental results show that the proposed method significantly suppresses discomfort by up to 16 and 9% for bending and curvature manipulations, respectively.

Head-Mounted Display with Increased Downward Field of View Improves Presence and Sense of Self-Location.

Common existing head-mounted displays (HMDs) for virtual reality (VR) provide users with a high presence and embodiment. However, the field of view (FoV) of a typical HMD for VR is about 90 to 110 [deg] in the diagonal direction and about 70 to 90 [deg] in the vertical direction, which is narrower than that of humans. Specifically, the downward FoV of conventional HMDs is too narrow to present the user avatar's body and feet. To address this problem, we have developed a novel HMD with a pair of additional display units to increase the downward FoV by approximately 60 ( 10+50) [deg]. We comprehensively investigated the effects of the increased downward FoV on the sense of immersion that includes presence, sense of self-location (SoSL), sense of agency (SoA), and sense of body ownership (SoBO) during VR experience and on patterns of head movements and cybersickness as its secondary effects. As a result, it was clarified that the HMD with an increased FoV improved presence and SoSL. Also, it was confirmed that the user could see the object below with a head movement pattern close to the real behavior, and did not suffer from cybersickness. Moreover, the effect of the increased downward FoV on SoBO and SoA was limited since it was easier to perceive the misalignment between the real and virtual bodies.

Virtual Co-Embodiment: Evaluation of the Sense of Agency While Sharing the Control of a Virtual Body Among Two Individuals.

In this article, we introduce a concept called "virtual co-embodiment", which enables a user to share their virtual avatar with another entity (e.g., another user, robot, or autonomous agent). We describe a proof-of-concept in which two users can be immersed from a first-person perspective in a virtual environment and can have complementary levels of control (total, partial, or none) over a shared avatar. In addition, we conducted an experiment to investigate the influence of users' level of control over the shared avatar and prior knowledge of their actions on the users' sense of agency and motor actions. The results showed that participants are good at estimating their real level of control but significantly overestimate their sense of agency when they can anticipate the motion of the avatar. Moreover, participants performed similar body motions regardless of their real control over the avatar. The results also revealed that the internal dimension of the locus of control, which is a personality trait, is negatively correlated with the user's perceived level of control. The combined results unfold a new range of applications in the fields of virtual-reality-based training and collaborative teleoperation, where users would be able to share their virtual body.

The relationship between the body and the environment in the virtual world: The interpupillary distance affects the body size perception.

Previous research suggests that the size of one's body is used as a metric to scale the external world. On the other hand, the influence of information from the external world on the perception of body size is unclear. It has been suggested that increased inter-pupillary distance (IPD) leads people to perceive the external world as smaller than it actually is. The present study investigated the effect of the IPD on body size perception, and the relationship between the perceived scale of the body and the external world when the IPD is manipulated. To this end, in a virtual environment, we manipulated the IPD as well as the size and presence of participants' hands, while participant's eye height was increased vertically. Results showed that, when participants' eye height was increased and their hands were enlarged, people with a fixed IPD perceived the size of their body to be large (like a giant) while the external world was perceived to be changed minimally. Alternatively, people with increased IPD perceived that the external world as having shrank, whereas their perception of their body size changed little. However, when a viewers' virtual hands were not shown, the IPD did not affect the individual's percept of body size, although the IPD did affect one's perception of the external world. These results suggest that, when the ratio of the size between one's body and the external world are explicit, the perceived size of one's body is affected by the IPD or perceived scale of the external world that is affected by the IPD.

Improvement of driver active interventions during automated driving by displaying trajectory pointers-A driving simulator study.

Objective: The handover of vehicle control from automated to manual operation is a critical aspect of interaction between drivers and automated driving systems (ADS). In some cases, it is possible that the ADS may fail to detect an object. In this event, the driver must be aware of the situation and resume control of the vehicle without assistance from the system. Consequently, the driver must fulfill the following 2 main roles while driving: (1) monitor the vehicle trajectory and surrounding traffic environment and (2) actively take over vehicle control if the driver identifies a potential issue along the trajectory. An effective human-machine interface (HMI) is required that enables the driver to fulfill these roles. This article proposes an HMI that constantly indicates the future position of the vehicle. Methods: This research used the Toyota Dynamic Driving Simulator to evaluate the effect of the proposed HMI and compares the proposed HMI with an HMI that notifies the driver when the vehicle trajectory changes. A total of 48 test subjects were divided into 2 groups of 24: One group used the HMI that constantly indicated the future position of the vehicle and the other group used the HMI that provided information when the vehicle trajectory changed. The following instructions were given to the test subjects: (1) to not hold the steering wheel and to allow the vehicle to drive itself, (2) to constantly monitor the surrounding traffic environment because the functions of the ADS are limited, and (3) to take over driving if necessary. The driving simulator experiments were composed of an initial 10-min acclimatization period and a 10-min evaluation period. Approximately 10 min after the start of the evaluation period, a scenario occurred in which the ADS failed to detect an object on the vehicle trajectory, potentially resulting in a collision if the driver did not actively take over control and manually avoid the object. Results: The collision avoidance rate of the HMI that constantly indicated the future position of the vehicle was higher than that of the HMI that notified the driver of trajectory changes, χ2 = 6.38, P < .05. The steering wheel hands-on and steering override timings were also faster with the proposed HMI (t test; P < .05). Conclusions: This research confirmed that constantly indicating the position of the vehicle several seconds in the future facilitates active driver intervention when an ADS is in operation.

Ascending and Descending in Virtual Reality: Simple and Safe System Using Passive Haptics.

This paper presents a novel interactive system that provides users with virtual reality (VR) experiences, wherein users feel as if they are ascending/descending stairs through passive haptic feedback. The passive haptic stimuli are provided by small bumps under the feet of users; these stimuli are provided to represent the edges of the stairs in the virtual environment. The visual stimuli of the stairs and shoes, provided by head-mounted displays, evoke a visuo-haptic interaction that modifies a user's perception of the floor shape. Our system enables users to experience all types of stairs, such as half-turn and spiral stairs, in a VR setting. We conducted a preliminary user study and two experiments to evaluate the proposed technique. The preliminary user study investigated the effectiveness of the basic idea associated with the proposed technique for the case of a user ascending stairs. The results demonstrated that the passive haptic feedback produced by the small bumps enhanced the user's feeling of presence and sense of ascending. We subsequently performed an experiment to investigate an improved viewpoint manipulation method and the interaction of the manipulation and haptics for both the ascending and descending cases. The experimental results demonstrated that the participants had a feeling of presence and felt a steep stair gradient under the condition of haptic feedback and viewpoint manipulation based on the characteristics of actual stair walking data. However, these results also indicated that the proposed system may not be as effective in providing a sense of descending stairs without an optimization of the haptic stimuli. We then redesigned the shape of the small bumps, and evaluated the design in a second experiment. The results indicated that the best shape to present haptic stimuli is a right triangle cross section in both the ascending and descending cases. Although it is necessary to install small protrusions in the determined direction, by using this optimized shape the users feeling of presence of the stairs and the sensation of walking up and down was enhanced.

Taste of breath: the temporal order of taste and smell synchronized with breathing as a determinant for taste and olfactory integration.

Many studies have reported that subjective taste intensity is enhanced by odors which are congruent, for example a sweet taste and a vanilla odor. Some reports have suggested that subjective taste is more strongly enhanced by retronasal than by orthonasal odors; others have suggested that taste enhancements by both odor routes are identical. Differences between the two routes include the direction of airflow accompanying breath. Thus, it is possible that the order of gustatory and olfactory stimuli when breathing through either route while drinking is a determining factor for taste-odor integration. To reveal the natural relationship between taste intensity enhancement by odors and breath, synchronization of odor stimulation with the breath is necessary. Here, we examined whether the enhancement of a sweet taste is induced by a vanilla odor presented in various combinations of odor routes, immediately before and immediately after drinking. The results showed that a retronasal odor after drinking enhanced taste, but an orthonasal odor before drinking did not. The retronasal odor before drinking and the orthonasal odor after drinking did not enhance the sweet taste. These results show that congruency with the natural order of stimulus and kinetic sensation is a determining factor for odor-induced taste enhancement.

Simplification of olfactory stimuli in pseudo-gustatory displays.

In this paper, we propose a technique that simplifies pseudo-gustatory systems by using the cross-modal effect between vision and olfaction to change only the visual and olfactory stimuli without altering the ingredients of the food. Conventional pseudo-gustatory simulations require one olfactory stimulus for each flavor. In contrast, we hypothesize that the cross-modal effect between vision and olfaction in a pseudo-gustatory presentation can be utilized to reduce the required number of olfactory stimuli, and verify this hypothesis using a pseudo-gustatory display with our proposed method. This pseudo-gustatory display uses the visual-olfactory cross-modal effect and the key scent components decided on the basis of similarity analysis of olfactory perception. We also investigate how users perceive the taste of a drink under various visual and olfactory conditions, both with and without the visual-olfactory interactions. Our results indicate that the cross-modal effect between vision and olfaction can effectively simplify pseudo-gustatory simulations.