RESUMEN
This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.
RESUMEN
This paper introduces an interaction method allowing virtual reality (VR) users to interact with virtual objects by blowing air. The proposed method allows users to interact with virtual objects in a physically plausible way by recognizing the intensity of the wind generated by the user's actual wind blowing activity in the physical world. This is expected to provide immersed VR experience since it enables users to interact with virtual objects in the same way they do in the real world. Three experiments were carried out to develop and improve this method. In the first experiment, we collected the user's blowing data and used it to model a formula to estimate the speed of the wind from the sound waves obtained through a microphone. In the second experiment, we investigated how much gain can be applied to the formula obtained in the first experiment. The aim is to reduce the lung capacity required to generate wind without compromising physical plausibility. In the third experiment, the advantages and disadvantages of the proposed method compared to the controller-based method were investigated in two scenarios of blowing a ball and a pinwheel. According to the experimental results and participant interview, participants felt a stronger sense of presence and found the VR experience more fun with the proposed blowing interaction method.
