Violating body movement semantics: Neural signatures of self-generated and external-generated errors.

How do we recognize ourselves as the agents of our actions? Do we use the same error detection mechanisms to monitor self-generated vs. externally imposed actions? Using event-related brain potentials (ERPs), we identified two different error-monitoring loops involved in providing a coherent sense of the agency of our actions. In the first ERP experiment, the participants were embodied in a virtual body (avatar) while performing an error-prone fast reaction time task. Crucially, in certain trials, participants were deceived regarding their own actions, i.e., the avatar movement did not match the participant's movement. Self-generated real errors and false (avatar) errors showed very different ERP signatures and with different processing latencies: while real errors showed a classical frontal-central error-related negativity (Ne/ERN), peaking 100ms after error commission, false errors elicited a larger and delayed parietal negative component (at about 350-400ms). The violation of the sense of agency elicited by false avatar errors showed a strong similarity to ERP signatures related to semantic or conceptual violations (N400 component). In a follow-up ERP control experiment, a subset of the same participants merely acted as observers of the avatar correct and error movements. This experimental situation did not elicit the N400 component associated with agency violation. Thus, the results show a clear neural dissociation between internal and external error-monitoring loops responsible for distinguishing our self-generated errors from those imposed externally, opening new avenues for the study of the mental processes underlying the integration of internal and sensory feedback information while being actors of our own actions.

A threat to a virtual hand elicits motor cortex activation.

We report an experiment where participants observed an attack on their virtual body as experienced in an immersive virtual reality (IVR) system. Participants sat by a table with their right hand resting upon it. In IVR, they saw a virtual table that was registered with the real one, and they had a virtual body that substituted their real body seen from a first person perspective. The virtual right hand was collocated with their real right hand. Event-related brain potentials were recorded in two conditions, one where the participant's virtual hand was attacked with a knife and a control condition where the knife only struck the virtual table. Significantly greater P450 potentials were obtained in the attack condition confirming our expectations that participants had a strong illusion of the virtual hand being their own, which was also strongly supported by questionnaire responses. Higher levels of subjective virtual hand ownership correlated with larger P450 amplitudes. Mu-rhythm event-related desynchronization in the motor cortex and readiness potential (C3-C4) negativity were clearly observed when the virtual hand was threatened-as would be expected, if the real hand was threatened and the participant tried to avoid harm. Our results support the idea that event-related potentials may provide a promising non-subjective measure of virtual embodiment. They also support previous experiments on pain observation and are placed into context of similar experiments and studies of body perception and body ownership within cognitive neuroscience.

Virtual reality for assessment of patients suffering chronic pain: a case study.

The study of body representation and ownership has been a very active research area in recent years. Synchronous multisensory stimulation has been used for the induction of the illusion of ownership over virtual body parts and even full bodies, and it has provided experimental paradigms for the understanding of the brain processing of body representation. However, the illusion of ownership of a virtual body has rarely been used for patient evaluation and diagnosis. Here we propose a method that exploits ownership of a virtual body in combination with a simple brain computer interface (BCI) and basic physiological measures to complement neurological assessment. A male patient presenting a fixed posture dystonia featuring a permanently closed left fist participated in this case study. The patient saw a virtual body that substituted his own after donning a head-mounted display and thereby entering the virtual reality. The left virtual hand had the same posture as his corresponding real hand. After inducing virtual hand ownership by correlated visuo-tactile stimulation and dynamic reflections in a virtual mirror, the virtual hand would open either automatically or through a cognitive task assessed through a BCI that required him to focus attention on the virtual hand. The results reveal that body ownership induced changes on electromyography and BCI performance in the patient that were different from those in five healthy controls. Overall, the case study shows that the induction of virtual body ownership combined with simple electrophysiological measures could be useful for the diagnosis of patients with neurological conditions.

Enhanced efficiency in visually guided online motor control for actions redirected towards the body midline.

Reaching objects in a dynamic environment requires fast online corrections that compensate for sudden object shifts or postural changes. Previous studies revealed the key role of visually monitoring the hand-to-target distance throughout action execution. In the current study, we investigate how sensorimotor asymmetries associated with space perception, brain lateralization and biomechanical constraints, affect the efficiency of online corrections. Participants performed reaching actions in virtual reality, where the virtual hand was progressively displaced from the real hand to trigger online corrections, for which it was possible to control the total amount of the redirection and the region of space in which the action unfolded. The efficiency of online corrections and the degree of awareness of the ensuing motor corrections were taken as assessment variables. Results revealed more efficient visuo-motor corrections for actions redirected towards, rather than away from the body midline. The effect is independent on the reaching hand and the hemispace of action, making explanations associated with laterality effects and biomechanical constraints improbable. The result cannot either be accounted for by the visual processing advantage in the straight-ahead region. An explanation may be found in the finer sensorimotor representations characterizing the frontal space proximal to body, where a preference for visual processing has been documented, and where high-value functional actions, like fine manipulative skills, typically take place. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

A mechatronic shape display based on auxetic materials.

Shape displays enable people to touch simulated surfaces. A common architecture of such devices uses a mechatronic pin-matrix. Besides their complexity and high cost, these matrix displays suffer from sharp edges due to the discreet representation which reduces their ability to render a large continuous surface when sliding the hand. We propose using an engineered auxetic material actuated by a smaller number of motors. The material bends in multiple directions, feeling smooth and rigid to touch. A prototype implementation uses nine actuators on a 220 mm square section of material. It can display a range of surface curvatures under the palm of a user without aliased edges. In this work we use an auxetic skeleton to provide rigidity on a soft material and demonstrate the potential of this class of surface through user experiments.

Rethinking GPS navigation: creating cognitive maps through auditory clues.

GPS navigation is commonplace in everyday life. While it has the capacity to make our lives easier, it is often used to automate functions that were once exclusively performed by our brain. Staying mentally active is key to healthy brain aging. Therefore, is GPS navigation causing more harm than good? Here we demonstrate that traditional turn-by-turn navigation promotes passive spatial navigation and ultimately, poor spatial learning of the surrounding environment. We propose an alternative form of GPS navigation based on sensory augmentation, that has the potential to fundamentally alter the way we navigate with GPS. By implementing a 3D spatial audio system similar to an auditory compass, users are directed towards their destination without explicit directions. Rather than being led passively through verbal directions, users are encouraged to take an active role in their own spatial navigation, leading to more accurate cognitive maps of space. Technology will always play a significant role in everyday life; however, it is important that we actively engage with the world around us. By simply rethinking the way we interact with GPS navigation, we can engage users in their own spatial navigation, leading to a better spatial understanding of the explored environment.

Using Facial Animation to Increase the Enfacement Illusion and Avatar Self-Identification.

Through avatar embodiment in Virtual Reality (VR) we can achieve the illusion that an avatar is substituting our body: the avatar moves as we move and we see it from a first person perspective. However, self-identification, the process of identifying a representation as being oneself, poses new challenges because a key determinant is that we see and have agency in our own face. Providing control over the face is hard with current HMD technologies because face tracking is either cumbersome or error prone. However, limited animation is easily achieved based on speaking. We investigate the level of avatar enfacement, that is believing that a picture of a face is one's own face, with three levels of facial animation: (i) one in which the facial expressions of the avatars are static, (ii) one in which we implement lip-sync motion and (iii) one in which the avatar presents lip-sync plus additional facial animations, with blinks, designed by a professional animator. We measure self-identification using a face morphing tool that morphs from the face of the participant to the face of a gender matched avatar. We find that self-identification on avatars can be increased through pre-baked animations even when these are not photorealistic nor look like the participant.

Avatar Embodiment Enhances Haptic Confidence on the Out-of-Body Touch Illusion.

In the real world, our bodies influence how we perceive ourselves and how others perceive us. Our body can also affect estimations of object sizes and distances. But how does our body affect our haptic experience? Here, we examined the modulation of a visuo-haptic illusion of touch on a virtual stick in virtual reality, when participants were embodied in an avatar and when they were not. During the experiments participants (n = 49) received successions of three taps delivered from two independent controllers while they saw visual stimuli presented sequentially along the virtual stick. The stimulation pattern resulted in a robust illusion of tapping directly on the virtual stick. After each trial, participants were asked to report where they perceived the taps. We found that participants in both the body and no-body conditions displaced the second tap toward the center of the stick, and reported similar levels of certainty about their reported location. However, the illusion of touch on the stick, as measured by the reported location of the tap, was significantly stronger for those who had a virtual body than those who did not. Therefore, our study shows that avatar embodiment can change haptic perception.

Avatar Embodiment. Towards a Standardized Questionnaire.

Inside virtual reality, users can embody avatars that are collocated from a first-person perspective. When doing so, participants have the feeling that the own body has been substituted by the self-avatar, and that the new body is the source of the sensations. Embodiment is complex as it includes not only body ownership over the avatar, but also agency, co-location, and external appearance. Despite the multiple variables that influence it, the illusion is quite robust, and it can be produced even if the self-avatar is of a different age, size, gender, or race from the participant's own body. Embodiment illusions are therefore the basis for many social VR experiences and a current active research area among the community. Researchers are interested both in the body manipulations that can be accepted, as well as studying how different self-avatars produce different attitudinal, social, perceptual, and behavioral effects. However, findings suggest that despite embodiment being strongly associated with the performance and reactions inside virtual reality, the extent to which the illusion is experienced varies between participants. In this paper, we review the questionnaires used in past experiments and propose a standardized embodiment questionnaire based on 25 questions that are prevalent in the literature. We encourage future virtual reality experiments that include first-person virtual avatars to administer this questionnaire in order to evaluate the degree of embodiment.

The uncanny valley of haptics.

During teleoperation and virtual reality experiences, enhanced haptic feedback incongruent with other sensory cues can reduce subjective realism, producing an uncanny valley of haptics.

Generic HRTFs May be Good Enough in Virtual Reality. Improving Source Localization through Cross-Modal Plasticity.

Auditory spatial localization in humans is performed using a combination of interaural time differences, interaural level differences, as well as spectral cues provided by the geometry of the ear. To render spatialized sounds within a virtual reality (VR) headset, either individualized or generic Head Related Transfer Functions (HRTFs) are usually employed. The former require arduous calibrations, but enable accurate auditory source localization, which may lead to a heightened sense of presence within VR. The latter obviate the need for individualized calibrations, but result in less accurate auditory source localization. Previous research on auditory source localization in the real world suggests that our representation of acoustic space is highly plastic. In light of these findings, we investigated whether auditory source localization could be improved for users of generic HRTFs via cross-modal learning. The results show that pairing a dynamic auditory stimulus, with a spatio-temporally aligned visual counterpart, enabled users of generic HRTFs to improve subsequent auditory source localization. Exposure to the auditory stimulus alone or to asynchronous audiovisual stimuli did not improve auditory source localization. These findings have important implications for human perception as well as the development of VR systems as they indicate that generic HRTFs may be enough to enable good auditory source localization in VR.

Participant concerns for the Learner in a Virtual Reality replication of the Milgram obedience study.

In Milgram's seminal obedience studies, participants' behaviour has traditionally been explained as a demonstration of people's tendency to enter into an 'agentic state' when in the presence of an authority figure: they attend only to the demands of that authority and are insensitive to the plight of their victims. There have been many criticisms of this view, but most rely on either indirect or anecdotal evidence. In this study, participants (n = 40) are taken through a Virtual Reality simulation of the Milgram paradigm. Compared to control participants (n = 20) who are not taken through the simulation, those in the experimental conditions are found to attempt to help the Learner more by putting greater emphasis on the correct word over the incorrect words. We also manipulate the extent to which participants identify with the science of the study and show that high identifiers both give more help, are less stressed, and are more hesitant to press the shock button than low identifiers. We conclude that these findings constitute a refutation of the 'agentic state' approach to obedience. Instead, we discuss implications for the alternative approaches such as 'engaged followership' which suggests that obedience is a function of relative identification with the science and with the victim in the study. Finally, we discuss the value of Virtual Reality as a technique for investigating hard-to-study psychological phenomena.

Model of Illusions and Virtual Reality.

In Virtual Reality (VR) it is possible to induce illusions in which users report and behave as if they have entered into altered situations and identities. The effect can be robust enough for participants to respond "realistically," meaning behaviors are altered as if subjects had been exposed to the scenarios in reality. The circumstances in which such VR illusions take place were first introduced in the 80's. Since then, rigorous empirical evidence has explored a wide set of illusory experiences in VR. Here, we compile this research and propose a neuroscientific model explaining the underlying perceptual and cognitive mechanisms that enable illusions in VR. Furthermore, we describe the minimum instrumentation requirements to support illusory experiences in VR, and discuss the importance and shortcomings of the generic model.

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues.

Humans are good at selectively listening to specific target conversations, even in the presence of multiple concurrent speakers. In our research, we study how auditory-visual cues modulate this selective listening. We do so by using immersive Virtual Reality technologies with spatialized audio. Exposing 32 participants to an Information Masking Task with concurrent speakers, we find significantly more errors in the decision-making processes triggered by asynchronous audiovisual speech cues. More precisely, the results show that lips on the Target speaker matched to a secondary (Mask) speaker's audio severely increase the participants' comprehension error rates. In a control experiment (n = 20), we further explore the influences of the visual modality over auditory selective attention. The results show a dominance of visual-speech cues, which effectively turn the Mask into the Target and vice-versa. These results reveal a disruption of selective attention that is triggered by bottom-up multisensory integration. The findings are framed in the sensory perception and cognitive neuroscience theories. The VR setup is validated by replicating previous results in this literature in a supplementary experiment.

The Neurological Traces of Look-Alike Avatars.

We designed an observational study where participants (n = 17) were exposed to pictures and look-alike avatars pictures of themselves, a familiar friend or an unfamiliar person. By measuring participants' brain activity with electroencephalography (EEG), we found face-recognition event related potentials (ERPs) in the visual cortex, around 200-250 ms, to be prominent for the different familiarity levels. A less positive component was found for self-recognized pictures (P200) than pictures of others, showing similar effects in both real faces and look-alike avatars. A rapid adaptation in the same component was found when comparing the neural processing of avatar faces vs. real faces, as if avatars in general were assimilated as real face representations over time. ERP results also showed that in the case of the self-avatar, the P200 component correlated with more complex conscious encodings of self-representation, i.e., the difference in voltage in the P200 between the self-avatar and the self-picture was reduced in participants that felt the avatar looked like them. This study is put into context within the literature of self-recognition and face recognition in the visual cortex. Additionally, the implications of these results on look-alike avatars are discussed both for future virtual reality (VR) and neuroscience studies.

Empowering patients to perform physical therapy at home.

In this paper we address the problem of patient adherence to physical therapy using a sensor-enabled virtual reality gaming interface that motivates users to complete their exercises while collecting quantitative data. The system also allows the therapist to monitor and interact with patients remotely providing reinforcing feedback and support with the CollaboRhythm care delivery platform. The data collected with this system enables the therapist and the patient to make informed decisions about patient treatment and exercise regimens based on the patient progress. The system is capable of supporting a wide array of rehabilitation scenarios with remote collaboration. A knee replacement scenario was tested with an experimental protocol involving 16 healthy participants. The results show both quantitatively and qualitatively that patients can learn intuitively to perform their physical therapy exercises on a remote environment without further human intervention.

Motor imagery based brain-computer interface: a study of the effect of positive and negative feedback.

Co-adaptation between the human brain and computers is an important issue in brain-computer interface (BCI) research. However, most of the research has focused on the computer side of BCI, such as developing powerful machine-learning algorithms, while less research has focused on investigating how BCI users may optimally adapt. This paper assesses the influences of positive and negative visual feedback on motor imagery (MI) skills by evaluating the performance. More precisely, a MI based BCI paradigm was employed with fake visual feedback, regardless of subjects' real performance. Subjects were exposed to two experimental conditions--one positive and one negative, in which 80% or 30% of the trials were associated with positive feedback, respectively. The main EEG feature for MI-BCI classification--the asymmetry of mu-rhythm between hemispheres--was more prominent only after the negative feedback session. In addition, the negative feedback condition was accompanied by larger heart rate variability compared to the positive feedback condition. Our results suggest that visual feedback is an important aspect to take into account when designing BCI skill acquisition sessions.