Visuo-Haptic Rendering of the Hand during 3D Manipulation in Augmented Reality.

Manipulating virtual objects with bare hands is a key interaction in Augmented Reality (AR) applications. However, there are still several limitations that affect the manipulation, including the lack of mutual visual occlusion between virtual and real content as well as the lack of haptic sensations. To address the two abovementioned matters, the role of the visuo-haptic rendering of the hand as sensory feedback is investigated. The first experiment explores the effect of showing the hand of the user as seen by the AR system through an avatar, comparing six visual hand rendering. The second experiment explores the effect of the visuo-haptic hand rendering by comparing two vibrotactile contact techniques provided at four delocalized positions on the hand and combined with the two most representative visual hand renderings from the first experiment. Results show that delocalized vibrotactile haptic hand rendering improved perceived effectiveness, realism, and usefulness when provided close to the contact point. However, the farthest rendering position, i.e., on the contralateral hand, gave the best performance even though it was largely disliked. The visual hand rendering was perceived as less necessary for manipulation when the haptic hand rendering was available, but still provided useful feedback on the hand tracking.

Bimanual Ultrasound Mid-Air Haptics for Virtual Reality Manipulation.

The ability to manipulate and physically feel virtual objects without any real object being present and without equipping the user has been a long-standing goal in virtual reality (VR). Emerging ultrasound mid-air haptics (UMH) technology could potentially address this challenge, as it enables remote tactile stimulation of unequipped users. However, to date, UMH has received limited attention in the field of haptic exploration and manipulation in virtual environments. Existing work has primarily focused on interactions requiring a single hand and thus the delivery of unimanual haptic feedback. Despite being fundamental to a large part of haptic interactions with our environments, bimanual tasks have rarely been studied in the field of UMH interaction in VR. In this paper, we propose the use of non-coplanar mid-air haptic devices for providing simultaneous tactile feedback to both hands during bimanual VR manipulation. We discuss coupling schemes and haptic rendering algorithms for providing bimanual haptic feedback in bimanual interactions with virtual environments. We then present two human participant studies, assessing the benefits of bimanual ultrasound haptic feedback in a two-handed grasping and holding task and in a shape exploration task. Results suggest that the use of multiple non-coplanar UMH devices could be an interesting approach for enriching unencumbered haptic manipulation in virtual environments.

When Tangibles Become Deformable: Studying Pseudo-Stiffness Perceptual Thresholds in a VR Grasping Task.

Pseudo-Haptic techniques, or visuo-haptic illusions, leverage user's visual dominance over haptics to alter the users' perception. As they create a discrepancy between virtual and physical interactions, these illusions are limited to a perceptual threshold. Many haptic properties have been studied using pseudo-haptic techniques, such as weight, shape or size. In this paper, we focus on estimating the perceptual thresholds for pseudo-stiffness in a virtual reality grasping task. We conducted a user study (n = 15) where we estimated if compliance can be induced on a non-compressible tangible object and to what extent. Our results show that (1) compliance can be induced in a rigid tangible object and that (2) pseudo-haptics can simulate beyond 24 N/cm stiffness ( k ≥ 24 N / cm, between a gummy bear and a raisin, up to rigid objects). Pseudo-stiffness efficiency is (3) enhanced by the objects' scales, but mostly (4) correlated to the user input force. Taken altogether, our results offer novel opportunities to simplify the design of future haptic interfaces, and extend the haptic properties of passive props in VR.

Deconstructing Haptic Feedback Information in Robot-Assisted Needle Insertion in Soft Tissues.

This paper evaluates the role and effectiveness of different types of haptic feedback in presenting relevant feedback information during needle insertion in soft tissues through a remotely operated robot. We carried out three experiments with human subjects to analyze the effect of grounded kinesthetic feedback, cutaneous vibrotactile feedback, and cutaneous pressure feedback for rendering the elastic and the viscous force components of a simplified needle-tissue interaction model in a simulated environment. Results showed that providing the two pieces of feedback information through different channels, i.e., kinesthetic and cutaneous, led to the best performance, yielding an improvement in detecting a different tissue layer with respect to providing both information through the same commercial grounded kinesthetic interface. Moreover, results indicate that cutaneous pressure feedback is more suited for rendering the elastic component of the interaction with respect to vibrotactile cutaneous sensations. Finally, results suggest that rendering this elastic component where the user holds the input interface is not so important, confirming that delocalized cutaneous sensations can be an effective solution.

Can we Effectively Combine Tangibles and Ultrasound Mid-Air Haptics? A Study of Acoustically Transparent Tangible Surfaces.

We propose to study the combination of acoustically transparent tangible objects (ATTs) and ultrasound mid-air haptic (UMH) feedback to support haptic interactions with digital content. Both these haptic feedback methods have the advantage of leaving users unencumbered, and present uniquely complementary strengths and weaknesses. In this article, we provide an overview of the design space for haptic interactions covered by this combination, as well as requirements for their technical implementation. Indeed, when imagining the concurrent manipulation of physical objects and delivery of mid-air haptic stimuli, reflection and absorption of sound by the tangibles may impede delivery of the UMH stimuli. To demonstrate the viability of our approach, we study the combination of single ATT surfaces, i.e. the basic building blocks for any tangible object, and UMH stimuli. We investigate attenuation of a focal point focused through various plates of acoustically transparent materials, and run three human subject experiments investigating the impact of acoustically transparent materials on detection thresholds, discrimination of motion, and localization of ultrasound haptic stimuli. Results show that tangible surfaces which do not significantly attenuate ultrasound can be fabricated with relative ease. The perception studies confirm that ATT surfaces do not impede perception of UMH stimulus properties, and thus that both may viably be combined in haptics applications.

Ultrasound Mid-Air Haptics for Hand Guidance in Virtual Reality.

This paper presents and evaluates a set of mid-air ultrasound haptic strategies to provide 2-degree-of-freedom position and orientation guidance in Virtual Reality (VR). We devised four strategies for providing position guidance and two for providing orientation guidance. A human subject study evaluated the effectiveness of the proposed techniques in guiding users towards objectives in static and dynamic environments in VR, both in position and orientation. Results show that, compared to visual feedback of the virtual environment alone, the considered strategies significantly improve positioning performance in the static scenario. On the other hand, orientation guidance led to significant improvements only in the dynamic scenario.

Investigating the Haptic Perception of Directional Information Within a Handle.

This paper studies the perception of 2-dimensional directional cues presented on a hand-held tangible interface that resembles a cylindrical handle. The tangible interface is designed to be comfortably held with one hand and houses five custom electromagnetic actuators composed of coils as stators and magnets as movers. We carried out a human subjects experiment enrolling 24 participants, analysing the recognition rate of directional cues using the actuators either to vibrate or tap in sequence across the user's palm. Results show an impact of the positioning/holding of the handle, the mode of stimulation, and the directional indication sent via the handle. There was also a correlation between the score and the confidence of the participants, showing that participants are more confident when recognising vibration patterns. Overall, results supported the potential of the haptic handle to provide accurate guidance, with recognition rates higher than 70 % in all conditions and higher than 75 % in the precane and power wheelchair configurations.

PalmEx: Adding Palmar Force-Feedback for 3D Manipulation with Haptic Exoskeleton Gloves.

Haptic exoskeleton gloves are a widespread solution for providing force-feedback in Virtual Reality (VR), especially for 3D object manipulations. However, they are still lacking an important feature regarding in-hand haptic sensations: the palmar contact. In this paper, we present PalmEx, a novel approach which incorporates palmar force-feedback into exoskeleton gloves to improve the overall grasping sensations and manual haptic interactions in VR. PalmEx's concept is demonstrated through a self-contained hardware system augmenting a hand exoskeleton with an encountered palmar contact interface - physically encountering the users' palm. We build upon current taxonomies to elicit PalmEx's capabilities for both the exploration and manipulation of virtual objects. We first conduct a technical evaluation optimising the delay between the virtual interactions and their physical counterparts. We then empirically evaluate PalmEx's proposed design space in a user study (n=12) to assess the potential of a palmar contact for augmenting an exoskeleton. Results show that PalmEx offers the best rendering capabilities to perform believable grasps in VR. PalmEx highlights the importance of the palmar stimulation, and provides a low-cost solution to augment existing high-end consumer hand exoskeletons.

Haptic Feedback, Performance and Arousal: A Comparison Study in an Immersive VR Motor Skill Training Task.

This paper investigates the relationship between fine motor skill training in VR, haptic feedback, and physiological arousal. To do so, we present the design and development of a motor skill task (buzzwire), along with a custom vibrotactile feedback attachment for the Geomagic Touch haptic device. A controlled experiment following a between-subjects design was conducted with 73 participants, studying the role of three feedback conditions - visual/kinesthetic, visual/vibrotactile and visual only - on the learning and performance of the considered task and the arousal levels of the participants. Results indicate that performance improved in all three feedback conditions after the considered training session. However, participants reported no change in self-efficacy and in terms of presence and task load (NASA-TLX). All three feedback conditions also showed similar arousal levels. Further analysis revealed that positive changes in performance were linked to higher arousal levels. These results suggest the potential of haptic feedback to affect arousal levels and encourage further research into using this relationship to improve motor skill training in VR.

Does Multi-Actuator Vibrotactile Feedback Within Tangible Objects Enrich VR Manipulation?

Rich, informative and realistic haptic feedback is key to enhancing Virtual Reality (VR) manipulation. Tangible objects provide convincing grasping and manipulation interactions with haptic feedback of e.g., shape, mass and texture properties. But these properties are static, and cannot respond to interactions in the virtual environment. On the other hand, vibrotactile feedback provides the opportunity for delivering dynamic cues rendering many different contact properties, such as impacts, object vibrations or textures. Handheld objects or controllers in VR are usually restricted to vibrating in a monolithic fashion. In this paper, we investigate how spatialiazing vibrotactile cues within handheld tangibles could enable a wider range of sensations and interactions. We conduct a set of perception studies, investigating the extent to which spatialization of vibrotactile feedback within tangible objects is possible as well as the benefits of proposed rendering schemes leveraging multiple actuators in VR. Results show that vibrotactile cues from localized actuators can be discriminated and are beneficial for certain rendering schemes.

A Wearable Haptic Device for the Hand with Interchangeable End-Effectors.

This paper presents a 4-degrees-of-freedom (4-DoF) hand wearable haptic device for Virtual Reality (VR). It is designed to support different end-effectors, that can be easily exchanged so as to provide a wide range of haptic sensations. The device is composed of a static upper body, secured to the back of the hand, and the (changeable) end-effector, placed in contact with the palm. The two parts of the device are connected by two articulated arms, actuated by four servo motors housed on the upper body and along the arms. The paper summarizes the design and kinematics of the wearable haptic device and presents a position control scheme able to actuate a broad range of end-effectors. As a proof of concept, we present and evaluate three representative end-effectors during interactions in VR, rendering the sensation of interacting (E1) with rigid slanted surfaces and sharp edges having different orientations, (E2) with curved surfaces having different curvatures, and (E3) with soft surfaces having different stiffness characteristics. A few additional end-effector designs are discussed. A human-subjects evaluation in immersive VR shows the broad applicability of the device, able to render rich interactions with a diverse set of virtual objects.

Electrotactile Feedback Applications for Hand and Arm Interactions: A Systematic Review, Meta-Analysis, and Future Directions.

Haptic feedback is critical in a broad range of human-machine/computer-interaction applications. However, the high cost and low portability/wearability of haptic devices remain unresolved issues, severely limiting the adoption of this otherwise promising technology. Electrotactile interfaces have the advantage of being more portable and wearable due to their reduced actuators' size, as well as their lower power consumption and manufacturing cost. The applications of electrotactile feedback have been explored in human-computer interaction and human-machine-interaction for facilitating hand-based interactions in applications, such as prosthetics, virtual reality, robotic teleoperation, surface haptics, portable devices, and rehabilitation. This article presents a technological overview of electrotactile feedback, as well a systematic review and meta-analysis of its applications for hand-based interactions. We discuss the different electrotactile systems according to the type of application. We also discuss over a quantitative congregation of the findings, to offer a high-level overview into the state-of-art and suggest future directions. Electrotactile feedback systems showed increased portability/wearability, and they were successful in rendering and/or augmenting most tactile sensations, eliciting perceptual processes, and improving performance in many scenarios. However, knowledge gaps (e.g., embodiment), technical (e.g., recurrent calibration, electrodes' durability) and methodological (e.g., sample size) drawbacks were detected, which should be addressed in future studies.

Action-Specific Perception & Performance on a Fitts's Law Task in Virtual Reality: The Role of Haptic Feedback.

While user's perception and performance are predominantly examined independently in virtual reality, the Action-Specific Perception (ASP) theory postulates that the performance of an individual on a task modulates this individual's spatial and time perception pertinent to the task's components and procedures. This paper examines the association between performance and perception and the potential effects that tactile feedback modalities could generate. This paper reports a user study (N=24), in which participants performed a standardized Fitts's law target acquisition task by using three feedback modalities: visual, visuo-electrotactile, and visuo-vibrotactile. The users completed 3 Target Sizes × 2 Distances × 3 feedback modalities = 18 trials. The size perception, distance perception, and (movement) time perception were assessed at the end of each trial. Performance-wise, the results showed that electrotactile feedback facilitates a significantly better accuracy compared to vibrotactile and visual feedback, while vibrotactile provided the worst accuracy. Electrotactile and visual feedback enabled a comparable reaction time, while the vibrotactile offered a substantially slower reaction time than visual feedback. Although amongst feedback types the pattern of differences in perceptual aspects were comparable to performance differences, none of them was statistically significant. However, performance indeed modulated perception. Significant action-specific effects on spatial and time perception were detected. Changes in accuracy modulate both size perception and time perception, while changes in movement speed modulate distance perception. Also, the index of difficulty was found to modulate all three perceptual aspects. However, individual differences appear to affect the magnitude of action-specific effects. These outcomes highlighted the importance of haptic feedback on performance, and importantly the significance of action-specific effects on spatial and time perception in VR, which should be considered in future VR studies.

Crowd Navigation in VR: Exploring Haptic Rendering of Collisions.

Virtual reality (VR) is a valuable experimental tool for studying human movement, including the analysis of interactions during locomotion tasks for developing crowd simulation algorithms. However, these studies are generally limited to distant interactions in crowds, due to the difficulty of rendering realistic sensations of collisions in VR. In this article, we explore the use of wearable haptics to render contacts during virtual crowd navigation. We focus on the behavioral changes occurring with or without haptic rendering during a navigation task in a dense crowd, as well as on potential after-effects introduced by the use haptic rendering. Our objective is to provide recommendations for designing VR setup to study crowd navigation behavior. To the end, we designed an experiment (N=23) where participants navigated in a crowded virtual train station without, then with, and then again without haptic feedback of their collisions with virtual characters. Results show that providing haptic feedback improved the overall realism of the interaction, as participants more actively avoided collisions. We also noticed a significant after-effect in the users' behavior when haptic rendering was once again disabled in the third part of the experiment. Nonetheless, haptic feedback did not have any significant impact on the users' sense of presence and embodiment.

Design of Personalized Wearable Haptic Interfaces to Account for Fingertip Size and shape.

The size and shape of fingertips vary significantly across humans, making it challenging to design wearable fingertip interfaces suitable for everyone. Although deemed important, this issue has often been neglected due to the difficulty of customizing devices for each different user. This article presents an innovative approach for automatically adapting the hardware design of a wearable haptic interface for a given user. We consider a three-DoF fingertip cutaneous device, composed of a static body and a mobile platform linked by three articulated legs. The mobile platform is capable of making and breaking contact with the finger pulp and re-angle to replicate contacts with arbitrarily-oriented surfaces. We analyze the performance of this device as a function of its main geometrical dimensions. Then, starting from the user's fingertip characteristics, we define a numerical procedure that best adapts the dimension of the device to: (i) maximize the range of renderable haptic stimuli; (ii) avoid unwanted contacts between the device and the skin; (iii) avoid singular configurations; and (iv) minimize the device encumbrance and weight. Together with the mechanical analysis and evaluation of the adapted design, we present a MATLAB script that calculates the device dimensions customized for a target fingertip as well as an online CAD utility for generating a ready-to-print STL file of the personalized design.

Capacitive Sensing for Improving Contact Rendering With Tangible Objects in VR.

We combine tracking information from a tangible object instrumented with capacitive sensors and an optical tracking system, to improve contact rendering when interacting with tangibles in VR. A human-subject study shows that combining capacitive sensing with optical tracking significantly improves the visuohaptic synchronization and immersion of the VR experience.

PUMAH: Pan-Tilt Ultrasound Mid-Air Haptics for Larger Interaction Workspace in Virtual Reality.

Mid-air haptic interfaces are promising tools for providing tactile feedback in Virtual Reality (VR) applications, as they do not require the user to be tethered to, hold, or wear any system or device. Currently, focused ultrasound phased arrays are the most mature solution for providing mid-air haptic feedback. They modulate the phase of an array of ultrasound emitters so as to generate focused points of oscillating high pressure, eliciting vibrotactile sensations when encountering a user's skin. While these arrays feature a reasonably large vertical workspace, they are not capable of displaying stimuli far beyond their horizontal limits, severely limiting their workspace in the lateral dimensions. In this paper, we propose an innovative low-cost solution for enlarging the workspace of focused ultrasound arrays. It features two degrees of freedom, rotating the array around the pan and tilt axes, thereby significantly increasing the usable workspace and enabling multi-directional feedback. Our hardware tests and human subject study in an ecological VR setting show a 14-fold increase in workspace volume, with focal point repositioning speeds over 0.85 m/s while delivering tactile feedback with positional accuracy below 18 mm. Finally, we propose a representative use case to exemplify the potential of our system for VR applications.

Compensating for Fingertip Size to Render Tactile Cues More Accurately.

Fingertip haptic feedback offers advantages in many applications, including robotic teleoperation, gaming, and training. However, fingertip size and shape vary significantly across humans, making it difficult to design fingertip interfaces and rendering techniques suitable for everyone. This article starts with an existing data-driven haptic rendering algorithm that ignores fingertip size, and it then develops two software-based approaches to personalize this algorithm for fingertips of different sizes using either additional data or geometry. We evaluate our algorithms in the rendering of pre-recorded tactile sensations onto rubber casts of six different fingertips as well as onto the real fingertips of 13 human participants. Results on the casts show that both approaches significantly improve performance, reducing force error magnitudes by an average of 78% with respect to the standard non-personalized rendering technique. Congruent results were obtained for real fingertips, with subjects rating each of the two personalized rendering techniques significantly better than the standard non-personalized method.

Altering the Stiffness, Friction, and Shape Perception of Tangible Objects in Virtual Reality Using Wearable Haptics.

Tangible objects are used in virtual reality (VR) and augmented reality (AR) to enhance haptic information on the general shape of virtual objects. However, they are often passive or unable to simulate rich varying mechanical properties. This article studies the effect of combining simple passive tangible objects and wearable haptics for improving the display of varying stiffness, friction, and shape sensations in these environments. By providing timely cutaneous stimuli through a wearable finger device, we can make an object feel softer or more slippery than it really is, and we can also create the illusion of encountering virtual bumps and holes. We evaluate the proposed approach carrying out three experiments with human subjects. Results confirm that we can increase the compliance of a tangible object by varying the pressure applied through a wearable device. We are also able to simulate the presence of bumps and holes by providing timely pressure and skin stretch sensations. Altering the friction of a tangible surface showed recognition rates above the chance level, albeit lower than those registered in the other experiments. Finally, we show the potential of our techniques in an immersive medical palpation use case in VR. These results pave the way for novel and promising haptic interactions in VR, better exploiting the multiple ways of providing simple, unobtrusive, and inexpensive haptic displays.

A Haptic Shared-Control Architecture for Guided Multi-Target Robotic Grasping.

Although robotic telemanipulation has always been a key technology for the nuclear industry, little advancement has been seen over the last decades. Despite complex remote handling requirements, simple mechanically linked master-slave manipulators still dominate the field. Nonetheless, there is a pressing need for more effective robotic solutions able to significantly speed up the decommissioning of legacy radioactive waste. This paper describes a novel haptic shared-control approach for assisting a human operator in the sort and segregation of different objects in a cluttered and unknown environment. A three-dimensional scan of the scene is used to generate a set of potential grasp candidates on the objects at hand. These grasp candidates are then used to generate guiding haptic cues, which assist the operator in approaching and grasping the objects. The haptic feedback is designed to be smooth and continuous as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. To validate our approach, we carried out two human-subject studies, enrolling 15 participants. We registered an average improvement of 20.8%, 20.1%, and 32.5% in terms of completion time, linear trajectory, and perceived effectiveness, respectively, between the proposed approach and standard teleoperation.