Skin-integrated wireless haptic interfaces for virtual and augmented reality.

Traditional technologies for virtual reality (VR) and augmented reality (AR) create human experiences through visual and auditory stimuli that replicate sensations associated with the physical world. The most widespread VR and AR systems use head-mounted displays, accelerometers and loudspeakers as the basis for three-dimensional, computer-generated environments that can exist in isolation or as overlays on actual scenery. In comparison to the eyes and the ears, the skin is a relatively underexplored sensory interface for VR and AR technology that could, nevertheless, greatly enhance experiences at a qualitative level, with direct relevance in areas such as communications, entertainment and medicine1,2. Here we present a wireless, battery-free platform of electronic systems and haptic (that is, touch-based) interfaces capable of softly laminating onto the curved surfaces of the skin to communicate information via spatio-temporally programmable patterns of localized mechanical vibrations. We describe the materials, device structures, power delivery strategies and communication schemes that serve as the foundations for such platforms. The resulting technology creates many opportunities for use where the skin provides an electronically programmable communication and sensory input channel to the body, as demonstrated through applications in social media and personal engagement, prosthetic control and feedback, and gaming and entertainment.

Large-area MRI-compatible epidermal electronic interfaces for prosthetic control and cognitive monitoring.

Skin-interfaced medical devices are critically important for diagnosing disease, monitoring physiological health and establishing control interfaces with prosthetics, computer systems and wearable robotic devices. Skin-like epidermal electronic technologies can support these use cases in soft and ultrathin materials that conformally interface with the skin in a manner that is mechanically and thermally imperceptible. Nevertheless, schemes so far have limited the overall sizes of these devices to less than a few square centimetres. Here, we present materials, device structures, handling and mounting methods, and manufacturing approaches that enable epidermal electronic interfaces that are orders of magnitude larger than previously realized. As a proof-of-concept, we demonstrate devices for electrophysiological recordings that enable coverage of the full scalp and the full circumference of the forearm. Filamentary conductive architectures in open-network designs minimize radio frequency-induced eddy currents, forming the basis for structural and functional compatibility with magnetic resonance imaging. We demonstrate the use of the large-area interfaces for the multifunctional control of a transhumeral prosthesis by patients who have undergone targeted muscle-reinnervation surgery, in long-term electroencephalography, and in simultaneous electroencephalography and structural and functional magnetic resonance imaging.

Publisher Correction: Large-area MRI-compatible epidermal electronic interfaces for prosthetic control and cognitive monitoring.

In Fig. 4c of this Article, the scale bar units were incorrectly stated as 'µV'; the correct units are 'mV'. The figure has now been amended accordingly.

Publisher Correction: Large-area MRI-compatible epidermal electronic interfaces for prosthetic control and cognitive monitoring.

In Fig. 4c of this Article originally published, the bottom y axis was incorrectly labelled as 'MRI-ECG (µV)'; the correct label is 'MRI/ECG'. In addition, in Fig. 4d, the bottom y axis was incorrectly labelled as 'ECG (µV)'; the correct label is 'ECG (mV)'. The scale bar units were also incorrectly stated as 'mV', the correct units are 'µV'. The figure has now been amended accordingly.

Controlling sensation intensity for electrotactile stimulation in human-machine interfaces.

A barrier to practical use of electrotactile stimulation for haptic feedback has been large variability in perceived sensation intensity due to changes in the impedance of the electrode-skin interface, such as when electrodes peel or users sweat. Here, we show how to significantly reduce this variability by modulating stimulation parameters in response to measurements of impedance. Our method derives from three contributions. First, we created a model between stimulation parameters and impedance at constant perceived sensation intensity by looking at the peak pulse energy and phase charge. Our model fits experimental data better than previous models (mean R2 > 0.9) and holds over a larger set of conditions (subjects, sessions, magnitudes of sensation, stimulation locations, electrode sizes). Second, we implemented a controller that regulates perceived sensation intensity by using our model to derive a new current amplitude and pulse duration in response to changes in impedance. Our controller accurately predicts subject-chosen stimulation parameters at constant sensation intensity (mean R2 > 0.9). Third, we demonstrated as a proof-of-concept on two subjects with below-elbow amputations-using a prosthesis with electrotactile touch feedback-that our controller can regulate sensation intensity in response to large impedance changes that occur in activities of daily living. These results make electrotactile stimulation for human-machine interfaces more reliable during activities of daily living.

A Compliant Four-bar Linkage Mechanism that Makes the Fingers of a Prosthetic Hand More Impact Resistant.

Repeated mechanical failure due to accidental impact is one of the main reasons why people with upper-limb amputations abandon commercially-available prosthetic hands. To address this problem, we present the design and evaluation of a compliant four-bar linkage mechanism that makes the fingers of a prosthetic hand more impact resistant. Our design replaces both the rigid input and coupler links with a monolithic compliant bone, and replaces the follower link with three layers of pre-stressed spring steel. This design behaves like a conventional four-bar linkage but adds lateral compliance and eliminates a pin joint, which is a main site of failure on impact. Results from free-end and fixed-end impact tests show that, compared to those made with a conventional four-bar linkage, fingers made with our design absorb up to 11% more energy on impact with no mechanical failure. We also show the integration of these fingers in a prosthetic hand that is low-cost, light-weight, and easy to assemble, and that has grasping performance comparable to commercially-available hands.

A Portable, Arbitrary Waveform, Multichannel Constant Current Electrotactile Stimulator.

In this paper, we present the design and performance of a portable, arbitrary waveform, multichannel constant current electrotactile stimulator that costs less than $30 in components. The stimulator consists of a stimulation controller and power supply that are less than half the size of a credit card and can produce ±15 mA at ±150 V. The design is easily extensible to multiple independent channels that can receive an arbitrary waveform input from a digital-to-analog converter, drawing only 0.9 W/channel (lasting 4-5 hours upon continuous stimulation using a 9 V battery). Finally, we compare the performance of our stimulator to similar stimulators both commercially available and developed in research.

Estimation of distal arm joint angles from EMG and shoulder orientation for transhumeral prostheses.

In this paper, we quantify the extent to which shoulder orientation, upper-arm electromyography (EMG), and forearm EMG are predictors of distal arm joint angles during reaching in eight subjects without disability as well as three subjects with a unilateral transhumeral amputation and targeted reinnervation. Prior studies have shown that shoulder orientation and upper-arm EMG, taken separately, are predictors of both elbow flexion/extension and forearm pronation/supination. We show that, for eight subjects without disability, shoulder orientation and upper-arm EMG together are a significantly better predictor of both elbow flexion/extension during unilateral (R2=0.72) and mirrored bilateral (R2=0.72) reaches and of forearm pronation/supination during unilateral (R2=0.77) and mirrored bilateral (R2=0.70) reaches. We also show that adding forearm EMG further improves the prediction of forearm pronation/supination during unilateral (R2=0.82) and mirrored bilateral (R2=0.75) reaches. In principle, these results provide the basis for choosing inputs for control of transhumeral prostheses, both by subjects with targeted motor reinnervation (when forearm EMG is available) and by subjects without target motor reinnervation (when forearm EMG is not available). In particular, we confirm that shoulder orientation and upper-arm EMG together best predict elbow flexion/extension (R2=0.72) for three subjects with unilateral transhumeral amputations and targeted motor reinnervation. However, shoulder orientation alone best predicts forearm pronation/supination (R2=0.88) for these subjects, a contradictory result that merits further study.

Automatic Grasp Selection using a Camera in a Hand Prosthesis.

In this paper, we demonstrate how automatic grasp selection can be achieved by placing a camera in the palm of a prosthetic hand and training a convolutional neural network on images of objects with corresponding grasp labels. Our labeled dataset is built from common graspable objects curated from the ImageNet dataset and from images captured from our own camera that is placed in the hand. We achieve a grasp classification accuracy of 93.2% and show through real-time grasp selection that using a camera to augment current electromyography controlled prosthetic hands may be useful.

A Low-Cost, Open-Source, Compliant Hand for Enabling Sensorimotor Control for People with Transradial Amputations.

In this paper, we describe the design and implementation of a low-cost, open-source prosthetic hand that enables both motor control and sensory feedback for people with transradial amputations. We integrate electromyographic pattern recognition for motor control along with contact reflexes and sensory substitution to provide feedback to the user. Compliant joints allow for robustness to impacts. The entire hand can be built for around $550. This low cost makes research and development of sensorimotor prosthetic hands more accessible to researchers worldwide, while also being affordable for people with amputations in developing nations. We evaluate the sensorimotor capabilites of our hand with a subject with a transradial amputation. We show that using contact reflexes and sensory substitution, when compared to standard myoelectric prostheses that lack these features, improves grasping of delicate objects like an eggshell and a cup of water both with and without visual feedback. Our hand is easily integrated into standard sockets, facilitating long-term testing of sensorimotor capabilities.

Flexible Electronics: An Epidermal Stimulation and Sensing Platform for Sensorimotor Prosthetic Control, Management of Lower Back Exertion, and Electrical Muscle Activation (Adv. Mater. 22/2016).

The design of an ultrathin, conformal electronic device that integrates electrotactile stimulation with electromyography, temperature, and strain sensing in a single, simple platform is reported by J. A. Rogers and co-workers on page 4462. Demonstrated application possibilities include prosthetic control with sensory feedback, monitors, and stimulation signals related to lower back exertion, and electrical muscle stimulation with feedback control.

An Epidermal Stimulation and Sensing Platform for Sensorimotor Prosthetic Control, Management of Lower Back Exertion, and Electrical Muscle Activation.

The design of an ultrathin, conformal electronic device that integrates electrotactile stimulation with electromyography, temperature, and strain sensing in a single, simple platform is reported. Experiments demonstrate simultaneous use of multiple modes of operation of this type of device in the sensorimotor control of robotic systems, in the monitoring of lower back exertion and in muscle stimulation.

Playing checkers with your mind: an interactive multiplayer hardware game platform for brain-computer interfaces.

In this paper we describe a multiplayer brain-computer interface (BCI) based on the classic game of checkers using steady-state visually evoked potentials (SSVEPs). Previous research in BCI gaming focuses mainly on the production of software-based games using a computer screen--few hardware-based BCI games using a physical board have been developed. Hardware-based games can present a unique set of challenges when compared to software-based games. Depending on where the user is sitting, some stimuli might be farther away from the player, at a steeper viewing angle, conflated with competing stimuli, or occluded by physical barriers. In our game, we light squares on a checkerboard with flickering LEDs to elicit SSVEP responses in the subjects. When a subject attends to a particular square, the resulting SSVEPs are classified and a robot arm moves the selected piece. In a set of pilot experiments we investigated the ability of two subjects to use the SSVEP-based hardware game platform, and assessed how interstimulus distance, interstimulus angle, distance between target stimulus and subject, number of competing stimuli, and visual occlusions of the stimuli influence classification accuracy.

Materials and optimized designs for human-machine interfaces via epidermal electronics.

Thin, soft, and elastic electronics with physical properties well matched to the epidermis can be conformally and robustly integrated with the skin. Materials and optimized designs for such devices are presented for surface electromyography (sEMG). The findings enable sEMG from wide ranging areas of the body. The measurements have quality sufficient for advanced forms of human-machine interface.

Prediction of distal arm joint angles from EMG and shoulder orientation for prosthesis control.

Current state-of-the-art upper limb myoelectric prostheses are limited by only being able to control a single degree of freedom at a time. However, recent studies have separately shown that the joint angles corresponding to shoulder orientation and upper arm EMG can predict the joint angles corresponding to elbow flexion/extension and forearm pronation/ supination, which would allow for simultaneous control over both degrees of freedom. In this preliminary study, we show that the combination of both upper arm EMG and shoulder joint angles may predict the distal arm joint angles better than each set of inputs alone. Also, with the advent of surgical techniques like targeted muscle reinnervation, which allows a person with an amputation intuitive muscular control over his or her prosthetic, our results suggest that including a set of EMG electrodes around the forearm increases performance when compared to upper arm EMG and shoulder orientation. We used a Time-Delayed Adaptive Neural Network to predict distal arm joint angles. Our results show that our network's root mean square error (RMSE) decreases and coefficient of determination (R(2)) increases when combining both shoulder orientation and EMG as inputs.