Flexible Electrostatic Transducer Array with Displacement Control for Haptic Sensing and Actuation.

We developed flexible electrostatic transducers with both a single element and a 2×2 array format to actuate at a precise displacement across a range of loads with a control circuitry and algorithm. The transducer, composed of a moving buckled film with an integrated electrode and a rigid electrode, can be used to simultaneously generate and sense displacements. A circuit and computer program were designed to demonstrate displacement control and quantify the sensing precision of the transducer. Specifically, we applied a range of voltage and load conditions to the transducer and array and measured the displacement while under loading through capacitive sensing. The change in capacitance was linear with respect to the area of the electrode in contact and matched theoretical predictions when described as a function of the displacement. The transducer was loaded with weights in the range of 5-27 mN and capacitance-driving voltage graphs were obtained. An 8Hz driving frequency was used to move the transducer, while a 10.8kHz signal was used to sense the capacitance. These were used to build a predictive model to correct for sensed load to maintain a average displacement. It was found that a transducer of dimensions 10mm × 40mm was able to maintain displacement under loads of 5-27mN, while a matrix composed of 10mm × 20mm transducers was able to maintain displacement under loads of 2.5-11mN. In general, the detection thresholds of human skin can range between 5-20mN of force and 2-20um of displacement for frequencies between 1Hz and 250Hz, so these values are in line with what is needed to build a functional haptic wearable device. The present work provides a method to quantitatively measure and control a new type of flexible transducer for a variety of haptic applications.

Sound-to-Touch Crossmodal Pitch Matching for Short Sounds.

This article explores how to relate sound and touch in terms of their spectral characteristics based on crossmodal congruence. The context is the audio-to-tactile conversion of short sounds frequently used for user experience improvement across various applications. For each short sound, a single-frequency amplitude-modulated vibration is synthesized so that their intensive and temporal characteristics are very similar. It leaves the vibration frequency, which determines the tactile pitch, as the only variable. Each sound is paired with many vibrations of different frequencies. The congruence between sound and vibration is evaluated for 175 pairs (25 sounds × 7 vibration frequencies). This dataset is employed to estimate a functional relationship from the sound loudness spectrum of sound to the most harmonious vibration frequency. Finally, this sound-to-touch crossmodal pitch mapping function is evaluated using cross-validation. To our knowledge, this is the first attempt to find general rules for spectral matching between sound and touch.

Haptic Communication of Language.

In this review, the development of communication systems and devices that convey language tactually is examined, first from an historical perspective focusing on the communities who use the tactile modality to substitute for impairments in vision and/or hearing. Then, the more recent developments in wearable tactile communication systems for conveying text and speech to those without sensory impairments are reviewed. The performance of tactile display technology developed for these user communities is discussed in the context of the proficiency achieved by skilled users of natural methods of tactile communication. In tracing the history of tactile devices used to convey language, it is evident that technological advances in other domains, such as screen readers and speech synthesizers for the visually impaired and cochlear implants for those with hearing loss, have had a profound impact on the requirements for effective tactile language systems. For some communities, such as the Deafblind, it is essential that the tactile communication platform is bi-directional so that the user can both send and receive language. Devices developed to address such needs have yet to achieve commercial success. Recent research on wearable tactile displays has highlighted the importance of extensive training for learning and retaining languages presented tactually.

Super-resolution wearable electrotactile rendering system.

The human somatosensory system is capable of extracting features with millimeter-scale spatial resolution and submillisecond temporal precision. Current technologies that can render tactile stimuli with such high definition are neither portable nor easily accessible. Here, we present a wearable electrotactile rendering system that elicits tactile stimuli with both high spatial resolution (76 dots/cm2) and rapid refresh rates (4 kHz), because of a previously unexplored current-steering super-resolution stimulation technique. For user safety, we present a high-frequency modulation method to reduce the stimulation voltage to as low as 13 V. The utility of our high spatiotemporal tactile rendering system is highlighted in applications such as braille display, virtual reality shopping, and digital virtual experiences. Furthermore, we integrate our setup with tactile sensors to transmit fine tactile features through thick gloves used by firefighters, allowing tiny objects to be localized based on tactile sensing alone.

Improving Tactile Codes for Increased Speech Communication Rates in a Phonemic-Based Tactile Display.

Previous research has shown evidence of tactile speech acquisition of up to 500 English words presented as tactile phonemic patterns using a 4-by-6 tactor array worn on the forearm. This article describes modifications to some of the tactile codes encoding the 39 English phonemes, and ten additional codes as abbreviated patterns for the ten most frequent phoneme pairs in spoken English. The re-design aimed to reduce the duration of phonemes and phoneme pairs that occur most frequently, with the goal to increase tactile speech transmission rates. Code identification experiments were conducted with ten participants over three weeks using a video game. The average identification rate of the 49 modified codes (39 phonemes plus 10 phoneme pairs) was 83.3% with an average learning time of 6.2 hours. The average identification rate of the 49 codes in a retention test with 7 of the 10 participants after more than 90 days of no exposure to the tactile codes was 75.7%. An analysis using ideal transmission rates showed a 58% increase in transmission rate with the modified tactile codes as compared to the original codes, demonstrating that the improved codes can speed up tactile speech communication.

Incidental Categorization of Vibrotactile Stimuli.

Past research has demonstrated incidental learning of task-irrelevant visual and auditory stimuli. Motivated by the possibility of similar evidence in the tactile domain and potential applications in tactile speech communication systems, we investigated incidental categorization of vibrotactile stimuli through a visuomotor task of shape identification. Two experiments were conducted where participants were exposed to position-based or movement-based vibrotactile stimuli prior to performing a speeded response to one of two targets. The two experiments differed only in the particular sets of such stimuli employed. Unbeknownst to the participants, the vibrotactile stimuli and visual targets were initially correlated perfectly to facilitate the incidental learning of their associations, briefly uncorrelated to check the cost in reaction time, and correlated again to re-establish the initial association. Finally, participants were asked to predict visual targets from novel position-based and movement-based stimuli. The results from both experiments provided evidence of incidental categorization of vibrotactile stimuli. The percent-correct scores and sensitivity indices for the overt categorization of novel stimuli from both experiments were well above chance, indicating generalization of learning. And while both experiments showed an increase in reaction time when the association between vibrotactile stimuli and visual targets was disrupted, this reaction time cost was significant only for the stimuli used in the second experiment. Our finding of incidental categorization in the tactile domain has important implications for the effective acquisition of speech in tactile speech communication systems.

Acquisition of 500 English Words through a TActile Phonemic Sleeve (TAPS).

Recently, a phonemic-based tactile speech communication system was developed with the goal to transmit speech through the skin for people with hearing impairments and those whose auditory and visual channels are overloaded or compromised. The display, called the TActile Phonemic Sleeve (TAPS), consisted of a 4-by-6 tactor array worn on the dorsal and volar surfaces of the forearm. Earlier work showed that people were able to learn the haptic symbols for 39 English phonemes and reach a mean phoneme recognition rate of 86% correct within one to four hours of training. The current research evaluated the acquisition of up to 500 words using TAPS. A total of 51 participants were trained and tested in three studies with increasing number of phonemes and vocabulary sizes. Individual achievements varied, but the results clearly demonstrate the potential of transmitting any English word using TAPS within a reasonable period of learning. Future work will include increasing the speech transmission rate with TAPS by improving the phonemic codes and reducing the inter-phoneme intervals, addressing the reception of words and sentences composed of strings of tactile phonemes, and assessing the performance of TAPS as a speech communication system for people with severe hearing impairments.

Controller design and consonantal contrast coding using a multi-finger tactual display.

This paper presents the design and evaluation of a new controller for a multi-finger tactual display in speech communication. A two-degree-of-freedom controller consisting of a feedback controller and a prefilter and its application in a consonant contrasting experiment are presented. The feedback controller provides stable, fast, and robust response of the fingerpad interface and the prefilter shapes the frequency-response of the closed-loop system to match with the human detection-threshold function. The controller is subsequently used in a speech communication system that extracts spectral features from recorded speech signals and presents them as vibrational-motional waveforms to three digits on a receiver's left hand. Performance from a consonantal contrast test suggests that participants are able to identify tactual cues necessary for discriminating consonants in the initial position of consonant-vowel-consonant (CVC) segments. The average sensitivity indices for contrasting voicing, place, and manner features are 3.5, 2.7, and 3.4, respectively. The results show that the consonantal features can be successfully transmitted by utilizing a broad range of the kinesthetic-cutaneous sensory system. The present study also demonstrates the validity of designing controllers that take into account not only the electromechanical properties of the hardware, but the sensory characteristics of the human user.

Visuo-Haptic Discrimination of Viscoelastic Materials.

In our daily lives, we interact with different types of deformable materials. Regarding their mechanical behavior, some of those materials lie in a range that is between purely elastic and purely viscous. This range of mechanical behavior is described as viscoelasticity. In certain types of haptic interactions, such as assessment of ripeness of fruit, firmness of cheese, and consistency of organ tissue, we rely heavily on our haptic perception of viscoelastic materials. The relationship between the mechanical behavior of viscoelastic materials and our perception of them has been investigated in the field of psychorheology. However, our knowledge on how we perceive viscoelastic materials is still quite limited though some research work has already been done on purely elastic and purely viscous materials. History- and frequency-dependent behavior of viscoelastic materials result in a complex time-dependent response, which requires relatively more sophisticated models to investigate their behavior than those of purely elastic and viscous materials. In this study, we model viscoelasticity using a "springpot" (i.e., fractional-order derivative element) and express its behavior in the frequency domain using two physical parameters-"magnitude" and "phase" of complex stiffness. In the frequency domain, we are able to devise signal detection experiments where we can investigate the perception of viscoelastic materials using the perceptual terms of "firmness" and "bounciness," corresponding to the physical parameters of "magnitude" and "phase." The results of our experiments show that the just-noticeable difference (JND) for bounciness increases linearly with increasing "phase," following Weber's law, while the JND for firmness is surprisingly independent of the level of "phase."

Detection and Identification of Pattern Information on an Electrostatic Friction Display.

An electrostatic friction modulation device based on a tablet computer was used to present pattern stimuli to the fingertip for two tasks: detecting patches of friction and matching a frictional pattern to the visual image that produced it. In the detection task, friction patterns were displayed on zero, one two or three cells in a matrix. Errors, whether misses or false alarms, were few. Duration of target-present trials was a linear function of the number of patterns in the display. The intercept indicated an average of under 1 sec to test a location for the presence of a friction patch. The slope was 1.0 sec per item, representing the time to confirm friction change, verify the location, and report. In contrast to fast and accurate detection of friction modulation, identification of patterns by matching to a visual display was at chance, although the patterns were differentiated by form and scale. Given that the patterns fall within the normal acuity of the fingertip, along with previous evidence that fingertip motion per se does not preclude pattern recognition, it appears that the failure to match tactual patterns to visual images resides in processes inherent in information pickup from friction-modulation displays.

A Phonemic-Based Tactile Display for Speech Communication.

Despite a long history of research, the development of synthetic tactual aids to support the communication of speech has proven to be a difficult task. The current paper describes a new tactile speech device based on the presentation of phonemic-based tactile codes. The device consists of 24 tactors under independent control for stimulation at the forearm. Using properties that include frequency and waveform of stimulation, amplitude, spatial location, and movement characteristics, unique tactile codes were designed for 39 consonant and vowel phonemes of the English language. The strategy for mapping the phonemes to tactile symbols is described, and properties of the individual phonemic codes are provided. Results are reported for an exploratory study of the ability of 10 young adults to identify the tactile symbols. The participants were trained to identify sets of consonants and vowels, before being tested on the full set of 39 tactile codes. The results indicate a mean recognition rate of 86 percent correct within one to four hours of training across participants. Thus, these results support the viability of a phonemic-based approach for conveying speech information through the tactile sense.

Short term memory for tactile stimuli.

Research has shown that unreported information stored in rapidly decaying visual representations may be accessed more accurately using partial report than using full report procedures (e.g., [Sperling, G., 1960. The information available in brief visual presentations. Psychological Monographs, 74, 1-29.]). In the 3 experiments reported here, we investigated whether unreported information regarding the actual number of tactile stimuli presented in parallel across the body surface can be accessed using a partial report procedure. In Experiment 1, participants had to report the total number of stimuli in a tactile display composed of up to 6 stimuli presented across their body (numerosity task), or else to detect whether or not a tactile stimulus had previously been presented in a position indicated by a visual probe given at a variable delay after offset of a tactile display (i.e., partial report). The results showed that participants correctly reported up to 3 stimuli in the numerosity judgment task, but their performance was significantly better than chance when up to 5 stimuli were presented in the partial report task. This result shows that short-lasting tactile representations can be accessed using partial report procedures similar to those used previously in visual studies. Experiment 2 showed that the duration of these representations (or the time available to consciously access them) depends on the number of stimuli presented in the display (the greater the number of stimuli that are presented, the faster their representation decays). Finally, the results of a third experiment showed that the differences in performance between the numerosity judgment and partial report tasks could not be explained solely in terms of any difference in task difficulty.

Tactile and visual distractors induce change blindness for tactile stimuli presented on the fingertips.

Recent studies of change detection have revealed that people are surprisingly poor at detecting changes between two consecutively-presented scenes, when they are separated by a distractor that masks the transients typically associated with change. This failure, known as 'change blindness', has been reported within vision, audition, and touch. In the three experiments reported here, we investigated people's ability to detect the change between two patterns of tactile stimuli presented to their fingertips. The two to-be-compared patterns were presented either consecutively, separated by an empty interval or else by a tactile, visual, or auditory mask. Participants' performance was impaired when an empty interval was inserted between the two consecutively-presented patterns as compared with the consecutive stimulus presentation. Participants' performance was further impaired not only when a tactile mask was introduced between the two to-be-compared displays, but also when a visual mask was used instead. Interestingly, however, the addition of an auditory mask to an empty interval did not have any effect on participants' performance. These results are discussed in relation to the multisensory/amodal nature of spatial attention.

Directing visual attention with spatially informative and spatially noninformative tactile cues.

We investigated the tactile cuing of visual spatial attention using spatially-informative (75% valid) and spatially-noninformative (25% valid) tactile cues. The participants performed a visual change detection task following the presentation of a tactile spatial cue on their back whose location corresponded to one of the four visual quadrants on a computer monitor. The participants were explicitly instructed to use the spatially-informative tactile cues but to ignore the spatially-noninformative cues. In addition to reaction time data, participants' eye-gaze was monitored as a measure of overt visual attention. The results showed that the spatially-informative tactile cues resulted in initial saccades toward the cued visual quadrants, and significantly reduced the visual change detection latencies. When spatially-noninformative tactile cues were used, the participants were largely successful at ignoring them as indicated by a saccade distribution that was independent of the quadrant that was cued, as well as the lack of a significant change in search time as compared to the baseline measure of no tactile cuing. The eye-gaze data revealed that the participants could not always completely ignore the spatially-noninformative tactile cues. Our results suggest that the tactile cuing of visual attention is natural but not automatic when the tactile cue and visual target are not collocated spatially, and that it takes effort to ignore the cues even when they are known to provide no useful information. In addition, our results confirm previous findings that spatially-informative tactile cues are especially effective at directing overt visual attention to locations that are not typically monitored visually, such as the bottom of a computer screen or the rearview mirror in an automobile.

Effect of Cutaneous Feedback on the Perceived Hardness of a Virtual Object.

We investigate the effect of adding cutaneous cues to kinesthetic feedback on the perception of a virtual object's hardness. A cutaneous haptic interface is designed to deliver hardness information to a user's fingertip along with a force-feedback interface, and the corresponding rendering strategy is implemented. Two sets of experiments are conducted to evaluate the proposed approach for hardness perception using one-finger touch and two-finger grasp. Experimental results indicate that the addition of cutaneous feedback can make the virtual surface feel significantly harder than the nominal stiffness delivered by force-feedback alone. In addition, the perceived hardness is significantly affected by the rate of hardness rendered with a cutaneous interface for the nominal stiffness K = 0.3 and 0.5 N/mm. For two-finger grip, the effect of a virtual object's thickness has a significant effect on the perceived hardness measured in stiffness. When the perceived hardness is converted to Young's modulus, the effect of thickness is insignificant.

Crossmodal change blindness between vision and touch.

Change blindness is the name given to people's inability to detect changes introduced between two consecutively-presented scenes when they are separated by a distractor that masks the transients that are typically associated with change. Change blindness has been reported within vision, audition, and touch, but has never before been investigated when successive patterns are presented to different sensory modalities. In the study reported here, we investigated change detection performance when the two to-be-compared stimulus patterns were presented in the same sensory modality (i.e., both visual or both tactile) and when one stimulus pattern was tactile while the other was presented visually or vice versa. The two to-be-compared patterns were presented consecutively, separated by an empty interval, or else separated by a masked interval. In the latter case, the masked interval could either be tactile or visual. The first experiment investigated visual-tactile and tactile-visual change detection performance. The results showed that in the absence of masking, participants detected changes in position accurately, despite the fact that the two to-be-compared displays were presented in different sensory modalities. Furthermore, when a mask was presented between the two to-be-compared displays, crossmodal change blindness was elicited no matter whether the mask was visual or tactile. The results of two further experiments showed that performance was better overall in the unimodal (visual or tactile) conditions than in the crossmodal conditions. These results suggest that certain of the processes underlying change blindness are multisensory in nature. We discuss these findings in relation to recent claims regarding the crossmodal nature of spatial attention.

Haptic interfaces for virtual environments: perceived instability and force constancy in haptic sensing of virtual surfaces.

Haptic interfaces are becoming more commonplace in virtual environment and teleoperation systems. There is a growing need to not only continue to improve hardware platforms and rendering algorithms, but evaluate human performance with haptic interfaces. This review summarizes two recent studies inspired by perception problems in using haptic interfaces to interact with virtual environments. The first study evaluated perceived quality of virtual haptic textures and discovered several types of perceived instability and their sources. We found that the buzzing type of perceived instability was most likely due to the mechanical resonance of the haptic interface hardware, and the aliveness type of perceived instability due to our inability to sense the slight movements of our hands in free space. The second study focused on the motor strategy employed during interaction with a virtual surface via a force-feedback haptic interface. We found that users tended to maintain a constant penetration force into a virtual surface when interacting with the surface. This can result in a reversal in perceived relative surface heights if the taller surface is rendered with a lower stiffness, thereby resulting in an erroneous perception of the virtual environment being rendered. For both studies, possible solutions to improving human perception of virtual and remote objects via hardware and/or software are discussed.

Haptic Perception of Edge Sharpness in Real and Virtual Environments.

We investigate the accuracy with which the haptic sharpness perception of a virtual edge is matched to that of a real edge and the effect of the virtual surface stiffness on the match. The perceived sharpness of virtual edges was estimated in terms of the point of subjective equality (PSE) when participants matched the sharpness of virtual edges to that of real edges with a radius of 0.5, 2.5, and 12.5 mm over a virtual stiffness range of 0.6 to 3.0 N/mm. The perceived sharpness of a real and a virtual edge of the same radius was significantly different under all but one of the experimental conditions and there was a significant effect of virtual surface stiffness on the accuracy of the match. The results suggest that the latter is presumably due to a constant penetration force employed by the participants that influenced the penetration depth and perceived sharpness of virtual edges at different surface stiffness levels. Our findings provide quantitative relations for appropriately offsetting the radii of virtual edges in order to achieve the desired perceived sharpness of virtual edges.

When visual transients impair tactile change detection: a novel case of crossmodal change blindness?

The inability of people to detect changes between consecutively presented visual displays, when separated by a blank screen or distractor, is known as "change blindness". This phenomenon has recently been reported to occur within the auditory and tactile modalities as well. To date, however, only distractors presented within the same sensory modality as the change have been demonstrated to produce change blindness. In the present experiment, we studied whether tactile change blindness might also be elicited by the presentation of a visual mask. Participants made same versus different judgments regarding two successively presented displays composed of two to three vibrotactile stimuli. While change detection performance was near-perfect when the two displays were presented one directly after the other, participants failed to detect many of the changes between the tactile displays when they were separated by an empty temporal interval. Critically, performance deteriorated still further when the presentation of a local (i.e., a mudsplash) or global visual transient coincided with the onset of the second tactile pattern. Analysis of the results using signal detection theory revealed that this crossmodal effect reflected a genuine perceptual impairment.

The failure to detect tactile change: a tactile analogue of visual change blindness.

A large body of empirical research now shows that people are surprisingly poor at detecting significant changes in visually presented scenes. This phenomenon is known as change blindness in vision. A similar phenomenon occurs in audition, but to date no such effect has been documented in touch. In the present study, we explored the ability of people to detect changes introduced between two consecutively presented vibrotactile patterns presented over the body surface. The patterns consisted of two or three vibrotactile stimuli presented for 200 msec. The position of one of the vibrotactile stimuli composing the display was repeatedly changed (alternating between two different positions) on 50% of the trials, but the same pattern was presented repeatedly on the remaining trials. Three conditions were investigated: No interval between the patterns, an empty interval between the patterns, and a masked interval between the patterns. Change detection was near perfect in the no-interval block. Performance deteriorated somewhat in the empty-interval block, but by far the worst change detection performance occurred in the masked-interval block. These results demonstrate that "change blindness" can also affect tactile perception.