Sound-to-Touch Crossmodal Pitch Matching for Short Sounds.

This article explores how to relate sound and touch in terms of their spectral characteristics based on crossmodal congruence. The context is the audio-to-tactile conversion of short sounds frequently used for user experience improvement across various applications. For each short sound, a single-frequency amplitude-modulated vibration is synthesized so that their intensive and temporal characteristics are very similar. It leaves the vibration frequency, which determines the tactile pitch, as the only variable. Each sound is paired with many vibrations of different frequencies. The congruence between sound and vibration is evaluated for 175 pairs (25 sounds × 7 vibration frequencies). This dataset is employed to estimate a functional relationship from the sound loudness spectrum of sound to the most harmonious vibration frequency. Finally, this sound-to-touch crossmodal pitch mapping function is evaluated using cross-validation. To our knowledge, this is the first attempt to find general rules for spectral matching between sound and touch.

Haptic Communication of Language.

In this review, the development of communication systems and devices that convey language tactually is examined, first from an historical perspective focusing on the communities who use the tactile modality to substitute for impairments in vision and/or hearing. Then, the more recent developments in wearable tactile communication systems for conveying text and speech to those without sensory impairments are reviewed. The performance of tactile display technology developed for these user communities is discussed in the context of the proficiency achieved by skilled users of natural methods of tactile communication. In tracing the history of tactile devices used to convey language, it is evident that technological advances in other domains, such as screen readers and speech synthesizers for the visually impaired and cochlear implants for those with hearing loss, have had a profound impact on the requirements for effective tactile language systems. For some communities, such as the Deafblind, it is essential that the tactile communication platform is bi-directional so that the user can both send and receive language. Devices developed to address such needs have yet to achieve commercial success. Recent research on wearable tactile displays has highlighted the importance of extensive training for learning and retaining languages presented tactually.

Super-resolution wearable electrotactile rendering system.

The human somatosensory system is capable of extracting features with millimeter-scale spatial resolution and submillisecond temporal precision. Current technologies that can render tactile stimuli with such high definition are neither portable nor easily accessible. Here, we present a wearable electrotactile rendering system that elicits tactile stimuli with both high spatial resolution (76 dots/cm2) and rapid refresh rates (4 kHz), because of a previously unexplored current-steering super-resolution stimulation technique. For user safety, we present a high-frequency modulation method to reduce the stimulation voltage to as low as 13 V. The utility of our high spatiotemporal tactile rendering system is highlighted in applications such as braille display, virtual reality shopping, and digital virtual experiences. Furthermore, we integrate our setup with tactile sensors to transmit fine tactile features through thick gloves used by firefighters, allowing tiny objects to be localized based on tactile sensing alone.

Improving Tactile Codes for Increased Speech Communication Rates in a Phonemic-Based Tactile Display.

Previous research has shown evidence of tactile speech acquisition of up to 500 English words presented as tactile phonemic patterns using a 4-by-6 tactor array worn on the forearm. This article describes modifications to some of the tactile codes encoding the 39 English phonemes, and ten additional codes as abbreviated patterns for the ten most frequent phoneme pairs in spoken English. The re-design aimed to reduce the duration of phonemes and phoneme pairs that occur most frequently, with the goal to increase tactile speech transmission rates. Code identification experiments were conducted with ten participants over three weeks using a video game. The average identification rate of the 49 modified codes (39 phonemes plus 10 phoneme pairs) was 83.3% with an average learning time of 6.2 hours. The average identification rate of the 49 codes in a retention test with 7 of the 10 participants after more than 90 days of no exposure to the tactile codes was 75.7%. An analysis using ideal transmission rates showed a 58% increase in transmission rate with the modified tactile codes as compared to the original codes, demonstrating that the improved codes can speed up tactile speech communication.

Flexible Electrostatic Transducer Array with Displacement Control for Haptic Sensing and Actuation.

We developed flexible electrostatic transducers with both a single element and a 2×2 array format to actuate at a precise displacement across a range of loads with a control circuitry and algorithm. The transducer, composed of a moving buckled film with an integrated electrode and a rigid electrode, can be used to simultaneously generate and sense displacements. A circuit and computer program were designed to demonstrate displacement control and quantify the sensing precision of the transducer. Specifically, we applied a range of voltage and load conditions to the transducer and array and measured the displacement while under loading through capacitive sensing. The change in capacitance was linear with respect to the area of the electrode in contact and matched theoretical predictions when described as a function of the displacement. The transducer was loaded with weights in the range of 5-27 mN and capacitance-driving voltage graphs were obtained. An 8Hz driving frequency was used to move the transducer, while a 10.8kHz signal was used to sense the capacitance. These were used to build a predictive model to correct for sensed load to maintain a average displacement. It was found that a transducer of dimensions 10mm × 40mm was able to maintain displacement under loads of 5-27mN, while a matrix composed of 10mm × 20mm transducers was able to maintain displacement under loads of 2.5-11mN. In general, the detection thresholds of human skin can range between 5-20mN of force and 2-20um of displacement for frequencies between 1Hz and 250Hz, so these values are in line with what is needed to build a functional haptic wearable device. The present work provides a method to quantitatively measure and control a new type of flexible transducer for a variety of haptic applications.

Acquisition of 500 English Words through a TActile Phonemic Sleeve (TAPS).

Recently, a phonemic-based tactile speech communication system was developed with the goal to transmit speech through the skin for people with hearing impairments and those whose auditory and visual channels are overloaded or compromised. The display, called the TActile Phonemic Sleeve (TAPS), consisted of a 4-by-6 tactor array worn on the dorsal and volar surfaces of the forearm. Earlier work showed that people were able to learn the haptic symbols for 39 English phonemes and reach a mean phoneme recognition rate of 86% correct within one to four hours of training. The current research evaluated the acquisition of up to 500 words using TAPS. A total of 51 participants were trained and tested in three studies with increasing number of phonemes and vocabulary sizes. Individual achievements varied, but the results clearly demonstrate the potential of transmitting any English word using TAPS within a reasonable period of learning. Future work will include increasing the speech transmission rate with TAPS by improving the phonemic codes and reducing the inter-phoneme intervals, addressing the reception of words and sentences composed of strings of tactile phonemes, and assessing the performance of TAPS as a speech communication system for people with severe hearing impairments.

Incidental Categorization of Vibrotactile Stimuli.

Past research has demonstrated incidental learning of task-irrelevant visual and auditory stimuli. Motivated by the possibility of similar evidence in the tactile domain and potential applications in tactile speech communication systems, we investigated incidental categorization of vibrotactile stimuli through a visuomotor task of shape identification. Two experiments were conducted where participants were exposed to position-based or movement-based vibrotactile stimuli prior to performing a speeded response to one of two targets. The two experiments differed only in the particular sets of such stimuli employed. Unbeknownst to the participants, the vibrotactile stimuli and visual targets were initially correlated perfectly to facilitate the incidental learning of their associations, briefly uncorrelated to check the cost in reaction time, and correlated again to re-establish the initial association. Finally, participants were asked to predict visual targets from novel position-based and movement-based stimuli. The results from both experiments provided evidence of incidental categorization of vibrotactile stimuli. The percent-correct scores and sensitivity indices for the overt categorization of novel stimuli from both experiments were well above chance, indicating generalization of learning. And while both experiments showed an increase in reaction time when the association between vibrotactile stimuli and visual targets was disrupted, this reaction time cost was significant only for the stimuli used in the second experiment. Our finding of incidental categorization in the tactile domain has important implications for the effective acquisition of speech in tactile speech communication systems.

Detection and Identification of Pattern Information on an Electrostatic Friction Display.

An electrostatic friction modulation device based on a tablet computer was used to present pattern stimuli to the fingertip for two tasks: detecting patches of friction and matching a frictional pattern to the visual image that produced it. In the detection task, friction patterns were displayed on zero, one two or three cells in a matrix. Errors, whether misses or false alarms, were few. Duration of target-present trials was a linear function of the number of patterns in the display. The intercept indicated an average of under 1 sec to test a location for the presence of a friction patch. The slope was 1.0 sec per item, representing the time to confirm friction change, verify the location, and report. In contrast to fast and accurate detection of friction modulation, identification of patterns by matching to a visual display was at chance, although the patterns were differentiated by form and scale. Given that the patterns fall within the normal acuity of the fingertip, along with previous evidence that fingertip motion per se does not preclude pattern recognition, it appears that the failure to match tactual patterns to visual images resides in processes inherent in information pickup from friction-modulation displays.

Visuo-Haptic Discrimination of Viscoelastic Materials.

In our daily lives, we interact with different types of deformable materials. Regarding their mechanical behavior, some of those materials lie in a range that is between purely elastic and purely viscous. This range of mechanical behavior is described as viscoelasticity. In certain types of haptic interactions, such as assessment of ripeness of fruit, firmness of cheese, and consistency of organ tissue, we rely heavily on our haptic perception of viscoelastic materials. The relationship between the mechanical behavior of viscoelastic materials and our perception of them has been investigated in the field of psychorheology. However, our knowledge on how we perceive viscoelastic materials is still quite limited though some research work has already been done on purely elastic and purely viscous materials. History- and frequency-dependent behavior of viscoelastic materials result in a complex time-dependent response, which requires relatively more sophisticated models to investigate their behavior than those of purely elastic and viscous materials. In this study, we model viscoelasticity using a "springpot" (i.e., fractional-order derivative element) and express its behavior in the frequency domain using two physical parameters-"magnitude" and "phase" of complex stiffness. In the frequency domain, we are able to devise signal detection experiments where we can investigate the perception of viscoelastic materials using the perceptual terms of "firmness" and "bounciness," corresponding to the physical parameters of "magnitude" and "phase." The results of our experiments show that the just-noticeable difference (JND) for bounciness increases linearly with increasing "phase," following Weber's law, while the JND for firmness is surprisingly independent of the level of "phase."

A Phonemic-Based Tactile Display for Speech Communication.

Despite a long history of research, the development of synthetic tactual aids to support the communication of speech has proven to be a difficult task. The current paper describes a new tactile speech device based on the presentation of phonemic-based tactile codes. The device consists of 24 tactors under independent control for stimulation at the forearm. Using properties that include frequency and waveform of stimulation, amplitude, spatial location, and movement characteristics, unique tactile codes were designed for 39 consonant and vowel phonemes of the English language. The strategy for mapping the phonemes to tactile symbols is described, and properties of the individual phonemic codes are provided. Results are reported for an exploratory study of the ability of 10 young adults to identify the tactile symbols. The participants were trained to identify sets of consonants and vowels, before being tested on the full set of 39 tactile codes. The results indicate a mean recognition rate of 86 percent correct within one to four hours of training across participants. Thus, these results support the viability of a phonemic-based approach for conveying speech information through the tactile sense.

Effect of Cutaneous Feedback on the Perceived Hardness of a Virtual Object.

We investigate the effect of adding cutaneous cues to kinesthetic feedback on the perception of a virtual object's hardness. A cutaneous haptic interface is designed to deliver hardness information to a user's fingertip along with a force-feedback interface, and the corresponding rendering strategy is implemented. Two sets of experiments are conducted to evaluate the proposed approach for hardness perception using one-finger touch and two-finger grasp. Experimental results indicate that the addition of cutaneous feedback can make the virtual surface feel significantly harder than the nominal stiffness delivered by force-feedback alone. In addition, the perceived hardness is significantly affected by the rate of hardness rendered with a cutaneous interface for the nominal stiffness K = 0.3 and 0.5 N/mm. For two-finger grip, the effect of a virtual object's thickness has a significant effect on the perceived hardness measured in stiffness. When the perceived hardness is converted to Young's modulus, the effect of thickness is insignificant.

Haptic Perception of Edge Sharpness in Real and Virtual Environments.

We investigate the accuracy with which the haptic sharpness perception of a virtual edge is matched to that of a real edge and the effect of the virtual surface stiffness on the match. The perceived sharpness of virtual edges was estimated in terms of the point of subjective equality (PSE) when participants matched the sharpness of virtual edges to that of real edges with a radius of 0.5, 2.5, and 12.5 mm over a virtual stiffness range of 0.6 to 3.0 N/mm. The perceived sharpness of a real and a virtual edge of the same radius was significantly different under all but one of the experimental conditions and there was a significant effect of virtual surface stiffness on the accuracy of the match. The results suggest that the latter is presumably due to a constant penetration force employed by the participants that influenced the penetration depth and perceived sharpness of virtual edges at different surface stiffness levels. Our findings provide quantitative relations for appropriately offsetting the radii of virtual edges in order to achieve the desired perceived sharpness of virtual edges.

Application of psychophysical techniques to haptic research.

Various psychophysical methods have been used to study human haptic perception, although the selection of a particular method is often based on convention, rather than an analysis of which technique is optimal for the question being addressed. In this review, classical psychophysical techniques used to measure sensory thresholds are described as well as more modern methods such as adaptive procedures and those associated with signal detection theory. Details are provided as to how these techniques should be implemented to measure absolute and difference thresholds and factors that influence subjects' responses are noted. In addition to the methods used to measure sensory thresholds, the techniques available for measuring the perception of suprathreshold stimuli are presented. These scaling methods are reviewed in the context of the various stimulus and response biases that influence how subjects respond to stimuli. The importance of understanding the factors that influence perceptual processing is highlighted throughout the review with reference to experimental studies of haptic perception.

Human detection and discrimination of tactile repeatability, mechanical backlash, and temporal delay in a combined tactile-kinesthetic haptic display system.

Many of the devices used in haptics research are over-engineered for the task and are designed with capabilities that go far beyond human perception levels. Designing devices that more closely match the limits of human perception will make them smaller, less expensive, and more useful. However, many device-centric perception thresholds have yet to be evaluated. To this end, three experiments were conducted, using one degree-of-freedom contact location feedback device in combination with a kinesthetic display, to provide a more explicit set of specifications for similar tactile-kinesthetic haptic devices. The first of these experiments evaluated the ability of humans to repeatedly localize tactile cues across the fingerpad. Subjects could localize cues to within 1.3 mm and showed bias toward the center of the fingerpad. The second experiment evaluated the minimum perceptible difference of backlash at the tactile element. Subjects were able to discriminate device backlash in excess of 0.46 mm on low-curvature models and 0.93 mm on high-curvature models. The last experiment evaluated the minimum perceptible difference of system delay between user action and device reaction. Subjects were able to discriminate delays in excess of 61 ms. The results from these studies can serve as the maximum (i.e., most demanding) device specifications for most tactile-kinesthetic haptic systems.

Visuohaptic discrimination of 3D gross shape.

Human sensitivity to 3D gross shape changes was measured for the visual and haptic sensory channels. Three volume-invariant affine transformations were defined: compressing, shearing and stretching. Participants discriminated a reference 3D object (cube or sphere) from its deformed shape under three experimental conditions: visual only (on a computer monitor), haptic only (through a point-contact force-feedback device) and visuohaptic simulations. The results indicate that vision is more sensitive to gross shape changes than point-based touch, and that vision dominated in the visuohaptic condition. In the haptic alone condition, thresholds were higher for shearing and stretching than for compressing. Thresholds were otherwise similar for the three transformations in the vision only or visuohaptic conditions. These trends were similar for the two shapes tested. A second experiment, conducted under similar conditions but preventing participants from manipulating object orientations, verified that the main conclusion of our research still holds when visual inspection can rely only on a single perspective view of the object. Our earlier studies on 3D visuohaptic watermarking showed that the haptic channel is more sensitive to surface texture and roughness changes than vision. The thresholds from the present and our earlier studies can potentially be used as the upper limits for selecting watermark strengths in order to ensure watermark imperceptibility in a 3D visuohaptic watermarking system.

Optimum Information Transfer Rates for Communication through Haptic and Other Sensory Modalities.

This paper is concerned with investigating the factors that contribute to optimizing information transfer (IT) rate in humans. With an increasing interest in designing complex haptic signals for a wide variety of applications, there is a need for a better understanding of how information can be displayed in an optimal way. Based on the results of several early studies from the 1950s, a general "rule of thumb" has arisen in the literature which suggests that IT rate is dependent primarily on the stimulus delivery rate and is optimized for presentation rates of 2-3 items/s. Thus, the key to maximizing IT rate is to maximize the information in the stimulus set. Recent data obtained with multidimensional tactual signals, however, appear to contradict these conclusions. In particular, these current results suggest that optimal delivery rate varies with stimulus information to yield a constant peak IT rate that depends on the degree of familiarity and training with a particular stimulus set. We discuss factors that may be responsible for the discrepancies in results across studies including procedural differences, training issues, and stimulus-response compatibility. These factors should be taken into account when designing haptic signals to yield optimal IT rates for communication devices.

Spatiotemporal integration in somatosensory perception: effects of sensory saltation on pointing at perceived positions on the body surface.

In the past, sensory saltation phenomena (Geldard and Sherrick, 1972) have been used repeatedly to analyze the spatiotemporal integration capacity of somatosensory and other sensory mechanisms by means of their psychophysical characteristic. The core phenomenon consists in a systematic mislocalization of one tactile stimulus (the attractee) toward another successive tactile stimulus (the attractant) presented at another location, increasing with shorter intervals. In a series of four experiments, sensory saltation characteristics were studied at the forearm and the abdomen. Participants reported the perceived positions of attractees, attractants, and reference stimuli by pointing. In general, saltation characteristics compared well to those reported in previous studies, but we were able to gain several new insights regarding this phenomenon: (a) the attractee-attractant interval did not exclusively affect the perceived attractee position, but also the perceived attractant position; (b) saltation characteristics were very similar at different body sites and orientations, but did show differences suggesting anisotropy (direction-dependency) in the underlying integration processes; (c) sensory saltation could be elicited with stimulation patterns crossing the body midline on the abdomen. In addition to the saltation-specific results, our experiments demonstrate that pointing reports of perceived positions on the body surface generally show pronounced systematic biases compared to veridical positions, moderate intraindividual consistency, and a high degree of inter-individual variability. Finally, we address methodological and terminological controversies concerning the sensory saltation paradigm and discuss its possible neurophysiological basis.

Changes in tactile sensitivity over the time-course of a goal-directed movement.

We report three experiments designed to investigate changes in tactile sensitivity over the time-course of goal-directed movements. A dual-task paradigm involving a speeded movement task and a non-speeded perceptual task was utilized. In the movement task, participants grasped a start computer mouse with their right hand (RH) and, at the go signal, reached for and grasped a goal mouse placed 25 cm in front of it. In the perceptual task, a tactile (standard) pulse was presented to the middle finger of the left hand (LH) which was kept at rest throughout the experiment. A comparison pulse was delivered to the middle finger of the RH. In Experiment 1, this was delivered in the motor preparation period, at the release of the start mouse, during the reaching phase, at the grasp of the goal mouse, or shortly after the grasping action. In Experiment 2, the comparison pulse was delivered in the preparation period, in the early, mid or late execution periods, or in the post-movement period. In Experiment 3, participants only performed the perceptual task. The participants made an intensity comparison regarding the second pulse (i.e., stronger vs. weaker than the first). Significant changes in tactile sensitivity were observed, with decreased thresholds (i.e., better performance) during the motor preparation and post-movement periods and increased thresholds (i.e., poorer performance) during the execution period. These results are discussed in terms of sensory suppression.

Driver reaction time to tactile and auditory rear-end collision warnings while talking on a cell phone.

OBJECTIVE: This study examined the effectiveness of rear-end collision warnings presented in different sensory modalities while drivers were engaged in cell phone conversations in a driving simulator. BACKGROUND: Tactile and auditory collision warnings have been shown to improve braking response time (RT) in rear-end collision situations. However, it is not clear how effective these warnings are when the driver is engaged in attentionally demanding secondary tasks, such as talking on a cell phone. METHOD: Sixteen participants in a driving simulator experienced three collision warning conditions (none, tactile, and auditory) in three conversation conditions (none, simple hands free, complex hands free). Driver RT was captured from warning onset to brake initiation (WON2B). RESULTS: WON2B times for auditory warnings were significantly larger for simple conversations compared with no conversation (+148 ms), whereas there was no significant difference between these conditions for tactile warnings (+53 ms). For complex conversations, WON2B times for both tactile (+146 ms) and auditory warnings (+221 ms) were significantly larger than during no conversation. During complex conversations, tactile warnings produced significantly shorter WON2B times than no warning (-141 ms). CONCLUSION: Tactile warnings are more effective than auditory warnings during both simple and complex conversations. APPLICATION: These results indicate that tactile rear-end collision warnings have the potential to offset some of the driving impairments caused by cell phone conversations.

Controller design and consonantal contrast coding using a multi-finger tactual display.

This paper presents the design and evaluation of a new controller for a multi-finger tactual display in speech communication. A two-degree-of-freedom controller consisting of a feedback controller and a prefilter and its application in a consonant contrasting experiment are presented. The feedback controller provides stable, fast, and robust response of the fingerpad interface and the prefilter shapes the frequency-response of the closed-loop system to match with the human detection-threshold function. The controller is subsequently used in a speech communication system that extracts spectral features from recorded speech signals and presents them as vibrational-motional waveforms to three digits on a receiver's left hand. Performance from a consonantal contrast test suggests that participants are able to identify tactual cues necessary for discriminating consonants in the initial position of consonant-vowel-consonant (CVC) segments. The average sensitivity indices for contrasting voicing, place, and manner features are 3.5, 2.7, and 3.4, respectively. The results show that the consonantal features can be successfully transmitted by utilizing a broad range of the kinesthetic-cutaneous sensory system. The present study also demonstrates the validity of designing controllers that take into account not only the electromechanical properties of the hardware, but the sensory characteristics of the human user.