The vision of Hsiao on somatosensation.

The goal of this review is to start to consolidate and distill the substantial body of research that comprises the published work of the late Professor Steven S. Hsiao. The studies of Hsiao began by demonstrating the receptive field properties of somatosensory neurons, progressed to describing cortical feature selectivity, and then eventually elevated the field to hopes of tapping into natural neural codes with artificial somatosensory feedback. With ongoing analogies to contemporaneous studies in visual neuroscience, the research results and writings of Hsiao have provided the fields of haptics and somatosensory neurophysiology with the conceptual tools needed to allow profound progress. Specifically, Hsiao suggested that slowly adapting tactile form perception could be restored with cortical microstimulation, rapidly adapting slip reflexes should be relegated to low-level, hard-wired prosthetic components, and Pacinian-corpuscle spatiotemporal population responses could potentially be decoded/encoded to provide information about interactions of hands and hand-held instruments with external objects. Future studies will be guided by these insightful reports from Hsiao.

Patterns of muscle activity for digital coarticulation.

Although piano playing is a highly skilled task, basic features of motor pattern generation may be shared across tasks involving fine movements, such as handling coins, fingering food, or using a touch screen. The scripted and sequential nature of piano playing offered the opportunity to quantify the neuromuscular basis of coarticulation, i.e., the manner in which the muscle activation for one sequential element is altered to facilitate production of the preceding and subsequent elements. Ten pianists were asked to play selected pieces with the right hand at a uniform tempo. Key-press times were recorded along with the electromyographic (EMG) activity from seven channels: thumb flexor and abductor muscles, a flexor for each finger, and the four-finger extensor muscle. For the thumb and index finger, principal components of EMG waveforms revealed highly consistent variations in the shape of the flexor bursts, depending on the type of sequence in which a particular central key press was embedded. For all digits, the duration of the central EMG burst scaled, along with slight variations across subjects in the duration of the interkeystroke intervals. Even within a narrow time frame (about 100 ms) centered on the central EMG burst, the exact balance of EMG amplitudes across multiple muscles depended on the nature of the preceding and subsequent key presses. This fails to support the idea of fixed burst patterns executed in sequential phases and instead provides evidence for neuromuscular coarticulation throughout the time course of a hand movement sequence.

Predictive mechanisms in the control of contour following.

In haptic exploration, when running a fingertip along a surface, the control system may attempt to anticipate upcoming changes in curvature in order to maintain a consistent level of contact force. Such predictive mechanisms are well known in the visual system, but have yet to be studied in the somatosensory system. Thus, the present experiment was designed to reveal human capabilities for different types of haptic prediction. A robot arm with a large 3D workspace was attached to the index fingertip and was programmed to produce virtual surfaces with curvatures that varied within and across trials. With eyes closed, subjects moved the fingertip around elliptical hoops with flattened regions or Limaçon shapes, where the curvature varied continuously. Subjects anticipated the corner of the flattened region rather poorly, but for the Limaçon shapes, they varied finger speed with upcoming curvature according to the two-thirds power law. Furthermore, although the Limaçon shapes were randomly presented in various 3D orientations, modulation of contact force also indicated good anticipation of upcoming changes in curvature. The results demonstrate that it is difficult to haptically anticipate the spatial location of an abrupt change in curvature, but smooth changes in curvature may be facilitated by anticipatory predictions.

Coordination of hand shape.

The neural control of hand movement involves coordination of the sensory, motor, and memory systems. Recent studies have documented the motor coordinates for hand shape, but less is known about the corresponding patterns of somatosensory activity. To initiate this line of investigation, the present study characterized the sense of hand shape by evaluating the influence of differences in the amount of grasping or twisting force, and differences in forearm orientation. Human subjects were asked to use the left hand to report the perceived shape of the right hand. In the first experiment, six commonly grasped items were arranged on the table in front of the subject: bottle, doorknob, egg, notebook, carton, and pan. With eyes closed, subjects used the right hand to lightly touch, forcefully support, or imagine holding each object, while 15 joint angles were measured in each hand with a pair of wired gloves. The forces introduced by supporting or twisting did not influence the perceptual report of hand shape, but for most objects, the report was distorted in a consistent manner by differences in forearm orientation. Subjects appeared to adjust the intrinsic joint angles of the left hand, as well as the left wrist posture, so as to maintain the imagined object in its proper spatial orientation. In a second experiment, this result was largely replicated with unfamiliar objects. Thus, somatosensory and motor information appear to be coordinated in an object-based, spatial-coordinate system, sensitive to orientation relative to gravitational forces, but invariant to grasp forcefulness.

Somatosensory comparison during haptic tracing.

Active sensing involves memory retrieval and updating as well as mechanisms that trigger corrections to the ongoing exploratory movement. The present study examined this process in a task where human subjects moved the index fingertip clockwise around the circumference of a virtual sphere created by a robotic device. The fingertip pressed into the sphere during the movement, and the subjects were to report slight differences in sphere size (or surface curvature), which occurred from trial to trial. During each 2- to 3-s trial, subjects gradually adjusted their speed and pressure according to the current surface curvature, achieving a consistent level of contact force in the last half of the exploration. The results demonstrate that subjects were gradually accumulating haptic information about curvature and, at the same time, gradually changing the motor commands for the movement. When subjects encountered an unexpected transition in curvature (from circular to flat), they reacted by abruptly decreasing contact force at a latency of about 50 ms. This short latency indicates that spinally mediated corrections are engaged during this task. The results support the hypothesis that during haptic exploration, the neural comparison between expected and actual somatosensory feedback takes places at multiple levels, including the spinal cord.

Hand kinematics of piano playing.

Dexterous use of the hand represents a sophisticated sensorimotor function. In behaviors such as playing the piano, it can involve strong temporal and spatial constraints. The purpose of this study was to determine fundamental patterns of covariation of motion across joints and digits of the human hand. Joint motion was recorded while 5 expert pianists played 30 excerpts from musical pieces, which featured ∼50 different tone sequences and fingering. Principal component analysis and cluster analysis using an expectation-maximization algorithm revealed that joint velocities could be categorized into several patterns, which help to simplify the description of the movements of the multiple degrees of freedom of the hand. For the thumb keystroke, two distinct patterns of joint movement covariation emerged and they depended on the spatiotemporal patterns of the task. For example, the thumb-under maneuver was clearly separated into two clusters based on the direction of hand translation along the keyboard. While the pattern of the thumb joint velocities differed between these clusters, the motions at the metacarpo-phalangeal and proximal-phalangeal joints of the four fingers were more consistent. For a keystroke executed with one of the fingers, there were three distinct patterns of joint rotations, across which motion at the striking finger was fairly consistent, but motion of the other fingers was more variable. Furthermore, the amount of movement spillover of the striking finger to the adjacent fingers was small irrespective of the finger used for the keystroke. These findings describe an unparalleled amount of independent motion of the fingers.

What is the biological basis of sensorimotor integration?

This Prospects presents the problems that must be solved by the vertebrate nervous system in the process of sensorimotor integration and motor control. The concepts of efference copy and inverse model are defined, and multiple biological mechanisms are described, including those that form the basis of integration, extrapolation, and comparison/cancellation operations. Open questions for future research include the biological basis of continuous and distributed versus modular control, and somatosensory-motor coordination.

Coarticulation in fluent fingerspelling.

In speech, the phenomenon of coarticulation (differentiation of phoneme production depending on the preceding or following phonemes) suggests an organization of movement sequences that is not strictly serial. In the skeletal motor system, however, evidence for comparable fluency has been lacking. Thus the present study was designed to quantify coarticulation in the hand movement sequences of sign language interpreters engaged in fingerspelling. Records of 17 measured joint angles were subjected to discriminant and correlation analyses to determine to what extent and in what manner the hand shape for a particular letter was influenced by the hand shapes for the preceding or the following letters. Substantial evidence of coarticulation was found, revealing both forward and reverse influences across letters. These influences could be further categorized as assimilation (tending to reduce the differences between sequential hand shapes) or dissimilation (tending to emphasize the differences between sequential hand shapes). The proximal interphalangeal (PIP) joints of the index and middle fingers tended to show dissimilation, whereas at the same time (i.e., during the spelling of the same letters) the joints of the wrist and thumb tended to show assimilation. The index and middle finger PIP joints have been shown previously to be among the most important joints for computer recognition of the 26 letter shapes, and therefore the dissimilation may have served to enhance visual discrimination. The simultaneous occurrence of dissimilation in some joints and assimilation in others demonstrates an unprecedented level of parallel control of individual joint rotations in an essentially serial task.

Patterns of hand motion during grasping and the influence of sensory guidance.

This study was aimed at describing temporal synergies of hand movement and determining the influence of sensory cues on the control of these synergies. Subjects were asked to reach to and grasp various objects under three experimental conditions: (1) memory-guided movements, in which the object was not in view during the movement; (2) virtual object, in which a virtual image of the object was in view but the object was not physically present; and (3) real object, in which the object was in view and physically present. Motion of the arm and of 15 degrees of freedom of the hand was recorded. A principal components analysis was developed to provide a concise description of the spatiotemporal patterns underlying the motion. Vision of the object during the reaching movement had no influence on the kinematics, and the effect of the physical presence of the object became manifest primarily after the fingers had contacted the object. Two principal components accounted for >75% of the variance. For both components, there was a strong positive correlation in the rotations of metacarpophalangeal and proximal interphalangeal joints of the fingers. The first principal component exhibited a pattern of finger extension reversing to flexion, whereas the second principal component became important only in the second half of the reaching movement.

Functional somatotopy in sensorimotor cortex.

In an effort to understand the highly distributed somatotopy of primary motor cortex, this review draws on principles derived from studies of auditory, visual and somatosensory cortical areas. In each case, a behaviorally important feature or function is overlaid in multiple locations on an underlying topographic map of the peripheral sensory surface. Recent studies of hand muscle synergies suggest the types of two-dimensional functional axes that might reasonably be mapped to the two-dimensional surface of the primary motor cortex. However, other research emphasizes that even a functional somatotopy must be extremely flexible.

Choosing a wavelet for single-trial EMG.

A wavelet analysis was developed to measure the timing of multiunit bursts in surface electromyograms (EMGs) from single trials. EMG data were taken from eleven elbow and/or shoulder muscles during reaching movements in six different directions, at a range of speeds. A relatively simple wavelet (db2) was chosen, and the analysis focused on wavelet coefficients at an intermediate scale (D3), where the wavelet length approximately matched the wavelengths present in EMG bursts. Burst times were identified from the peaks of the coefficient traces and were plotted as a function of movement time. Linear regression revealed significant relations in most cases, and thus served to validate the wavelet burst identification. With a few exceptions, burst timing scaled in a manner approximately similar to the scaling of movement time. As shown previously with different analytical methods, both within and across joints, EMG bursts were not confined to distinct 'agonist' and 'antagonist' time frames, but instead showed a variety of phases relative to speed or joint torque.

Time constants in the perception of a change in the direction of motion in humans.

Motion signals are subject to spatio-temporal filtering at early stages of processing. In general, motion can be characterized by two parameters: speed and direction. This study sought to determine the time constants for the filtering of the directional component of the motion signal. In a forced-choice discrimination task, subjects were asked to choose the more abrupt change in direction of a target that moved through two 90 degrees corners. At each corner, direction of motion was low-pass filtered. Subjects were able to reliably perform this task if the filter time constants differed by >20 ms.

Biological constraints simplify the recognition of hand shapes.

This study sought to identify constraints that might lead to a concise system of recognizing fingerspelling hand shapes. Previous studies of grasping suggested that hand shape is controlled using combinations of a small number of neuromuscular synergies, but fingerspelling shapes appear to be more highly individuated and, therefore, might require a larger number of degrees of freedom. Static hand postures of the American Sign Language manual alphabet were recorded by measuring 17 joint angles. Principal components (PCs) analysis was compared to the use of subsets of individual variables (i.e., joint angles) for reduction in degrees of freedom. The first four PCs were similar across subjects. Classification using weightings from these four components was 86.6% accurate, while classification using four individual variables was 88.5% accurate (thumb abduction, as well as flexion at the index and middle finger proximal interphalangeal joints and the ring finger metacarpalphalangeal joint). When chosen for each subject, particular four-variable subsets yielded correct rates above 95%. This superior performance of variable subsets over PC weighting vectors suggests that the reduction in degrees of freedom is due to biomechanical and neuromuscular constraints rather than synergistic control. Thus, in future application to dynamic fingerspelling, reasonable recognition accuracy might be achieved with a significant reduction in both computational and measured degrees of freedom.

Multiple Factors Underlying Haptic Perception of Length and Orientation.

Information about the shape and spatial orientation of an object can be gathered during exploratory hand and arm movements, and then must be synthesized into a unified percept. During the robotically guided exploration of virtual polygons or triangles, the perception of the lengths of two adjoining segments is not always geometrically consistent with the perception of the internal angles between these segments. The present study further characterized this established inconsistency, and also found that subjects' internal angle judgments were influenced by the spatial orientations of the segments, especially the segment that was explored last in the sequence. Internal angle judgments were also biased by the subjects' own active forces, applied in the direction perpendicular to the programmed handle motion. For the last segment, but not for the earlier segments, subjects produced more outward force when they reported larger angles and more inward force when they reported smaller angles. Thus, the haptic synthesis of object shape is influenced by multiple geometric, spatial, and self-produced factors.

Effects of object compliance on three-digit grasping.

Compared with rigid objects, grasping and lifting compliant objects presents additional uncertainties. For any static grasp, forces at the fingertips depend on factors including the locations of the contact points and the contact forces must be coordinated to maintain equilibrium. For compliant objects, the locations and orientations of the contact surfaces change in a force-dependent manner, thus changing the force requirements. Furthermore, every force adjustment then results in additional changes in object shape. This study characterized force and muscle activation patterns in this situation. Fingertip forces were measured as subjects grasped and lifted a 200-g object using their thumb, index, and ring fingers. A spring was sometimes placed under the index and/or ring finger contact surface. Surface electromyographic activity was recorded from ten hand muscles and one proximal arm muscle. The patterns of grip (normal) force and muscle activity were similar across conditions during the load and lift phases, but their amplitude depended on whether the contact surface was compliant. Specifically, the grip force increased smoothly during the load phase of the task under all conditions. To the contrary, the tangential contact (load) force did not increase monotonically when one or more of the contact surfaces were compliant, resulting in a decoupling of the grip and load forces.

Extrapolation of visual motion for manual interception.

A frequent goal of hand movement is to touch a moving target or to make contact with a stationary object that is in motion relative to the moving head and body. This process requires a prediction of the target's motion, since the initial direction of the hand movement anticipates target motion. This experiment was designed to define the visual motion parameters that are incorporated in this prediction of target motion. On seeing a go signal (a change in target color), human subjects slid the right index finger along a touch-sensitive computer monitor to intercept a target moving along an unseen circular or oval path. The analysis focused on the initial direction of the interception movement, which was found to be influenced by the time required to intercept the target and the target's distance from the finger's starting location. Initial direction also depended on the curvature of the target's trajectory in a manner that suggested that this parameter was underestimated during the process of extrapolation. The pattern of smooth pursuit eye movements suggests that the extrapolation of visual target motion was based on local motion cues around the time of the onset of hand movement, rather than on a cognitive synthesis of the target's pattern of motion.

Sensorimotor control of contact force.

Interacting with objects in the environment introduces several new challenges for motor control: the potential for instability, external constraints on possible motions and novel dynamics. Grasping and manipulating objects provide the most elaborate examples of such motor tasks. We review each of these topics and suggest that when sensory feedback is reliable, it is used to adapt the motion to the requirements imposed by the object. When sensory feedback is unreliable, subjects adapt the stiffness of muscles and joints to the task's requirements. One of the simplifications introduced in the control of such movements is a reduction in the effective number of degrees of freedom (sensorimotor axes and muscle synergies) and recent findings and methodological considerations relevant to this topic are discussed.

Factors Influencing Haptic Perception of Complex Shapes.

Exploration of an object by arm movement and somatosensation is a serial process that relies on memories and expectations. The present experiments tested the hypothesis that this process involves breaking the object into component shapes (primitives). This was tested by having human subjects explore shapes composed of semicircular arcs, as well as quarter circles or quarter ellipses. The subjects' perception was reported using a visual display. In the first experiment, in which a series of semicircular arcs was presented, with offsets that differed from trial to trial, performance was consistent with the perception of two (left and right) semicircles. In the second experiment, subjects often failed to detect the quarter circles or quarter ellipses and again behaved as if the object was composed of two (top and bottom) semicircles. The results suggest that the synthesis of haptically sensed shapes is biased toward simple geometric objects and that it can be strongly influenced by expectations.

Multi-digit control of contact forces during rotation of a hand-held object.

Rotation of an object held with three fingers is produced by modulation of force amplitude and direction at one or more contact points. Changes in the moment arm through which these forces act can also contribute to the modulation of the rotational moment. Therefore force amplitude and direction as well as the center of pressure on each contact surface must be carefully coordinated to produce a rotation. Because there is not a single solution, this study sought to describe consistent strategies for simple position-to-position rotations in the pitch, roll, and yaw axes. Force amplitude and direction, and center of pressure on the contact surfaces (and thus the moment arm), were measured as human subjects rotated a 420 g force-transducer instrumented object, grasped with the thumb, index and ring fingers (average movement time: 500 ms). Electromyographic (EMG) activity was recorded from five intrinsic and three extrinsic hand muscles and two wrist muscles. Principal components analysis of force and EMG revealed just two main temporal patterns: the main one followed rotational position and the secondary one had a time course that resembled that of rotational velocity. Although the task could have been accomplished by dynamic modulation of the activity of wrist muscles alone, these two main dynamic EMG patterns were seen in intrinsic hand muscles as well. In contrast to previous reports of shifting in time of the phasic activity bursts of various muscles, in this task, all EMG records were well described by just two temporal patterns, resembling the position and velocity traces.

Intrinsic Hand Muscle Activation for Grasp and Horizontal Transport.

During object manipulation, the hand and arm muscles produce internal forces on the object (grasping forces) and forces that result in external translation or rotation of the object in space (transport forces). The present study tested whether the intrinsic hand muscles are actively involved in transport as well as grasping. Intrinsic hand muscle activity increased with increasing demands for grasp stability, but also showed the timing and directional tuning patterns appropriate for actively transmitting external forces to the object, during the translational acceleration and deceleration of object transport.