The science and engineering behind sensitized brain-controlled bionic hands.

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

The neural mechanisms of manual dexterity.

The hand endows us with unparalleled precision and versatility in our interactions with objects, from mundane activities such as grasping to extraordinary ones such as virtuoso pianism. The complex anatomy of the human hand combined with expansive and specialized neuronal control circuits allows a wide range of precise manual behaviours. To support these behaviours, an exquisite sensory apparatus, spanning the modalities of touch and proprioception, conveys detailed and timely information about our interactions with objects and about the objects themselves. The study of manual dexterity provides a unique lens into the sensorimotor mechanisms that endow the nervous system with the ability to flexibly generate complex behaviour.

Sensory computations in the cuneate nucleus of macaques.

Tactile nerve fibers fall into a few classes that can be readily distinguished based on their spatiotemporal response properties. Because nerve fibers reflect local skin deformations, they individually carry ambiguous signals about object features. In contrast, cortical neurons exhibit heterogeneous response properties that reflect computations applied to convergent input from multiple classes of afferents, which confer to them a selectivity for behaviorally relevant features of objects. The conventional view is that these complex response properties arise within the cortex itself, implying that sensory signals are not processed to any significant extent in the two intervening structures-the cuneate nucleus (CN) and the thalamus. To test this hypothesis, we recorded the responses evoked in the CN to a battery of stimuli that have been extensively used to characterize tactile coding in both the periphery and cortex, including skin indentations, vibrations, random dot patterns, and scanned edges. We found that CN responses are more similar to their cortical counterparts than they are to their inputs: CN neurons receive input from multiple classes of nerve fibers, they have spatially complex receptive fields, and they exhibit selectivity for object features. Contrary to consensus, then, the CN plays a key role in processing tactile information.

Frequency Shapes the Quality of Tactile Percepts Evoked through Electrical Stimulation of the Nerves.

Electrical stimulation of the peripheral nerves of human participants provides a unique opportunity to study the neural determinants of perceptual quality using a causal manipulation. A major challenge in the study of neural coding of touch has been to isolate the role of spike timing-at the scale of milliseconds or tens of milliseconds-in shaping the sensory experience. In the present study, we address this question by systematically varying the pulse frequency (PF) of electrical stimulation pulse trains delivered to the peripheral nerves of seven participants with upper and lower extremity limb loss via chronically implanted neural interfaces. We find that increases in PF lead to systematic increases in perceived frequency, up to ∼50 Hz, at which point further changes in PF have little to no impact on sensory quality. Above this transition frequency, ratings of perceived frequency level off, the ability to discriminate changes in PF is abolished, and verbal descriptors selected to characterize the sensation change abruptly. We conclude that sensation quality is shaped by temporal patterns of neural activation, even if these patterns are imposed on a fixed neural population, but this temporal patterning can only be resolved up to ∼50 Hz. These findings highlight the importance of spike timing in shaping the quality of a sensation and will contribute to the development of encoding strategies for conveying touch feedback through bionic hands and feet.SIGNIFICANCE STATEMENT A major challenge in the study of neural coding of touch has been to understand how temporal patterns in neuronal responses shape the sensory experience. We address this question by varying the pulse frequency (PF) of electrical pulse trains delivered through implanted nerve interfaces in seven amputees. We concomitantly vary pulse width to separate the effect of changing PF on sensory quality from its effect on perceived magnitude. We find that increases in PF lead to increases in perceived frequency, a qualitative dimension, up to ∼50 Hz, beyond which changes in PF have little impact on quality. We conclude that temporal patterning in the neuronal response can shape quality and discuss the implications for restoring touch via neural interfaces.

The frequency of cortical microstimulation shapes artificial touch.

Intracortical microstimulation (ICMS) of the somatosensory cortex evokes vivid tactile sensations and can be used to convey sensory feedback from brain-controlled bionic hands. Changes in ICMS frequency lead to changes in the resulting sensation, but the discriminability of frequency has only been investigated over a narrow range of low frequencies. Furthermore, the sensory correlates of changes in ICMS frequency remain poorly understood. Specifically, it remains to be elucidated whether changes in frequency only modulate sensation magnitude-as do changes in amplitude-or whether they also modulate the quality of the sensation. To fill these gaps, we trained monkeys to discriminate the frequency of ICMS pulse trains over a wide range of frequencies (from 10 to 400 Hz). ICMS amplitude also varied across stimuli to dissociate sensation magnitude from ICMS frequency and ensure that animals could not make frequency judgments based on magnitude. We found that animals could consistently discriminate ICMS frequency up to ∼200 Hz but that the sensory correlates of frequency were highly electrode dependent: On some electrodes, changes in frequency were perceptually distinguishable from changes in amplitude-seemingly giving rise to a change in sensory quality; on others, they were not. We discuss the implications of our findings for neural coding and for brain-controlled bionic hands.

Prehension kinematics in humans and macaques.

Nonhuman primates, especially rhesus macaques, have been a dominant model to study sensorimotor control of the upper limbs. Indeed, human and macaques have similar hands and homologous neural circuits to mediate manual behavior. However, few studies have systematically and quantitatively compared the manual behaviors of the two species. Such comparison is critical for assessing the validity of using the macaque sensorimotor system as a model of its human counterpart. In this study, we systematically compared the prehensile behaviors of humans and rhesus macaques using an identical experimental setup. We found human and macaque prehension kinematics to be generally similar with a few subtle differences. Although the structure of the preshaping hand postures is similar in humans and macaques, human postures are more object-specific and human joints are less intercorrelated. Conversely, monkeys demonstrate more stereotypical preshaping behaviors that are common across all objects and more variability in their postures across repeated presentations of the same object. Despite these subtle differences in manual behavior between humans and monkeys, our results bolster the use of the macaque model to understand the neural mechanisms of manual dexterity in humans.NEW & NOTEWORTHY Macaques have been a dominant animal model to study the neural mechanisms of human dexterity because they exhibit complex manual behavior. We show that the kinematics of prehension, a critical dexterous behavior, are largely similar in humans and macaques. However, human preshaping postures are more object-specific and the movement of human digits is less correlated with each other. The thumb, index, and wrist are major drivers of these interspecies differences.

Proprioceptive sensitivity to imposed finger deflections.

Hand proprioception, the sense of the posture and movements of the wrist and digits, is critical to dexterous manual behavior and to stereognosis, the ability to sense the three-dimensional structure of objects held in the hand. To better understand this sensory modality and its role in hand function, we sought to characterize the acuity with which the postures and movements of finger joints are sensed. To this end, we measured the ability of human subjects to discriminate changes in posture and speed around the three joints of the index finger. In these experiments, we isolated the sensory component by imposing the postures on an otherwise still hand, to complement other studies in which subjects made judgments on actively achieved postures. We found that subjects could reliably sense 12-16% changes in joint angle and 18-32% changes in joint speed. Furthermore, the acuity for posture and speed was comparable across the three joints of the finger. Finally, task performance was unaffected by the presence of a vibratory stimulus, calling into question the role of cutaneous cues in hand proprioception.NEW & NOTEWORTHY Manual dexterity and stereognosis are supported by two exquisite sensory systems, namely touch and proprioception. Here, we measure the sensitivity of hand proprioception and show that humans can sense the posture and movements of the fingers with great accuracy. We also show that application of a skin vibration does not impair sensitivity, suggesting that proprioceptive acuity relies primarily on receptors in the muscles (and possibly tendons) rather than the skin.

Feeling fooled: Texture contaminates the neural code for tactile speed.

Motion is an essential component of everyday tactile experience: most manual interactions involve relative movement between the skin and objects. Much of the research on the neural basis of tactile motion perception has focused on how direction is encoded, but less is known about how speed is. Perceived speed has been shown to be dependent on surface texture, but previous studies used only coarse textures, which span a restricted range of tangible spatial scales and provide a limited window into tactile coding. To fill this gap, we measured the ability of human observers to report the speed of natural textures-which span the range of tactile experience and engage all the known mechanisms of texture coding-scanned across the skin. In parallel experiments, we recorded the responses of single units in the nerve and in the somatosensory cortex of primates to the same textures scanned at different speeds. We found that the perception of speed is heavily influenced by texture: some textures are systematically perceived as moving faster than are others, and some textures provide a more informative signal about speed than do others. Similarly, the responses of neurons in the nerve and in cortex are strongly dependent on texture. In the nerve, although all fibers exhibit speed-dependent responses, the responses of Pacinian corpuscle-associated (PC) fibers are most strongly modulated by speed and can best account for human judgments. In cortex, approximately half of the neurons exhibit speed-dependent responses, and this subpopulation receives strong input from PC fibers. However, speed judgments seem to reflect an integration of speed-dependent and speed-independent responses such that the latter help to partially compensate for the strong texture dependence of the former.

High-dimensional representation of texture in somatosensory cortex of primates.

In the somatosensory nerves, the tactile perception of texture is driven by spatial and temporal patterns of activation distributed across three populations of afferents. These disparate streams of information must then be integrated centrally to achieve a unified percept of texture. To investigate the representation of texture in somatosensory cortex, we scanned a wide range of natural textures across the fingertips of rhesus macaques and recorded the responses evoked in Brodmann's areas 3b, 1, and 2. We found that texture identity is reliably encoded in the idiosyncratic responses of populations of cortical neurons, giving rise to a high-dimensional representation of texture. Cortical neurons fall along a continuum in their sensitivity to fine vs. coarse texture, and neurons at the extrema of this continuum seem to receive their major input from different afferent populations. Finally, we show that cortical responses can account for several aspects of texture perception in humans.

Encoding of limb state by single neurons in the cuneate nucleus of awake monkeys.

The cuneate nucleus (CN) is among the first sites along the neuraxis where proprioceptive signals can be integrated, transformed, and modulated. The objective of the study was to characterize the proprioceptive representations in CN. To this end, we recorded from single CN neurons in three monkeys during active reaching and passive limb perturbation. We found that many neurons exhibited responses that were tuned approximately sinusoidally to limb movement direction, as has been found for other sensorimotor neurons. The distribution of their preferred directions (PDs) was highly nonuniform and resembled that of muscle spindles within individual muscles, suggesting that CN neurons typically receive inputs from only a single muscle. We also found that the responses of proprioceptive CN neurons tended to be modestly amplified during active reaching movements compared to passive limb perturbations, in contrast to cutaneous CN neurons whose responses were not systematically different in the active and passive conditions. Somatosensory signals thus seem to be subject to a "spotlighting" of relevant sensory information rather than uniform suppression as has been suggested previously.NEW & NOTEWORTHY The cuneate nucleus (CN) is the somatosensory gateway into the brain, and only recently has it been possible to record these signals from an awake animal. We recorded single CN neurons in monkeys. Proprioceptive CN neurons appear to receive input from very few muscles, and their sensitivity to movement changes reliably during reaching relative to passive arm perturbations. Sensitivity is generally increased, but not exclusively so, as though CN "spotlights" critical proprioceptive information during reaching.

Information about contact force and surface texture is mixed in the firing rates of cutaneous afferent neurons.

Cutaneous mechanoreceptors in our hands gather information about the objects we handle. Tactile fibers encode mixed information about contact events and object properties. Neural coding in tactile afferents is typically studied by varying a single aspect of tactile stimuli, avoiding the confounds of real-world haptic interactions. We instead record responses of small populations of dorsal root ganglia (DRG) neurons to variable tactile stimuli and find that neurons primarily respond to force, though some texture information can be detected. Tactile nerve fibers convey information about many features of haptic interactions, including the force and speed of contact, as well as the texture and shape of the objects being handled. How we perceive these object features is relatively unaffected by the forces and movements we use when interacting with the object. Because signals related to contact events and object properties are mixed in the responses of tactile fibers, our ability to disentangle these different components of our tactile experience implies that they are demultiplexed as they propagate along the neuraxis. To understand how texture and contact mechanics are encoded together by tactile fibers, we studied the activity of multiple neurons recorded simultaneously in the cervical DRG of two anesthetized rhesus monkeys while textured surfaces were applied to the glabrous skin of the fingers and palm using a handheld probe. A transducer at the tip of the textured probe measured contact forces as tactile stimuli were applied at different locations on the finger-pads and palm. We examined how a sample population of DRG neurons encode force and texture and found that firing rates of individual neurons are modulated by both force and texture. In particular, slowly adapting (SA) neurons were more responsive to force than texture, and rapidly adapting (RA) neurons were more responsive to texture than force. Although force could be decoded accurately throughout the entire contact interval, texture signals were most salient during onset and offset phases of the contact interval.NEW & NOTEWORTHY Cutaneous mechanoreceptors in our hands gather information about the objects we handle. Tactile fibers encode mixed information about contact events and object properties. Neural coding in tactile afferents is typically studied by varying a single aspect of tactile stimuli, avoiding the confounds of real-world haptic interactions. We instead record responses of small populations of DRG neurons to variable tactile stimuli and find that neurons primarily respond to force, though some texture information can be detected.

Emergence of an Invariant Representation of Texture in Primate Somatosensory Cortex.

A major function of sensory processing is to achieve neural representations of objects that are stable across changes in context and perspective. Small changes in exploratory behavior can lead to large changes in signals at the sensory periphery, thus resulting in ambiguous neural representations of objects. Overcoming this ambiguity is a hallmark of human object recognition across sensory modalities. Here, we investigate how the perception of tactile texture remains stable across exploratory movements of the hand, including changes in scanning speed, despite the concomitant changes in afferent responses. To this end, we scanned a wide range of everyday textures across the fingertips of rhesus macaques at multiple speeds and recorded the responses evoked in tactile nerve fibers and somatosensory cortical neurons (from Brodmann areas 3b, 1, and 2). We found that individual cortical neurons exhibit a wider range of speed-sensitivities than do nerve fibers. The resulting representations of speed and texture in cortex are more independent than are their counterparts in the nerve and account for speed-invariant perception of texture. We demonstrate that this separation of speed and texture information is a natural consequence of previously described cortical computations.

Neural Coding of Contact Events in Somatosensory Cortex.

Manual interactions with objects require precise and rapid feedback about contact events. These tactile signals are integrated with motor plans throughout the neuraxis to achieve dexterous object manipulation. To better understand the role of somatosensory cortex in interactions with objects, we measured, using chronically implanted arrays of electrodes, the responses of populations of somatosensory neurons to skin indentations designed to simulate the initiation, maintenance, and termination of contact with an object. First, we find that the responses of somatosensory neurons to contact onset and offset dwarf their responses to maintenance of contact. Second, we show that these responses rapidly and reliably encode features of the simulated contact events-their timing, location, and strength-and can account for the animals' performance in an amplitude discrimination task. Third, we demonstrate that the spatiotemporal dynamics of the population response in cortex mirror those of the population response in the nerves. We conclude that the responses of populations of somatosensory neurons are well suited to encode contact transients and are consistent with a role of somatosensory cortex in signaling transitions between task subgoals.

Simulating tactile signals from the whole hand with millisecond precision.

When we grasp and manipulate an object, populations of tactile nerve fibers become activated and convey information about the shape, size, and texture of the object and its motion across the skin. The response properties of tactile fibers have been extensively characterized in single-unit recordings, yielding important insights into how individual fibers encode tactile information. A recurring finding in this extensive body of work is that stimulus information is distributed over many fibers. However, our understanding of population-level representations remains primitive. To fill this gap, we have developed a model to simulate the responses of all tactile fibers innervating the glabrous skin of the hand to any spatiotemporal stimulus applied to the skin. The model first reconstructs the stresses experienced by mechanoreceptors when the skin is deformed and then simulates the spiking response that would be produced in the nerve fiber innervating that receptor. By simulating skin deformations across the palmar surface of the hand and tiling it with receptors at their known densities, we reconstruct the responses of entire populations of nerve fibers. We show that the simulated responses closely match their measured counterparts, down to the precise timing of the evoked spikes, across a wide variety of experimental conditions sampled from the literature. We then conduct three virtual experiments to illustrate how the simulation can provide powerful insights into population coding in touch. Finally, we discuss how the model provides a means to establish naturalistic artificial touch in bionic hands.

Rapid geometric feature signaling in the simulated spiking activity of a complete population of tactile nerve fibers.

Tactile feature extraction is essential to guide the dexterous manipulation of objects. The longstanding theory is that geometric features at each location of contact between hand and object are extracted from the spatial layout of the response of populations of tactile nerve fibers. However, recent evidence suggests that some features (e.g., edge orientation) are extracted very rapidly (<200 ms), casting doubt that this information relies on a spatial code, which ostensibly requires integrating responses over time. An alternative hypothesis is that orientation is conveyed in precise temporal spiking patterns. Here we simulate, using a recently developed and validated model, the responses of the two relevant subpopulations of tactile fibers from the entire human fingertip (~800 afferents) to edges indented into the skin. We show that edge orientation can be quickly (<50 ms) and accurately (<3°) decoded from the spatial pattern of activation across the afferent population, starting with the very first spike. Next, we implement a biomimetic decoder of edge orientation, consisting of a bank of oriented Gabor filters, designed to mimic the documented responses of cortical neurons. We find that the biomimetic approach leads to orientation decoding performance that approaches the limit set by optimal decoders and is actually more robust to changes in other stimulus features. Finally, we show that orientation signals, measured from single units in the somatosensory cortex of nonhuman primates (2 macaque monkeys, 1 female), follow a time course consistent with that of their counterparts in the nerve. We conclude that a spatial code is fast and accurate enough to support object manipulation. NEW & NOTEWORTHY The dexterous manipulation of objects relies on the rapid and accurate extraction of the objects' geometric features by the sense of touch. Here we simulate the responses of all the nerve fibers that innervate the fingertip when an edge is indented into the skin and characterize the time course over which signals about its orientation evolve in this neural population. We show that orientation can be rapidly and accurately decoded from the spatial pattern of afferent activation using spatial filters that mimic the response properties of neurons in cortical somatosensory neurons along a time course consistent with that observed in cortex. We conclude that the classical model of tactile feature extraction is rapid and accurate enough to support object manipulation.

Restoring sensorimotor function through intracortical interfaces: progress and looming challenges.

The loss of a limb or paralysis resulting from spinal cord injury has devastating consequences on quality of life. One approach to restoring lost sensory and motor abilities in amputees and patients with tetraplegia is to supply them with implants that provide a direct interface with the CNS. Such brain-machine interfaces might enable a patient to exert voluntary control over a prosthetic or robotic limb or over the electrically induced contractions of paralysed muscles. A parallel interface could convey sensory information about the consequences of these movements back to the patient. Recent developments in the algorithms that decode motor intention from neuronal activity and in approaches to convey sensory feedback by electrically stimulating neurons, using biomimetic and adaptation-based approaches, have shown the promise of invasive interfaces with sensorimotor cortices, although substantial challenges remain.

Seeing and Feeling Motion: Canonical Computations in Vision and Touch.

While the different sensory modalities are sensitive to different stimulus energies, they are often charged with extracting analogous information about the environment. Neural systems may thus have evolved to implement similar algorithms across modalities to extract behaviorally relevant stimulus information, leading to the notion of a canonical computation. In both vision and touch, information about motion is extracted from a spatiotemporal pattern of activation across a sensory sheet (in the retina and in the skin, respectively), a process that has been extensively studied in both modalities. In this essay, we examine the processing of motion information as it ascends the primate visual and somatosensory neuraxes and conclude that similar computations are implemented in the two sensory systems.

Behavioral assessment of sensitivity to intracortical microstimulation of primate somatosensory cortex.

Intracortical microstimulation (ICMS) is a powerful tool to investigate the functional role of neural circuits and may provide a means to restore sensation for patients for whom peripheral stimulation is not an option. In a series of psychophysical experiments with nonhuman primates, we investigate how stimulation parameters affect behavioral sensitivity to ICMS. Specifically, we deliver ICMS to primary somatosensory cortex through chronically implanted electrode arrays across a wide range of stimulation regimes. First, we investigate how the detectability of ICMS depends on stimulation parameters, including pulse width, frequency, amplitude, and pulse train duration. Then, we characterize the degree to which ICMS pulse trains that differ in amplitude lead to discriminable percepts across the range of perceptible and safe amplitudes. We also investigate how discriminability of pulse amplitude is modulated by other stimulation parameters-namely, frequency and duration. Perceptual judgments obtained across these various conditions will inform the design of stimulation regimes for neuroscience and neuroengineering applications.

Speed invariance of tactile texture perception.

The nervous system achieves stable perceptual representations of objects despite large variations in the activity patterns of sensory receptors. Here, we explore perceptual constancy in the sense of touch. Specifically, we investigate the invariance of tactile texture perception across changes in scanning speed. Texture signals in the nerve have been shown to be highly dependent on speed: temporal spiking patterns in nerve fibers that encode fine textural features contract or dilate systematically with increases or decreases in scanning speed, respectively, resulting in concomitant changes in response rate. Nevertheless, texture perception has been shown, albeit with restricted stimulus sets and limited perceptual assays, to be independent of scanning speed. Indeed, previous studies investigated the effect of scanning speed on perceived roughness, only one aspect of texture, often with impoverished stimuli, namely gratings and embossed dot patterns. To fill this gap, we probe the perceptual constancy of a wide range of textures using two different paradigms: one that probes texture perception along well-established sensory dimensions independently and one that probes texture perception as a whole. We find that texture perception is highly stable across scanning speeds, irrespective of the texture or the perceptual assay. Any speed-related effects are dwarfed by differences in percepts evoked by different textures. This remarkable speed invariance of texture perception stands in stark contrast to the strong dependence of the texture responses of nerve fibers on scanning speed. Our results imply neural mechanisms that compensate for scanning speed to achieve stable representations of surface texture.NEW & NOTEWORTHY Our brain forms stable representations of objects regardless of viewpoint, a phenomenon known as invariance that has been described in several sensory modalities. Here, we explore invariance in the sense of touch and show that the tactile perception of texture does not depend on scanning speed. This perceptual constancy implies neural mechanisms that extract information about texture from the response of nerve fibers such that the resulting neural representation is stable across speeds.

The neural code for tactile roughness in the somatosensory nerves.

Roughness is the most salient perceptual dimension of surface texture but has no well-defined physical basis. We seek to determine the neural determinants of tactile roughness in the somatosensory nerves. Specifically, we record the patterns of activation evoked in tactile nerve fibers of anesthetized Rhesus macaques to a large and diverse set of natural textures and assess what aspect of these patterns of activation can account for psychophysical judgments of roughness, obtained from human observers. We show that perceived roughness is determined by the variation in the population response, weighted by fiber type. That is, a surface will feel rough to the extent that the activity varies across nerve fibers and varies across time within nerve fibers. We show that this variation-based neural code can account not only for magnitude estimates of roughness but also for roughness discrimination performance.NEW & NOTEWORTHY Our sense of touch endows us with an exquisite sensitivity to the microstructure of surfaces, the most salient aspect of which is roughness. We analyze the responses evoked in tactile fibers of monkeys by natural textures and compare them to judgments of roughness obtained for the same textures from human observers. We then describe how texture signals from three populations of nerve fibers are integrated to culminate in a percept of roughness.