A high-performance neuroprosthesis for speech decoding and avatar control.

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participant's pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

Nonhuman primate vestibuloocular reflex responses to prosthetic vestibular stimulation are robust to pulse timing errors caused by temporal discretization.

Electrical stimulation of vestibular afferent neurons to partially restore semicircular canal sensation of head rotation and the stabilizing reflexes that sensation supports has potential to effectively treat individuals disabled by bilateral vestibular hypofunction. Ideally, a vestibular implant system using this approach would be integrated with a cochlear implant, which would provide clinicians with a means to simultaneously treat loss of both vestibular and auditory sensation. Despite obvious similarities, merging these technologies poses several challenges, including stimulus pulse timing errors that arise when a system must implement a pulse frequency modulation-encoding scheme (as is used in vestibular implants to mimic normal vestibular nerve encoding of head movement) within fixed-rate continuous interleaved sampling (CIS) strategies used in cochlear implants. Pulse timing errors caused by temporal discretization inherent to CIS create stair step discontinuities of the vestibular implant's smooth mapping of head velocity to stimulus pulse frequency. In this study, we assayed electrically evoked vestibuloocular reflex responses in two rhesus macaques using both a smooth pulse frequency modulation map and a discretized map corrupted by temporal errors typical of those arising in a combined cochlear-vestibular implant. Responses were measured using three-dimensional scleral coil oculography for prosthetic electrical stimuli representing sinusoidal head velocity waveforms that varied over 50-400°/s and 0.1-5 Hz. Pulse timing errors produced negligible effects on responses across all canals in both animals, indicating that temporal discretization inherent to implementing a pulse frequency modulation-coding scheme within a cochlear implant's CIS fixed pulse timing framework need not sacrifice performance of the combined system's vestibular implant portion. NEW & NOTEWORTHY Merging a vestibular implant system with existing cochlear implant technology can provide clinicians with a means to restore both vestibular and auditory sensation. Pulse timing errors inherent to integration of pulse frequency modulation vestibular stimulation with fixed-rate, continuous interleaved sampling cochlear implant stimulation would discretize the smooth head velocity encoding of a combined device. In this study, we show these pulse timing errors produce negligible effects on electrically evoked vestibulo-ocular reflex responses in two rhesus macaques.

Motor Neuroprostheses.

Neuroprostheses (NPs) are electrical stimulators that activate nerves, either to provide sensory input to the central nervous system (sensory NPs), or to activate muscles (motor NPs: MNPs). The first MNPs were belts with inbuilt batteries and electrodes developed in the 1850s to exercise the abdominal muscles. They became enormously popular among the general public, but as a result of exaggerated therapeutic claims they were soon discredited by the medical community. In the 1950s, MNPs reemerged for the serious purpose of activating paralyzed muscles. Neuromuscular electrical stimulation (NMES), when applied in a preset sequence, is called therapeutic electrical stimulation (TES). NMES timed so that it enhances muscle contraction in intended voluntary movements is called functional electrical stimulation (FES) or functional neuromuscular stimulation (FNS). It has been 50 years since the first FES device, a foot-drop stimulator, was described and 40 years since the first implantable version was tested in humans. A commercial foot-drop stimulator became available in the 1970s, but for various reasons, it failed to achieve widespread use. With advances in technology, such devices are now more convenient and reliable. Enhancing upper limb function is a more difficult task, but grasp-release stimulators have been shown to provide significant benefits. This chapter deals with the technical aspects of NMES, the therapeutic and functional benefits of TES and FES, delayed-onset and carryover effects attributable to "neuromodulation" and the barriers and opportunities in this rapidly developing field. © 2019 American Physiological Society. Compr Physiol 9:127-148, 2019.

A new psychometric questionnaire for reporting of somatosensory percepts.

OBJECTIVE: There have been remarkable advances over the past decade in neural prostheses to restore lost motor function. However, restoration of somatosensory feedback, which is essential for fine motor control and user acceptance, has lagged behind. With an increasing interest in using electrical stimulation to restore somatosensory sensations within the peripheral (PNS) and central nervous systems (CNS), it is critical to characterize the percepts evoked by electrical stimulation in a standardized manner with a validated psychometric questionnaire. This will allow comparison of results from applications at various nervous system levels in multiple settings. APPROACH: We compiled a summary of published reports of somatosensory percepts that were elicited by electrical stimulation in humans and used these to develop a new psychometric questionnaire. RESULTS: This new questionnaire was able to characterize subjective evoked sensations with good test-retest reliability (Spearman's correlation coefficients ranging 0.716 ⩽ ρ ⩽ 1.000, p ⩽ 0.005) in 13 subjects receiving stimulation through neural implants in both the CNS and PNS. Furthermore, the new questionnaire captured more descriptors (M = 2.65, SD = 0.91) that would have been missed by being categorized as 'other sensations', using a previous questionnaire (M = 1.40, SD = 0.77, t(12) = -10.24, p < 0.001). Lastly, the new questionnaire was able to capture different descriptors within subjects using different patterns of electrical stimulation (Wilk's Lambda = 0.42, F(3, 10) = 4.58, p = 0.029). SIGNIFICANCE: This new somatosensory psychometric questionnaire will aid in establishing consistency and standardization of reporting in future studies of somatosensory neural prostheses.

Reactivation of emergent task-related ensembles during slow-wave sleep after neuroprosthetic learning.

Brain-machine interfaces can allow neural control over assistive devices. They also provide an important platform for studying neural plasticity. Recent studies have suggested that optimal engagement of learning is essential for robust neuroprosthetic control. However, little is known about the neural processes that may consolidate a neuroprosthetic skill. On the basis of the growing body of evidence linking slow-wave activity (SWA) during sleep to consolidation, we examined whether there is 'offline' processing after neuroprosthetic learning. Using a rodent model, we found that, after successful learning, task-related units specifically experienced increased locking and coherency to SWA during sleep. Moreover, spike-spike coherence among these units was substantially enhanced. These changes were not present with poor skill acquisition or after control awake periods, demonstrating the specificity of our observations to learning. Notably, the time spent in SWA predicted the performance gains. Thus, SWA appears to be involved in offline processing after neuroprosthetic learning.

Towards autonomous neuroprosthetic control using Hebbian reinforcement learning.

OBJECTIVE: Our goal was to design an adaptive neuroprosthetic controller that could learn the mapping from neural states to prosthetic actions and automatically adjust adaptation using only a binary evaluative feedback as a measure of desirability/undesirability of performance. APPROACH: Hebbian reinforcement learning (HRL) in a connectionist network was used for the design of the adaptive controller. The method combines the efficiency of supervised learning with the generality of reinforcement learning. The convergence properties of this approach were studied using both closed-loop control simulations and open-loop simulations that used primate neural data from robot-assisted reaching tasks. MAIN RESULTS: The HRL controller was able to perform classification and regression tasks using its episodic and sequential learning modes, respectively. In our experiments, the HRL controller quickly achieved convergence to an effective control policy, followed by robust performance. The controller also automatically stopped adapting the parameters after converging to a satisfactory control policy. Additionally, when the input neural vector was reorganized, the controller resumed adaptation to maintain performance. SIGNIFICANCE: By estimating an evaluative feedback directly from the user, the HRL control algorithm may provide an efficient method for autonomous adaptation of neuroprosthetic systems. This method may enable the user to teach the controller the desired behavior using only a simple feedback signal.

Determination of relevant neuron-neuron connections for neural prosthetics using time-delayed mutual information: tutorial and preliminary results.

BACKGROUND: Identification of functional dependence among neurons is a necessary component in both the rational design of neural prostheses as well as in the characterization of network physiology. The objective of this article is to provide a tutorial for neurosurgeons regarding information theory, specifically time-delayed mutual information, and to compare time-delayed mutual information, an information theoretic quantity based on statistical dependence, with cross-correlation, a commonly used metric for this task in a preliminary analysis of rat hippocampal neurons. METHODS: Spike trains were recorded from rats performing delayed nonmatch-to-sample task using an array of electrodes surgically implanted into the hippocampus of each hemisphere of the brain. In addition, spike train simulations of positively correlated neurons, negatively correlated neurons, and neurons correlated by nonlinear functions were generated. These were evaluated by time-delayed mutual information (MI) and cross-correlation. RESULTS: Application of time-delayed MI to experimental data indicated the optimal bin size for information capture in the CA3-CA1 system was 40 ms, which may provide some insight into the spatiotemporal nature of encoding in the rat hippocampus. On simulated data, time-delayed MI showed peak values at appropriate time lags in positively correlated, negatively correlated, and complexly correlated data. Cross-correlation showed peak and troughs with positively correlated and negatively correlated data, but failed to capture some higher order correlations. CONCLUSIONS: Comparison of time-delayed MI to cross-correlation in identification of functionally dependent neurons indicates that the methods are not equivalent. Time-delayed MI appeared to capture some interactions between CA3-CA1 neurons at physiologically plausible time delays missed by cross-correlation. It should be considered as a method for identification of functional dependence between neurons and may be useful in the development of neural prosthetics.

Adaptive decoding for brain-machine interfaces through Bayesian parameter updates.

Brain-machine interfaces (BMIs) transform the activity of neurons recorded in motor areas of the brain into movements of external actuators. Representation of movements by neuronal populations varies over time, during both voluntary limb movements and movements controlled through BMIs, due to motor learning, neuronal plasticity, and instability in recordings. To ensure accurate BMI performance over long time spans, BMI decoders must adapt to these changes. We propose the Bayesian regression self-training method for updating the parameters of an unscented Kalman filter decoder. This novel paradigm uses the decoder's output to periodically update its neuronal tuning model in a Bayesian linear regression. We use two previously known statistical formulations of Bayesian linear regression: a joint formulation, which allows fast and exact inference, and a factorized formulation, which allows the addition and temporary omission of neurons from updates but requires approximate variational inference. To evaluate these methods, we performed offline reconstructions and closed-loop experiments with rhesus monkeys implanted cortically with microwire electrodes. Offline reconstructions used data recorded in areas M1, S1, PMd, SMA, and PP of three monkeys while they controlled a cursor using a handheld joystick. The Bayesian regression self-training updates significantly improved the accuracy of offline reconstructions compared to the same decoder without updates. We performed 11 sessions of real-time, closed-loop experiments with a monkey implanted in areas M1 and S1. These sessions spanned 29 days. The monkey controlled the cursor using the decoder with and without updates. The updates maintained control accuracy and did not require information about monkey hand movements, assumptions about desired movements, or knowledge of the intended movement goals as training signals. These results indicate that Bayesian regression self-training can maintain BMI control accuracy over long periods, making clinical neuroprosthetics more viable.

A novel command signal for motor neuroprosthetic control.

BACKGROUND: Neuroprostheses can restore functions such as hand grasp or standing to individuals with spinal cord injury (SCI) using electrical stimulation to elicit movements in paralyzed muscles. Implanted neuroprostheses currently use electromyographic (EMG) activity from muscles above the lesion that remain under volitional control as a command input. Systems in development use a networked approach and will allow for restoration of multiple functions but will require additional command signals to control the system, especially in individuals with high-level tetraplegia. OBJECTIVE: The objective of this study was to investigate the feasibility of using muscles innervated below the injury level as command sources for a neuroprosthesis. Recent anatomical and physiological studies have demonstrated the presence of intact axons across the lesion, even in those diagnosed with a clinically complete SCI; hence, EMG activity may be present in muscles with no sign of movement. METHODS: Twelve participants with motor complete SCI were enrolled and EMG was recorded with surface electrodes from 8 muscles below the knee in each leg. RESULTS: Significant activity was evident in 89% of the 192 muscles studied during attempted movements of the foot and lower limb. At least 2 muscles from each participant were identified as potential command signals for a neuroprosthesis based on 2-state, threshold classification. CONCLUSIONS: Results suggest that voluntary activity is present and recordable in below lesion muscles even after clinically complete SCI.

Patient preferences for next generation neural prostheses to restore bladder function.

STUDY DESIGN: A survey administered to 66 individuals with spinal cord injury (SCI) implementing a choice-based conjoint (CBC) analysis. Six attributes with three levels each were defined and used to generate choice sets with treatment scenarios. Patients were asked to choose the scenario that they preferred most. OBJECTIVES: To determine the utility weights for treatment characteristics as well as the overall preference for the three types of neural prostheses (NP), that is Brindley, rhizotomy-free Brindley, and pudendal nerve stimulation. Earlier studies have revealed the importance of restoration of bladder function, but no studies have been performed to determine the importance of NP features. SETTING: Two academic affiliated medical systems' SCI outpatient and inpatient rehabilitation programs, Cleveland, OH. METHODS: CBC analysis followed by multinomial logit modeling. Individual part-worth utilities were estimated using hierarchical Bayes. RESULTS: Side effects had the greatest significant impact on subject choices, followed by the effectiveness on continence and voiding. NPs with rhizotomy-free sacral root stimulation were preferred (45% first choice) over pudendal afferent nerve stimulation (39% second choice) and sacral root stimulation with rhizotomy (53% third choice). Almost 20% did not want to have an NP at all times. CONCLUSION: CBC has shown to be a valuable tool to support design choices. The data showed that persons would prefer a bladder NP with minimally invasive electrodes, which would give them complete bladder function, with no side effects and that can be operated by pushing a button and they do not have to recharge themselves.