A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages.

Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.

Information flow between motor cortex and striatum reverses during skill learning.

The coordination of neural activity across brain areas during a specific behavior is often interpreted as neural communication involved in controlling the behavior. However, whether information relevant to the behavior is actually transferred between areas is often untested. Here, we used information-theoretic tools to quantify how motor cortex and striatum encode and exchange behaviorally relevant information about specific reach-to-grasp movement features during skill learning in rats. We found a temporal shift in the encoding of behaviorally relevant information during skill learning, as well as a reversal in the primary direction of behaviorally relevant information flow, from cortex-to-striatum during naive movements to striatum-to-cortex during skilled movements. Standard analytical methods that quantify the evolution of overall neural activity during learning-such as changes in neural signal amplitude or the overall exchange of information between areas-failed to capture these behaviorally relevant information dynamics. Using these standard methods, we instead found a consistent coactivation of overall neural signals during movement production and a bidirectional increase in overall information propagation between areas during learning. Our results show that skill learning is achieved through a transformation in how behaviorally relevant information is routed across cortical and subcortical brain areas and that isolating the components of neural activity relevant to and informative about behavior is critical to uncover directional interactions within a coactive and coordinated network.

Emergence of task-related spatiotemporal population dynamics in transplanted neurons.

Loss of nervous system tissue after severe brain injury is a main determinant of poor functional recovery. Cell transplantation is a promising method to restore lost tissue and function, yet it remains unclear if transplanted neurons can demonstrate the population level dynamics important for movement control. Here we present a comprehensive approach for long-term single neuron monitoring and manipulation of transplanted embryonic cortical neurons after cortical injury in adult male mice performing a prehension task. The observed patterns of population activity in the transplanted network strongly resembled that of healthy networks. Specifically, the task-related spatiotemporal activity patterns of transplanted neurons could be represented by latent factors that evolve within a low dimensional manifold. We also demonstrate reliable modulation of the transplanted networks using minimally invasive epidural stimulation. Our approach may allow greater insight into how restoration of cell-type specific network dynamics in vivo can restore motor function.

Terminal organization of the corticospinal projection from the arm/hand region of the rostral primary motor cortex (M1r or old M1) to the cervical enlargement (C5-T1) in rhesus monkey.

High-resolution anterograde tracers and stereology were used to study the terminal organization of the corticospinal projection (CSP) from the rostral portion of the primary motor cortex (M1r) to spinal levels C5-T1. Most of this projection (90%) terminated contralaterally within laminae V-IX, with the densest distribution in lamina VII. Moderate bouton numbers occurred in laminae VI, VIII, and IX with few in lamina V. Within lamina VII, labeling occurred over the distal-related dorsolateral subsectors and proximal-related ventromedial subsectors. Within motoneuron lamina IX, most terminations occurred in the proximal-related dorsomedial quadrant, followed by the distal-related dorsolateral quadrant. Segmentally, the contralateral lamina VII CSP gradually declined from C5-T1 but was consistently distributed at C5-C7 in lamina IX. The ipsilateral CSP ended in axial-related lamina VIII and adjacent ventromedial region of lamina VII. These findings demonstrate the M1r CSP influences distal and proximal/axial-related spinal targets. Thus, the M1r CSP represents a transitional CSP, positioned between the caudal M1 (M1c) CSP, which is 98% contralateral and optimally organized to mediate distal upper extremity movements (Morecraft et al., 2013), and dorsolateral premotor (LPMCd) CSP being 79% contralateral and optimally organized to mediate proximal/axial movements (Morecraft et al., 2019). This distal to proximal CSP gradient corresponds to the clinical deficits accompanying caudal to rostral motor cortex injury. The lamina IX CSP is considered in the light of anatomical and neurophysiological evidence which suggests M1c gives rise to the major proportion of the cortico-motoneuronal (CM) projection, while there is a limited M1r CM projection.

Flexible regulation of representations on a drifting manifold enables long-term stable complex neuroprosthetic control.

The nervous system needs to balance the stability of neural representations with plasticity. It is unclear what is the representational stability of simple actions, particularly those that are well-rehearsed in humans, and how it changes in new contexts. Using an electrocorticography brain-computer interface (BCI), we found that the mesoscale manifold and relative representational distances for a repertoire of simple imagined movements were remarkably stable. Interestingly, however, the manifold's absolute location demonstrated day-to-day drift. Strikingly, representational statistics, especially variance, could be flexibly regulated to increase discernability during BCI control without somatotopic changes. Discernability strengthened with practice and was specific to the BCI, demonstrating remarkable contextual specificity. Accounting for drift, and leveraging the flexibility of representations, allowed neuroprosthetic control of a robotic arm and hand for over 7 months without recalibration. Our study offers insight into how electrocorticography can both track representational statistics across long periods and allow long-term complex neuroprosthetic control.

A high-performance neuroprosthesis for speech decoding and avatar control.

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participant's pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

Emergence of preparatory dynamics in VIP interneurons during motor learning.

To determine what actions to perform in each context, animals must learn how to execute motor programs in response to sensory cues. In rodents, the interface between sensory processing and motor planning occurs in the secondary motor cortex (M2). Here, we investigate dynamics in vasointestinal peptide (VIP) and somatostatin (SST) interneurons in M2 during acquisition of a cue-based, reach-to-grasp (RTG) task in mice. We observe the emergence of preparatory activity consisting of sensory responses and ramping activation in a subset of VIP interneurons during motor learning. We show that preparatory and movement activities in VIP neurons exhibit compartmentalized dynamics, with principal component 1 (PC1) and PC2 reflecting primarily movement and preparatory activity, respectively. In contrast, we observe later and more synchronous activation of SST neurons during the movement epoch with learning. Our results reveal how VIP population dynamics might support sensorimotor learning and compartmentalization of sensory processing and movement execution.

Measuring Arm and Hand Joint Kinematics to Estimate Impairment During a Functional Reach and Grasp Task after Stroke.

BACKGROUND: Current approaches to characterizing deficits in upper limb movements after stroke typically focus either on changes in a functional measure, for example, how well a patient can complete a task, or changes in impairment, for example, isolated measurements of joint range of motion. However, there can be notable dissociations between static measures of impairment versus those of function. OBJECTIVE: We develop a method to measure upper limb joint angles during performance of a functional task and use measurements to characterize joint impairment in the context of a functional task. METHODS: We developed a sensorized glove that can precisely measure select finger, hand, and arm joints while participants complete a functional reach-to-grasp task involving manipulation of a sensorized object. RESULTS: We first characterized the accuracy and precision of the glove's joint angle measurements. We then measured joint angles in neurologically intact participants (n = 4 participants, 8 limbs) to define the expected distribution of joint angle variation during task execution. These distributions were used to normalize finger, hand, and arm joint angles in stroke participants (n = 6) as they performed the task. We present a participant-specific visualization of functional joint angle variance which illustrated that stroke participants with nearly identical clinical scores exhibited unique patterns of joint angle variation. CONCLUSIONS: Overall, measuring individual joint angles in the context of a functional task may inform whether changes in functional scores over recovery or rehabilitation are driven by changes in impairment or the development of compensatory strategies, and provide a quantified path toward personalized rehabilitative therapy.

Cortical-hippocampal coupling during manifold exploration in motor cortex.

Systems consolidation-a process for long-term memory stabilization-has been hypothesized to occur in two stages1-4. Whereas new memories require the hippocampus5-9, they become integrated into cortical networks over time10-12, making them independent of the hippocampus. How hippocampal-cortical dialogue precisely evolves during this and how cortical representations change in concert is unknown. Here, we use a skill learning task13,14 to monitor the dynamics of cross-area coupling during non-rapid eye movement sleep along with changes in primary motor cortex (M1) representational stability. Our results indicate that precise cross-area coupling between hippocampus, prefrontal cortex and M1 can demarcate two distinct stages of processing. We specifically find that each animal demonstrates a sharp increase in prefrontal cortex and M1 sleep slow oscillation coupling with stabilization of performance. This sharp increase then predicts a drop in hippocampal sharp-wave ripple (SWR)-M1 slow oscillation coupling-suggesting feedback to inform hippocampal disengagement and transition to a second stage. Notably, the first stage shows significant increases in hippocampal SWR-M1 slow oscillation coupling in the post-training sleep and is closely associated with rapid learning and variability of the M1 low-dimensional manifold. Strikingly, even after consolidation, inducing new manifold exploration by changing task parameters re-engages hippocampal-M1 coupling. We thus find evidence for dynamic hippocampal-cortical dialogue associated with manifold exploration during learning and adaptation.

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis.

Neuroprostheses have the potential to restore communication to people who cannot speak or type due to paralysis. However, it is unclear if silent attempts to speak can be used to control a communication neuroprosthesis. Here, we translated direct cortical signals in a clinical-trial participant (ClinicalTrials.gov; NCT03698149) with severe limb and vocal-tract paralysis into single letters to spell out full sentences in real time. We used deep-learning and language-modeling techniques to decode letter sequences as the participant attempted to silently spell using code words that represented the 26 English letters (e.g. "alpha" for "a"). We leveraged broad electrode coverage beyond speech-motor cortex to include supplemental control signals from hand cortex and complementary information from low- and high-frequency signal components to improve decoding accuracy. We decoded sentences using words from a 1,152-word vocabulary at a median character error rate of 6.13% and speed of 29.4 characters per minute. In offline simulations, we showed that our approach generalized to large vocabularies containing over 9,000 words (median character error rate of 8.23%). These results illustrate the clinical viability of a silently controlled speech neuroprosthesis to generate sentences from a large vocabulary through a spelling-based approach, complementing previous demonstrations of direct full-word decoding.

Selective modulation of cortical population dynamics during neuroprosthetic skill learning.

Brain-machine interfaces (BMIs) provide a framework for studying how cortical population dynamics evolve over learning in a task in which the mapping between neural activity and behavior is precisely defined. Learning to control a BMI is associated with the emergence of coordinated neural dynamics in populations of neurons whose activity serves as direct input to the BMI decoder (direct subpopulation). While previous work shows differential modification of firing rate modulation in this population relative to a population whose activity was not directly input to the BMI decoder (indirect subpopulation), little is known about how learning-related changes in cortical population dynamics within these groups compare.To investigate this, we monitored both direct and indirect subpopulations as two macaque monkeys learned to control a BMI. We found that while the combined population increased coordinated neural dynamics, this increase in coordination was primarily driven by changes in the direct subpopulation. These findings suggest that motor cortex refines cortical dynamics by increasing neural variance throughout the entire population during learning, with a more pronounced coordination of firing activity in subpopulations that are causally linked to behavior.

Modulation of neural co-firing to enhance network transmission and improve motor function after stroke.

Stroke is a leading cause of disability. While neurotechnology has shown promise for improving upper limb recovery after stroke, efficacy in clinical trials has been variable. Our central thesis is that to improve clinical translation, we need to develop a common neurophysiological framework for understanding how neurotechnology alters network activity. Our perspective discusses principles for how motor networks, both healthy and those recovering from stroke, subserve reach-to-grasp movements. We focus on neural processing at the resolution of single movements, the timescale at which neurotechnologies are applied, and discuss how this activity might drive long-term plasticity. We propose that future studies should focus on cross-area communication and bridging our understanding of timescales ranging from single trials within a session to across multiple sessions. We hope that this perspective establishes a combined path forward for preclinical and clinical research with the goal of more robust clinical translation of neurotechnology.

Transition from predictable to variable motor cortex and striatal ensemble patterning during behavioral exploration.

Animals can capitalize on invariance in the environment by learning and automating highly consistent actions; however, they must also remain flexible and adapt to environmental changes. It remains unclear how primary motor cortex (M1) can drive precise movements, yet also support behavioral exploration when faced with consistent errors. Using a reach-to-grasp task in rats, along with simultaneous electrophysiological monitoring in M1 and dorsolateral striatum (DLS), we find that behavioral exploration to overcome consistent task errors is closely associated with tandem increases in M1 and DLS neural variability; subsequently, consistent ensemble patterning returns with convergence to a new successful strategy. We also show that compared to reliably patterned intracranial microstimulation in M1, variable stimulation patterns result in significantly greater movement variability. Our results thus indicate that motor and striatal areas can flexibly transition between two modes, reliable neural pattern generation for automatic and precise movements versus variable neural patterning for behavioral exploration.

Recovery of consolidation after sleep following stroke-interaction of slow waves, spindles, and GABA.

Sleep is known to promote recovery after stroke. Yet it remains unclear how stroke affects neural processing during sleep. Using an experimental stroke model in rats along with electrophysiological monitoring of neural firing and sleep microarchitecture, here we show that sleep processing is altered by stroke. We find that the precise coupling of spindles to global slow oscillations (SOs), a phenomenon that is known to be important for memory consolidation, is disrupted by a pathological increase in "isolated" local delta waves. The transition from this pathological to a physiological state-with increased spindle coupling to SO-is associated with sustained performance gains during recovery. Interestingly, post-injury sleep could be pushed toward a physiological state via a pharmacological reduction of tonic γ-aminobutyric acid (GABA). Together, our results suggest that sleep processing after stroke is impaired due to an increase in delta waves and that its restoration can be important for recovery.

Compartmentalized dynamics within a common multi-area mesoscale manifold represent a repertoire of human hand movements.

The human hand is unique in the animal kingdom for unparalleled dexterity, ranging from complex prehension to fine finger individuation. How does the brain represent such a diverse repertoire of movements? We evaluated mesoscale neural dynamics across the human "grasp network," using electrocorticography and dimensionality reduction methods, for a repertoire of hand movements. Strikingly, we found that the grasp network represented both finger and grasping movements alike. Specifically, the manifold characterizing the multi-areal neural covariance structure was preserved during all movements across this distributed network. In contrast, latent neural dynamics within this manifold were surprisingly specific to movement type. Aligning latent activity to kinematics further uncovered distinct submanifolds despite similarities in synergistic coupling of joints between movements. We thus find that despite preserved neural covariance at the distributed network level, mesoscale dynamics are compartmentalized into movement-specific submanifolds; this mesoscale organization may allow flexible switching between a repertoire of hand movements.

Timescales of Local and Cross-Area Interactions during Neuroprosthetic Learning.

How does the brain integrate signals with different timescales to drive purposeful actions? Brain-machine interfaces (BMIs) offer a powerful tool to causally test how distributed neural networks achieve specific neural patterns. During neuroprosthetic learning, actuator movements are causally linked to primary motor cortex (M1) neurons, i.e., "direct" neurons that project to the decoder and whose firing is required to successfully perform the task. However, it is unknown how such direct M1 neurons interact with both "indirect" local (in M1 but not part of the decoder) and across area neural populations (e.g., in premotor cortex/M2), all of which are embedded in complex biological recurrent networks. Here, we trained male rats to perform a M1-BMI task and simultaneously recorded the activity of indirect neurons in both M2 and M1. We found that both M2 and M1 indirect neuron populations could be used to predict the activity of the direct neurons (i.e., "BMI-potent activity"). Interestingly, compared with M1 indirect activity, M2 neural activity was correlated with BMI-potent activity across a longer set of time lags, and the timescale of population activity patterns evolved more slowly. M2 units also predicted the activity of both M1 direct and indirect neural populations, suggesting that M2 population dynamics provide a continuous modulatory influence on M1 activity as a whole, rather than a moment-by-moment influence solely on neurons most relevant to a task. Together, our results indicate that longer timescale M2 activity provides modulatory influence over extended time lags on shorter-timescale control signals in M1.SIGNIFICANCE STATEMENT A central question in the study of motor control is whether primary motor cortex (M1) and premotor cortex (M2) interact through task-specific subpopulations of neurons, or whether tasks engage broader correlated networks. Brain-machine interfaces (BMIs) are powerful tools to study cross-area interactions. Here, we performed simultaneous recordings of M1 and M2 in a BMI task using a subpopulation of M1 neurons (direct neurons). We found that activity outside of direct neurons in M1 and M2 was predictive of M1-BMI task activity, and that M2 activity evolved at slower timescales than M1. These findings suggest that M2 provides a continuous modulatory influence on M1 as a whole, supporting a model of interactions through broad correlated networks rather than task-specific neural subpopulations.

Coupling between motor cortex and striatum increases during sleep over long-term skill learning.

The strength of cortical connectivity to the striatum influences the balance between behavioral variability and stability. Learning to consistently produce a skilled action requires plasticity in corticostriatal connectivity associated with repeated training of the action. However, it remains unknown whether such corticostriatal plasticity occurs during training itself or 'offline' during time away from training, such as sleep. Here, we monitor the corticostriatal network throughout long-term skill learning in rats and find that non-rapid-eye-movement (NREM) sleep is a relevant period for corticostriatal plasticity. We first show that the offline activation of striatal NMDA receptors is required for skill learning. We then show that corticostriatal functional connectivity increases offline, coupled to emerging consistent skilled movements, and coupled cross-area neural dynamics. We then identify NREM sleep spindles as uniquely poised to mediate corticostriatal plasticity, through interactions with slow oscillations. Our results provide evidence that sleep shapes cross-area coupling required for skill learning.

Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria.

BACKGROUND: Technology to restore the ability to communicate in paralyzed persons who cannot speak has the potential to improve autonomy and quality of life. An approach that decodes words and sentences directly from the cerebral cortical activity of such patients may represent an advancement over existing methods for assisted communication. METHODS: We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke. Over the course of 48 sessions, we recorded 22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words. We used deep-learning algorithms to create computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to say them. RESULTS: We decoded sentences from the participant's cortical activity in real time at a median rate of 15.2 words per minute, with a median word error rate of 25.6%. In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period. CONCLUSIONS: In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from cortical activity during attempted speech with the use of deep-learning models and a natural-language model. (Funded by Facebook and others; ClinicalTrials.gov number, NCT03698149.).

Coordinated increase of reliable cortical and striatal ensemble activations during recovery after stroke.

Skilled movements rely on a coordinated cortical and subcortical network, but how this network supports motor recovery after stroke is unknown. Previous studies focused on the perilesional cortex (PLC), but precisely how connected subcortical areas reorganize and coordinate with PLC is unclear. The dorsolateral striatum (DLS) is of interest because it receives monosynaptic inputs from motor cortex and is important for learning and generation of fast reliable actions. Using a rat focal stroke model, we perform chronic electrophysiological recordings in motor PLC and DLS during long-term recovery of a dexterous skill. We find that recovery is associated with the simultaneous emergence of reliable movement-related single-trial ensemble spiking in both structures along with increased cross-area alignment of spiking. Our study highlights the importance of consistent neural activity patterns across brain structures during recovery and suggests that modulation of cross-area coordination can be a therapeutic target for enhancing motor function post-stroke.