The science and engineering behind sensitized brain-controlled bionic hands.

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Identifying Distinct Neural Features between the Initial and Corrective Phases of Precise Reaching Using AutoLFADS.

Many initial movements require subsequent corrective movements, but how the motor cortex transitions to make corrections and how similar the encoding is to initial movements is unclear. In our study, we explored how the brain's motor cortex signals both initial and corrective movements during a precision reaching task. We recorded a large population of neurons from two male rhesus macaques across multiple sessions to examine the neural firing rates during not only initial movements but also subsequent corrective movements. AutoLFADS, an autoencoder-based deep-learning model, was applied to provide a clearer picture of neurons' activity on individual corrective movements across sessions. Decoding of reach velocity generalized poorly from initial to corrective submovements. Unlike initial movements, it was challenging to predict the velocity of corrective movements using traditional linear methods in a single, global neural space. We identified several locations in the neural space where corrective submovements originated after the initial reaches, signifying firing rates different than the baseline before initial movements. To improve corrective movement decoding, we demonstrate that a state-dependent decoder incorporating the population firing rates at the initiation of correction improved performance, highlighting the diverse neural features of corrective movements. In summary, we show neural differences between initial and corrective submovements and how the neural activity encodes specific combinations of velocity and position. These findings are inconsistent with assumptions that neural correlations with kinematic features are global and independent, emphasizing that traditional methods often fall short in describing these diverse neural processes for online corrective movements.

A large-scale neural network training framework for generalized estimation of single-trial population dynamics.

Achieving state-of-the-art performance with deep neural population dynamics models requires extensive hyperparameter tuning for each dataset. AutoLFADS is a model-tuning framework that automatically produces high-performing autoencoding models on data from a variety of brain areas and tasks, without behavioral or task information. We demonstrate its broad applicability on several rhesus macaque datasets: from motor cortex during free-paced reaching, somatosensory cortex during reaching with perturbations, and dorsomedial frontal cortex during a cognitive timing task.

Inferring single-trial neural population dynamics using sequential auto-encoders.

Neuroscience is experiencing a revolution in which simultaneous recording of thousands of neurons is revealing population dynamics that are not apparent from single-neuron responses. This structure is typically extracted from data averaged across many trials, but deeper understanding requires studying phenomena detected in single trials, which is challenging due to incomplete sampling of the neural population, trial-to-trial variability, and fluctuations in action potential timing. We introduce latent factor analysis via dynamical systems, a deep learning method to infer latent dynamics from single-trial neural spiking data. When applied to a variety of macaque and human motor cortical datasets, latent factor analysis via dynamical systems accurately predicts observed behavioral variables, extracts precise firing rate estimates of neural dynamics on single trials, infers perturbations to those dynamics that correlate with behavioral choices, and combines data from non-overlapping recording sessions spanning months to improve inference of underlying dynamics.

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces.

In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.

Volitional control of single-electrode high gamma local field potentials by people with paralysis.

Intracortical brain-computer interfaces (BCIs) can enable individuals to control effectors, such as a computer cursor, by directly decoding the user's movement intentions from action potentials and local field potentials (LFPs) recorded within the motor cortex. However, the accuracy and complexity of effector control achieved with such "biomimetic" BCIs will depend on the degree to which the intended movements used to elicit control modulate the neural activity. In particular, channels that do not record distinguishable action potentials and only record LFP modulations may be of limited use for BCI control. In contrast, a biofeedback approach may surpass these limitations by letting the participants generate new control signals and learn strategies that improve the volitional control of signals used for effector control. Here, we show that, by using a biofeedback paradigm, three individuals with tetraplegia achieved volitional control of gamma LFPs (40-400 Hz) recorded by a single microelectrode implanted in the precentral gyrus. Control was improved over a pair of consecutive sessions up to 3 days apart. In all but one session, the channel used to achieve control lacked distinguishable action potentials. Our results indicate that biofeedback LFP-based BCIs may potentially contribute to the neural modulation necessary to obtain reliable and useful control of effectors. NEW & NOTEWORTHY Our study demonstrates that people with tetraplegia can volitionally control individual high-gamma local-field potential (LFP) channels recorded from the motor cortex, and that this control can be improved using biofeedback. Motor cortical LFP signals are thought to be both informative and stable intracortical signals and, thus, of importance for future brain-computer interfaces.

Stable long-term BCI-enabled communication in ALS and locked-in syndrome using LFP signals.

Restoring communication for people with locked-in syndrome remains a challenging clinical problem without a reliable solution. Recent studies have shown that people with paralysis can use brain-computer interfaces (BCIs) based on intracortical spiking activity to efficiently type messages. However, due to neuronal signal instability, most intracortical BCIs have required frequent calibration and continuous assistance of skilled engineers to maintain performance. Here, an individual with locked-in syndrome due to brain stem stroke and an individual with tetraplegia secondary to amyotrophic lateral sclerosis (ALS) used a simple communication BCI based on intracortical local field potentials (LFPs) for 76 and 138 days, respectively, without recalibration and without significant loss of performance. BCI spelling rates of 3.07 and 6.88 correct characters/minute allowed the participants to type messages and write emails. Our results indicate that people with locked-in syndrome could soon use a slow but reliable LFP-based BCI for everyday communication without ongoing intervention from a technician or caregiver. NEW & NOTEWORTHY This study demonstrates, for the first time, stable repeated use of an intracortical brain-computer interface by people with tetraplegia over up to four and a half months. The approach uses local field potentials (LFPs), signals that may be more stable than neuronal action potentials, to decode participants' commands. Throughout the several months of evaluation, the decoder remained unchanged; thus no technical interventions were required to maintain consistent brain-computer interface operation.

Recruitment of PfSET2 by RNA polymerase II to variant antigen encoding loci contributes to antigenic variation in P. falciparum.

Histone modifications are important regulators of gene expression in all eukaryotes. In Plasmodium falciparum, these epigenetic marks regulate expression of genes involved in several aspects of host-parasite interactions, including antigenic variation. While the identities and genomic positions of many histone modifications have now been cataloged, how they are targeted to defined genomic regions remains poorly understood. For example, how variant antigen encoding loci (var) are targeted for deposition of unique histone marks is a mystery that continues to perplex the field. Here we describe the recruitment of an ortholog of the histone modifier SET2 to var genes through direct interactions with the C-terminal domain (CTD) of RNA polymerase II. In higher eukaryotes, SET2 is a histone methyltransferase recruited by RNA pol II during mRNA transcription; however, the ortholog in P. falciparum (PfSET2) has an atypical architecture and its role in regulating transcription is unknown. Here we show that PfSET2 binds to the unphosphorylated form of the CTD, a property inconsistent with its recruitment during mRNA synthesis. Further, we show that H3K36me3, the epigenetic mark deposited by PfSET2, is enriched at both active and silent var gene loci, providing additional evidence that its recruitment is not associated with mRNA production. Over-expression of a dominant negative form of PfSET2 designed to disrupt binding to RNA pol II induced rapid var gene expression switching, confirming both the importance of PfSET2 in var gene regulation and a role for RNA pol II in its recruitment. RNA pol II is known to transcribe non-coding RNAs from both active and silent var genes, providing a possible mechanism by which it could recruit PfSET2 to var loci. This work unifies previous reports of histone modifications, the production of ncRNAs, and the promoter activity of var introns into a mechanism that contributes to antigenic variation by malaria parasites.

Retinal prosthetic strategy with the capacity to restore normal vision.

Retinal prosthetics offer hope for patients with retinal degenerative diseases. There are 20-25 million people worldwide who are blind or facing blindness due to these diseases, and they have few treatment options. Drug therapies are able to help a small fraction of the population, but for the vast majority, their best hope is through prosthetic devices [reviewed in Chader et al. (2009) Prog Brain Res 175:317-332]. Current prosthetics, however, are still very limited in the vision that they provide: for example, they allow for perception of spots of light and high-contrast edges, but not natural images. Efforts to improve prosthetic capabilities have focused largely on increasing the resolution of the device's stimulators (either electrodes or optogenetic transducers). Here, we show that a second factor is also critical: driving the stimulators with the retina's neural code. Using the mouse as a model system, we generated a prosthetic system that incorporates the code. This dramatically increased the system's capabilities--well beyond what can be achieved just by increasing resolution. Furthermore, the results show, using 9,800 optogenetically stimulated ganglion cell responses, that the combined effect of using the code and high-resolution stimulation is able to bring prosthetic capabilities into the realm of normal image representation.

Spinal interneuron population dynamics underlying flexible pattern generation.

The mammalian spinal locomotor network is composed of diverse populations of interneurons that collectively orchestrate and execute a range of locomotor behaviors. Despite the identification of many classes of spinal interneurons constituting the locomotor network, it remains unclear how the network's collective activity computes and modifies locomotor output on a step-by-step basis. To investigate this, we analyzed lumbar interneuron population recordings and multi-muscle electromyography from spinalized cats performing air stepping and used artificial intelligence methods to uncover state space trajectories of spinal interneuron population activity on single step cycles and at millisecond timescales. Our analyses of interneuron population trajectories revealed that traversal of specific state space regions held millisecond-timescale correspondence to the timing adjustments of extensor-flexor alternation. Similarly, we found that small variations in the path of state space trajectories were tightly linked to single-step, microvolt-scale adjustments in the magnitude of muscle output. One sentence summary: Features of spinal interneuron state space trajectories capture variations in the timing and magnitude of muscle activations across individual step cycles, with precision on the scales of milliseconds and microvolts respectively.

Identifying distinct neural features between the initial and corrective phases of precise reaching using AutoLFADS.

Many initial movements require subsequent corrective movements, but how motor cortex transitions to make corrections and how similar the encoding is to initial movements is unclear. In our study, we explored how the brain's motor cortex signals both initial and corrective movements during a precision reaching task. We recorded a large population of neurons from two male rhesus macaques across multiple sessions to examine the neural firing rates during not only initial movements but also subsequent corrective movements. AutoLFADS, an auto-encoder-based deep-learning model, was applied to provide a clearer picture of neurons' activity on individual corrective movements across sessions. Decoding of reach velocity generalized poorly from initial to corrective submovements. Unlike initial movements, it was challenging to predict the velocity of corrective movements using traditional linear methods in a single, global neural space. We identified several locations in the neural space where corrective submovements originated after the initial reaches, signifying firing rates different than the baseline before initial movements. To improve corrective movement decoding, we demonstrate that a state-dependent decoder incorporating the population firing rates at the initiation of correction improved performance, highlighting the diverse neural features of corrective movements. In summary, we show neural differences between initial and corrective submovements and how the neural activity encodes specific combinations of velocity and position. These findings are inconsistent with assumptions that neural correlations with kinematic features are global and independent, emphasizing that traditional methods often fall short in describing these diverse neural processes for online corrective movements. Significance Statement: We analyzed submovement neural population dynamics during precision reaching. Using an auto- encoder-based deep-learning model, AutoLFADS, we examined neural activity on a single-trial basis. Our study shows distinct neural dynamics between initial and corrective submovements. We demonstrate the existence of unique neural features within each submovement class that encode complex combinations of position and reach direction. Our study also highlights the benefit of state-specific decoding strategies, which consider the neural firing rates at the onset of any given submovement, when decoding complex motor tasks such as corrective submovements.

BRAND: a platform for closed-loop experiments with deep network models.

Objective.Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g. Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g. C and C++).Approach.To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termednodes, which communicate with each other in agraphvia streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis, an in-memory database, to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes.Main results.In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1 ms chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 ms of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial (ClinicalTrials.gov Identifier: NCT00912041) performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems.Significance.By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

Expressive architectures enhance interpretability of dynamics-based neural population models.

Artificial neural networks that can recover latent dynamics from recorded neural activity may provide a powerful avenue for identifying and interpreting the dynamical motifs underlying biological computation. Given that neural variance alone does not uniquely determine a latent dynamical system, interpretable architectures should prioritize accurate and low-dimensional latent dynamics. In this work, we evaluated the performance of sequential autoencoders (SAEs) in recovering latent chaotic attractors from simulated neural datasets. We found that SAEs with widely-used recurrent neural network (RNN)-based dynamics were unable to infer accurate firing rates at the true latent state dimensionality, and that larger RNNs relied upon dynamical features not present in the data. On the other hand, SAEs with neural ordinary differential equation (NODE)-based dynamics inferred accurate rates at the true latent state dimensionality, while also recovering latent trajectories and fixed point structure. Ablations reveal that this is mainly because NODEs (1) allow use of higher-capacity multi-layer perceptrons (MLPs) to model the vector field and (2) predict the derivative rather than the next state. Decoupling the capacity of the dynamics model from its latent dimensionality enables NODEs to learn the requisite low-D dynamics where RNN cells fail. Additionally, the fact that the NODE predicts derivatives imposes a useful autoregressive prior on the latent states. The suboptimal interpretability of widely-used RNN-based dynamics may motivate substitution for alternative architectures, such as NODE, that enable learning of accurate dynamics in low-dimensional latent spaces.

Expressive dynamics models with nonlinear injective readouts enable reliable recovery of latent features from neural activity.

The advent of large-scale neural recordings has enabled new approaches that aim to discover the computational mechanisms of neural circuits by understanding the rules that govern how their state evolves over time. While these neural dynamics cannot be directly measured, they can typically be approximated by low-dimensional models in a latent space. How these models represent the mapping from latent space to neural space can affect the interpretability of the latent representation. We show that typical choices for this mapping (e.g., linear or MLP) often lack the property of injectivity, meaning that changes in latent state are not obligated to affect activity in the neural space. During training, non-injective readouts incentivize the invention of dynamics that misrepresent the underlying system and the computation it performs. Combining our injective Flow readout with prior work on interpretable latent dynamics models, we created the Ordinary Differential equations autoencoder with Injective Nonlinear readout (ODIN), which learns to capture latent dynamical systems that are nonlinearly embedded into observed neural activity via an approximately injective nonlinear mapping. We show that ODIN can recover nonlinearly embedded systems from simulated neural activity, even when the nature of the system and embedding are unknown. Additionally, we show that ODIN enables the unsupervised recovery of underlying dynamical features (e.g., fixed points) and embedding geometry. When applied to biological neural recordings, ODIN can reconstruct neural activity with comparable accuracy to previous state-of-the-art methods while using substantially fewer latent dimensions. Overall, ODIN's accuracy in recovering ground-truth latent features and ability to accurately reconstruct neural activity with low dimensionality make it a promising method for distilling interpretable dynamics that can help explain neural computation.

Pseudo-linear Summation explains Neural Geometry of Multi-finger Movements in Human Premotor Cortex.

How does the motor cortex combine simple movements (such as single finger flexion/extension) into complex movements (such hand gestures or playing piano)? Motor cortical activity was recorded using intracortical multi-electrode arrays in two people with tetraplegia as they attempted single, pairwise and higher order finger movements. Neural activity for simultaneous movements was largely aligned with linear summation of corresponding single finger movement activities, with two violations. First, the neural activity was normalized, preventing a large magnitude with an increasing number of moving fingers. Second, the neural tuning direction of weakly represented fingers (e.g. middle) changed significantly as a result of the movement of other fingers. These deviations from linearity resulted in non-linear methods outperforming linear methods for neural decoding. Overall, simultaneous finger movements are thus represented by the combination of individual finger movements by pseudo-linear summation.

BRAND: A platform for closed-loop experiments with deep network models.

Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g., Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g., C and C++). To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termed nodes , which communicate with each other in a graph via streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes. In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1-millisecond chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 milliseconds of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems. By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

Myomatrix arrays for high-definition muscle recording.

Neurons coordinate their activity to produce an astonishing variety of motor behaviors. Our present understanding of motor control has grown rapidly thanks to new methods for recording and analyzing populations of many individual neurons over time. In contrast, current methods for recording the nervous system's actual motor output - the activation of muscle fibers by motor neurons - typically cannot detect the individual electrical events produced by muscle fibers during natural behaviors and scale poorly across species and muscle groups. Here we present a novel class of electrode devices ("Myomatrix arrays") that record muscle activity at unprecedented resolution across muscles and behaviors. High-density, flexible electrode arrays allow for stable recordings from the muscle fibers activated by a single motor neuron, called a "motor unit", during natural behaviors in many species, including mice, rats, primates, songbirds, frogs, and insects. This technology therefore allows the nervous system's motor output to be monitored in unprecedented detail during complex behaviors across species and muscle morphologies. We anticipate that this technology will allow rapid advances in understanding the neural control of behavior and in identifying pathologies of the motor system.

Myomatrix arrays for high-definition muscle recording.

Neurons coordinate their activity to produce an astonishing variety of motor behaviors. Our present understanding of motor control has grown rapidly thanks to new methods for recording and analyzing populations of many individual neurons over time. In contrast, current methods for recording the nervous system's actual motor output - the activation of muscle fibers by motor neurons - typically cannot detect the individual electrical events produced by muscle fibers during natural behaviors and scale poorly across species and muscle groups. Here we present a novel class of electrode devices ('Myomatrix arrays') that record muscle activity at unprecedented resolution across muscles and behaviors. High-density, flexible electrode arrays allow for stable recordings from the muscle fibers activated by a single motor neuron, called a 'motor unit,' during natural behaviors in many species, including mice, rats, primates, songbirds, frogs, and insects. This technology therefore allows the nervous system's motor output to be monitored in unprecedented detail during complex behaviors across species and muscle morphologies. We anticipate that this technology will allow rapid advances in understanding the neural control of behavior and identifying pathologies of the motor system.

A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution.

In many areas of the brain, neural populations act as a coordinated network whose state is tied to behavior on a millisecond timescale. Two-photon (2p) calcium imaging is a powerful tool to probe such network-scale phenomena. However, estimating the network state and dynamics from 2p measurements has proven challenging because of noise, inherent nonlinearities and limitations on temporal resolution. Here we describe Recurrent Autoencoder for Discovering Imaged Calcium Latents (RADICaL), a deep learning method to overcome these limitations at the population level. RADICaL extends methods that exploit dynamics in spiking activity for application to deconvolved calcium signals, whose statistics and temporal dynamics are quite distinct from electrophysiologically recorded spikes. It incorporates a new network training strategy that capitalizes on the timing of 2p sampling to recover network dynamics with high temporal precision. In synthetic tests, RADICaL infers the network state more accurately than previous methods, particularly for high-frequency components. In 2p recordings from sensorimotor areas in mice performing a forelimb reach task, RADICaL infers network state with close correspondence to single-trial variations in behavior and maintains high-quality inference even when neuronal populations are substantially reduced.