Deep learning approaches for neural decoding across architectures and recording modalities.

Decoding behavior, perception or cognitive state directly from neural signals is critical for brain-computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

From Prior Information to Saccade Selection: Evolution of Frontal Eye Field Activity during Natural Scene Search.

Prior knowledge about our environment influences our actions. How does this knowledge evolve into a final action plan and how does the brain represent this? Here, we investigated this question in the monkey oculomotor system during self-guided search of natural scenes. In the frontal eye field (FEF), we found a subset of neurons, "Early neurons," that contain information about the upcoming saccade long before it is executed, often before the previous saccade had even ended. Crucially, much of this early information did not relate to the actual saccade that would eventually be selected. Rather, it related to prior information about the probabilities of possible upcoming saccades based on the presaccade fixation location. Nearer to the time of saccade onset, a greater proportion of these neurons' activities related to the saccade selection, although prior information continued to influence activity throughout. A separate subset of FEF neurons, "Late neurons," only represented the final action plan near saccade onset and not prior information. Our results demonstrate how, across the population of FEF neurons, prior information evolves into definitive saccade plans.

A muscle-activity-dependent gain between motor cortex and EMG.

Whether one is delicately placing a contact lens on the surface of the eye or lifting a heavy weight from the floor, the motor system must produce a wide range of forces under different dynamical loads. How does the motor cortex, with neurons that have a limited activity range, function effectively under these widely varying conditions? In this study, we explored the interaction of activity in primary motor cortex (M1) and muscles (electromyograms, EMGs) of two male rhesus monkeys for wrist movements made during three tasks requiring different dynamical loads and forces. Despite traditionally providing adequate predictions in single tasks, in our experiments, a single linear model failed to account for the relation between M1 activity and EMG across conditions. However, a model with a gain parameter that increased with the target force remained accurate across forces and dynamical loads. Surprisingly, this model showed that a greater proportion of EMG changes were explained by the nonlinear gain than the linear mapping from M1. In addition to its theoretical implications, the strength of this nonlinearity has important implications for brain-computer interfaces (BCIs). If BCI decoders are to be used to control movement dynamics (including interaction forces) directly, they will need to be nonlinear and include training data from broad data sets to function effectively across tasks. Our study reinforces the need to investigate neural control of movement across a wide range of conditions to understand its basic characteristics as well as translational implications. NEW & NOTEWORTHY We explored the motor cortex-to-electromyogram (EMG) mapping across a wide range of forces and loading conditions, which we found to be highly nonlinear. A greater proportion of EMG was explained by a nonlinear gain than a linear mapping. This nonlinearity allows motor cortex to control the wide range of forces encountered in the real world. These results unify earlier observations and inform the next-generation brain-computer interfaces that will control movement dynamics and interaction forces.

Role of expected reward in frontal eye field during natural scene search.

When a saccade is expected to result in a reward, both neural activity in oculomotor areas and the saccade itself (e.g., its vigor and latency) are altered (compared with when no reward is expected). As such, it is unclear whether the correlations of neural activity with reward indicate a representation of reward beyond a movement representation; the modulated neural activity may simply represent the differences in motor output due to expected reward. Here, to distinguish between these possibilities, we trained monkeys to perform a natural scene search task while we recorded from the frontal eye field (FEF). Indeed, when reward was expected (i.e., saccades to the target), FEF neurons showed enhanced responses. Moreover, when monkeys accidentally made eye movements to the target, firing rates were lower than when they purposively moved to the target. Thus, neurons were modulated by expected reward rather than simply the presence of the target. We then fit a model that simultaneously included components related to expected reward and saccade parameters. While expected reward led to shorter latency and higher velocity saccades, these behavioral changes could not fully explain the increased FEF firing rates. Thus, FEF neurons appear to encode motivational factors such as reward expectation, above and beyond the kinematic and behavioral consequences of imminent reward.

Feature-based attention and spatial selection in frontal eye fields during natural scene search.

When we search for visual objects, the features of those objects bias our attention across the visual landscape (feature-based attention). The brain uses these top-down cues to select eye movement targets (spatial selection). The frontal eye field (FEF) is a prefrontal brain region implicated in selecting eye movements and is thought to reflect feature-based attention and spatial selection. Here, we study how FEF facilitates attention and selection in complex natural scenes. We ask whether FEF neurons facilitate feature-based attention by representing search-relevant visual features or whether they are primarily involved in selecting eye movement targets in space. We show that search-relevant visual features are weakly predictive of gaze in natural scenes and additionally have no significant influence on FEF activity. Instead, FEF activity appears to primarily correlate with the direction of the upcoming eye movement. Our result demonstrates a concrete need for better models of natural scene search and suggests that FEF activity during natural scene search is explained primarily by spatial selection.

Serotonin affects movement gain control in the spinal cord.

A fundamental challenge for the nervous system is to encode signals spanning many orders of magnitude with neurons of limited bandwidth. To meet this challenge, perceptual systems use gain control. However, whether the motor system uses an analogous mechanism is essentially unknown. Neuromodulators, such as serotonin, are prime candidates for gain control signals during force production. Serotonergic neurons project diffusely to motor pools, and, therefore, force production by one muscle should change the gain of others. Here we present behavioral and pharmaceutical evidence that serotonin modulates the input-output gain of motoneurons in humans. By selectively changing the efficacy of serotonin with drugs, we systematically modulated the amplitude of spinal reflexes. More importantly, force production in different limbs interacts systematically, as predicted by a spinal gain control mechanism. Psychophysics and pharmacology suggest that the motor system adopts gain control mechanisms, and serotonin is a primary driver for their implementation in force production.

Statistical analysis of molecular signal recording.

A molecular device that records time-varying signals would enable new approaches in neuroscience. We have recently proposed such a device, termed a "molecular ticker tape", in which an engineered DNA polymerase (DNAP) writes time-varying signals into DNA in the form of nucleotide misincorporation patterns. Here, we define a theoretical framework quantifying the expected capabilities of molecular ticker tapes as a function of experimental parameters. We present a decoding algorithm for estimating time-dependent input signals, and DNAP kinetic parameters, directly from misincorporation rates as determined by sequencing. We explore the requirements for accurate signal decoding, particularly the constraints on (1) the polymerase biochemical parameters, and (2) the amplitude, temporal resolution, and duration of the time-varying input signals. Our results suggest that molecular recording devices with kinetic properties similar to natural polymerases could be used to perform experiments in which neural activity is compared across several experimental conditions, and that devices engineered by combining favorable biochemical properties from multiple known polymerases could potentially measure faster phenomena such as slow synchronization of neuronal oscillations. Sophisticated engineering of DNAPs is likely required to achieve molecular recording of neuronal activity with single-spike temporal resolution over experimentally relevant timescales.

Identifying Interpretable Latent Factors with Sparse Component Analysis.

In many neural populations, the computationally relevant signals are posited to be a set of 'latent factors' - signals shared across many individual neurons. Understanding the relationship between neural activity and behavior requires the identification of factors that reflect distinct computational roles. Methods for identifying such factors typically require supervision, which can be suboptimal if one is unsure how (or whether) factors can be grouped into distinct, meaningful sets. Here, we introduce Sparse Component Analysis (SCA), an unsupervised method that identifies interpretable latent factors. SCA seeks factors that are sparse in time and occupy orthogonal dimensions. With these simple constraints, SCA facilitates surprisingly clear parcellations of neural activity across a range of behaviors. We applied SCA to motor cortex activity from reaching and cycling monkeys, single-trial imaging data from C. elegans, and activity from a multitask artificial network. SCA consistently identified sets of factors that were useful in describing network computations.

Flexible neural control of motor units.

Voluntary movement requires communication from cortex to the spinal cord, where a dedicated pool of motor units (MUs) activates each muscle. The canonical description of MU function rests upon two foundational tenets. First, cortex cannot control MUs independently but supplies each pool with a common drive. Second, MUs are recruited in a rigid fashion that largely accords with Henneman's size principle. Although this paradigm has considerable empirical support, a direct test requires simultaneous observations of many MUs across diverse force profiles. In this study, we developed an isometric task that allowed stable MU recordings, in a rhesus macaque, even during rapidly changing forces. Patterns of MU activity were surprisingly behavior-dependent and could be accurately described only by assuming multiple drives. Consistent with flexible descending control, microstimulation of neighboring cortical sites recruited different MUs. Furthermore, the cortical population response displayed sufficient degrees of freedom to potentially exert fine-grained control. Thus, MU activity is flexibly controlled to meet task demands, and cortex may contribute to this ability.

Area 2 of primary somatosensory cortex encodes kinematics of the whole arm.

Proprioception, the sense of body position, movement, and associated forces, remains poorly understood, despite its critical role in movement. Most studies of area 2, a proprioceptive area of somatosensory cortex, have simply compared neurons' activities to the movement of the hand through space. Using motion tracking, we sought to elaborate this relationship by characterizing how area 2 activity relates to whole arm movements. We found that a whole-arm model, unlike classic models, successfully predicted how features of neural activity changed as monkeys reached to targets in two workspaces. However, when we then evaluated this whole-arm model across active and passive movements, we found that many neurons did not consistently represent the whole arm over both conditions. These results suggest that 1) neural activity in area 2 includes representation of the whole arm during reaching and 2) many of these neurons represented limb state differently during active and passive movements.

Machine Learning for Neural Decoding.

Despite rapid advances in machine learning tools, the majority of neural decoding approaches still use traditional methods. Modern machine learning tools, which are versatile and easy to use, have the potential to significantly improve decoding performance. This tutorial describes how to effectively apply these algorithms for typical decoding problems. We provide descriptions, best practices, and code for applying common machine learning methods, including neural networks and gradient boosting. We also provide detailed comparisons of the performance of various methods at the task of decoding spiking activity in motor cortex, somatosensory cortex, and hippocampus. Modern methods, particularly neural networks and ensembles, significantly outperform traditional approaches, such as Wiener and Kalman filters. Improving the performance of neural decoding algorithms allows neuroscientists to better understand the information contained in a neural population and can help to advance engineering applications such as brain-machine interfaces. Our code package is available at github.com/kordinglab/neural_decoding.

The roles of supervised machine learning in systems neuroscience.

Over the last several years, the use of machine learning (ML) in neuroscience has been rapidly increasing. Here, we review ML's contributions, both realized and potential, across several areas of systems neuroscience. We describe four primary roles of ML within neuroscience: (1) creating solutions to engineering problems, (2) identifying predictive variables, (3) setting benchmarks for simple models of the brain, and (4) serving itself as a model for the brain. The breadth and ease of its applicability suggests that machine learning should be in the toolbox of most systems neuroscientists.

Population coding of conditional probability distributions in dorsal premotor cortex.

Our bodies and the environment constrain our movements. For example, when our arm is fully outstretched, we cannot extend it further. More generally, the distribution of possible movements is conditioned on the state of our bodies in the environment, which is constantly changing. However, little is known about how the brain represents such distributions, and uses them in movement planning. Here, we record from dorsal premotor cortex (PMd) and primary motor cortex (M1) while monkeys reach to randomly placed targets. The hand's position within the workspace creates probability distributions of possible upcoming targets, which affect movement trajectories and latencies. PMd, but not M1, neurons have increased activity when the monkey's hand position makes it likely the upcoming movement will be in the neurons' preferred directions. Across the population, PMd activity represents probability distributions of individual upcoming reaches, which depend on rapidly changing information about the body's state in the environment.

The Development and Analysis of Integrated Neuroscience Data.

There is a strong emphasis on developing novel neuroscience technologies, in particular on recording from more neurons. There has thus been increasing discussion about how to analyze the resulting big datasets. What has received less attention is that over the last 30 years, papers in neuroscience have progressively integrated more approaches, such as electrophysiology, anatomy, and genetics. As such, there has been little discussion on how to combine and analyze this multimodal data. Here, we describe the growth of multimodal approaches, and discuss the needed analysis advancements to make sense of this data.

Puzzle Imaging: Using Large-Scale Dimensionality Reduction Algorithms for Localization.

Current high-resolution imaging techniques require an intact sample that preserves spatial relationships. We here present a novel approach, "puzzle imaging," that allows imaging a spatially scrambled sample. This technique takes many spatially disordered samples, and then pieces them back together using local properties embedded within the sample. We show that puzzle imaging can efficiently produce high-resolution images using dimensionality reduction algorithms. We demonstrate the theoretical capabilities of puzzle imaging in three biological scenarios, showing that (1) relatively precise 3-dimensional brain imaging is possible; (2) the physical structure of a neural network can often be recovered based only on the neural connectivity matrix; and (3) a chemical map could be reproduced using bacteria with chemosensitive DNA and conjugative transfer. The ability to reconstruct scrambled images promises to enable imaging based on DNA sequencing of homogenized tissue samples.

Modeling the effect of selection history on pop-out visual search.

While attentional effects in visual selection tasks have traditionally been assigned "top-down" or "bottom-up" origins, more recently it has been proposed that there are three major factors affecting visual selection: (1) physical salience, (2) current goals and (3) selection history. Here, we look further into selection history by investigating Priming of Pop-out (POP) and the Distractor Preview Effect (DPE), two inter-trial effects that demonstrate the influence of recent history on visual search performance. Using the Ratcliff diffusion model, we model observed saccadic selections from an oddball search experiment that included a mix of both POP and DPE conditions. We find that the Ratcliff diffusion model can effectively model the manner in which selection history affects current attentional control in visual inter-trial effects. The model evidence shows that bias regarding the current trial's most likely target color is the most critical parameter underlying the effect of selection history. Our results are consistent with the view that the 3-item color-oddball task used for POP and DPE experiments is best understood as an attentional decision making task.

Adoption of a High-Impact Innovation in a Homogeneous Population.

Adoption of innovations, whether new ideas, technologies, or products, is crucially important to knowledge societies. The landmark studies of adoption dealt with innovations having great societal impact (such as antibiotics or hybrid crops) but where determining the utility of the innovation was straightforward (such as fewer side effects or greater yield). Recent large-scale studies of adoption were conducted within heterogeneous populations and focused on products with little societal impact. Here, we focus on a case with great practical significance: adoption by small groups of highly trained individuals of innovations with large societal impact but for which it is impractical to determine the true utility of the innovation. Specifically, we study experimentally the adoption by critical care physicians of a diagnostic assay that complements current protocols for the diagnosis of life-threatening bacterial infections and for which a physician cannot estimate the true accuracy of the assay based on personal experience. We show through computational modeling of the experiment that infection-spreading models-which have been formalized as generalized contagion processes-are not consistent with the experimental data, while a model inspired by opinion models is able to reproduce the empirical data. Our modeling approach enables us to investigate the efficacy of different intervention schemes on the rate and robustness of innovation adoption in the real world. While our study is focused on critical care physicians, our findings have implications for other settings in education, research, and business, where small groups of highly qualified peers make decisions about the adoption of innovations whose utility is difficult if not impossible to gauge.

Spatial information in large-scale neural recordings.

To record from a given neuron, a recording technology must be able to separate the activity of that neuron from the activity of its neighbors. Here, we develop a Fisher information based framework to determine the conditions under which this is feasible for a given technology. This framework combines measurable point spread functions with measurable noise distributions to produce theoretical bounds on the precision with which a recording technology can localize neural activities. If there is sufficient information to uniquely localize neural activities, then a technology will, from an information theoretic perspective, be able to record from these neurons. We (1) describe this framework, and (2) demonstrate its application in model experiments. This method generalizes to many recording devices that resolve objects in space and should be useful in the design of next-generation scalable neural recording systems.

Physical principles for scalable neural recording.

Simultaneously measuring the activities of all neurons in a mammalian brain at millisecond resolution is a challenge beyond the limits of existing techniques in neuroscience. Entirely new approaches may be required, motivating an analysis of the fundamental physical constraints on the problem. We outline the physical principles governing brain activity mapping using optical, electrical, magnetic resonance, and molecular modalities of neural recording. Focusing on the mouse brain, we analyze the scalability of each method, concentrating on the limitations imposed by spatiotemporal resolution, energy dissipation, and volume displacement. Based on this analysis, all existing approaches require orders of magnitude improvement in key parameters. Electrical recording is limited by the low multiplexing capacity of electrodes and their lack of intrinsic spatial resolution, optical methods are constrained by the scattering of visible light in brain tissue, magnetic resonance is hindered by the diffusion and relaxation timescales of water protons, and the implementation of molecular recording is complicated by the stochastic kinetics of enzymes. Understanding the physical limits of brain activity mapping may provide insight into opportunities for novel solutions. For example, unconventional methods for delivering electrodes may enable unprecedented numbers of recording sites, embedded optical devices could allow optical detectors to be placed within a few scattering lengths of the measured neurons, and new classes of molecularly engineered sensors might obviate cumbersome hardware architectures. We also study the physics of powering and communicating with microscale devices embedded in brain tissue and find that, while radio-frequency electromagnetic data transmission suffers from a severe power-bandwidth tradeoff, communication via infrared light or ultrasound may allow high data rates due to the possibility of spatial multiplexing. The use of embedded local recording and wireless data transmission would only be viable, however, given major improvements to the power efficiency of microelectronic devices.