Few-shot Algorithms for Consistent Neural Decoding (FALCON) Benchmark.

Intracortical brain-computer interfaces (iBCIs) can restore movement and communication abilities to individuals with paralysis by decoding their intended behavior from neural activity recorded with an implanted device. While this activity yields high-performance decoding over short timescales, neural data are often nonstationary, which can lead to decoder failure if not accounted for. To maintain performance, users must frequently recalibrate decoders, which requires the arduous collection of new neural and behavioral data. Aiming to reduce this burden, several approaches have been developed that either limit recalibration data requirements (few-shot approaches) or eliminate explicit recalibration entirely (zero-shot approaches). However, progress is limited by a lack of standardized datasets and comparison metrics, causing methods to be compared in an ad hoc manner. Here we introduce the FALCON benchmark suite (Few-shot Algorithms for COnsistent Neural decoding) to standardize evaluation of iBCI robustness. FALCON curates five datasets of neural and behavioral data that span movement and communication tasks to focus on behaviors of interest to modern-day iBCIs. Each dataset includes calibration data, optional few-shot recalibration data, and private evaluation data. We implement a flexible evaluation platform which only requires user-submitted code to return behavioral predictions on unseen data. We also seed the benchmark by applying baseline methods spanning several classes of possible approaches. FALCON aims to provide rigorous selection criteria for robust iBCI decoders, easing their translation to real-world devices. https://snel-repo.github.io/falcon/.

Neural population dynamics in songbird RA and HVC during learned motor-vocal behavior.

Complex, learned motor behaviors involve the coordination of large-scale neural activity across multiple brain regions, but our understanding of the population-level dynamics within different regions tied to the same behavior remains limited. Here, we investigate the neural population dynamics underlying learned vocal production in awake-singing songbirds. We use Neuropixels probes to record the simultaneous extracellular activity of populations of neurons in two regions of the vocal motor pathway. In line with observations made in non-human primates during limb-based motor tasks, we show that the population-level activity in both the premotor nucleus HVC and the motor nucleus RA is organized on low-dimensional neural manifolds upon which coordinated neural activity is well described by temporally structured trajectories during singing behavior. Both the HVC and RA latent trajectories provide relevant information to predict vocal sequence transitions between song syllables. However, the dynamics of these latent trajectories differ between regions. Our state-space models suggest a unique and continuous-over-time correspondence between the latent space of RA and vocal output, whereas the corresponding relationship for HVC exhibits a higher degree of neural variability. We then demonstrate that comparable high-fidelity reconstruction of continuous vocal outputs can be achieved from HVC and RA neural latents and spiking activity. Unlike those that use spiking activity, however, decoding models using neural latents generalize to novel sub-populations in each region, consistent with the existence of preserved manifolds that confine vocal-motor activity in HVC and RA.

Consistent spectro-spatial features of human ECoG successfully decode naturalistic behavioral states.

Objective: Understanding the neural correlates of naturalistic behavior is critical for extending and confirming the results obtained from trial-based experiments and designing generalizable brain-computer interfaces that can operate outside laboratory environments. In this study, we aimed to pinpoint consistent spectro-spatial features of neural activity in humans that can discriminate between naturalistic behavioral states. Approach: We analyzed data from five participants using electrocorticography (ECoG) with broad spatial coverage. Spontaneous and naturalistic behaviors such as "Talking" and "Watching TV" were labeled from manually annotated videos. Linear discriminant analysis (LDA) was used to classify the two behavioral states. The parameters learned from the LDA were then used to determine whether the neural signatures driving classification performance are consistent across the participants. Main results: Spectro-spatial feature values were consistently discriminative between the two labeled behavioral states across participants. Mainly, θ, α, and low and high γ in the postcentral gyrus, precentral gyrus, and temporal lobe showed significant classification performance and feature consistency across participants. Subject-specific performance exceeded 70%. Combining neural activity from multiple cortical regions generally does not improve decoding performance, suggesting that information regarding the behavioral state is non-additive as a function of the cortical region. Significance: To the best of our knowledge, this is the first attempt to identify specific spectro-spatial neural correlates that consistently decode naturalistic and active behavioral states. The aim of this work is to serve as an initial starting point for developing brain-computer interfaces that can be generalized in a realistic setting and to further our understanding of the neural correlates of naturalistic behavior in humans.

High-density transparent graphene arrays for predicting cellular calcium activity at depth from surface potential recordings.

Optically transparent neural microelectrodes have facilitated simultaneous electrophysiological recordings from the brain surface with the optical imaging and stimulation of neural activity. A remaining challenge is to scale down the electrode dimensions to the single-cell size and increase the density to record neural activity with high spatial resolution across large areas to capture nonlinear neural dynamics. Here we developed transparent graphene microelectrodes with ultrasmall openings and a large, transparent recording area without any gold extensions in the field of view with high-density microelectrode arrays up to 256 channels. We used platinum nanoparticles to overcome the quantum capacitance limit of graphene and to scale down the microelectrode diameter to 20 µm. An interlayer-doped double-layer graphene was introduced to prevent open-circuit failures. We conducted multimodal experiments, combining the recordings of cortical potentials of microelectrode arrays with two-photon calcium imaging of the mouse visual cortex. Our results revealed that visually evoked responses are spatially localized for high-frequency bands, particularly for the multiunit activity band. The multiunit activity power was found to be correlated with cellular calcium activity. Leveraging this, we employed dimensionality reduction techniques and neural networks to demonstrate that single-cell and average calcium activities can be decoded from surface potentials recorded by high-density transparent graphene arrays.

Spatiotemporal dynamics of human high gamma discriminate naturalistic behavioral states.

In analyzing the neural correlates of naturalistic and unstructured behaviors, features of neural activity that are ignored in a trial-based experimental paradigm can be more fully studied and investigated. Here, we analyze neural activity from two patients using electrocorticography (ECoG) and stereo-electroencephalography (sEEG) recordings, and reveal that multiple neural signal characteristics exist that discriminate between unstructured and naturalistic behavioral states such as "engaging in dialogue" and "using electronics". Using the high gamma amplitude as an estimate of neuronal firing rate, we demonstrate that behavioral states in a naturalistic setting are discriminable based on long-term mean shifts, variance shifts, and differences in the specific neural activity's covariance structure. Both the rapid and slow changes in high gamma band activity separate unstructured behavioral states. We also use Gaussian process factor analysis (GPFA) to show the existence of salient spatiotemporal features with variable smoothness in time. Further, we demonstrate that both temporally smooth and stochastic spatiotemporal activity can be used to differentiate unstructured behavioral states. This is the first attempt to elucidate how different neural signal features contain information about behavioral states collected outside the conventional experimental paradigm.

Unsupervised Channel Compression Methods in Motor Prostheses Design.

The development of high performance brain machine interfaces (BMIs) requires scaling recording channel count to enable simultaneous recording from large populations of neurons. Unfortunately, proposed implantable neural interfaces have power requirements that scale linearly with channel count. To facilitate the design of interfaces with reduced power requirements, we propose and evaluate an unsupervised-learning-based compressed sensing strategy. This strategy suggests novel neural interface architectures which compress neural data by methodically combining channels of spiking activity. We develop an entropy-based compression strategy that models the population of neurons as being generated from a lower dimensional set of latent variables and aims to minimize the loss of information in the latent variables due to compression. We evaluate compressed features by inferring the latent variables from these features and measuring the accuracy with which the activity of held out neurons and arm movements can be estimated. We apply these methods to different cortical regions (PMd and M1) and compare the proposed compression methods to a random projections strategy often employed for compressed sensing and to a supervised regression based channel dropping strategy traditionally applied in BMI applications.

Affective response to volitional input perturbations in obstacle avoidance and target tracking games.

We present the use of two game-like tasks, Catnip and Dinorun, to explore affective responses to volitional control perturbations. We analyze behavioral and physiological measures with the self-assessment manikin (SAM), pupillometry, and electroencephalography (EEG) responses to provide intratrial emotional state as well as inter-trial correlates with selfreported survey responses. We find that subject gameplay characteristics significantly correlate with valence and dominance scores for both games, and that perturbations to the games produce a measurable decrease in response scores for Dinorun. During perturbation events, pupillometry analysis reveals considerable SAM-agnostic dilation, with stronger responses in more rigid trialized event structures. Furthermore, analyses of neural activity from central and parietal regions demonstrate significant measurable evoked responses to perturbed events across the majority of subjects for both games. By introducing perturbations, this set of experiments and analyses inform and enable further studies of affective responses to the loss of volitional control during engaging, game-like tasks.

High γ Activity in Cortex and Hippocampus Is Correlated with Autonomic Tone during Sleep.

Studies in animals have demonstrated a strong relationship between cortical and hippocampal activity, and autonomic tone. However, the extent, distribution, and nature of this relationship have not been investigated with intracranial recordings in humans during sleep. Cortical and hippocampal population neuronal firing was estimated from high γ band activity (HG) from 70 to 110 Hz in local field potentials (LFPs) recorded from 15 subjects (nine females) during nonrapid eye movement (NREM) sleep. Autonomic tone was estimated from heart rate variability (HRV). HG and HRV were significantly correlated in the hippocampus and multiple cortical sites in NREM stages N1-N3. The average correlation between HG and HRV could be positive or negative across patients given anatomic location and sleep stage and was most profound in lateral temporal lobe in N3, suggestive of greater cortical activity associated with sympathetic tone. Patient-wide correlation was related to δ band activity (1-4 Hz), which is known to be correlated with high γ activity during sleep. The percentage of statistically correlated channels was weaker in N1 and N2 as compared with N3, and was strongest in regions that have previously been associated with autonomic processes, such as anterior hippocampus and insula. The anatomic distribution of HRV-HG correlations during sleep did not reproduce those usually observed with positron emission tomography (PET) or functional magnetic resonance imaging (fMRI) during waking. This study aims to characterize the relationship between autonomic tone and neuronal firing rate during sleep and further studies are needed to investigate finer temporal resolutions, denser coverages, and different frequency bands in both waking and sleep.

Local field potentials in a pre-motor region predict learned vocal sequences.

Neuronal activity within the premotor region HVC is tightly synchronized to, and crucial for, the articulate production of learned song in birds. Characterizations of this neural activity detail patterns of sequential bursting in small, carefully identified subsets of neurons in the HVC population. The dynamics of HVC are well described by these characterizations, but have not been verified beyond this scale of measurement. There is a rich history of using local field potentials (LFP) to extract information about behavior that extends beyond the contribution of individual cells. These signals have the advantage of being stable over longer periods of time, and they have been used to study and decode human speech and other complex motor behaviors. Here we characterize LFP signals presumptively from the HVC of freely behaving male zebra finches during song production to determine if population activity may yield similar insights into the mechanisms underlying complex motor-vocal behavior. Following an initial observation that structured changes in the LFP were distinct to all vocalizations during song, we show that it is possible to extract time-varying features from multiple frequency bands to decode the identity of specific vocalization elements (syllables) and to predict their temporal onsets within the motif. This demonstrates the utility of LFP for studying vocal behavior in songbirds. Surprisingly, the time frequency structure of HVC LFP is qualitatively similar to well-established oscillations found in both human and non-human mammalian motor areas. This physiological similarity, despite distinct anatomical structures, may give insight into common computational principles for learning and/or generating complex motor-vocal behaviors.

Neurally driven synthesis of learned, complex vocalizations.

Brain machine interfaces (BMIs) hold promise to restore impaired motor function and serve as powerful tools to study learned motor skill. While limb-based motor prosthetic systems have leveraged nonhuman primates as an important animal model,1-4 speech prostheses lack a similar animal model and are more limited in terms of neural interface technology, brain coverage, and behavioral study design.5-7 Songbirds are an attractive model for learned complex vocal behavior. Birdsong shares a number of unique similarities with human speech,8-10 and its study has yielded general insight into multiple mechanisms and circuits behind learning, execution, and maintenance of vocal motor skill.11-18 In addition, the biomechanics of song production bear similarity to those of humans and some nonhuman primates.19-23 Here, we demonstrate a vocal synthesizer for birdsong, realized by mapping neural population activity recorded from electrode arrays implanted in the premotor nucleus HVC onto low-dimensional compressed representations of song, using simple computational methods that are implementable in real time. Using a generative biomechanical model of the vocal organ (syrinx) as the low-dimensional target for these mappings allows for the synthesis of vocalizations that match the bird's own song. These results provide proof of concept that high-dimensional, complex natural behaviors can be directly synthesized from ongoing neural activity. This may inspire similar approaches to prosthetics in other species by exploiting knowledge of the peripheral systems and the temporal structure of their output.

Microscale Physiological Events on the Human Cortical Surface.

Despite ongoing advances in our understanding of local single-cellular and network-level activity of neuronal populations in the human brain, extraordinarily little is known about their "intermediate" microscale local circuit dynamics. Here, we utilized ultra-high-density microelectrode arrays and a rare opportunity to perform intracranial recordings across multiple cortical areas in human participants to discover three distinct classes of cortical activity that are not locked to ongoing natural brain rhythmic activity. The first included fast waveforms similar to extracellular single-unit activity. The other two types were discrete events with slower waveform dynamics and were found preferentially in upper cortical layers. These second and third types were also observed in rodents, nonhuman primates, and semi-chronic recordings from humans via laminar and Utah array microelectrodes. The rates of all three events were selectively modulated by auditory and electrical stimuli, pharmacological manipulation, and cold saline application and had small causal co-occurrences. These results suggest that the proper combination of high-resolution microelectrodes and analytic techniques can capture neuronal dynamics that lay between somatic action potentials and aggregate population activity. Understanding intermediate microscale dynamics in relation to single-cell and network dynamics may reveal important details about activity in the full cortical circuit.

The Argo: a high channel count recording system for neural recording in vivo.

OBJECTIVE: Decoding neural activity has been limited by the lack of tools available to record from large numbers of neurons across multiple cortical regions simultaneously with high temporal fidelity. To this end, we developed the Argo system to record cortical neural activity at high data rates. APPROACH: Here we demonstrate a massively parallel neural recording system based on platinum-iridium microwire electrode arrays bonded to a CMOS voltage amplifier array. The Argo system is the highest channel count in vivo neural recording system, supporting simultaneous recording from 65 536 channels, sampled at 32 kHz and 12-bit resolution. This system was designed for cortical recordings, compatible with both penetrating and surface microelectrodes. MAIN RESULTS: We validated this system through initial bench testing to determine specific gain and noise characteristics of bonded microwires, followed by in-vivo experiments in both rat and sheep cortex. We recorded spiking activity from 791 neurons in rats and surface local field potential activity from over 30 000 channels in sheep. SIGNIFICANCE: These are the largest channel count microwire-based recordings in both rat and sheep. While currently adapted for head-fixed recording, the microwire-CMOS architecture is well suited for clinical translation. Thus, this demonstration helps pave the way for a future high data rate intracortical implant.

Impact of Brain Surface Boundary Conditions on Electrophysiology and Implications for Electrocorticography.

Volume conduction of electrical potentials in the brain is highly influenced by the material properties and geometry of the tissue and recording devices implanted into the tissue. These effects are very large in EEG due to the volume conduction through the skull and scalp but are often neglected in intracranial electrophysiology. When considering penetrating electrodes deep in the brain, the assumption of an infinite and homogenous medium can be used when the sources are far enough from the brain surface and the electrodes to minimize the boundary effect. When the electrodes are recording from the brain's surface the effect of the boundary cannot be neglected, and the large surface area and commonly used insulating materials in surface electrode arrays may further increase the effect by altering the nature of the boundary in the immediate vicinity of the electrodes. This gives the experimenter some control over the spatial profiles of the potentials by appropriate design of the electrode arrays. We construct a simple three-layer model to describe the effect of material properties and geometry above the brain surface on the electric potentials and conduct empirical experiments to validate this model. A laminar electrode array is used to measure the effect of insulating and relatively conducting layers above the cortical surface by recording evoked potentials alternating between a dried surface and saline covering layer, respectively. Empirically, we find that an insulating boundary amplifies the potentials relative to conductive saline by about a factor of 4, and that the effect is not constrained to potentials that originate near the surface. The model is applied to predict the influence of array design and implantation procedure on the recording amplitude and spatial selectivity of the surface electrode arrays.

Stimulus Driven Single Unit Activity From Micro-Electrocorticography.

High-fidelity measurements of neural activity can enable advancements in our understanding of the neural basis of complex behaviors such as speech, audition, and language, and are critical for developing neural prostheses that address impairments to these abilities due to disease or injury. We develop a novel high resolution, thin-film micro-electrocorticography (micro-ECoG) array that enables high-fidelity surface measurements of neural activity from songbirds, a well-established animal model for studying speech behavior. With this device, we provide the first demonstration of sensory-evoked modulation of surface-recorded single unit responses. We establish that single unit activity is consistently sensed from micro-ECoG electrodes over the surface of sensorimotor nucleus HVC (used as a proper name) in anesthetized European starlings, and validate responses with correlated firing in single units recorded simultaneously at surface and depth. The results establish a platform for high-fidelity recording from the surface of subcortical structures that will accelerate neurophysiological studies, and development of novel electrode arrays and neural prostheses.

Use of Accelerometry for Long Term Monitoring of Stroke Patients.

Stroke patients are monitored hourly by physicians and nurses in an attempt to better understand their physical state. To quantify the patients' level of mobility, hourly movement (i.e. motor) assessment scores are performed, which can be taxing and time-consuming for nurses and physicians. In this paper, we attempt to find a correlation between patient motor scores and continuous accelerometer data recorded in subjects who are unilaterally impaired due to stroke. The accelerometers were placed on both upper and lower extremities of four severely unilaterally impaired patients and their movements were recorded continuously for 7 to 14 days. Features that incorporate movement smoothness, strength, and characteristic movement patterns were extracted from the accelerometers using time-frequency analysis. Support vector classifiers were trained with the extracted features to test the ability of the long term accelerometer recordings in predicting dependent and antigravity sides, and significantly above baseline performance was obtained in most instances ([Formula: see text]). Finally, a leave-one-subject-out approach was carried out to assess the generalizability of the proposed methodology, and above baseline performance was obtained in two out of the three tested subjects. The methodology presented in this paper provides a simple, yet effective approach to perform long term motor assessment in neurocritical care patients.

Selective Formation of Porous Pt Nanorods for Highly Electrochemically Efficient Neural Electrode Interfaces.

The enhanced electrochemical activity of nanostructured materials is readily exploited in energy devices, but their utility in scalable and human-compatible implantable neural interfaces can significantly advance the performance of clinical and research electrodes. We utilize low-temperature selective dealloying to develop scalable and biocompatible one-dimensional platinum nanorod (PtNR) arrays that exhibit superb electrochemical properties at various length scales, stability, and biocompatibility for high performance neurotechnologies. PtNR arrays record brain activity with cellular resolution from the cortical surfaces in birds and nonhuman primates. Significantly, strong modulation of surface recorded single unit activity by auditory stimuli is demonstrated in European Starling birds as well as the modulation of local field potentials in the visual cortex by light stimuli in a nonhuman primate and responses to electrical stimulation in mice. PtNRs record behaviorally and physiologically relevant neuronal dynamics from the surface of the brain with high spatiotemporal resolution, which paves the way for less invasive brain-machine interfaces.

Correlation Structure in Micro-ECoG Recordings is Described by Spatially Coherent Components.

Electrocorticography (ECoG) is becoming more prevalent due to improvements in fabrication and recording technology as well as its ease of implantation compared to intracortical electrophysiology, larger cortical coverage, and potential advantages for use in long term chronic implantation. Given the flexibility in the design of ECoG grids, which is only increasing, it remains an open question what geometry of the electrodes is optimal for an application. Conductive polymer, PEDOT:PSS, coated microelectrodes have an advantage that they can be made very small without losing low impedance. This makes them suitable for evaluating the required granularity of ECoG recording in humans and experimental animals. We used two-dimensional (2D) micro-ECoG grids to record intra-operatively in humans and during acute implantations in mouse with separation distance between neighboring electrodes (i.e., pitch) of 0.4 mm and 0.2/0.25 mm respectively. To assess the spatial properties of the signals, we used the average correlation between electrodes as a function of the pitch. In agreement with prior studies, we find a strong frequency dependence in the spatial scale of correlation. By applying independent component analysis (ICA), we find that the spatial pattern of correlation is largely due to contributions from multiple spatially extended, time-locked sources present at any given time. Our analysis indicates the presence of spatially structured activity down to the sub-millimeter spatial scale in ECoG despite the effects of volume conduction, justifying the use of dense micro-ECoG grids.

Coarse behavioral context decoding.

OBJECTIVE: Current brain-computer interface (BCI) studies demonstrate the potential to decode neural signals obtained from structured and trial-based tasks to drive actuators with high performance within the context of these tasks. Ideally, to maximize utility, such systems will be applied to a wide range of behavioral settings or contexts. Thus, we explore the potential to augment such systems with the ability to decode abstract behavioral contextual states from neural activity. APPROACH: To demonstrate the feasibility of such context decoding, we used electrocorticography (ECoG) and stereo-electroencephalography (sEEG) data recorded from the cortical surface and deeper brain structures, respectively, continuously across multiple days from three subjects. During this time, the subjects were engaged in a range of naturalistic behaviors in a hospital environment. Behavioral contexts were labeled manually from video and audio recordings; four states were considered: engaging in dialogue, rest, using electronics, and watching television. We decode these behaviors using a factor analysis and support vector machine (SVM) approach. MAIN RESULTS: We demonstrate that these general behaviors can be decoded with high accuracies of 73% for a four-class classifier for one subject and 71% and 62% for a three-class classifier for two subjects. SIGNIFICANCE: To our knowledge, this is the first demonstration of the potential to disambiguate abstract naturalistic behavioral contexts from neural activity recorded throughout the day from implanted electrodes. This work motivates further study of context decoding for BCI applications using continuously recorded naturalistic activity in the clinical setting.

Autoencoders for learning template spectrograms in electrocorticographic signals.

OBJECTIVE: Electrocorticography (ECoG) based studies generally analyze features from specific frequency bands selected by manual evaluation of spectral power. However, the definition of these features can vary across subjects, cortical areas, tasks and across time for a given subject. We propose an autoencoder based approach for summarizing ECoG data with 'template spectrograms', i.e. informative time-frequency (t-f) patterns, and demonstrate their efficacy in two contexts: brain-computer interfaces (BCIs) and functional brain mapping. APPROACH: We use a publicly available dataset wherein subjects perform a finger flexion task in response to a visual cue. We train autoencoders to learn t-f patterns and use them in a deep neural network to decode finger flexions. Additionally, we propose and evaluate an unsupervised method for clustering electrode channels based on their aggregated activity. MAIN RESULTS: We show that the learnt t-f patterns can be used to classify individual finger movements with consisentently higher accuracy than with traditional spectral features. Furthermore, electrodes within automatically generated clusters tend to demonstrate functionally similar activity. SIGNIFICANCE: With increasing interest in and active development towards higher spatial resolution ECoG, along with the availability of large scale datasets from epilepsy monitoring units, there is an opportunity to develop automated and scalable unsupervised methods to learn effective summaries of spatial, temporal and frequency patterns in these data. The proposed methods reduce the effort required by neural engineers to develop effective features for BCI decoders. The clustering approach has applications in functional mapping studies for identifying brain regions associated with behavioral changes.