Learning neural decoders without labels using multiple data streams.

Objective.Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding.Approach.We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models.Main results.We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models.Significance.We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

AJILE12: Long-term naturalistic human intracranial neural recordings and pose.

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

Behavioral and Neural Variability of Naturalistic Arm Movements.

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed "in the wild." Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled interevent variability using 10 behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior.

Mining naturalistic human behaviors in long-term video and neural recordings.

BACKGROUND: Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. NEW METHOD: Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video. RESULTS: Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7-9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain-computer interfacing applications, and we show decoding results from a movement initiation detection task. COMPARISON WITH EXISTING METHODS: Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution. CONCLUSIONS: Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.

Generalized neural decoders for transfer learning across participants and recording modalities.

Objective. Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.Approach. We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (a) a Hilbert transform that computes spectral power at data-driven frequencies and (b) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main results. HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features.Significance. By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Human electrocortical, electromyographical, ocular, and kinematic data during perturbed walking and standing.

Active balance control is critical for performing many of our everyday activities. Our nervous systems rely on multiple sensory inputs to inform cortical processing, leading to coordinated muscle actions that maintain balance. However, such cortical processing can be challenging to record during mobile balance tasks due to limitations in noninvasive neuroimaging and motion artifact contamination. Here, we present a synchronized, multi-modal dataset from 30 healthy, young human participants during standing and walking while undergoing brief sensorimotor perturbations. Our dataset includes 20 total hours of high-density electroencephalography (EEG) recorded from 128 scalp electrodes, along with surface electromyography (EMG) from 10 neck and leg electrodes, electrooculography (EOG) recorded from 3 electrodes, and 3D body position from 2 sensors. In addition, we include ∼ 18000 total balance perturbation events across participants. To facilitate data reuse, we share this dataset in the Brain Imaging Data Structure (BIDS) data standard and publicly release code that replicates our previous event-related findings.

Unsupervised Sleep and Wake State Identification in Long-Term Electrocorticography Recordings.

Studying the neural correlates of sleep can lead to revelations in our understanding of sleep and its interplay with different neurological disorders. Sleep research relies on manual annotation of sleep stages based on rules developed for healthy adults. Automating sleep stage annotation can expedite sleep research and enable us to better understand atypical sleep patterns. Our goal was to create a fully unsupervised approach to label sleep and wake states in human electro-corticography (ECoG) data from epilepsy patients. Here, we demonstrate that with continuous data from a single ECoG electrode, hidden semi-Markov models (HSMM) perform best in classifying sleep/wake states without excessive transitions, with a mean accuracy (n=4) of 85.2% compared to using K-means clustering (72.2%) and hidden Markov models (81.5%). Our results confirm that HSMMs produce meaningful labels for ECoG data and establish the groundwork to apply this model to cluster sleep stages and potentially other behavioral states.

Group-level cortical and muscular connectivity during perturbations to walking and standing balance.

Maintaining balance is a complex process requiring multisensory processing and coordinated muscle activation. Previous studies have indicated that the cortex is directly involved in balance control, but less information is known about cortical flow of signals for balance. We studied source-localized electrocortical effective connectivity dynamics of healthy young subjects (29 subjects: 14 male and 15 female) walking and standing with both visual and physical perturbations to their balance. The goal of this study was to quantify differences in group-level corticomuscular connectivity responses to sensorimotor perturbations during walking and standing. We hypothesized that perturbed visual input during balance would transiently decrease connectivity between occipital and parietal areas due to disruptive visual input during sensory processing. We also hypothesized that physical pull perturbations would increase cortical connections to central sensorimotor areas because of higher sensorimotor integration demands. Our findings show decreased occipito-parietal connectivity during visual rotations and widespread increases in connectivity during pull perturbations focused on central areas, as expected. We also found evidence for communication from cortex to muscles during perturbed balance. These results show that sensorimotor perturbations to balance alter cortical networks and can be quantified using effective connectivity estimation.

Combined head phantom and neural mass model validation of effective connectivity measures.

OBJECTIVE: Due to its high temporal resolution, electroencephalography (EEG) has become a promising tool for quantifying cortical dynamics and effective connectivity in a mobile setting. While many connectivity estimators are available, the efficacy of these measures has not been rigorously validated in real-world scenarios. The goal of this study was to quantify the accuracy of independent component analysis and multiple connectivity measures on ground-truth connections while exposed real-world volume conduction and head motion. APPROACH: We collected high-density EEG from a phantom head with embedded antennae, using neural mass models to generate transiently interconnected signals. The head was mounted upon a motion platform that mimicked recorded human head motion at various walking speeds. We used cross-correlation and signal to noise ratio to determine how well independent component analysis recovered the original antenna signals. For connectivity measures, we computed the average and standard deviation across frequency of each estimated connectivity peak. MAIN RESULTS: Independent component analysis recovered most antenna signals, as evidenced by cross-correlations primarily above 0.8, and maintained consistent signal to noise ratio values near 10 dB across walking speeds compared to scalp channel data, which had decreased signal to noise ratios of ~2 dB at fast walking speeds. The connectivity measures used were generally able to identify the true interconnections, but some measures were susceptible to spurious high-frequency connections inducing large standard deviations of ~10 Hz. SIGNIFICANCE: Our results indicate that independent component analysis and some connectivity measures can be effective at recovering underlying connections among brain areas. These results highlight the utility of validating EEG processing techniques with a combination of complex signals, phantom head use, and realistic head motion.

Differentiation in Theta and Beta Electrocortical Activity between Visual and Physical Perturbations to Walking and Standing Balance.

Human balance is a complex process in healthy adults, requiring precisely timed coordination among sensory information, cognitive processing, and motor control. It has been difficult to quantify brain dynamics during human balance control due to limitations in brain-imaging modalities. The goal of this study was to determine whether by using high-density electroencephalography (EEG) and independent component analysis, we can identify common cortical responses to visual and physical balance perturbations during walking and standing. We studied the responses of 30 healthy young adults to sensorimotor perturbations that challenged their balance. Subjects performed four 10 min trials of beam walking and tandem stance while either being mediolaterally pulled at the waist or viewing brief 20° field-of-view rotations in virtual reality. We recorded high-density EEG, motion capture, lower leg electromyography (EMG), and neck EMG. We hypothesized that both physical pull and visual rotation perturbations would elicit time-frequency fluctuations in theta (4-8 Hz) and beta (13-30 Hz) bands, with increased occipito-parietal activity during visual rotations compared with pull perturbations. Our results confirmed this hypothesis. For both perturbations, we found early theta synchronization and late alpha-beta (8-30 Hz) desynchronization following perturbation onset. This pattern was strongest in occipito-parietal areas during visual perturbations and strongest in sensorimotor areas during pull perturbations. These results suggest a similar time-frequency electrocortical pattern when humans respond to sensorimotor conflict, but with substantive differences in the brain areas involved for visual versus physical perturbations. Our findings may have important implications for assessing and training balance in individuals with and without motor disabilities.

Effects of virtual reality high heights exposure during beam-walking on physiological stress and cognitive loading.

Virtual reality has been increasingly used in research on balance rehabilitation because it provides robust and novel sensory experiences in controlled environments. We studied 19 healthy young subjects performing a balance beam walking task in two virtual reality conditions and with unaltered view (15 minutes each) to determine if virtual reality high heights exposure induced stress. We recorded number of steps off the beam, heart rate, electrodermal activity, response time to an auditory cue, and high-density electroencephalography (EEG). We hypothesized that virtual high heights exposure would increase measures of physiological stress compared to unaltered viewing at low heights. We found that the virtual high height condition increased heart rate variability and heart rate frequency power relative to virtual low heights. Virtual reality use resulted in increased number of step-offs, heart rate, electrodermal activity, and response time compared to the unaltered viewing at low heights condition. Our results indicated that virtual reality decreased dynamic balance performance and increased physical and cognitive loading compared to unaltered viewing at low heights. In virtual reality, we found significant decreases in source-localized EEG peak amplitude relative to unaltered viewing in the anterior cingulate, which is considered important in sensing loss of balance. Our findings indicate that virtual reality provides realistic experiences that can induce physiological stress in humans during dynamic balance tasks, but virtual reality use impairs physical and cognitive performance during balance.

Transient visual perturbations boost short-term balance learning in virtual reality by modulating electrocortical activity.

Immersive virtual reality can expose humans to novel training and sensory environments, but motor training with virtual reality has not been able to improve motor performance as much as motor training in real-world conditions. An advantage of immersive virtual reality that has not been fully leveraged is that it can introduce transient visual perturbations on top of the visual environment being displayed. The goal of this study was to determine whether transient visual perturbations introduced in immersive virtual reality modify electrocortical activity and behavioral outcomes in human subjects practicing a novel balancing task during walking. We studied three groups of healthy young adults (5 male and 5 female for each) while they learned a balance beam walking task for 30 min under different conditions. Two groups trained while wearing a virtual reality headset, and one of those groups also had half-second visual rotation perturbations lasting ~10% of the training time. The third group trained without virtual reality. We recorded high-density electroencephalography (EEG) and movement kinematics. We hypothesized that virtual reality training with perturbations would increase electrocortical activity and improve balance performance compared with virtual reality training without perturbations. Our results confirmed the hypothesis. Brief visual perturbations induced increased theta spectral power and decreased alpha spectral power in parietal and occipital regions and improved balance performance in posttesting. Our findings indicate that transient visual perturbations during immersive virtual reality training can boost short-term motor learning by inducing a cognitive change, minimizing the negative effects of virtual reality on motor training. NEW & NOTEWORTHY We found that transient visual perturbations in virtual reality during balance training can boost short-term motor learning by inducing a cognitive change, overcoming the negative effects of immersive virtual reality. As a result, subjects training in immersive virtual reality with visual perturbations have equivalent performance improvement as training in real-world conditions. Visual perturbations elicited cortical responses in occipital and parietal regions and may have improved the brain's ability to adapt to variations in sensory input.

Independent Component Analysis and Source Localization on Mobile EEG Data Can Identify Increased Levels of Acute Stress.

Mobile electroencephalography (EEG) is a very useful tool to investigate the physiological basis of cognition under real-world conditions. However, as we move experimentation into less-constrained environments, the influence of state changes increases. The influence of stress on cortical activity and cognition is an important example. Monitoring of modulation of cortical activity by EEG measurements is a promising tool for assessing acute stress. In this study, we test this hypothesis and combine EEG with independent component analysis and source localization to identify cortical differences between a control condition and a stressful condition. Subjects performed a stationary shooting task using an airsoft rifle with and without the threat of an experimenter firing a different airsoft rifle in their direction. We observed significantly higher skin conductance responses and salivary cortisol levels (p < 0.05 for both) during the stressful conditions, indicating that we had successfully induced an adequate level of acute stress. We located independent components in five regions throughout the cortex, most notably in the dorsolateral prefrontal cortex, a region previously shown to be affected by increased levels of stress. This area showed a significant decrease in spectral power in the theta and alpha bands less than a second after the subjects pulled the trigger. Overall, our results suggest that EEG with independent component analysis and source localization has the potential of monitoring acute stress in real-world environments.

Effects of linking a soil-water-balance model with a groundwater-flow model.

A previously published regional groundwater-flow model in north-central Nebraska was sequentially linked with the recently developed soil-water-balance (SWB) model to analyze effects to groundwater-flow model parameters and calibration results. The linked models provided a more detailed spatial and temporal distribution of simulated recharge based on hydrologic processes, improvement of simulated groundwater-level changes and base flows at specific sites in agricultural areas, and a physically based assessment of the relative magnitude of recharge for grassland, nonirrigated cropland, and irrigated cropland areas. Root-mean-squared (RMS) differences between the simulated and estimated or measured target values for the previously published model and linked models were relatively similar and did not improve for all types of calibration targets. However, without any adjustment to the SWB-generated recharge, the RMS difference between simulated and estimated base-flow target values for the groundwater-flow model was slightly smaller than for the previously published model, possibly indicating that the volume of recharge simulated by the SWB code was closer to actual hydrogeologic conditions than the previously published model provided. Groundwater-level and base-flow hydrographs showed that temporal patterns of simulated groundwater levels and base flows were more accurate for the linked models than for the previously published model at several sites, particularly in agricultural areas.