Decoding neural activity to assess individual latent state in ecologically valid contexts.

Objective.Currently, there exists very few ways to isolate cognitive processes, historically defined via highly controlled laboratory studies, in more ecologically valid contexts. Specifically, it remains unclear as to what extent patterns of neural activity observed under such constraints actually manifest outside the laboratory in a manner that can be used to make accurate inferences about latent states, associated cognitive processes, or proximal behavior. Improving our understanding of when and how specific patterns of neural activity manifest in ecologically valid scenarios would provide validation for laboratory-based approaches that study similar neural phenomena in isolation and meaningful insight into the latent states that occur during complex tasks.Approach.Domain generalization methods, borrowed from the work of the brain-computer interface community, have the potential to capture high-dimensional patterns of neural activity in a way that can be reliably applied across experimental datasets in order to address this specific challenge. We previously used such an approach to decode phasic neural responses associated with visual target discrimination. Here, we extend that work to more tonic phenomena such as internal latent states. We use data from two highly controlled laboratory paradigms to train two separate domain-generalized models. We apply the trained models to an ecologically valid paradigm in which participants performed multiple, concurrent driving-related tasks while perched atop a six-degrees-of-freedom ride-motion simulator.Main Results.Using the pretrained models, we estimate latent state and the associated patterns of neural activity. As the patterns of neural activity become more similar to those patterns observed in the training data, we find changes in behavior and task performance that are consistent with the observations from the original, laboratory-based paradigms.Significance.These results lend ecological validity to the original, highly controlled, experimental designs and provide a methodology for understanding the relationship between neural activity and behavior during complex tasks.

Decoding P300 Variability Using Convolutional Neural Networks.

Deep convolutional neural networks (CNN) have previously been shown to be useful tools for signal decoding and analysis in a variety of complex domains, such as image processing and speech recognition. By learning from large amounts of data, the representations encoded by these deep networks are often invariant to moderate changes in the underlying feature spaces. Recently, we proposed a CNN architecture that could be applied to electroencephalogram (EEG) decoding and analysis. In this article, we train our CNN model using data from prior experiments in order to later decode the P300 evoked response from an unseen, hold-out experiment. We analyze the CNN output as a function of the underlying variability in the P300 response and demonstrate that the CNN output is sensitive to the experiment-induced changes in the neural response. We then assess the utility of our approach as a means of improving the overall signal-to-noise ratio in the EEG record. Finally, we show an example of how CNN-based decoding can be applied to the analysis of complex data.

Analyzing P300 Distractors for Target Reconstruction.

P300-based brain-computer interfaces (BCIs) are often trained per-user and per-application space. Training such models requires ground truth knowledge of target and nontarget stimulus categories during model training, which imparts bias into the model. Additionally, not all non-targets are created equal; some may contain visual features that resemble targets or may otherwise be visually salient. Current research has indicated that non-target distractors may elicit attenuated P300 responses based on the perceptual similarity of these distractors to the target category. To minimize this bias, and enable a more nuanced analysis, we use a generalized BCI approach that is fit to neither user nor task. We do not seek to improve the overall accuracy of the BCI with our generalized approach; we instead demonstrate the utility of our approach for identifying targetrelated image features. When combined with other intelligent agents, such as computer vision systems, the performance of the generalized model equals that of the user-specific models, without any user specific data.

Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials.

OBJECTIVE: Steady-state visual evoked potentials (SSVEPs) are neural oscillations from the parietal and occipital regions of the brain that are evoked from flickering visual stimuli. SSVEPs are robust signals measurable in the electroencephalogram (EEG) and are commonly used in brain-computer interfaces (BCIs). However, methods for high-accuracy decoding of SSVEPs usually require hand-crafted approaches that leverage domain-specific knowledge of the stimulus signals, such as specific temporal frequencies in the visual stimuli and their relative spatial arrangement. When this knowledge is unavailable, such as when SSVEP signals are acquired asynchronously, such approaches tend to fail. APPROACH: In this paper, we show how a compact convolutional neural network (Compact-CNN), which only requires raw EEG signals for automatic feature extraction, can be used to decode signals from a 12-class SSVEP dataset without the need for user-specific calibration. MAIN RESULTS: The Compact-CNN demonstrates across subject mean accuracy of approximately 80%, out-performing current state-of-the-art, hand-crafted approaches using canonical correlation analysis (CCA) and Combined-CCA. Furthermore, the Compact-CNN approach can reveal the underlying feature representation, revealing that the deep learner extracts additional phase- and amplitude-related features associated with the structure of the dataset. SIGNIFICANCE: We discuss how our Compact-CNN shows promise for BCI applications that allow users to freely gaze/attend to any stimulus at any time (e.g. asynchronous BCI) as well as provides a method for analyzing SSVEP signals in a way that might augment our understanding about the basic processing in the visual cortex.

EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces.

OBJECTIVE: Brain-computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct characteristics of its expected EEG control signal, limiting its application to that specific signal. Convolutional neural networks (CNNs), which have been used in computer vision and speech recognition to perform automatic feature extraction and classification, have successfully been applied to EEG-based BCIs; however, they have mainly been applied to single BCI paradigms and thus it remains unclear how these architectures generalize to other paradigms. Here, we ask if we can design a single CNN architecture to accurately classify EEG signals from different BCI paradigms, while simultaneously being as compact as possible. APPROACH: In this work we introduce EEGNet, a compact convolutional neural network for EEG-based BCIs. We introduce the use of depthwise and separable convolutions to construct an EEG-specific model which encapsulates well-known EEG feature extraction concepts for BCI. We compare EEGNet, both for within-subject and cross-subject classification, to current state-of-the-art approaches across four BCI paradigms: P300 visual-evoked potentials, error-related negativity responses (ERN), movement-related cortical potentials (MRCP), and sensory motor rhythms (SMR). MAIN RESULTS: We show that EEGNet generalizes across paradigms better than, and achieves comparably high performance to, the reference algorithms when only limited training data is available across all tested paradigms. In addition, we demonstrate three different approaches to visualize the contents of a trained EEGNet model to enable interpretation of the learned features. SIGNIFICANCE: Our results suggest that EEGNet is robust enough to learn a wide variety of interpretable features over a range of BCI tasks. Our models can be found at: https://github.com/vlawhern/arl-eegmodels.

Isolating Discriminant Neural Activity in the Presence of Eye Movements and Concurrent Task Demands.

A growing number of studies use the combination of eye-tracking and electroencephalographic (EEG) measures to explore the neural processes that underlie visual perception. In these studies, fixation-related potentials (FRPs) are commonly used to quantify early and late stages of visual processing that follow the onset of each fixation. However, FRPs reflect a mixture of bottom-up (sensory-driven) and top-down (goal-directed) processes, in addition to eye movement artifacts and unrelated neural activity. At present there is little consensus on how to separate this evoked response into its constituent elements. In this study we sought to isolate the neural sources of target detection in the presence of eye movements and over a range of concurrent task demands. Here, participants were asked to identify visual targets (Ts) amongst a grid of distractor stimuli (Ls), while simultaneously performing an auditory N-back task. To identify the discriminant activity, we used independent components analysis (ICA) for the separation of EEG into neural and non-neural sources. We then further separated the neural sources, using a modified measure-projection approach, into six regions of interest (ROIs): occipital, fusiform, temporal, parietal, cingulate, and frontal cortices. Using activity from these ROIs, we identified target from non-target fixations in all participants at a level similar to other state-of-the-art classification techniques. Importantly, we isolated the time course and spectral features of this discriminant activity in each ROI. In addition, we were able to quantify the effect of cognitive load on both fixation-locked potential and classification performance across regions. Together, our results show the utility of a measure-projection approach for separating task-relevant neural activity into meaningful ROIs within more complex contexts that include eye movements.

EEG-Based User Reaction Time Estimation Using Riemannian Geometry Features.

Riemannian geometry has been successfully used in many brain-computer interface (BCI) classification problems and demonstrated superior performance. In this paper, for the first time, it is applied to BCI regression problems, an important category of BCI applications. More specifically, we propose a new feature extraction approach for electroencephalogram (EEG)-based BCI regression problems: a spatial filter is first used to increase the signal quality of the EEG trials and also to reduce the dimensionality of the covariance matrices, and then Riemannian tangent space features are extracted. We validate the performance of the proposed approach in reaction time estimation from EEG signals measured in a large-scale sustained-attention psychomotor vigilance task, and show that compared with the traditional powerband features, the tangent space features can reduce the root mean square estimation error by 4.30%-8.30%, and increase the estimation correlation coefficient by 6.59%-11.13%.

Spectral Transfer Learning Using Information Geometry for a User-Independent Brain-Computer Interface.

Recent advances in signal processing and machine learning techniques have enabled the application of Brain-Computer Interface (BCI) technologies to fields such as medicine, industry, and recreation; however, BCIs still suffer from the requirement of frequent calibration sessions due to the intra- and inter-individual variability of brain-signals, which makes calibration suppression through transfer learning an area of increasing interest for the development of practical BCI systems. In this paper, we present an unsupervised transfer method (spectral transfer using information geometry, STIG), which ranks and combines unlabeled predictions from an ensemble of information geometry classifiers built on data from individual training subjects. The STIG method is validated in both off-line and real-time feedback analysis during a rapid serial visual presentation task (RSVP). For detection of single-trial, event-related potentials (ERPs), the proposed method can significantly outperform existing calibration-free techniques as well as outperform traditional within-subject calibration techniques when limited data is available. This method demonstrates that unsupervised transfer learning for single-trial detection in ERP-based BCIs can be achieved without the requirement of costly training data, representing a step-forward in the overall goal of achieving a practical user-independent BCI system.

A Permutation Test for Unbalanced Paired Comparisons of Global Field Power.

Global field power is a valuable summary of multi-channel electroencephalography data. However, global field power is biased by the noise typical of electroencephalography experiments, so comparisons of global field power on data with unequal noise are invalid. Here, we demonstrate the relationship between the number of trials that contribute to a global field power measure and the expected value of that global field power measure. We also introduce a statistical testing procedure that can be used for multi-subject, repeated-measures (also called within-subjects) comparisons of global field power when the number of trials per condition is unequal across conditions. Simulations demonstrate the effect of unequal trial numbers on global field power comparisons and show the validity of the proposed test in contrast to conventional approaches. Finally, the proposed test and two alternative tests are applied to data collected in a rapid serial visual presentation target detection experiment. The results show that the proposed test finds global field power differences in the classical P3 range; the other tests find differences in that range but also at other times including at times before stimulus onset. These results are interpreted as showing that the proposed test is valid and sensitive to real within-subject differences in global field power in multi-subject unbalanced data.

Switching EEG Headsets Made Easy: Reducing Offline Calibration Effort Using Active Weighted Adaptation Regularization.

Electroencephalography (EEG) headsets are the most commonly used sensing devices for brain-computer interface. In real-world applications, there are advantages to extrapolating data from one user session to another. However, these advantages are limited if the data arise from different hardware systems, which often vary between application spaces. Currently, this creates a need to recalibrate classifiers, which negatively affects people's interest in using such systems. In this paper, we employ active weighted adaptation regularization (AwAR), which integrates weighted adaptation regularization (wAR) and active learning, to expedite the calibration process. wAR makes use of labeled data from the previous headset and handles class-imbalance, and active learning selects the most informative samples from the new headset to label. Experiments on single-trial event-related potential classification show that AwAR can significantly increase the classification accuracy, given the same number of labeled samples from the new headset. In other words, AwAR can effectively reduce the number of labeled samples required from the new headset, given a desired classification accuracy, suggesting value in collating data for use in wide scale transfer-learning applications.

Improved Neural Signal Classification in a Rapid Serial Visual Presentation Task Using Active Learning.

The application space for brain-computer interface (BCI) technologies is rapidly expanding with improvements in technology. However, most real-time BCIs require extensive individualized calibration prior to use, and systems often have to be recalibrated to account for changes in the neural signals due to a variety of factors including changes in human state, the surrounding environment, and task conditions. Novel approaches to reduce calibration time or effort will dramatically improve the usability of BCI systems. Active Learning (AL) is an iterative semi-supervised learning technique for learning in situations in which data may be abundant, but labels for the data are difficult or expensive to obtain. In this paper, we apply AL to a simulated BCI system for target identification using data from a rapid serial visual presentation (RSVP) paradigm to minimize the amount of training samples needed to initially calibrate a neural classifier. Our results show AL can produce similar overall classification accuracy with significantly less labeled data (in some cases less than 20%) when compared to alternative calibration approaches. In fact, AL classification performance matches performance of 10-fold cross-validation (CV) in over 70% of subjects when training with less than 50% of the data. To our knowledge, this is the first work to demonstrate the use of AL for offline electroencephalography (EEG) calibration in a simulated BCI paradigm. While AL itself is not often amenable for use in real-time systems, this work opens the door to alternative AL-like systems that are more amenable for BCI applications and thus enables future efforts for developing highly adaptive BCI systems.

The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.