Eliminating or Shortening the Calibration for a P300 Brain-Computer Interface Based on a Convolutional Neural Network and Big Electroencephalography Data: An Online Study.

A brain-computer interface (BCI) measures and analyzes brain activity and converts it into computer commands to control external devices. Traditional BCIs usually require full calibration, which is time-consuming and makes BCI systems inconvenient to use. In this study, we propose an online P300 BCI spelling system with zero or shortened calibration based on a convolutional neural network (CNN) and big electroencephalography (EEG) data. Specifically, three methods are proposed to train CNNs for the online detection of P300 potentials: (i) training a subject-independent CNN with data collected from 150 subjects; (ii) adapting the CNN online via a semisupervised learning/self-training method based on unlabeled data collected during the user's online operation; and (iii) fine-tuning the CNN with a transfer learning method based on a small quantity of labeled data collected before the user's online operation. Note that the calibration process is eliminated in the first two methods and dramatically shortened in the third method. Based on these methods, an online P300 spelling system is developed. Twenty subjects participated in our online experiments. Average accuracies of 89.38%, 94.00% and 93.50% were obtained by the subject-independent CNN, the self-training-based CNN and the transfer learning-based CNN, respectively. These results demonstrate the effectiveness of our methods, and thus, the convenience of the online P300-based BCI system is substantially improved.

Electromagnetic Source Imaging With a Combination of Sparse Bayesian Learning and Deep Neural Network.

Accurate reconstruction of the brain activities from electroencephalography and magnetoencephalography (E/MEG) remains a long-standing challenge for the intrinsic ill-posedness in the inverse problem. In this study, to address this issue, we propose a novel data-driven source imaging framework based on sparse Bayesian learning and deep neural network (SI-SBLNN). Within this framework, the variational inference in conventional algorithm, which is built upon sparse Bayesian learning, is compressed via constructing a straightforward mapping from measurements to latent sparseness encoding parameters using deep neural network. The network is trained with synthesized data derived from the probabilistic graphical model embedded in the conventional algorithm. We achieved a realization of this framework with the algorithm, source imaging based on spatio-temporal basis function (SI-STBF), as backbone. In numerical simulations, the proposed algorithm validated its availability for different head models and robustness against distinct intensities of the noise. Meanwhile, it acquired superior performance compared to SI-STBF and several benchmarks in a variety of source configurations. Additionally, in real data experiments, it obtained the concordant results with the prior studies.

Associating Latent Representations With Cognitive Maps via Hyperspherical Space for Neural Population Spikes.

Recently, there has been a focus on drawing progress on representation learning to obtain more identifiable and interpretable latent representations for spike trains, which helps analyze neural population activity and understand neural mechanisms. Most existing deep generative models adopt carefully designed constraints to capture meaningful latent representations. For neural data involving navigation in cognitive space, based on insights from studies on cognitive maps, we argue that the good representations should reflect such directional nature. Due to manifold mismatch, models utilizing the Euclidean space learn a distorted geometric structure that is difficult to interpret. In the present work, we explore capturing the directional nature in a simpler yet more efficient way by introducing hyperspherical neural latent variable models (SNLVM). SNLVM is an improved deep latent variable model modeling neural activity and behavioral variables simultaneously with hyperspherical latent space. It bridges cognitive maps and latent variable models. We conduct experiments on modeling a static unidirectional task. The results show that while SNLVM has competitive performance, a hyperspherical prior naturally provides more informative and significantly better latent structures that can be interpreted as spatial cognitive maps.

Electromagnetic Source Imaging via a Data-Synthesis-Based Convolutional Encoder-Decoder Network.

Electromagnetic source imaging (ESI) requires solving a highly ill-posed inverse problem. To seek a unique solution, traditional ESI methods impose various forms of priors that may not accurately reflect the actual source properties, which may hinder their broad applications. To overcome this limitation, in this article, a novel data-synthesized spatiotemporally convolutional encoder-decoder network (DST-CedNet) method is proposed for ESI. The DST-CedNet recasts ESI as a machine learning problem, where discriminative learning and latent-space representations are integrated in a CedNet to learn a robust mapping from the measured electroencephalography/magnetoencephalography (E/MEG) signals to the brain activity. In particular, by incorporating prior knowledge regarding dynamical brain activities, a novel data synthesis strategy is devised to generate large-scale samples for effectively training CedNet. This stands in contrast to traditional ESI methods where the prior information is often enforced via constraints primarily aimed for mathematical convenience. Extensive numerical experiments as well as analysis of a real MEG and epilepsy EEG dataset demonstrate that the DST-CedNet outperforms several state-of-the-art ESI methods in robustly estimating source signals under a variety of source configurations.

Enhancement of lower limb motor imagery ability via dual-level multimodal stimulation and sparse spatial pattern decoding method.

Recently, motor imagery brain-computer interfaces (MI-BCIs) with stimulation systems have been developed in the field of motor function assistance and rehabilitation engineering. An efficient stimulation paradigm and Electroencephalogram (EEG) decoding method have been designed to enhance the performance of MI-BCI systems. Therefore, in this study, a multimodal dual-level stimulation paradigm is designed for lower-limb rehabilitation training, whereby visual and auditory stimulations act on the sensory organ while proprioceptive and functional electrical stimulations are provided to the lower limb. In addition, upper triangle filter bank sparse spatial pattern (UTFB-SSP) is proposed to automatically select the optimal frequency sub-bands related to desynchronization rhythm during enhanced imaginary movement to improve the decoding performance. The effectiveness of the proposed MI-BCI system is demonstrated on an the in-house experimental dataset and the BCI competition IV IIa dataset. The experimental results show that the proposed system can effectively enhance the MI performance by inducing the α, ß and γ rhythms in lower-limb movement imagery tasks.

Electromagnetic Source Imaging via Bayesian Modeling With Smoothness in Spatial and Temporal Domains.

Accurate reconstruction of cortical activation from electroencephalography and magnetoencephalography (E/MEG) is a long-standing challenge because of the inherently ill-posed inverse problem. In this paper, a novel algorithm under the empirical Bayesian framework, source imaging with smoothness in spatial and temporal domains (SI-SST), is proposed to address this issue. In SI-SST, current sources are decomposed into the product of spatial smoothing kernel, sparseness encoding coefficients, and temporal basis functions (TBFs). Further smoothness is integrated in the temporal domain with the employment of an underlying autoregressive model. Because sparseness encoding coefficients are constructed depending on overlapped clusters over cortex in this model, we derived a novel update rule based on fixed-point criterion instead of the convexity based approach which becomes invalid in this scenario. Entire variables and hyper parameters are updated alternatively in the variational inference procedure. SI-SST was assessed by multiple metrics with both simulated and experimental datasets. In practice, SI-SST had the superior reconstruction performance in both spatial extents and temporal profiles compared to the benchmarks.

Single-Trial EEG Classification via Orthogonal Wavelet Decomposition-Based Feature Extraction.

Achieving high classification performance is challenging due to non-stationarity and low signal-to-noise ratio (low SNR) characteristics of EEG signals. Spatial filtering is commonly used to improve the SNR yet the individual differences in the underlying temporal or frequency information is often ignored. This paper investigates motor imagery signals via orthogonal wavelet decomposition, by which the raw signals are decomposed into multiple unrelated sub-band components. Furthermore, channel-wise spectral filtering via weighting the sub-band components are implemented jointly with spatial filtering to improve the discriminability of EEG signals, with an l 2-norm regularization term embedded in the objective function to address the underlying over-fitting issue. Finally, sparse Bayesian learning with Gaussian prior is applied to the extracted power features, yielding an RVM classifier. The classification performance of SEOWADE is significantly better than those of several competing algorithms (CSP, FBCSP, CSSP, CSSSP, and shallow ConvNet). Moreover, scalp weight maps of the spatial filters optimized by SEOWADE are more neurophysiologically meaningful. In summary, these results demonstrate the effectiveness of SEOWADE in extracting relevant spatio-temporal information for single-trial EEG classification.

Learning Invariant Patterns Based on a Convolutional Neural Network and Big Electroencephalography Data for Subject-Independent P300 Brain-Computer Interfaces.

A brain-computer interface (BCI) measures and analyzes brain activity and converts this activity into computer commands to control external devices. In contrast to traditional BCIs that require a subject-specific calibration process before being operated, a subject-independent BCI learns a subject-independent model and eliminates subject-specific calibration for new users. However, building subject-independent BCIs remains difficult because electroencephalography (EEG) is highly noisy and varies by subject. In this study, we propose an invariant pattern learning method based on a convolutional neural network (CNN) and big EEG data for subject-independent P300 BCIs. The CNN was trained using EEG data from a large number of subjects, allowing it to extract subject-independent features and make predictions for new users. We collected EEG data from 200 subjects in a P300-based spelling task using two different types of amplifiers. The offline analysis showed that almost all subjects obtained significant cross-subject and cross-amplifier effects, with an average accuracy of more than 80%. Furthermore, more than half of the subjects achieved accuracies above 85%. These results indicated that our method was effective for building a subject-independent P300 BCI, with which more than 50% of users could achieve high accuracies without subject-specific calibration.

Cross-correlation based discriminant criterion for channel selection in motor imagery BCI systems.

Objective.Many electroencephalogram (EEG)-based brain-computer interface (BCI) systems use a large amount of channels for higher performance, which is time-consuming to set up and inconvenient for practical applications. Finding an optimal subset of channels without compromising the performance is a necessary and challenging task.Approach.In this article, we proposed a cross-correlation based discriminant criterion (XCDC) which assesses the importance of a channel for discriminating the mental states of different motor imagery (MI) tasks. Channels are ranked and selected according to the proposed criterion. The efficacy of XCDC is evaluated on two MI EEG datasets.Main results.On the two datasets, the proposed method reduces the channel number from 71 and 15 to under 18 and 11 respectively without compromising the classification accuracy on unseen data. Under the same constraint of accuracy, the proposed method requires fewer channels than existing channel selection methods based on Pearson's correlation coefficient and common spatial pattern. Visualization of XCDC shows consistent results with neurophysiological principles.Significance.This work proposes a quantitative criterion for assessing and ranking the importance of EEG channels in MI tasks and provides a practical method for selecting the ranked channels in the calibration phase of MI BCI systems, which alleviates the computational complexity and configuration difficulty in the subsequent steps, leading to real-time and more convenient BCI systems.

Capsule Network for ERP Detection in Brain-Computer Interface.

Event-related potential (ERP) is bioelectrical activity that occurs in the brain in response to specific events or stimuli, reflecting the electrophysiological changes in the brain during cognitive processes. ERP is important in cognitive neuroscience and has been applied to brain-computer interfaces (BCIs). However, because ERP signals collected on the scalp are weak, mixed with spontaneous electroencephalogram (EEG) signals, and their temporal and spatial features are complex, accurate ERP detection is challenging. Compared to traditional neural networks, the capsule network (CapsNet) replaces scalar-output neurons with vector-output capsules, allowing the various input information to be well preserved in the capsules. In this study, we expect to utilize CapsNet to extract the discriminative spatial-temporal features of ERP and encode them in capsules to reduce the loss of valuable information, thereby improving the ERP detection performance for BCI. Therefore, we propose ERP-CapsNet to perform ERP detection in a BCI speller application. The experimental results on BCI Competition datasets and the Akimpech dataset show that ERP-CapsNet achieves better classification performances than do the state-of-the-art techniques. We also use a decoder to investigate the attributes of ERPs encoded in capsules. The results show that ERP-CapsNet relies on the P300 and P100 components to detect ERP. Therefore, ERP-CapsNet not only acts as an outstanding method for ERP detection, but also provides useful insights into the ERP detection mechanism.

Spatiotemporal-Filtering-Based Channel Selection for Single-Trial EEG Classification.

Achieving high classification performance in electroencephalogram (EEG)-based brain-computer interfaces (BCIs) often entails a large number of channels, which impedes their use in practical applications. Despite the previous efforts, it remains a challenge to determine the optimal subset of channels in a subject-specific manner without heavily compromising the classification performance. In this article, we propose a new method, called spatiotemporal-filtering-based channel selection (STECS), to automatically identify a designated number of discriminative channels by leveraging the spatiotemporal information of the EEG data. In STECS, the channel selection problem is cast under the framework of spatiotemporal filter optimization by incorporating a group sparsity constraints, and a computationally efficient algorithm is developed to solve the optimization problem. The performance of STECS is assessed on three motor imagery EEG datasets. Compared with state-of-the-art spatiotemporal filtering algorithms using full EEG channels, STECS yields comparable classification performance with only half of the channels. Moreover, STECS significantly outperforms the existing channel selection methods. These results suggest that this algorithm holds promise for simplifying BCI setups and facilitating practical utility.

A Bayesian Shared Control Approach for Wheelchair Robot With Brain Machine Interface.

To enhance the performance of the brain-actuated robot system, a novel shared controller based on Bayesian approach is proposed for intelligently combining robot automatic control and brain-actuated control, which takes into account the uncertainty of robot perception, action and human control. Based on maximum a posteriori probability (MAP), this method establishes the probabilistic models of human and robot control commands to realize the optimal control of a brain-actuated shared control system. Application on an intelligent Bayesian shared control system based on steady-state visual evoked potential (SSVEP)-based brain machine interface (BMI) is presented for all-time continuous wheelchair navigation task. Moreover, to obtain more accurate brain control commands for shared controller and adapt the proposed system to the uncertainty of electroencephalogram (EEG), a hierarchical brain control mechanism with feedback rule is designed. Experiments have been conducted to verify the proposed system in several scenarios. Eleven subjects participated in our experiments and the results illustrate the effectiveness of the proposed method.

EEG- and EOG-Based Asynchronous Hybrid BCI: A System Integrating a Speller, a Web Browser, an E-Mail Client, and a File Explorer.

This paper presents a new asynchronous hybrid brain-computer interface (BCI) system that integrates a speller, a web browser, an e-mail client, and a file explorer using electroencephalographic (EEG) and electrooculography (EOG) signals. More specifically, an EOG-based button selection method, which requires the user to blink his/her eyes synchronously with the target button's flashes during button selection, is first presented. Next, we propose a mouse control method by combining EEG and EOG signals, in which the left-/right-hand motor imagery (MI)-related EEG is used to control the horizontal movement of the mouse and the blink-related EOG is used to control the vertical movement of the mouse and to select/reject a target. These two methods are further combined to develop the integrated hybrid BCI system. With the hybrid BCI, users can input text, access the internet, communicate with others via e-mail, and manage files in their computer using only EEG and EOG without any body movements. Ten healthy subjects participated in a comprehensive online experiment, and superior performance was achieved compared with our previously developed P300- and MI-based BCI and some other asynchronous BCIs, therefore demonstrating the system's effectiveness.

Dilated-Inception Net: Multi-Scale Feature Aggregation for Cardiac Right Ventricle Segmentation.

Segmentation of cardiac ventricle from magnetic resonance images is significant for cardiac disease diagnosis, progression assessment, and monitoring cardiac conditions. Manual segmentation is so time consuming, tedious, and subjective that automated segmentation methods are highly desired in practice. However, conventional segmentation methods performed poorly in cardiac ventricle, especially in the right ventricle. Compared with the left ventricle, whose shape is a simple thick-walled circle, the structure of the right ventricle is more complex due to ambiguous boundary, irregular cavity, and variable crescent shape. Hence, effective feature extractors and segmentation models are preferred. In this paper, we propose a dilated-inception net (DIN) to extract and aggregate multi-scale features for right ventricle segmentation. The DIN outperforms many state-of-the-art models on the benchmark database of right ventricle segmentation challenge. In addition, the experimental results indicate that the proposed model has potential to reach expert-level performance in right ventricular epicardium segmentation. More importantly, DIN behaves similarly to clinical expert with high correlation coefficients in four clinical cardiac indices. Therefore, the proposed DIN is promising for automated cardiac right ventricle segmentation in clinical applications.

Bayesian Electromagnetic Spatio-Temporal Imaging of Extended Sources Based on Matrix Factorization.

Accurate estimation of the locations and extents of neural sources from electroencephalography and magnetoencephalography (E/MEG) is challenging, especially for deep and highly correlated neural activities. In this study, we proposed a new fully data-driven source imaging method, source imaging based on spatio-temporal basis function (SI-STBF), which is built upon a Bayesian framework, to address this issue. The SI-STBF is based on the factorization of a source matrix as a product of a sparse coding matrix and a temporal basis function (TBF) matrix, which includes a few TBFs. The prior of the TBF is set in the empirical Bayesian manner. Similarly, for the spatial constraint, the SI-STBF assumes the prior covariance of the coding matrix as a weighted sum of several spatial covariance components. Both the TBFs and the coding matrix are learned from E/MEG simultaneously through variational Bayesian inference. To enable inference on high-resolution source space, we derived a scalable algorithm using convex analysis. The performance of the SI-STBF was assessed using both simulated and experimental E/MEG recordings. Compared with L2-norm constrained methods, the SI-STBF is superior in reconstructing extended sources with less spatial diffusion and less localization error. By virtue of the spatio-temporal factorization of source matrix, the SI-STBF also produces more accurate estimations than spatial-only constraint method for high correlated and deep sources.

An Online Interactive Paradigm for P300 Brain-Computer Interface Speller.

For each brain-computer interface system, efficiency is a key issue that considers both accuracy and speed. The P300 spellers built upon oddball paradigm are usually less efficient due to the repetitive stimulation of multiple characters for reliable detection. In this paper, based on the online EEG signal, we propose an interactive paradigm for P300 speller to improve its efficiency, primarily focusing within the single characterP300 paradigm. Specifically, after each stimulation, we first evaluate the posterior probability of each character in the stimuli set to be the target. The lowprobability characters are then removed fromthe stimuli set in the subsequent round(s), as character flash continues until the probability of any character surpasses a predefined threshold. Then, the character is selected as the target and data collection for the trial terminates. By reducing stimulus sequence characters, the system efficiency can be substantially improved. The spelling accuracy is insignificantly affected as the characters being removed have low probability to be the target. The online experimental results from a total of eight subjects show that an average practical information transfer rate of 50.26 bits/min (i.e. 9.07 characters/min) has been achieved, with 91% average spelling accuracy rate.

Spatial-Temporal Discriminative Restricted Boltzmann Machine for Event-Related Potential Detection and Analysis.

Detecting event-related potential (ERP) is a challenging problem because of its low signal-to-noise ratio and complex spatial-temporal features. Conventional detection methods usually rely on the ensemble averaging technique, which may eliminate subtle but important information in ERP signals and lead to poor detection performance. Inspired by the good performance of discriminative restricted Boltzmann machine (DRBM) in feature extraction and classification, we propose a spatial-temporal DRBM (ST-DRBM) to extract spatial and temporal features for ERP detection. The experimental results and statistical analyses demonstrate that the proposed method is able to achieve state-of-the-art ERP detection performance. The ST-DRBM is not only an effective ERP detector, but also a practical tool for ERP analysis. Based on the proposed model, similar scalp distribution and temporal variations were found in the ERP signals of different sessions, which indicated the feasibility of cross-session ERP detection. Given its state-of-the-art performance and effective analytical technique, ST-DRBM is promising for ERP-based brain-computer interfaces and neuroscience research.

Automatic System for Obstructive Sleep Apnea Events Detection Using Convolutional Neural Network.

Obstructive Sleep Apnea (OSA) is characterized by repetitive episodes of airflow reduction (hypopnea) or cessation (apnea), which, as a prevalent sleep disorder, can cause people to stop breathing for 10 to 30 seconds at a time and lead to serious problems such as daytime fatigue, impaired memory, and depression. This work intends to explore automatic detection of OSA events with 1-second annotation based on blood oxygen saturation, oronasal airflow, and ribcage and abdomen movements. Deep Learning (DL) technology, specifically, Convolutional Neural Network (CNN), is employed as a feature detector to learn the characteristics of the highorder correlation among visible data and corresponding labels. A fully-connected layer in the last stage of the CNN is connected to the output layer and constructs the desired number of outputs for sleep apnea events classification. A leave-one-out cross-validation has been conducted on the PhysioNet Sleep Database provided by St. Vincents University Hospital and University College Dublin, and an average accuracy of $79 .61$% across normal, hypopnea, and apnea, classes is achieved.

A Hybrid Network for ERP Detection and Analysis Based on Restricted Boltzmann Machine.

Detecting and Please provide the correct one analyzing the event-related potential (ERP) remains an important problem in neuroscience. Due to the low signal-to-noise ratio and complex spatio-temporal patterns of ERP signals, conventional methods usually rely on ensemble averaging technique for reliable detection, which may obliterate subtle but important information in each trial of ERP signals. Inspired by deep learning methods, we propose a novel hybrid network termed ERP-NET. With hybrid deep structure, the proposed network is able to learn complex spatial and temporal patterns from single-trial ERP signals. To verify the effectiveness of ERP-NET, we carried out a few ERP detection experiments that the proposed model achieved cutting-edge performance. The experimental results demonstrate that the patterns learned by the ERP-NET are discriminative ERP components in which the ERP signals are properly characterized. More importantly, as an effective approach to single-trial analysis, ERP-NET is able to discover new ERP patterns which are significant to neuroscience study as well as BCI applications. Therefore, the proposed ERP-NET is a promising tool for the research on ERP signals.

Bilinear Regularized Locality Preserving Learning on Riemannian Graph for Motor Imagery BCI.

In off-line training of motor imagery-based brain-computer interfaces (BCIs), to enhance the generalization performance of the learned classifier, the local information contained in test data could be used to improve the performance of motor imagery as well. Further considering that the covariance matrices of electroencephalogram (EEG) signal lie on Riemannian manifold, in this paper, we construct a Riemannian graph to incorporate the information of training and test data into processing. The adjacency and weight in Riemannian graph are determined by the geodesic distance of Riemannian manifold. Then, a new graph embedding algorithm, called bilinear regularized locality preserving (BRLP), is derived upon the Riemannian graph for addressing the problems of high dimensionality frequently arising in BCIs. With a proposed regularization term encoding prior information of EEG channels, the BRLP could obtain more robust performance. Finally, an efficient classification algorithm based on extreme learning machine is proposed to perform on the tangent space of learned embedding. Experimental evaluations on the BCI competition and in-house data sets reveal that the proposed algorithms could obtain significantly higher performance than many competition algorithms after using same filter process.