BCI-Based Rehabilitation on the Stroke in Sequela Stage.

Background: Stroke is the leading cause of serious and long-term disability worldwide. Survivors may recover some motor functions after rehabilitation therapy. However, many stroke patients missed the best time period for recovery and entered into the sequela stage of chronic stroke. Method: Studies have shown that motor imagery- (MI-) based brain-computer interface (BCI) has a positive effect on poststroke rehabilitation. This study used both virtual limbs and functional electrical stimulation (FES) as feedback to provide patients with a closed-loop sensorimotor integration for motor rehabilitation. An MI-based BCI system acquired, analyzed, and classified motor attempts from electroencephalogram (EEG) signals. The FES system would be activated if the BCI detected that the user was imagining wrist dorsiflexion on the instructed side of the body. Sixteen stroke patients in the sequela stage were randomly assigned to a BCI group and a control group. All of them participated in rehabilitation training for four weeks and were assessed by the Fugl-Meyer Assessment (FMA) of motor function. Results: The average improvement score of the BCI group was 3.5, which was higher than that of the control group (0.9). The active EEG patterns of the four patients in the BCI group whose FMA scores increased gradually became centralized and shifted to sensorimotor areas and premotor areas throughout the study. Conclusions: Study results showed evidence that patients in the BCI group achieved larger functional improvements than those in the control group and that the BCI-FES system is effective in restoring motor function to upper extremities in stroke patients. This study provides a more autonomous approach than traditional treatments used in stroke rehabilitation.

Music-induced emotions can be predicted from a combination of brain activity and acoustic features.

It is widely acknowledged that music can communicate and induce a wide range of emotions in the listener. However, music is a highly-complex audio signal composed of a wide range of complex time- and frequency-varying components. Additionally, music-induced emotions are known to differ greatly between listeners. Therefore, it is not immediately clear what emotions will be induced in a given individual by a piece of music. We attempt to predict the music-induced emotional response in a listener by measuring the activity in the listeners electroencephalogram (EEG). We combine these measures with acoustic descriptors of the music, an approach that allows us to consider music as a complex set of time-varying acoustic features, independently of any specific music theory. Regression models are found which allow us to predict the music-induced emotions of our participants with a correlation between the actual and predicted responses of up to r=0.234,p<0.001. This regression fit suggests that over 20% of the variance of the participant's music induced emotions can be predicted by their neural activity and the properties of the music. Given the large amount of noise, non-stationarity, and non-linearity in both EEG and music, this is an encouraging result. Additionally, the combination of measures of brain activity and acoustic features describing the music played to our participants allows us to predict music-induced emotions with significantly higher accuracies than either feature type alone (p<0.01).

MOCNN: A Multiscale Deep Convolutional Neural Network for ERP-Based Brain-Computer Interfaces.

Event-related potentials (ERPs) reflect neurophysiological changes of the brain in response to external events and their associated underlying complex spatiotemporal feature information is governed by ongoing oscillatory activity within the brain. Deep learning methods have been increasingly adopted for ERP-based brain-computer interfaces (BCIs) due to their excellent feature representation abilities, which allow for deep analysis of oscillatory activity within the brain. Features with higher spatiotemporal frequencies usually represent detailed and localized information, while features with lower spatiotemporal frequencies usually represent global structures. Mining EEG features from multiple spatiotemporal frequencies is conducive to obtaining more discriminative information. A multiscale feature fusion octave convolution neural network (MOCNN) is proposed in this article. MOCNN divides the ERP signals into high-, medium-and low-frequency components corresponding to different resolutions and processes them in different branches. By adding mid-and low-frequency components, the feature information used by MOCNN can be enriched, and the required amount of calculations can be reduced. After successive feature mapping using temporal and spatial convolutions, MOCNN realizes interactive learning among different components through the exchange of feature information among branches. Classification is accomplished by feeding the fused deep spatiotemporal features from various components into a fully connected layer. The results, obtained on two public datasets and a self-collected ERP dataset, show that MOCNN can achieve state-of-the-art ERP classification performance. In this study, the generalized concept of octave convolution is introduced into the field of ERP-BCI research, which allows effective spatiotemporal features to be extracted from multiscale networks through branch width optimization and information interaction at various scales.

Testing for significance of phase synchronisation dynamics in the EEG.

A number of tests exist to check for statistical significance of phase synchronisation within the Electroencephalogram (EEG); however, the majority suffer from a lack of generality and applicability. They may also fail to account for temporal dynamics in the phase synchronisation, regarding synchronisation as a constant state instead of a dynamical process. Therefore, a novel test is developed for identifying the statistical significance of phase synchronisation based upon a combination of work characterising temporal dynamics of multivariate time-series and Markov modelling. We show how this method is better able to assess the significance of phase synchronisation than a range of commonly used significance tests. We also show how the method may be applied to identify and classify significantly different phase synchronisation dynamics in both univariate and multivariate datasets.

Neural decoding of music from the EEG.

Neural decoding models can be used to decode neural representations of visual, acoustic, or semantic information. Recent studies have demonstrated neural decoders that are able to decode accoustic information from a variety of neural signal types including electrocortiography (ECoG) and the electroencephalogram (EEG). In this study we explore how functional magnetic resonance imaging (fMRI) can be combined with EEG to develop an accoustic decoder. Specifically, we first used a joint EEG-fMRI paradigm to record brain activity while participants listened to music. We then used fMRI-informed EEG source localisation and a bi-directional long-term short term deep learning network to first extract neural information from the EEG related to music listening and then to decode and reconstruct the individual pieces of music an individual was listening to. We further validated our decoding model by evaluating its performance on a separate dataset of EEG-only recordings. We were able to reconstruct music, via our fMRI-informed EEG source analysis approach, with a mean rank accuracy of 71.8% ([Formula: see text], [Formula: see text]). Using only EEG data, without participant specific fMRI-informed source analysis, we were able to identify the music a participant was listening to with a mean rank accuracy of 59.2% ([Formula: see text], [Formula: see text]). This demonstrates that our decoding model may use fMRI-informed source analysis to aid EEG based decoding and reconstruction of acoustic information from brain activity and makes a step towards building EEG-based neural decoders for other complex information domains such as other acoustic, visual, or semantic information.

An Evaluation of Hybrid Deep Learning Models for Classifying Multiple Lower Limb Actions.

Brain-computer Interfaces (BCIs) interpret electroencephalography (EEG) signals and translate them into control commands for operating external devices. The motor imagery (MI) paradigm is popular in this context. Recent research has demonstrated that deep learning models, such as convolutional neural network (CNN) and long short-term memory (LSTM), are successful in a wide range of classification applications. This is because CNN has the property of spatial invariance, and LSTM can capture temporal associations among features. A combination of CNN and LSTM could enhance the classification performance of EEG signals due to the complementation of their strengths. Such a combination has been applied to MI classification based on EEG. However, most studies focused on either the upper limbs or treated both lower limbs as a single class, with only limited research performed on separate lower limbs. We, therefore, explored hybrid models (different combinations of CNN and LSTM) and evaluated them in the case of individual lower limbs. In addition, we classified multiple actions: MI, real movements and movement observations using four typical hybrid models and aimed to identify which model was the most suitable. The comparison results demonstrated that no model was significantly better than the others in terms of classification accuracy, but all of them were better than the chance level. Our study informs the possibility of the use of multiple actions in BCI systems and provides useful information for further research into the classification of separate lower limb actions.

Classification of Motor Imagery Based on Multi-Scale Feature Extraction and the Channel-Temporal Attention Module.

Motor imagery (MI) is a popular paradigm for controlling electroencephalogram (EEG) based Brain-Computer Interface (BCI) systems. Many methods have been developed to attempt to accurately classify MI-related EEG activity. Recently, the development of deep learning has begun to draw increasing attention in the BCI research community because it does not need to use sophisticated signal preprocessing and can automatically extract features. In this paper, we propose a deep learning model for use in MI-based BCI systems. Our model makes use of a convolutional neural network based on a multi-scale and channel-temporal attention module (CTAM), which called MSCTANN. The multi-scale module is able to extract a large number of features, while the attention module includes both a channel attention module and a temporal attention module, which together allow the model to focus attention on the most important features extracted from the data. The multi-scale module and the attention module are connected by a residual module, which avoids the degradation of the network. Our network model is built from these three core modules, which combine to improve the recognition ability of the network for EEG signals. Our experimental results on three datasets (BCI competition IV 2a, III IIIa and IV 1) show that our proposed method has better performance than other state-of-the-art methods, with accuracy rates of 80.6%, 83.56% and 79.84%. Our model has stable performance in decoding EEG signals and achieves efficient classification performance while using fewer network parameters than other comparable state-of-the-art methods.

Novel channel selection model based on graph convolutional network for motor imagery.

Multi-channel electroencephalography (EEG) is used to capture features associated with motor imagery (MI) based brain-computer interface (BCI) with a wide spatial coverage across the scalp. However, redundant EEG channels are not conducive to improving BCI performance. Therefore, removing irrelevant channels can help improve the classification performance of BCI systems. We present a new method for identifying relevant EEG channels. Our method is based on the assumption that useful channels share related information and that this can be measured by inter-channel connectivity. Specifically, we treat all candidate EEG channels as a graph and define channel selection as the problem of node classification on a graph. Then we design a graph convolutional neural network (GCN) model for channels classification. Channels are selected based on the outputs of our GCN model. We evaluate our proposed GCN-based channel selection (GCN-CS) method on three MI datasets. On three datasets, GCN-CS achieves performance improvements by reducing the number of channels. Specifically, we achieve classification accuracies of 79.76% on Dataset 1, 89.14% on Dataset 2 and 87.96% on Dataset 3, which outperform competing methods significantly.

Feature learning framework based on EEG graph self-attention networks for motor imagery BCI systems.

Learning distinguishable features from raw EEG signals is crucial for accurate classification of motor imagery (MI) tasks. To incorporate spatial relationships between EEG sources, we developed a feature set based on an EEG graph. In this graph, EEG channels represent the nodes, with power spectral density (PSD) features defining their properties, and the edges preserving the spatial information. We designed an EEG based graph self-attention network (EGSAN) to learn low-dimensional embedding vector for EEG graph, which can be used as distinguishable features for motor imagery task classification. We evaluated our EGSAN model on two publicly available MI EEG datasets, each containing different types of motor imagery tasks. Our experiments demonstrate that our proposed model effectively extracts distinguishable features from EEG graphs, achieving significantly higher classification accuracies than existing state-of-the-art methods.

Relationship between electrical brain responses to motor imagery and motor impairment in stroke.

BACKGROUND AND PURPOSE: New strategies like motor imagery based brain-computer interfaces, which use brain signals such as event-related desynchronization (ERD) or event-related synchronization (ERS) for motor rehabilitation after a stroke, are undergoing investigation. However, little is known about the relationship between ERD and ERS patterns and the degree of stroke impairment. The aim of this work was to clarify this relationship. METHODS: EEG during motor imagery and execution were measured in 29 patients with first-ever monolateral stroke causing any degree of motor deficit in the upper limb. The strength and laterality of the ERD or ERS patterns were correlated with the scores of the European Stroke Scale, the Medical Research Council, and the Modified Ashworth Scale. RESULTS: Mean age of the patients was 58 ± 15 years; mean time from the incident was 4 ± 4 months. Stroke lesions were cortical (n=8), subcortical (n=11), or mixed (n=10), attributable to either an ischemic event (n=26) or a hemorrhage (n=3), affecting the right (n=16) or left (n=13) hemisphere. Higher impairment was related to stronger ERD in the unaffected hemisphere and higher spasticity was related to stronger ERD in the affected hemisphere. Both were related to a relatively stronger ERS in the affected hemisphere. CONCLUSIONS: The results of this study may have implications for the design of potential poststroke rehabilitation interventions based on brain-computer interface technologies that use neurophysiological signals like ERD or ERS as neural substrates for the mutual interaction between brain and machine and, ultimately, help stroke patients to regain motor control.

Neural decoding of semantic concepts: a systematic literature review.

Objective.Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding.Approach.We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity.Main results.Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area.Significance.Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Feature Extraction Method Based on Filter Banks and Riemannian Tangent Space in Motor-Imagery BCI.

Optimal feature extraction for multi-category motor imagery brain-computer interfaces (MI-BCIs) is a research hotspot. The common spatial pattern (CSP) algorithm is one of the most widely used methods in MI-BCIs. However, its performance is adversely affected by variance in the operational frequency band and noise interference. Furthermore, the performance of CSP is not satisfactory when addressing multi-category classification problems. In this work, we propose a fusion method combining Filter Banks and Riemannian Tangent Space (FBRTS) in multiple time windows. FBRTS uses multiple filter banks to overcome the problem of variance in the operational frequency band. It also applies the Riemannian method to the covariance matrix extracted by the spatial filter to obtain more robust features in order to overcome the problem of noise interference. In addition, we use a One-Versus-Rest support vector machine (OVR-SVM) model to classify multi-category features. We evaluate our FBRTS method using BCI competition IV dataset 2a and 2b. The experimental results show that the average classification accuracy of our FBRTS method is 77.7% and 86.9% in datasets 2a and 2b respectively. By analyzing the influence of the different numbers of filter banks and time windows on the performance of our FBRTS method, we can identify the optimal number of filter banks and time windows. Additionally, our FBRTS method can obtain more distinctive features than the filter banks common spatial pattern (FBCSP) method in two-dimensional embedding space. These results show that our proposed method can improve the performance of MI-BCIs.

Bispectrum-based hybrid neural network for motor imagery classification.

BACKGROUND: The performance of motor imagery electroencephalogram (MI-EEG) decoding systems is easily affected by noise. As a higher-order spectra (HOS), the bispectrum is capable of suppressing Gaussian noise and increasing the signal-to-noise ratio of signals. However, the sum of logarithmic amplitudes (SLA) and the first order spectral moment (FOSM) features extracted from the bispectrum only use the numerical values of the bispectrum, ignoring the related information between different frequency bins. NEW METHOD: In this study, we proposed a novel framework, termed a bispectrum-based hybrid neural network (BHNN), to make full use of bispectrum for improving the performance of the MI-based brain-computer interface (BCI). Specifically, the BHNN consisted of a convolutional neural network (CNN), gated recurrent units (GRU), and squeeze-and-excitation (SE) modules. The SE modules and CNNs are first used to learn the deep relation between frequency bins of the bispectrum estimated from different time window segmentations of the MI-EEG. Then, we used GRU to seek the overlooked sequential information of the bispectrum. RESULTS: To validate the effectiveness of the proposed BHNN, three public BCI competition datasets were used in this study. The results demonstrated that the BHNN can achieve promising performance in decoding MI-EEG. COMPARISON WITH EXISTING METHODS: The statistical test results demonstrated that the proposed BHNN can significantly outperform other competing methods (p < =0.05). CONCLUSION: The proposed BHNN is a novel bispectrum-based neural network, which can enhance the decoding performance of MI-based BCIs.

A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery Based Brain-Computer Interface.

The motor imagery (MI) based brain-computer interfaces (BCIs) have been proposed as a potential physical rehabilitation technology. However, the low classification accuracy achievable with MI tasks is still a challenge when building effective BCI systems. We propose a novel MI classification model based on measurement of functional connectivity between brain regions and graph theory. Specifically, motifs describing local network structures in the brain are extracted from functional connectivity graphs. A graph embedding model called Ego-CNNs is then used to build a classifier, which can convert the graph from a structural representation to a fixed-dimensional vector for detecting critical structure in the graph. We validate our proposed method on four datasets, and the results show that our proposed method produces high classification accuracies in two-class classification tasks (92.8% for dataset 1, 93.4% for dataset 2, 96.5% for dataset 3, and 80.2% for dataset 4) and multiclass classification tasks (90.33% for dataset 1). Our proposed method achieves a mean Kappa value of 0.88 across nine participants, which is superior to other methods we compared it to. These results indicate that there is a local structural difference in functional connectivity graphs extracted under different motor imagery tasks. Our proposed method has great potential for motor imagery classification in future studies.

SincNet-Based Hybrid Neural Network for Motor Imagery EEG Decoding.

It is difficult to identify optimal cut-off frequencies for filters used with the common spatial pattern (CSP) method in motor imagery (MI)-based brain-computer interfaces (BCIs). Most current studies choose filter cut-frequencies based on experience or intuition, resulting in sub-optimal use of MI-related spectral information in the electroencephalography (EEG). To improve information utilization, we propose a SincNet-based hybrid neural network (SHNN) for MI-based BCIs. First, raw EEG is segmented into different time windows and mapped into the CSP feature space. Then, SincNets are used as filter bank band-pass filters to automatically filter the data. Next, we used squeeze-and-excitation modules to learn a sparse representation of the filtered data. The resulting sparse data were fed into convolutional neural networks to learn deep feature representations. Finally, these deep features were fed into a gated recurrent unit module to seek sequential relations, and a fully connected layer was used for classification. We used the BCI competition IV datasets 2a and 2b to verify the effectiveness of our SHNN method. The mean classification accuracies (kappa values) of our SHNN method are 0.7426 (0.6648) on dataset 2a and 0.8349 (0.6697) on dataset 2b, respectively. The statistical test results demonstrate that our SHNN can significantly outperform other state-of-the-art methods on these datasets.

Sonic enhancement of virtual exhibits.

Museums have widely embraced virtual exhibits. However, relatively little attention is paid to how sound may create a more engaging experience for audiences. To begin addressing this lacuna, we conducted an online experiment to explore how sound influences the interest level, emotional response, and engagement of individuals who view objects within a virtual exhibit. As part of this experiment, we designed a set of different soundscapes, which we presented to participants who viewed museum objects virtually. We then asked participants to report their felt affect and level of engagement with the exhibits. Our results show that soundscapes customized to exhibited objects significantly enhance audience engagement. We also found that more engaged audience members were more likely to want to learn additional information about the object(s) they viewed and to continue viewing these objects for longer periods of time. Taken together, our findings suggest that virtual museum exhibits can improve visitor engagement through forms of customized soundscape design.

Removal of physiological artifacts from simultaneous EEG and fMRI recordings.

OBJECTIVE: Simultaneous recording of the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) allows a combination of eletrophysiological and haemodynamic information to be used to form a more complete picture of cerebral dynamics. However, EEG recorded within the MRI scanner is contaminated by both imaging artifacts and physiological artifacts. The majority of the techniques used to pre-process such EEG focus on removal of the imaging and balistocardiogram artifacts, with some success, but don't remove all other physiological artifacts. METHODS: We propose a new offline EEG artifact removal method based upon a combination of independent component analysis and fMRI-based head movement estimation to aid the removal of physiological artifacts from EEG recorded during EEG-fMRI recordings. Our method makes novel use of head movement trajectories estimated from the fMRI recording in order to assist with identifying physiological artifacts in the EEG and is designed to be used after removal of the fMRI imaging artifact from the EEG. RESULTS: We evaluate our method on EEG recorded during a joint EEG-fMRI session from healthy adult participants. Our method significantly reduces the influence of all types of physiological artifacts on the EEG. We also compare our method with a state-of-the-art physiological artifact removal method and demonstrate superior performance removing physiological artifacts. CONCLUSIONS: Our proposed method is able to remove significantly more physiological artifact components from the EEG, recorded during a joint EEG-fMRI session, than other state-of-the-art methods. SIGNIFICANCE: Our proposed method represents a marked improvement over current processing pipelines for removing physiological noise from EEG recorded during a joint EEG-fMRI session.

Neural component analysis: A spatial filter for electroencephalogram analysis.

BACKGROUND: Spatial filtering and source separation are valuable tools in the analysis of EEG data. However, despite the well-known spatial localisation of individual cognitive processes within the brain, the available methods for source separation, such as the widely used blind source separation technique, do not take into account the spatial distributions and locations of sources. This can result in sub-optimal source identification. NEW METHOD: We present a new method for deriving a spatial filter for EEG data that attempts to identify sources that are maximally spatially distinct from one another in terms of the spatial distributions of their projections. RESULTS: We first evaluate our method with simulated EEG and show that it is able to separate EEG signals into components with distinct spatial distributions that closely resemble the original simulated sources. We also evaluate our method with real EEG and show it is able to identify a spatial filter that can be used to significantly improve classification accuracy of the P300 event-related potential (ERP). COMPARISON WITH EXISTING METHODS: We compare our method to a state of the art blind source separation methods, fast independent component analysis (ICA) and common spatial patterns (CSP). We evaluate the methods suitability for a common source separation application, analysis of ERPs. CONCLUSIONS: Our results show that our method is well suited to identifying spatial filters for EEG analysis. This has potential applications in a wide range of EEG signal processing applications.

How to build a fast and accurate code-modulated brain-computer interface.

Objective.In the last decade, the advent of code-modulated brain-computer interfaces (BCIs) has allowed the implementation of systems with high information transfer rates (ITRs) and increased the possible practicality of such interfaces. In this paper, we evaluate the effect of different numbers of targets in the stimulus display, modulation sequences generators, and signal processing algorithms on the accuracy and ITR of code-modulated BCIs.Approach.We use both real and simulated electroencephalographic (EEG) data, to evaluate these parameters and methods. Then, we compared numerous different setups to assess their performance and identify the best configurations. We also evaluated the dependability of our simulated evaluation approach.Main results.Our results show that Golay, almost perfect, and deBruijn sequence-based visual stimulus modulations provide the best results, significantly outperforming the commonly used m-sequences in all cases. We conclude that artificial neural network processing algorithms offer the best processing pipeline for this type of BCI, achieving a maximum classification accuracy of 94.7% on real EEG data while obtaining a maximum ITR of 127.2 bits min-1in a simulated 64-target system.Significance.We used a simulated framework that demonstrated previously unattainable flexibility and convenience while staying reasonably realistic. Furthermore, our findings suggest several new considerations which can be used to guide further code-based BCI development.

Decoding of semantic categories of imagined concepts of animals and tools in fNIRS.

Objective.Semantic decoding refers to the identification of semantic concepts from recordings of an individual's brain activity. It has been previously reported in functional magnetic resonance imaging and electroencephalography. We investigate whether semantic decoding is possible with functional near-infrared spectroscopy (fNIRS). Specifically, we attempt to differentiate between the semantic categories of animals and tools. We also identify suitable mental tasks for potential brain-computer interface (BCI) applications.Approach.We explore the feasibility of a silent naming task, for the first time in fNIRS, and propose three novel intuitive mental tasks based on imagining concepts using three sensory modalities: visual, auditory, and tactile. Participants are asked to visualize an object in their minds, imagine the sounds made by the object, and imagine the feeling of touching the object. A general linear model is used to extract hemodynamic responses that are then classified via logistic regression in a univariate and multivariate manner.Main results.We successfully classify all tasks with mean accuracies of 76.2% for the silent naming task, 80.9% for the visual imagery task, 72.8% for the auditory imagery task, and 70.4% for the tactile imagery task. Furthermore, we show that consistent neural representations of semantic categories exist by applying classifiers across tasks.Significance.These findings show that semantic decoding is possible in fNIRS. The study is the first step toward the use of semantic decoding for intuitive BCI applications for communication.