Cross-dataset transfer learning for motor imagery signal classification via multi-task learning and pre-training.

Objective.Deep learning (DL) models have been proven to be effective in decoding motor imagery (MI) signals in Electroencephalogram (EEG) data. However, DL models' success relies heavily on large amounts of training data, whereas EEG data collection is laborious and time-consuming. Recently, cross-dataset transfer learning has emerged as a promising approach to meet the data requirements of DL models. Nevertheless, transferring knowledge across datasets involving different MI tasks remains a significant challenge in cross-dataset transfer learning, limiting the full utilization of valuable data resources. APPROACH: This study proposes a pre-training-based cross-dataset transfer learning method inspired by Hard Parameter Sharing in multi-task learning. Different datasets with distinct MI paradigms are considered as different tasks, classified with shared feature extraction layers and individual task-specific layers to allow cross-dataset classification with one unified model. Then, Pre-training and fine-tuning are employed to transfer knowledge across datasets. We also designed four fine-tuning schemes and conducted extensive experiments on them. MAIN RESULTS: The results showed that compared to models without pre-training, models with pre-training achieved a maximum increase in accuracy of 7.76%. Moreover, when limited training data were available, the pre-training method significantly improved DL model's accuracy by 27.34% at most. The experiments also revealed that pre-trained models exhibit faster convergence and remarkable robustness. The training time per subject could be reduced by up to 102.83 s, and the variance of classification accuracy decreased by 75.22% at best. SIGNIFICANCE: This study represents the first comprehensive investigation of the cross-dataset transfer learning method between two datasets with different MI tasks. The proposed pre-training method requires only minimal fine-tuning data when applying DL models to new MI paradigms, making MI-Brain-computer interface more practical and user-friendly.

Study of the Correlation between the Motor Ability of the Individual Upper Limbs and Motor Imagery Induced Neural Activities.

Motor imagery based brain-computer interfaces (MI-BCIs) have excellent application prospects in motor enhancement and rehabilitation. However, MI-induced electroencephalogram features applied to MI-BCI usually vary from person to person. This study aimed to investigate whether the motor ability of the individual upper limbs was associated with these features, which helps understand the causes of inter-subject variability. We focused on the behavioral and psychological factors reflecting motor abilities. We first obtained the behavioral scale scores from Edinburgh Handedness Questionnaire, Maximum Grip Strength Test, and Purdue Pegboard Test assessments to evaluate the motor execution ability. We also required the subjects to complete the psychological Movement Imagery Questionnaire-3 estimate, representing MI ability. Then we recorded EEG signals from all twenty-two subjects during MI tasks. Pearson correlation coefficient and stepwise regression were used to analyze the relationships between MI-induced relative event-related desynchronization (rERD) patterns and motor abilities. Both Purdue Pegboard Test and Movement Imagery Questionnaire-3 scores had significant correlations with MI-induced neural oscillation patterns. Notably, the Purdue Pegboard Test of the left hand had the most significant correlation with the alpha rERD. The results of stepwise multiple regression analysis showed that the Purdue Pegboard Test and Movement Imagery Questionnaire-3 could best predict the MI-induced rERD. The results demonstrate that hand dexterity and fine motor coordination are significantly related to MI-induced neural activities. In addition, the method of imagining is also relevant to MI features. Therefore, this study is meaningful for understanding individual differences and the design of user-centered MI-BCI.

Incorporating EEG and fNIRS Patterns to Evaluate Cortical Excitability and MI-BCI Performance During Motor Training.

As electroencephalography (EEG) is nonlinear and nonstationary in nature, an imperative challenge for brain-computer interfaces (BCIs) is to construct a robust classifier that can survive for a long time and monitor the brain state stably. To this end, this research aims to improve BCI performance by incorporation of electroencephalographic and cerebral hemodynamic patterns. A motor imagery (MI)-BCI based visual-haptic neurofeedback training (NFT) experiment was designed with sixteen participants. EEG and functional near infrared spectroscopy (fNIRS) signals were simultaneously recorded before and after this transient NFT. Cortical activation was significantly improved after repeated and continuous NFT through time-frequency and topological analysis. A classifier calibration strategy, weighted EEG-fNIRS patterns (WENP), was proposed, in which elementary classifiers were constructed by using both the EEG and fNIRS information and then integrated into a strong classifier with their independent accuracy-based weight assessment. The results revealed that the classifier constructed on integrating EEG and fNIRS patterns was significantly superior to that only with independent information (  âˆ¼  10% and  âˆ¼  18% improvement respectively), reaching  âˆ¼  89% in mean classification accuracy. The WENP is a classifier calibration strategy that can effectively improve the performance of the MI-BCI and could also be used to other BCI paradigms. These findings validate that our proposed methods are feasible and promising for optimizing conventional motor training methods and clinical rehabilitation.

Motor Imagery and Action Observation Induced Electroencephalographic Activations to Guide Subject-Specific Training Paradigm: A Pilot Study.

Brain-computer interface (BCI)-based motor rehabilitation feedback training system can facilitate motor function reconstruction, but its rehabilitation mechanism with suitable training protocol is unclear, which affects the application effect. To this end, we probed the electroencephalographic (EEG) activations induced by motor imagery (MI) and action observation (AO) to provide an effective method to optimize motor feedback training. We grouped subjects according to their alpha-band sensorimotor cortical excitability under MI and AO conditions, and investigated the EEG response under the same paradigm between groups and different motor paradigms within group, respectively. The results showed that there were significant differences in sensorimotor activations between two groups of subjects. Specifically, the group with weaker MI induced EEG features, could achieve stronger sensorimotor activations in AO than that of other conditions. The group with stronger MI induced EEG features, could achieve stronger sensorimotor activations in the MI+AO than that of other conditions. We also explored their classification and brain network differences, which might try to explain the EEG mechanism in different individuals and help stroke patients to choose appropriate subject-specific motor training paradigm for their rehabilitation and better treatment outcomes.

Effects of Transcutaneous Vagus Nerve Stimulation (tVNS) on Action Planning: A Behavioural and EEG Study.

Action planning is an important decision-making process, which can be specially affected by environment. Response selection during action planning has been demonstrated to be modulated by tVNS. Therefore, tVNS shows a great potential for modulating the action planning process. We aimed to explore the tVNS-induced effect on action planning in behavioural and electrophysiology. Twenty-eight participants were randomly divided into two groups (active group and sham group). A single-blind, sham-controlled between-subject design was applied to explore the effect of online-tVNS (i.e., tVNS overlapping with the task) on action planning paradigm. We measured and compared reaction time (RT) and movement-related cortical potentials (MRCPs) before and after tVNS between active and sham groups. As compared to sham group, for the ipsilateral hand/contralateral hemisphere relative to the stimulated side, active tVNS significantly reduced the reaction time and decreased the MRCP amplitude mainly in the challenging tasks. Our results indicate that tVNS can produce a lateralization effect on action planning, especially plays an important role in the more challenging tasks as reflected both in the behavioural and electrophysiological results.

A Tensor-Based Frequency Features Combination Method for Brain-Computer Interfaces.

With the development of the brain-computer interface (BCI) community, motor imagery-based BCI system using electroencephalogram (EEG) has attracted increasing attention because of its portability and low cost. Concerning the multi-channel EEG, the frequency component is one of the most critical features. However, insufficient extraction hinders the development and application of MI-BCIs. To deeply mine the frequency information, we proposed a method called tensor-based frequency feature combination (TFFC). It combined tensor-to-vector projection (TVP), fast fourier transform (FFT), common spatial pattern (CSP) and feature fusion to construct a new feature set. With two datasets, we used different classifiers to compare TFFC with the state-of-the-art feature extraction methods. The experimental results showed that our proposed TFFC could robustly improve the classification accuracy of about 5% ( ). Moreover, visualization analysis implied that the TFFC was a generalization of CSP and Filter Bank CSP (FBCSP). Also, a complementarity between weighted narrowband features (wNBFs) and broadband features (BBFs) was observed from the averaged fusion ratio. This article certificates the importance of frequency information in the MI-BCI system and provides a new direction for designing a feature set of MI-EEG.

[Progress of classification algorithms for motor imagery electroencephalogram signals].

Motor imagery (MI), motion intention of the specific body without actual movements, has attracted wide attention in fields as neuroscience. Classification algorithms for motor imagery electroencephalogram (MI-EEG) signals are able to distinguish different MI tasks based on the physiological information contained by the EEG signals, especially the features extracted from them. In recent years, there have been some new advances in classification algorithms for MI-EEG signals in terms of classifiers versus machine learning strategies. In terms of classifiers, traditional machine learning classifiers have been improved by some researchers, deep learning and Riemannian geometry classifiers have been widely applied as well. In terms of machine learning strategies, ensemble learning, adaptive learning, and transfer learning strategies have been utilized to improve classification accuracies or reach other targets. This paper reviewed the progress of classification algorithms for MI-EEG signals, summarized and evaluated the existing classifiers and machine learning strategies, to provide new ideas for developing classification algorithms with higher performance.

Enhancing transfer performance across datasets for brain-computer interfaces using a combination of alignment strategies and adaptive batch normalization.

Objective. Recently, transfer learning (TL) and deep learning (DL) have been introduced to solve intra- and inter-subject variability problems in brain-computer interfaces (BCIs). However, current TL and DL algorithms are usually validated within a single dataset, assuming that data of the test subjects are acquired under the same condition as that of training (source) subjects. This assumption is generally violated in practice because of different acquisition systems and experimental settings across studies and datasets. Thus, the generalization ability of these algorithms needs further validations in a cross-dataset scenario, which is closer to the actual situation. This study compared the transfer performance of pre-trained deep-learning models with different preprocessing strategies in a cross-dataset scenario.Approach. This study used four publicly available motor imagery datasets, each was successively selected as a source dataset, and the others were used as target datasets. EEGNet and ShallowConvNet with four preprocessing strategies, namely channel normalization, trial normalization, Euclidean alignment, and Riemannian alignment, were trained with the source dataset. The transfer performance of pre-trained models was validated on the target datasets. This study also used adaptive batch normalization (AdaBN) for reducing interval covariate shift across datasets. This study compared the transfer performance of using the four preprocessing strategies and that of a baseline approach based on manifold embedded knowledge transfer (MEKT). This study also explored the possibility and performance of fusing MEKT and EEGNet.Main results. The results show that DL models with alignment strategies had significantly better transfer performance than the other two preprocessing strategies. As an unsupervised domain adaptation method, AdaBN could also significantly improve the transfer performance of DL models. The transfer performance of DL models that combined AdaBN and alignment strategies significantly outperformed MEKT. Moreover, the generalizability of EEGNet models that combined AdaBN and alignment strategies could be further improved via the domain adaptation step in MEKT, achieving the best generalization ability among multiple datasets (BNCI2014001: 0.788, PhysionetMI: 0.679, Weibo2014: 0.753, Cho2017: 0.650).Significance. The combination of alignment strategies and AdaBN could easily improve the generalizability of DL models without fine-tuning. This study may provide new insights into the design of transfer neural networks for BCIs by separating source and target batch normalization layers in the domain adaptation process.

Optimization of Task Allocation for Collaborative Brain-Computer Interface Based on Motor Imagery.

OBJECTIVE: Collaborative brain-computer interfaces (cBCIs) can make the BCI output more credible by jointly decoding concurrent brain signals from multiple collaborators. Current cBCI systems usually require all collaborators to execute the same mental tasks (common-work strategy). However, it is still unclear whether the system performance will be improved by assigning different tasks to collaborators (division-of-work strategy) while keeping the total tasks unchanged. Therefore, we studied a task allocation scheme of division-of-work and compared the corresponding classification accuracies with common-work strategy's. APPROACH: This study developed an electroencephalograph (EEG)-based cBCI which had six instructions related to six different motor imagery tasks (MI-cBCI), respectively. For the common-work strategy, all five subjects as a group had the same whole instruction set and they were required to conduct the same instruction at a time. For the division-of-work strategy, every subject's instruction set was a subset of the whole one and different from each other. However, their union set was equal to the whole set. Based on the number of instructions in a subset, we divided the division-of-work strategy into four types, called "2 Tasks" … "5 Tasks." To verify the effectiveness of these strategies, we employed EEG data collected from 19 subjects who independently performed six types of MI tasks to conduct the pseudo-online classification of MI-cBCI. MAIN RESULTS: Taking the number of tasks performed by one collaborator as the horizontal axis (two to six), the classification accuracy curve of MI-cBCI was mountain-like. The curve reached its peak at "4 Tasks," which means each subset contained four instructions. It outperformed the common-work strategy ("6 Tasks") in classification accuracy (72.29 ± 4.43 vs. 58.53 ± 4.36%). SIGNIFICANCE: The results demonstrate that our proposed task allocation strategy effectively enhanced the cBCI classification performance and reduced the individual workload.

EEG-controlled functional electrical stimulation rehabilitation for chronic stroke: system design and clinical application.

Stroke is one of the most serious diseases that threaten human life and health. It is a major cause of death and disability in the clinic. New strategies for motor rehabilitation after stroke are undergoing exploration. We aimed to develop a novel artificial neural rehabilitation system, which integrates brain-computer interface (BCI) and functional electrical stimulation (FES) technologies, for limb motor function recovery after stroke. We conducted clinical trials (including controlled trials) in 32 patients with chronic stroke. Patients were randomly divided into the BCI-FES group and the neuromuscular electrical stimulation (NMES) group. The changes in outcome measures during intervention were compared between groups, and the trends of ERD values based on EEG were analyzed for BCI-FES group. Results showed that the increase in Fugl Meyer Assessment of the Upper Extremity (FMA-UE) and Kendall Manual Muscle Testing (Kendall MMT) scores of the BCI-FES group was significantly higher than that in the sham group, which indicated the practicality and superiority of the BCI-FES system in clinical practice. The change in the laterality coefficient (LC) values based on µ-ERD (ΔLCm-ERD) had high significant positive correlation with the change in FMA-UE(r = 0.6093, P = 0.012), which provides theoretical basis for exploring novel objective evaluation methods.

[Research advancements of motor imagery for motor function recovery after stroke].

Neurological damage caused by stroke is one of the main causes of motor dysfunction in patients, which brings great spiritual and economic burdens for society and families. Motor imagery is an important assisting method for the rehabilitation of patients after stroke, which is easy to learn with low cost and has great significance in improving the motor function and the quality of patient's life. This paper mainly summarizes the positive effects of motor imagery on post-stroke rehabilitation, outlines the physiological performance and theoretical model of motor imagery, the influencing factors of motor imagery, the scoring criteria of motor imagery and analyzes the shortcomings such as the few kinds of experimental subject, the subjective evaluation method and the low resolution of the experimental equipment in the process of rehabilitation of motor function in post-stroke patients. It is hopeful that patients with stroke will be more scientifically and effectively using motor imagery therapy.

A BCI based visual-haptic neurofeedback training improves cortical activations and classification performance during motor imagery.

OBJECTIVE: We proposed a brain-computer interface (BCI) based visual-haptic neurofeedback training (NFT) by incorporating synchronous visual scene and proprioceptive electrical stimulation feedback. The goal of this work was to improve sensorimotor cortical activations and classification performance during motor imagery (MI). In addition, their correlations and brain network patterns were also investigated respectively. APPROACH: 64-channel electroencephalographic (EEG) data were recorded in nineteen healthy subjects during MI before and after NFT. During NFT sessions, the synchronous visual-haptic feedbacks were driven by real-time lateralized relative event-related desynchronization (lrERD). MAIN RESULTS: By comparison between previous and posterior control sessions, the cortical activations measured by multi-band (i.e. alpha_1: 8-10 Hz, alpha_2: 11-13 Hz, beta_1: 15-20 Hz and beta_2: 22-28 Hz) absolute ERD powers and lrERD patterns were significantly enhanced after the NFT. The classification performance was also significantly improved, achieving a ~9% improvement and reaching ~85% in mean classification accuracy from a relatively poor performance. Additionally, there were significant correlations between lrERD patterns and classification accuracies. The partial directed coherence based functional connectivity (FC) networks covering the sensorimotor area also showed an increase after the NFT. SIGNIFICANCE: These findings validate the feasibility of our proposed NFT to improve sensorimotor cortical activations and BCI performance during motor imagery. And it is promising to optimize conventional NFT manner and evaluate the effectiveness of motor training.

BCI Monitor Enhances Electroencephalographic and Cerebral Hemodynamic Activations During Motor Training.

Motor imagery-based brain-computer interface (MI-BCI) controlling functional electrical stimulation (FES) is promising for disabled patients to restore their motor functions. However, it remains unclear how much the BCI part can contribute to the functional coupling between the brain and muscle. Specifically, whether it can enhance the cerebral activation for motor training? Here, we investigate the electroencephalographic and cerebral hemodynamic responses for MI-BCI-FES training and MI-FES training, respectively. Twelve healthy subjects were recruited in the motor training study when concurrent electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were recorded. Compared with the MI-FES training conditions, the MI-BCI-FES could induce significantly stronger event-related desynchronization (ERD) and blood oxygen response, which demonstrates that BCI indeed plays a functional role in the closed-loop motor training. Therefore, this paper verifies the feasibility of using BCI to train motor functions in a closed-loop manner.

A visual-haptic neurofeedback training improves sensorimotor cortical activations and BCI performance.

Neurofeedback training (NFT) could provide a novel way to investigate or restore the impaired brain function and neuroplasticity. However, it remains unclear how much the different feedback modes can contribute to NFT training. Specifically, whether they can enhance the cortical activations for motor training. To this end, our study proposed a brain-computer interface (BCI) based visual-haptic NFT incorporating synchronous visual scene and proprioceptive electrical stimulation feedback. By comparison between previous and posterior control sessions, the cortical activations measured by multi-band (i.e. alpha_1: 8-10Hz, alpha_2: 11-13Hz, beta_1: 15-20Hz and beta_2: 22-28Hz) lateralized relative event-related desynchronization (lrERD) patterns were significantly enhanced after NFT. And the classification performance was also significantly improved, achieving a ~9% improvement and reaching ~85% in mean classification accuracy from a relatively low MI-BCI performance. These findings validate the feasibility of our proposed visual- haptic NFT approach to improve sensorimotor cortical activations and BCI performance during motor training.

Analysis and Classification for EEG Patterns of Force Motor Imagery Using Movement Related Cortical Potentials.

Motor imagery-based BCIs are the most natural human-computer interaction paradigms. In recent years, researchers have tried to decode the kinetic information of motor imagery. In this paper, we analyzed and discriminated the EEG patterns of different force levels motor imagery using MRCPs. In the experiment, nine healthy subjects were required to perform the hand force motor imagery tasks (30% MVC and 10% MVC). From the view of MRCPs, the most significant discrimination between the two levels of mental tasks was the manifestation of motor planning. The average classification accuracy for features involving both MRCP and CSP was 78.3%, which was 8.5% higher than the CSP-based features (p¡0.001) and 2% higher than the MRCP-based features. The results demonstrated the feasibility of using MRCPs for hand force motor imagery classification.

Randomly dividing homologous samples leads to overinflated accuracies for emotion recognition.

There are numerous studies measuring the brain emotional status by analyzing EEGs under the emotional stimuli that have occurred. However, they often randomly divide the homologous samples into training and testing groups, known as randomly dividing homologous samples (RDHS), despite considering the impact of the non-emotional information among them, which would inflate the recognition accuracy. This work proposed a modified method, the integrating homologous samples (IHS), where the homologous samples were either used to build a classifier, or to be tested. The results showed that the classification accuracy was much lower for the IHS than for the RDHS. Furthermore, a positive correlation was found between the accuracy and the overlapping rate of the homologous samples. These findings implied that the overinflated accuracy did exist in those previous studies where the RDHS method was employed for emotion recognition. Moreover, this study performed a feature selection for the IHS condition based on the support vector machine-recursive feature elimination, after which the average accuracies were greatly improved to 85.71% and 77.18% in the picture-induced and video-induced tasks, respectively.