Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning.

Human activity recognition (HAR) based on the wearable device has attracted more attention from researchers with sensor technology development in recent years. However, personalized HAR requires high accuracy of recognition, while maintaining the model's generalization capability is a major challenge in this field. This paper designed a compact wireless wearable sensor node, which combines an air pressure sensor and inertial measurement unit (IMU) to provide multi-modal information for HAR model training. To solve personalized recognition of user activities, we propose a new transfer learning algorithm, which is a joint probability domain adaptive method with improved pseudo-labels (IPL-JPDA). This method adds the improved pseudo-label strategy to the JPDA algorithm to avoid cumulative errors due to inaccurate initial pseudo-labels. In order to verify our equipment and method, we use the newly designed sensor node to collect seven daily activities of 7 subjects. Nine different HAR models are trained by traditional machine learning and transfer learning methods. The experimental results show that the multi-modal data improve the accuracy of the HAR system. The IPL-JPDA algorithm proposed in this paper has the best performance among five HAR models, and the average recognition accuracy of different subjects is 93.2%.

Motor Imagery Classification for Asynchronous EEG-Based Brain-Computer Interfaces.

Motor imagery (MI) based brain-computer interfaces (BCIs) enable the direct control of external devices through the imagined movements of various body parts. Unlike previous systems that used fixed-length EEG trials for MI decoding, asynchronous BCIs aim to detect the user's MI without explicit triggers. They are challenging to implement, because the algorithm needs to first distinguish between resting-states and MI trials, and then classify the MI trials into the correct task, all without any triggers. This paper proposes a sliding window prescreening and classification (SWPC) approach for MI-based asynchronous BCIs, which consists of two modules: a prescreening module to screen MI trials out of the resting-state, and a classification module for MI classification. Both modules are trained with supervised learning followed by self-supervised learning, which refines the feature extractors. Within-subject and cross-subject asynchronous MI classifications on four different EEG datasets validated the effectiveness of SWPC, i.e., it always achieved the highest average classification accuracy, and outperformed the best state-of-the-art baseline on each dataset by about 2%.

Alignment-Based Adversarial Training (ABAT) for Improving the Robustness and Accuracy of EEG-Based BCIs.

Machine learning has achieved great success in electroencephalogram (EEG) based brain-computer interfaces (BCIs). Most existing BCI studies focused on improving the decoding accuracy, with only a few considering the adversarial security. Although many adversarial defense approaches have been proposed in other application domains such as computer vision, previous research showed that their direct extensions to BCIs degrade the classification accuracy on benign samples. This phenomenon greatly affects the applicability of adversarial defense approaches to EEG-based BCIs. To mitigate this problem, we propose alignment-based adversarial training (ABAT), which performs EEG data alignment before adversarial training. Data alignment aligns EEG trials from different domains to reduce their distribution discrepancies, and adversarial training further robustifies the classification boundary. The integration of data alignment and adversarial training can make the trained EEG classifiers simultaneously more accurate and more robust. Experiments on five EEG datasets from two different BCI paradigms (motor imagery classification, and event related potential recognition), three convolutional neural network classifiers (EEGNet, ShallowCNN and DeepCNN) and three different experimental settings (offline within-subject cross-block/-session classification, online cross-session classification, and pre-trained classifiers) demonstrated its effectiveness. It is very intriguing that adversarial attacks, which are usually used to damage BCI systems, can be used in ABAT to simultaneously improve the model accuracy and robustness.

Deep Source Semi-Supervised Transfer Learning (DS3TL) for Cross-Subject EEG Classification.

OBJECTIVE: An electroencephalogram (EEG) based brain-computer interface (BCI) maps the user's EEG signals into commands for external device control. Usually a large amount of labeled EEG trials are required to train a reliable EEG recognition model. However, acquiring labeled EEG data is time-consuming and user-unfriendly. Semi-supervised learning (SSL) and transfer learning can be used to exploit the unlabeled data and the auxiliary data, respectively, to reduce the amount of labeled data for a new subject. METHODS: This paper proposes deep source semi-supervised transfer learning (DS3TL) for EEG-based BCIs, which assumes the source subject has a small number of labeled EEG trials and a large number of unlabeled ones, whereas all EEG trials from the target subject are unlabeled. DS3TL mainly includes a hybrid SSL module, a weakly-supervised contrastive module, and a domain adaptation module. The hybrid SSL module integrates pseudo-labeling and consistency regularization for SSL. The weakly-supervised contrastive module performs contrastive learning by using the true labels of the labeled data and the pseudo-labels of the unlabeled data. The domain adaptation module reduces the individual differences by uncertainty reduction. RESULTS: Experiments on three EEG datasets from different tasks demonstrated that DS3TL outperformed a supervised learning baseline with many more labeled training data, and multiple state-of-the-art SSL approaches with the same number of labeled data. SIGNIFICANCE: To our knowledge, this is the first approach in EEG-based BCIs that exploits the unlabeled source data for more accurate target classifier training.

T-TIME: Test-Time Information Maximization Ensemble for Plug-and-Play BCIs.

OBJECTIVE: An electroencephalogram (EEG)-based brain-computer interface (BCI) enables direct communication between the human brain and a computer. Due to individual differences and non-stationarity of EEG signals, such BCIs usually require a subject-specific calibration session before each use, which is time-consuming and user-unfriendly. Transfer learning (TL) has been proposed to shorten or eliminate this calibration, but existing TL approaches mainly consider offline settings, where all unlabeled EEG trials from the new user are available. METHODS: This article proposes Test-Time Information Maximization Ensemble (T-TIME) to accommodate the most challenging online TL scenario, where unlabeled EEG data from the new user arrive in a stream, and immediate classification is performed. T-TIME initializes multiple classifiers from the aligned source data. When an unlabeled test EEG trial arrives, T-TIME first predicts its labels using ensemble learning, and then updates each classifier by conditional entropy minimization and adaptive marginal distribution regularization. Our code is publicized. RESULTS: Extensive experiments on three public motor imagery based BCI datasets demonstrated that T-TIME outperformed about 20 classical and state-of-the-art TL approaches. SIGNIFICANCE: To our knowledge, this is the first work on test time adaptation for calibration-free EEG-based BCIs, making plug-and-play BCIs possible.

Channel reflection: Knowledge-driven data augmentation for EEG-based brain-computer interfaces.

A brain-computer interface (BCI) enables direct communication between the human brain and external devices. Electroencephalography (EEG) based BCIs are currently the most popular for able-bodied users. To increase user-friendliness, usually a small amount of user-specific EEG data are used for calibration, which may not be enough to develop a pure data-driven decoding model. To cope with this typical calibration data shortage challenge in EEG-based BCIs, this paper proposes a parameter-free channel reflection (CR) data augmentation approach that incorporates prior knowledge on the channel distributions of different BCI paradigms in data augmentation. Experiments on eight public EEG datasets across four different BCI paradigms (motor imagery, steady-state visual evoked potential, P300, and seizure classifications) using different decoding algorithms demonstrated that: (1) CR is effective, i.e., it can noticeably improve the classification accuracy; (2) CR is robust, i.e., it consistently outperforms existing data augmentation approaches in the literature; and, (3) CR is flexible, i.e., it can be combined with other data augmentation approaches to further improve the performance. We suggest that data augmentation approaches like CR should be an essential step in EEG-based BCIs. Our code is available online.

Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification.

Cross-subject electroencephalogram (EEG) based seizure subtype classification is very important in precise epilepsy diagnostics. Deep learning is a promising solution, due to its ability to automatically extract latent patterns. However, it usually requires a large amount of training data, which may not always be available in clinical practice. This paper proposes Multi-Branch Mutual-Distillation (MBMD) Transformer for cross-subject EEG-based seizure subtype classification, which can be effectively trained from small labeled data. MBMD Transformer replaces all even-numbered encoder blocks of the vanilla Vision Transformer by our designed multi-branch encoder blocks. A mutual-distillation strategy is proposed to transfer knowledge between the raw EEG data and its wavelets of different frequency bands. Experiments on two public EEG datasets demonstrated that our proposed MBMD Transformer outperformed several traditional machine learning and state-of-the-art deep learning approaches. To our knowledge, this is the first work on knowledge distillation for EEG-based seizure subtype classification.

Source-Free Domain Adaptation (SFDA) for Privacy-Preserving Seizure Subtype Classification.

Electroencephalogram (EEG) based seizure subtype classification is very important in clinical diagnostics. Source-free domain adaptation (SFDA) uses a pre-trained source model, instead of the source data, for privacy-preserving transfer learning. SFDA is useful in seizure subtype classification, which can protect the privacy of the source patients, while reducing the amount of labeled calibration data for a new patient. This paper introduces semi-supervised transfer boosting (SS-TrBoosting), a boosting-based SFDA approach for seizure subtype classification. We further extend it to unsupervised transfer boosting (U-TrBoosting) for unsupervised SFDA, i.e., the new patient does not need any labeled EEG data. Experiments on three public seizure datasets demonstrated that SS-TrBoosting and U-TrBoosting outperformed multiple classical and state-of-the-art machine learning approaches in cross-dataset/cross-patient seizure subtype classification.

Small Data Least-Squares Transformation (sd-LST) for Fast Calibration of SSVEP-Based BCIs.

Steady-state visual evoked potential (SSVEP) is one of the most popular brain-computer interface (BCI) paradigms, with high information transmission rate and signal-to-noise ratio. Many calibration-free and calibration-based approaches have been proposed to improve the performance of SSVEP-based BCIs. This paper considers a quick calibration scenario, where there are plenty of data from multiple source subjects, but only a small number of calibration trials from a subset of stimulus frequencies for the new subject. We propose small data least-squares transformation (sd-LST) to solve this problem. Experiments on three publicly available SSVEP datasets demonstrated that sd-LST outperformed several classical or state-of-the-art approaches, with only about 10 calibration trials for 40-target SSVEP-based BCI spellers.

User Identity Protection in EEG-Based Brain-Computer Interfaces.

A brain-computer interface (BCI) establishes a direct communication pathway between the brain and an external device. Electroencephalogram (EEG) is the most popular input signal in BCIs, due to its convenience and low cost. Most research on EEG-based BCIs focuses on the accurate decoding of EEG signals; however, EEG signals also contain rich private information, e.g., user identity, emotion, and so on, which should be protected. This paper first exposes a serious privacy problem in EEG-based BCIs, i.e., the user identity in EEG data can be easily learned so that different sessions of EEG data from the same user can be associated together to more reliably mine private information. To address this issue, we further propose two approaches to convert the original EEG data into identity-unlearnable EEG data, i.e., removing the user identity information while maintaining the good performance on the primary BCI task. Experiments on seven EEG datasets from five different BCI paradigms showed that on average the generated identity-unlearnable EEG data can reduce the user identification accuracy from 70.01% to at most 21.36%, greatly facilitating user privacy protection in EEG-based BCIs.

Front-End Replication Dynamic Window (FRDW) for Online Motor Imagery Classification.

Motor imagery (MI) is a classical paradigm in electroencephalogram (EEG) based brain-computer interfaces (BCIs). Online accurate and fast decoding is very important to its successful applications. This paper proposes a simple yet effective front-end replication dynamic window (FRDW) algorithm for this purpose. Dynamic windows enable the classification based on a test EEG trial shorter than those used in training, improving the decision speed; front-end replication fills a short test EEG trial to the length used in training, improving the classification accuracy. Within-subject and cross-subject online MI classification experiments on three public datasets, with three different classifiers and three different data augmentation approaches, demonstrated that FRDW can significantly increase the information transfer rate in MI decoding. Additionally, FR can also be used in training data augmentation. FRDW helped win national champion of the China BCI Competition in 2022.

Unsupervised domain adaptation for cross-patient seizure classification.

Objective. Epileptic seizure is a chronic neurological disease affecting millions of patients. Electroencephalogram (EEG) is the gold standard in epileptic seizure classification. However, its low signal-to-noise ratio, strong non-stationarity, and large individual difference nature make it difficult to directly extend the seizure classification model from one patient to another. This paper considers multi-source unsupervised domain adaptation for cross-patient EEG-based seizure classification, i.e. there are multiple source patients with labeled EEG data, which are used to label the EEG trials of a new patient.Approach. We propose an source domain selection (SDS)-global domain adaptation (GDA)-target agent subdomain adaptation (TASA) approach, which includes SDS to filter out dissimilar source domains, GDA to align the overall distributions of the selected source domains and the target domain, and TASA to identify the most similar source domain to the target domain so that its labels can be utilized.Main results. Experiments on two public seizure datasets demonstrated that SDS-GDA-TASA outperformed 13 existing approaches in unsupervised cross-patient seizure classification.Significance. Our approach could save clinicians plenty of time in labeling EEG data for epilepsy patients, greatly increasing the efficiency of seizure diagnostics.

Mixture of Experts for EEG-Based Seizure Subtype Classification.

Epilepsy is a pervasive neurological disorder affecting approximately 50 million individuals worldwide. Electroencephalogram (EEG) based seizure subtype classification plays a crucial role in epilepsy diagnosis and treatment. However, automatic seizure subtype classification faces at least two challenges: 1) class imbalance, i.e., certain seizure types are considerably less common than others; and 2) no a priori knowledge integration, so that a large number of labeled EEG samples are needed to train a machine learning model, particularly, deep learning. This paper proposes two novel Mixture of Experts (MoE) models, Seizure-MoE and Mix-MoE, for EEG-based seizure subtype classification. Particularly, Mix-MoE adequately addresses the above two challenges: 1) it introduces a novel imbalanced sampler to address significant class imbalance; and 2) it incorporates a priori knowledge of manual EEG features into the deep neural network to improve the classification performance. Experiments on two public datasets demonstrated that the proposed Seizure-MoE and Mix-MoE outperformed multiple existing approaches in cross-subject EEG-based seizure subtype classification. Our proposed MoE models may also be easily extended to other EEG classification problems with severe class imbalance, e.g., sleep stage classification.

BoostTree and BoostForest for Ensemble Learning.

Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This article proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It increases the randomness (diversity) by drawing the cut-points randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest generally outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 35 classification and regression datasets. Remarkably, BoostForest tunes its parameters by simply sampling them randomly from a parameter pool, which can be easily specified, and its ensemble learning framework can also be used to combine many other base learners.

Multi-Modality Fusion & Inductive Knowledge Transfer Underlying Non-Sparse Multi-Kernel Learning and Distribution Adaption.

With the development of sensors, more and more multimodal data are accumulated, especially in biomedical and bioinformatics fields. Therefore, multimodal data analysis becomes very important and urgent. In this study, we combine multi-kernel learning and transfer learning, and propose a feature-level multi-modality fusion model with insufficient training samples. To be specific, we firstly extend kernel Ridge regression to its multi-kernel version under the lp-norm constraint to explore complementary patterns contained in multimodal data. Then we use marginal probability distribution adaption to minimize the distribution differences between the source domain and the target domain to solve the problem of insufficient training samples. Based on epilepsy EEG data provided by the University of Bonn, we construct 12 multi-modality & transfer scenarios to evaluate our model. Experimental results show that compared with baselines, our model performs better on most scenarios.

EEG-Based Brain-Computer Interfaces are Vulnerable to Backdoor Attacks.

Research and development of electroencephalogram (EEG) based brain-computer interfaces (BCIs) have advanced rapidly, partly due to deeper understanding of the brain and wide adoption of sophisticated machine learning approaches for decoding the EEG signals. However, recent studies have shown that machine learning algorithms are vulnerable to adversarial attacks. This paper proposes to use narrow period pulse for poisoning attack of EEG-based BCIs, which makes adversarial attacks much easier to implement. One can create dangerous backdoors in the machine learning model by injecting poisoning samples into the training set. Test samples with the backdoor key will then be classified into the target class specified by the attacker. What most distinguishes our approach from previous ones is that the backdoor key does not need to be synchronized with the EEG trials, making it very easy to implement. The effectiveness and robustness of the backdoor attack approach is demonstrated, highlighting a critical security concern for EEG-based BCIs and calling for urgent attention to address it.

Privacy-Preserving Domain Adaptation for Motor Imagery-Based Brain-Computer Interfaces.

OBJECTIVE: Electroencephalogram (EEG) is one of the most widely used signals in motor imagery (MI) based brain-computer interfaces (BCIs). Domain adaptation has been frequently used to improve the accuracy of EEG-based BCIs for a new user (target domain), by making use of labeled data from a previous user (source domain). However, this raises privacy concerns, as EEG contains sensitive health and mental information. It is very important to perform privacy-preserving domain adaptation, which simultaneously improves the classification accuracy for a new user and protects the privacy of a previous user. METHODS: We propose augmentation-based source-free adaptation (ASFA), which consists of two parts: 1) source model training, where a novel data augmentation approach is proposed for MI EEG signals to improve the cross-subject generalization performance of the source model; and, 2) target model training, which simultaneously considers uncertainty reduction for domain adaptation and consistency regularization for robustness. ASFA only needs access to the source model parameters, instead of the raw EEG data, thus protecting the privacy of the source domain. We further extend ASFA to a stricter privacy-preserving scenario, where the source model's parameters are also inaccessible. RESULTS: Experimental results on four MI datasets demonstrated that ASFA outperformed 15 classical and state-of-the-art MI classification approaches. SIGNIFICANCE: This is the first work on completely source-free domain adaptation for EEG-based BCIs. Our proposed ASFA achieves high classification accuracy and strong privacy protection simultaneously, important for the commercial applications of EEG-based BCIs.

Transfer learning for motor imagery based brain-computer interfaces: A tutorial.

A brain-computer interface (BCI) enables a user to communicate directly with an external device, e.g., a computer, using brain signals. It can be used to research, map, assist, augment, or repair human cognitive or sensory-motor functions. A closed-loop BCI system performs signal acquisition, temporal filtering, spatial filtering, feature engineering and classification, before sending out the control signal to an external device. Transfer learning (TL) has been widely used in motor imagery (MI) based BCIs to reduce the calibration effort for a new subject, greatly increasing their utility. This tutorial describes how TL can be considered in as many components of a BCI system as possible, and introduces a complete TL pipeline for MI-based BCIs. Examples on two MI datasets demonstrated the advantages of considering TL in multiple components of MI-based BCIs. Especially, integrating data alignment and sophisticated TL approaches can significantly improve the classification performance, and hence greatly reduces the calibration effort.

Multi-Source Decentralized Transfer for Privacy-Preserving BCIs.

Transfer learning, which utilizes labeled source domains to facilitate the learning in a target model, is effective in alleviating high intra- and inter-subject variations in electroencephalogram (EEG) based brain-computer interfaces (BCIs). Existing transfer learning approaches usually use the source subjects' EEG data directly, leading to privacy concerns. This paper considers a decentralized privacy-preserving transfer learning scenario: there are multiple source subjects, whose data and computations are kept local, and only the parameters or predictions of their pre-trained models can be accessed for privacy-protection; then, how to perform effective cross-subject transfer for a new subject with unlabeled EEG trials? We propose an offline unsupervised multi-source decentralized transfer (MSDT) approach, which first generates a pre-trained model from each source subject, and then performs decentralized transfer using the source model parameters (in gray-box settings) or predictions (in black-box settings). Experiments on two datasets from two BCI paradigms, motor imagery and affective BCI, demonstrated that MSDT outperformed several existing approaches, which do not consider privacy-protection at all. In other words, MSDT achieved both high privacy-protection and better classification performance.

Intelligent Gait Analysis and Evaluation System Based on Cane Robot.

Gait analysis and evaluation are vital for disease diagnosis and rehabilitation. Current gait analysis technologies require wearable devices or high-resolution vision systems within a limited usage space. To facilitate gait analysis and quantitative walking-ability evaluation in daily environments without using wearable devices, a mobile gait analysis and evaluation system is proposed based on a cane robot. Two laser range finders (LRFs) are mounted to obtain the leg motion data. An effective high-dimensional Takagi-Sugeno-Kang (HTSK) fuzzy system, which is suitable for high-dimensional data by solving the saturation problem caused by softmax function in defuzzification, is proposed to recognize the walking states using only the motion data acquired from LRFs. The gait spatial-temporal parameters are then extracted based on the gait cycle segmented by different walking states. Besides, a quantitative walking-ability evaluation index is proposed in terms of the conventional Tinetti scale. The plantar pressure sensing system records the walking states to label training data sets. Experiments were conducted with seven healthy subjects and four patients. Compared with five classical classification algorithms, the proposed method achieves the average accuracy rate of 96.57%, which is improved more than 10%, compared with conventional Takagi-Sugeno-Kang (TSK) fuzzy system. Compared with the gait parameters extracted by the motion capture system OptiTrack, the average errors of step length and gait cycle are only 0.02 m and 1.23 s, respectively. The comparison between the evaluation results of the robot system and the scores given by the physician also validates that the proposed method can effectively evaluate the walking ability.