Low-Quality Video Target Detection Based on EEG Signal Using Eye Movement Alignment.

The target detection based on electroencephalogram (EEG) signals is a new target detection method. This method recognizes the target by decoding the specific neural response when an operator observes the target, which has important theoretical and application values. This paper focuses on the EEG detection of low-quality video targets, which breaks through the limitation of previous target detection based on EEG signals only for high-quality video targets. We first design an experimental paradigm for EEG-based low-quality video target detection and propose an epoch extraction method based on eye movement signals to solve the asynchronous problem faced by low-quality video target detection. Then, the neural representation in the process of operator recognition is analyzed based on the time domain, frequency domain, and source space domain, respectively. We design the time-frequency features based on continuous wavelet transform according to the neural representation and obtain an average decoding test accuracy of 84.56%. The research results of this paper lay the foundation for the development of a video target detection system based on EEG signals in the future.

Vision-Based Estimation of Force Balance of Near-Suspended Melt Pool for Drooping and Collapsing Prediction.

Wire-arc additive manufacturing (WAAM) is favored by the industry for its high material utilization rate and low cost. However, wire-arc additive manufacturing of lattice structures faces problems with forming accuracy such as broken rod and surface morphology defects, which cannot meet the industrial demand. This article innovatively combines the melt pool stress theory with visual perception algorithms to visually study the force balance of the near-suspended melt pool to predict the state of the melt pool. First, the method for melt pool segmentation was studied. The results show that the optimized U-net achieved high accuracy in melt pool segmentation tasks, with accuracies of 98.18%, MIOU 96.64%, and Recall 98.34%. In addition, a method for estimating melt pool force balance and predicting normal, sagging, and collapsing states of the melt pool is proposed. By combining experimental testing with computer vision technology, an analysis of the force balance of the melt pool during the inclined rod forming process was conducted, showing a prediction rate as high as 90% for the testing set. By using this method, monitoring and predicting the state of the melt pool is achieved, preemptively avoiding issues of broken rods during the printing process. This approach can effectively assist in adjusting process parameters and improving welding quality. The application of this method will further promote the development of intelligent unmanned WAAM and provide some references for the development of artificial intelligence monitoring systems in the manufacturing field.

Neural Correlate and Movement Decoding of Simultaneous-and-Sequential Bimanual Movements Using EEG Signals.

Bimanual coordination is important for developing a natural motor brain-computer interface (BCI) from electroencephalogram (EEG) signals, covering the aspects of bilateral arm training for rehabilitation, bimanual coordination for daily-life assistance, and also improving the multidimensional control of BCIs. For the same task targets of both hands, simultaneous and sequential bimanual movements are two different bimanual coordination manners. Planning and performing motor sequences are the fundamental abilities of humans, and it is more natural to execute sequential movements compared to simultaneous movements in many complex tasks. However, to date, for these two different manners in which two hands coordinated to reach the same task targets, the differences in the neural correlate and also the feasibility of movement discrimination have not been explored. In this study, we aimed to investigate these two issues based on a bimanual reaching task for the first time. Finally, neural correlates in the view of the movement-related cortical potentials, event-related oscillations, and source imaging showed unique neural encoding patterns of sequential movements. Besides, for the same task targets of both hands, the simultaneous and sequential bimanual movements were successfully discriminated in both pre-movement and movement execution periods. This study revealed the neural encoding patterns of sequential bimanual movements and presented its values in developing a more natural and good-performance motor BCI.

Robust Decoding of Upper-Limb Movement Direction Under Cognitive Distraction With Invariant Patterns in Embedding Manifold.

Motor brain-computer interfaces (BCIs) have gained growing research interest in motor rehabilitation, restoration, and prostheses control. Decoding upper-limb movement direction with noninvasive BCIs has been extensively investigated. However, few of them address the intervention of cognitive distraction that impairs decoding performance in practice. In this study, we propose a novel decoding model with invariant patterns in embedding manifold on a mixed dataset pooled from electroencephalograph (EEG) signals under different attentional states. We reconstruct an embedding low-dimensional manifold that intrinsically characterizes movements of the upper limb and transfer patterns of neural activities decomposed from brain functional connectivity (FC) to the manifold subspace to further preserve movement-related information. Experimental results showed that the proposed decoding model had higher robustness on the mixed dataset of attentive and distracted states compared to the baseline method. Our research provides insights into modeling a uniform underlying mechanism of movement-related EEG signals and can help enhance the practicability of BCI systems under real-world situations.

EEG-Based Motor BCIs for Upper Limb Movement: Current Techniques and Future Insights.

Motor brain-computer interface (BCI) refers to the BCI that decodes voluntary motion intentions from brain signals directly and outputs corresponding control commands without activating peripheral nerves and muscles. Motor BCIs can be used for the restoration, compensation, and augmentation of motor function by activating the neuromuscular circuit and facilitating neural plasticity. The essential applications of motor BCIs include neurorehabilitation and daily-life assistance for motor-impaired patients. In recent years, studies on motor BCIs mainly concentrate on neural signatures, movement decoding, and its applications. In this review, we aim to provide a comprehensive review of the state-of-the-art research of electroencephalography (EEG) signals-based motor BCIs for the first time. We also aim to give some insights into advancing motor BCIs to a more natural and practical application scenario. In particular, we focus on the motor BCIs for the movements of the upper limbs. Specifically, the experimental paradigms, techniques, and application systems of upper-limb BCIs are reviewed. Several vital issues in developing more natural and practical upper-limb motor BCIs, including developing target-users-oriented, distraction-robust, and multi-limbs motor BCIs, and applying fusion techniques to promote the natural and practical motor BCIs, are discussed.

A Novel Asynchronous Brain Signals-Based Driver-Vehicle Interface for Brain-Controlled Vehicles.

Directly applying brain signals to operate a mobile manned platform, such as a vehicle, may help people with neuromuscular disorders regain their driving ability. In this paper, we developed a novel electroencephalogram (EEG) signal-based driver-vehicle interface (DVI) for the continuous and asynchronous control of brain-controlled vehicles. The proposed DVI consists of the user interface, the command decoding algorithm, and the control model. The user interface is designed to present the control commands and induce the corresponding brain patterns. The command decoding algorithm is developed to decode the control command. The control model is built to convert the decoded commands to control signals. Offline experimental results show that the developed DVI can generate a motion control command with an accuracy of 83.59% and a detection time of about 2 s, while it has a recognition accuracy of 90.06% in idle states. A real-time brain-controlled simulated vehicle based on the DVI was developed and tested on a U-turn road. Experimental results show the feasibility of the DVI for continuously and asynchronously controlling a vehicle. This work not only advances the research on brain-controlled vehicles but also provides valuable insights into driver-vehicle interfaces, multimodal interaction, and intelligent vehicles.

MRCPs-and-ERS/D-Oscillations-Driven Deep Learning Models for Decoding Unimanual and Bimanual Movements.

Motor brain-computer interface (BCI) can intend to restore or compensate for central nervous system functionality. In the motor-BCI, motor execution (ME), which relies on patients' residual or intact movement functions, is a more intuitive and natural paradigm. Based on the ME paradigm, we can decode voluntary hand movement intentions from electroencephalography (EEG) signals. Numerous studies have investigated EEG-based unimanual movement decoding. Moreover, some studies have explored bimanual movement decoding since bimanual coordination is important in daily-life assistance and bilateral neurorehabilitation therapy. However, the multi-class classification of the unimanual and bimanual movements shows weak performance. To address this problem, in this work, we propose a neurophysiological signatures-driven deep learning model utilizing the movement-related cortical potentials (MRCPs) and event-related synchronization/ desynchronization (ERS/D) oscillations for the first time, inspired by the finding that brain signals encode motor-related information with both evoked potentials and oscillation components in ME. The proposed model consists of a feature representation module, an attention-based channel-weighting module, and a shallow convolutional neural network module. Results show that our proposed model has superior performance to the baseline methods. Six-class classification accuracies of unimanual and bimanual movements achieved 80.3%. Besides, each feature module of our model contributes to the performance. This work is the first to fuse the MRCPs and ERS/D oscillations of ME in deep learning to enhance the multi-class unimanual and bimanual movements' decoding performance. This work can facilitate the neural decoding of unimanual and bimanual movements for neurorehabilitation and assistance.

Effects of Cognitive Distraction on Upper Limb Movement Decoding From EEG Signals.

OBJECTIVE: Hand movement decoding from electroencephalograms (EEG) signals is vital to the rehabilitation and assistance of upper limb-impaired patients. Few existing studies on hand movement decoding from EEG signals consider any distractions. However, in practice, patients can be distracted while using the hand movement decoding systems in real life. In this paper, we aim to investigate the effects of cognitive distraction on movement decoding performance. METHODS: We first propose a robust decoding method of hand movement directions to cognitive distraction from EEG signals by using the Riemannian Manifold to extract affine invariant features and Gaussian Naive Bayes classifier (named RM-GNBC). Then, we use the experimental and simulated EEG data under conditions without and with distraction to compare the decoding performance of three decoding methods (including the proposed method, tangent space linear discriminant analysis (TSLDA), and baseline method)). RESULTS: The simulation and experimental results show that the Riemannian-based methods (i.e., RM-GNBC and TSLDA) have higher accuracy under the conditions without and with cognitive distraction and smaller decreases in decoding accuracy between the conditions without and with cognitive distraction than the baseline method. Furthermore, the RM-GNBC method has 6% (paired t-test, p = 0.026) and 5% (paired t-test, p = 0.137) higher accuracies than the TSLDA method under the conditions without and with cognitive distraction, respectively. CONCLUSION: The results show that the Riemannian-based methods have higher robustness to cognitive distraction. SIGNIFICANCE: This work contributes to developing a brain-computer interface (BCI) to improve the rehabilitation and assistance of hand-impaired patients in real life and open an avenue to the studies on the effects of distraction on other BCI paradigms.

Sound Target Detection Under Noisy Environment Using Brain-Computer Interface.

As an important means of environmental reconnaissance and regional security protection, sound target detection (STD) has been widely studied in the field of machine learning for a long time. Considering the shortcomings of the robustness and generalization performance of existing methods based on machine learning, we proposed a target detection method by an auditory brain-computer interface (BCI). We designed the experimental paradigm according to the actual application scenarios of STD, recorded the changes in Electroencephalogram (EEG) signals during the process of detecting target sound, and further extracted the features used to decode EEG signals through the analysis of neural representations, including Event-Related Potential (ERP) and Event-Related Spectral Perturbation (ERSP). Experimental results showed that the proposed method achieved good detection performance under noisy environment. As the first study of BCI applied to STD, this study shows the feasibility of this scheme in BCI and can serve as the foundation for future related applications.

EEG-Based Continuous Hand Movement Decoding Using Improved Center-Out Paradigm.

The continuous decoding of human movement intention based on electroencephalogram (EEG) signals is valuable for developing a more natural motor augmented or assistive system instead of its discrete classifications. The classic center-out paradigm has been widely used to study discrete and continuous hand movement parameter decoding. However, when applying it in studying continuous movement decoding, the classic paradigm needs to be improved to increase the decoding performance, especially generalization performance. In this paper, we first discuss the limitations of the classic center-out paradigm in exploring the hand movement's continuous decoding. Then, an improved paradigm is proposed to enhance the continuous decoding performance. Besides, an adaptive decoder-ensemble framework is developed for continuous kinematic parameter decoding. Finally, with the improved center-out paradigm and the ensemble decoding framework, the average Pearson's correlation coefficients between the predicted and recorded movement kinematic parameters improve significantly by about 75 percent for the directional parameters and about 10 percent for the non-directional parameters. Furthermore, its generalization performance improves significantly by about 20 percent for the directional parameters. This study indicates the advantage of the improved paradigm in predicting the hand movement's kinematic information from low-frequency scalp EEG signals. It can advance the applications of the noninvasive motor brain-computer interface (BCI) in rehabilitation, daily assistance, and human augmentation areas.

Using Non-linear Dynamics of EEG Signals to Classify Primary Hand Movement Intent Under Opposite Hand Movement.

Decoding human hand movement from electroencephalograms (EEG) signals is essential for developing an active human augmentation system. Although existing studies have contributed much to decoding single-hand movement direction from EEG signals, decoding primary hand movement direction under the opposite hand movement condition remains open. In this paper, we investigated the neural signatures of the primary hand movement direction from EEG signals under the opposite hand movement and developed a novel decoding method based on non-linear dynamics parameters of movement-related cortical potentials (MRCPs). Experimental results showed significant differences in MRCPs between hand movement directions under an opposite hand movement. Furthermore, the proposed method performed well with an average binary decoding accuracy of 89.48 ± 5.92% under the condition of the opposite hand movement. This study may lay a foundation for the future development of EEG-based human augmentation systems for upper limbs impaired patients and healthy people and open a new avenue to decode other hand movement parameters (e.g., velocity and position) from EEG signals.

Sliding-Mode Nonlinear Predictive Control of Brain-Controlled Mobile Robots.

In this article, we develop a robust sliding-mode nonlinear predictive controller for brain-controlled robots with enhanced performance, safety, and robustness. First, the kinematics and dynamics of a mobile robot are built. After that, the proposed controller is developed by cascading a predictive controller and a smooth sliding-mode controller. The predictive controller integrates the human intention tracking with safety guarantee objectives into an optimization problem to minimize the invasion to human intention while maintaining robot safety. The smooth sliding-mode controller is designed to achieve robust desired velocity tracking. The results of human-in-the-loop simulation and robotic experiments both show the efficacy and robust performance of the proposed controller. This work provides an enabling design to enhance the future research and development of brain-controlled robots.

Hierarchical Decoding Model of Upper Limb Movement Intention From EEG Signals Based on Attention State Estimation.

Decoding the motion intention of the human upper limb from electroencephalography (EEG) signals has important practical values. However, existing decoding models are built under the attended state while subjects perform motion tasks. In practice, people are often distracted by other tasks or environmental factors, which may impair decoding performance. To address this problem, in this paper, we propose a hierarchical decoding model of human upper limb motion intention from EEG signals based on attention state estimation. The proposed decoding model includes two components. First, the attention state detection (ASD) component estimates the attention state during the upper limb movement. Next, the motion intention recognition (MIR) component decodes the motion intention by using the decoding models built under the attended and distracted states. The experimental results show that the proposed hierarchical decoding model performs well under the attended and distracted states. This work can advance the application of human movement intention decoding and provides new insights into the study of brain-machine interfaces.

EMG-Based 3D Hand Motor Intention Prediction for Information Transfer from Human to Robot.

(1) Background: Three-dimensional (3-D) hand position is one of the kinematic parameters that can be inferred from Electromyography (EMG) signals. The inferred parameter is used as a communication channel in human-robot collaboration applications. Although its application from the perspective of rehabilitation and assistive technologies are widely studied, there are few papers on its application involving healthy subjects such as intelligent manufacturing and skill transfer. In this regard, for tasks associated with complex hand trajectories without the consideration of the degree of freedom (DOF), the prediction of 3-D hand position from EMG signal alone has not been addressed. (2) Objective: The primary aim of this study is to propose a model to predict human motor intention that can be used as information from human to robot. Therefore, the prediction of a 3-D hand position directly from the EMG signal for complex trajectories of hand movement, without the direct consideration of joint movements, is studied. In addition, the effects of slow and fast motions on the accuracy of the prediction model are analyzed. (3) Methods: This study used the EMG signal that is collected from the upper limb of healthy subjects, and the position signal of the hand while the subjects manipulate complex trajectories. We considered and analyzed two types of tasks with complex trajectories, each with quick and slow motions. A recurrent fuzzy neural network (RFNN) model was constructed to predict the 3-D position of the hand from the features of EMG signals alone. We used the Pearson correlation coefficient (CC) and normalized root mean square error (NRMSE) as performance metrics. (4) Results: We found that 3-D hand positions of the complex movement can be predicted with the mean performance of CC = 0.85 and NRMSE = 0.105. The 3-D hand position can be predicted well within a future time of 250 ms, from the EMG signal alone. Even though tasks performed under quick motion had a better prediction performance; the statistical difference in the accuracy of prediction between quick and slow motion was insignificant. Concerning the prediction model, we found that RFNN has a good performance in decoding for the time-varying system. (5) Conclusions: In this paper, irrespective of the speed of the motion, the 3-D hand position is predicted from the EMG signal alone. The proposed approach can be used in human-robot collaboration applications to enhance the natural interaction between a human and a robot.

Decoding Single-Hand and Both-Hand Movement Directions From Noninvasive Neural Signals.

Decoding human movement parameters from electroencephalograms (EEG) signals is of great value for human-machine collaboration. However, existing studies on hand movement direction decoding concentrate on the decoding of a single-hand movement direction from EEG signals given the opposite hand is maintained still. In practice, the cooperative movement of both hands is common. In this paper, we investigated the neural signatures and decoding of single-hand and both-hand movement directions from EEG signals. The potentials of EEG signals and power sums in the low frequency band of EEG signals from 24 channels were used as decoding features. The linear discriminant analysis (LDA) and support vector machine (SVM) classifiers were used for decoding. Experimental results showed a significant difference in the negative offset maximums of movement-related cortical potentials (MRCPs) at electrode Cz between single-hand and both-hand movements. The recognition accuracies for six-class classification, including two single-hand and four both-hand movement directions, reached 70.29%± 10.85% by using EEG potentials as features with the SVM classifier. These findings showed the feasibility of decoding single-hand and both-hand movement directions. This work can lay a foundation for the future development of an active human-machine collaboration system based on EEG signals and open a new research direction in the field of decoding hand movement parameters from EEG signals.

Modeling of Human Operator Behavior for Brain-Actuated Mobile Robots Steering.

Human operator control of brain-actuated robot steering based on electroencephalograph (EEG)-signals is a complex behavior consisting of surroundings perceiving, decision making, and commands issuing and differs among individual operators. However, no existing models allow decoupling the user from the loop to improve the system design and testing process, which can capture such behavior of a brain-actuated robot. To address this problem, in this paper, we propose an operator brain-controlled steering model consisting of an operator decision model based on the queuing network (QN) cognitive architecture and a brain-machine interface (BMI) performance model. The QN-based operator decision model can mimic the human decision process with the individual operator differences considered. The new BMI performance model is built to represent the varied accuracy of BMI during brain-controlled direction operations. Furthermore, the model is simulated and validated against the results of human operator-in-the-loop experiments. The results show that the proposed model can reproduce the behavior of human operators thanks to its similar direction control performance.

EEG-Based Adaptive Driver-Vehicle Interface Using Variational Autoencoder and PI-TSVM.

Event-related potential (ERP)-based driver-vehicle interfaces (DVIs) have been developed to provide a communication channel for people with disabilities to drive a vehicle. However, they require a tedious and time-consuming training procedure to build the decoding model, which can translate EEG signals into commands. In this paper, to address this problem, we propose an adaptive DVI by using a new semi-supervised algorithm. The decoding model of the proposed DVI is first built with a small labeled training set, and then gradually improved by updating the proposed semi-supervised decoding model with new collected unlabeled EEG signals. In our semi-supervised algorithm, independent component analysis (ICA) and Kalman smoother are first used to improve the signal-to-noise ratio (SNR). After that, variational autoencoder is applied to provide a robust feature representation of EEG signals. Finally, a prior information-based transductive support vector machine (PI-TSVM) classifier is developed to translate these features into commands. Experimental results show that the proposed DVI can significantly reduce the training effort. After a short updating, its performance can be close to that of the supervised DVI requiring a lengthy training procedure. This work is vital for advancing the application of these DVIs.

Combined Lateral and Longitudinal Control of EEG Signals-Based Brain-Controlled Vehicles.

Using brain signals rather than limbs to control a vehicle may not only help persons with disabilities to acquire driving ability, but also provide healthy persons with a new alternative way to drive. In this paper, we propose a combined lateral and longitudinal control system for electroencephalogram (EEG) signals-based brain-controlled vehicles (BCVs). The proposed system is designed by integrating a user interface, a brain-computer interface (BCI), a control interface model, a lateral controller, and a longitudinal controller. We conduct driver-and-hardware-in-the-loop experiments under two control conditions (i.e., the brain- and manual-control conditions) with different subjects and three driving tests (i.e., the lane-changing, path-selection, and car-following tests). Experimental results show the feasibility of using brain signals to continuously perform both the lateral and longitudinal control of a vehicle. This study not only promotes the development of BCVs, but also provides some insights on how to apply BCIs in conjunction with assistant controllers to control other dynamic systems.

EEG Signals-Based Longitudinal Control System for a Brain-Controlled Vehicle.

Directly using brain signals to drive a vehicle may not only help persons with disabilities to regain driving ability but also provide a new alternative way for healthy people to control a vehicle. In this paper, we propose a new longitudinal control system based on electroencephalogram signals for brain-controlled vehicles (BCVs) by combining a user interface, a brain-computer interface (BCI) system, and a longitudinal control module. Driver-in-the-loop experiments were conducted by using two driving tests (i.e., the destination-approaching and car-following tests) with different subjects under two control conditions, i.e., the brain and manual control conditions. Experimental results show the feasibility of alone using brain signals to continuously perform the longitudinal control of a vehicle at a relatively high speed, at least for some users. This paper not only promotes the development of BCVs but also provides some insights into the research on how to apply BCIs to control other high-speed dynamic systems.