Feel Your Reach: An EEG-Based Framework to Continuously Detect Goal-Directed Movements and Error Processing to Gate Kinesthetic Feedback Informed Artificial Arm Control.

Establishing the basic knowledge, methodology, and technology for a framework for the continuous decoding of hand/arm movement intention was the aim of the ERC-funded project "Feel Your Reach". In this work, we review the studies and methods we performed and implemented in the last 6 years, which build the basis for enabling severely paralyzed people to non-invasively control a robotic arm in real-time from electroencephalogram (EEG). In detail, we investigated goal-directed movement detection, decoding of executed and attempted movement trajectories, grasping correlates, error processing, and kinesthetic feedback. Although we have tested some of our approaches already with the target populations, we still need to transfer the "Feel Your Reach" framework to people with cervical spinal cord injury and evaluate the decoders' performance while participants attempt to perform upper-limb movements. While on the one hand, we made major progress towards this ambitious goal, we also critically discuss current limitations.

On the interpretation of linear Riemannian tangent space model parameters in M/EEG.

Riemannian tangent space methods offer state-of-the-art performance in magnetoencephalography (MEG) and electroencephalography (EEG) based applications such as brain-computer interfaces and biomarker development. One limitation, particularly relevant for biomarker development, is limited model interpretability compared to established component-based methods. Here, we propose a method to transform the parameters of linear tangent space models into interpretable patterns. Using typical assumptions, we show that this approach identifies the true patterns of latent sources, encoding a target signal. In simulations and two real MEG and EEG datasets, we demonstrate the validity of the proposed approach and investigate its behavior when the model assumptions are violated. Our results confirm that Riemannian tangent space methods are robust to differences in the source patterns across observations. We found that this robustness property also transfers to the associated patterns.

Long-Term Mutual Training for the CYBATHLON BCI Race With a Tetraplegic Pilot: A Case Study on Inter-Session Transfer and Intra-Session Adaptation.

CYBATHLON is an international championship where people with severe physical disabilities compete with the aid of state-of-the-art assistive technology. In one of the disciplines, the BCI Race, tetraplegic pilots compete in a computer game race by controlling an avatar with a brain-computer interface (BCI). This competition offers a perfect opportunity for BCI researchers to study long-term training effects in potential end-users, and to evaluate BCI performance in a realistic environment. In this work, we describe the BCI system designed by the team Mirage91 for participation in the CYBATHLON BCI Series 2019, as well as in the CYBATHLON 2020 Global Edition. Furthermore, we present the BCI's interface with the game and the main methodological strategies, along with a detailed evaluation of its performance over the course of the training period, which lasted 14 months. The developed system was a 4-class BCI relying on task-specific modulations of brain rhythms. We implemented inter-session transfer learning to reduce calibration time, and to reinforce the stability of the brain patterns. Additionally, in order to compensate for potential intra-session shifts in the features' distribution, normalization parameters were continuously adapted in an unsupervised fashion. Across the aforementioned 14 months, we recorded 26 game-based training sessions. Between the first eight sessions, and the final eight sessions leading up to the CYBATHLON 2020 Global Edition, the runtimes significantly improved from 255 ± 23 s (mean ± std) to 225 ± 22 s, respectively. Moreover, we observed a significant increase in the classifier's accuracy from 46 to 53%, driven by more distinguishable brain patterns. Compared to conventional single session, non-adaptive BCIs, the inter-session transfer learning and unsupervised intra-session adaptation techniques significantly improved the performance. This long-term study demonstrates that regular training helped the pilot to significantly increase the distance between task-specific patterns, which resulted in an improvement of performance, both with respect to class separability in the calibration data, and with respect to the game. Furthermore, it shows that our methodological approaches were beneficial in transferring the performance across sessions, and most importantly to the CYBATHLON competitions.

Distance- and speed-informed kinematics decoding improves M/EEG based upper-limb movement decoder accuracy.

OBJECTIVE: One of the main goals in brain-computer interface (BCI) research is the replacement or restoration of lost function in individuals with paralysis. One line of research investigates the inference of movement kinematics from brain activity during different volitional states. A growing number of electroencephalography (EEG) and magnetoencephalography (MEG) studies suggest that information about directional (e.g. velocity) and nondirectional (e.g. speed) movement kinematics is accessible noninvasively. We sought to assess if the neural information associated with both types of kinematics can be combined to improve the decoding accuracy. APPROACH: In an offline analysis, we reanalyzed the data of two previous experiments containing the recordings of 34 healthy participants (15 EEG, 19 MEG). We decoded 2D movement trajectories from low-frequency M/EEG signals in executed and observed tracking movements, and compared the accuracy of an unscented Kalman filter (UKF) that explicitly modeled the nonlinear relation between directional and nondirectional kinematics to the accuracies of linear Kalman (KF) and Wiener filters which did not combine both types of kinematics. MAIN RESULTS: At the group level, posterior-parietal and parieto-occipital (executed and observed movements) and sensorimotor areas (executed movements) encoded kinematic information. Correlations between the recorded position and velocity trajectories and the UKF decoded ones were on average 0.49 during executed and 0.36 during observed movements. Compared to the other filters, the UKF could achieve the best trade-off between maximizing the signal to noise ratio and minimizing the amplitude mismatch between the recorded and decoded trajectories. SIGNIFICANCE: We present direct evidence that directional and nondirectional kinematic information is simultaneously detectable in low-frequency M/EEG signals. Moreover, combining directional and nondirectional kinematic information significantly improves the decoding accuracy upon a linear KF.

Non-linear online low-frequency EEG decoding of arm movements during a pursuit tracking task.

Decoding upper-limb movements in invasive recordings has become a reality, but neural tuning in non-invasive low-frequency recordings is still under discussion. Recent studies managed to decode movement positions and velocities using linear decoders, even developing an online system. The decoded signals, however, exhibited smaller amplitudes than actual movements, affecting feedback and user experience. Recently, we showed that a non-linear offline decoder can combine directional (e.g., velocity) and non-directional (e.g., speed) information. In this study, it is assessed if the non-linear decoder can be used online to provide real-time feedback. Five healthy subjects were asked to track a moving target by controlling a robotic arm. Initially, the robot was controlled by their right hand; then, the control was gradually switched until it was entirely controlled by the electroencephalogram (EEG). Correlations between actual and decoded movements were generally above chance level. Results suggest that information about speed was also encoded in the EEG, demonstrating that the proposed non-linear decoder is suitable for decoding real-time arm movements.

Continuous low-frequency EEG decoding of arm movement for closed-loop, natural control of a robotic arm.

OBJECTIVE: Continuous decoding of voluntary movement is desirable for closed-loop, natural control of neuroprostheses. Recent studies showed the possibility to reconstruct the hand trajectories from low-frequency (LF) electroencephalographic (EEG) signals. So far this has only been performed offline. Here, we attempt for the first time continuous online control of a robotic arm with LF-EEG-based decoded movements. APPROACH: The study involved ten healthy participants, asked to track a moving target by controlling a robotic arm. At the beginning of the experiment, the robot was fully controlled by the participant's hand trajectories. After calibrating the decoding model, that control was gradually replaced by LF-EEG-based decoded trajectories, first with 33%, 66% and finally 100% EEG control. Likewise with other offline studies, we regressed the movement parameters (two-dimensional positions, velocities, and accelerations) from the EEG with partial least squares (PLS) regression. To integrate the information from the different movement parameters, we introduced a combined PLS and Kalman filtering approach (named PLSKF). MAIN RESULTS: We obtained moderate yet overall significant (α = 0.05) online correlations between hand kinematics and PLSKF-decoded trajectories of 0.32 on average. With respect to PLS regression alone, the PLSKF had a stable correlation increase of Δr = 0.049 on average, demonstrating the successful integration of different models. Parieto-occipital activations were highlighted for the velocity and acceleration decoder patterns. The level of robot control was above chance in all conditions. Participants finally reported to feel enough control to be able to improve with training, even in the 100% EEG condition. SIGNIFICANCE: Continuous LF-EEG-based movement decoding for the online control of a robotic arm was achieved for the first time. The potential bottlenecks arising when switching from offline to online decoding, and possible solutions, were described. The effect of the PLSKF and its extensibility to different experimental designs were discussed.

Distinct cortical networks for hand movement initiation and directional processing: An EEG study.

Movement preparation and initiation have been shown to involve large scale brain networks. Recent findings suggest that movement preparation and initiation are represented in functionally distinct cortical networks. In electroencephalographic (EEG) recordings, movement initiation is reflected as a strong negative potential at medial central channels that is phase-locked to the movement onset - the movement-related cortical potential (MRCP). Movement preparation describes the process of transforming high level movement goals to low level commands. An integral part of this transformation process is directional processing (i.e., where to move). The processing of movement direction during visuomotor and oculomotor tasks is associated with medial parieto-occipital cortex (PO) activity, phase-locked to the presentation of potential movement goals. We surmised that the network generating the MRCP (movement initiation) would encode less information about movement direction than the parieto-occipital network processing movement direction. Here, we studied delta band EEG activity during center-out reaching movements (2D; 4 directions) in visuomotor and oculomotor tasks. In 15 healthy participants, we found a consistent representation of movement direction in PO 300-400 â€‹ms after the direction cue irrespective of the task. Despite generating the MRCP, sensorimotor areas (SM) encoded less information about the movement direction than PO. Moreover, the encoded directional information in SM was less consistent across participants and specific to the visuomotor task. In a classification approach, we could infer the four movement directions from the delta band EEG activity with moderate accuracies up to 55.9%. The accuracies for cue-aligned data were significantly higher than for movement onset-aligned data in either task, which also suggests a stronger representation of movement direction during movement preparation. Therefore, we present direct evidence that EEG delta band amplitude modulations carry information about both arm movement initiation and movement direction, and that they are represented in two distinct cortical networks.

Corneo-retinal-dipole and eyelid-related eye artifacts can be corrected offline and online in electroencephalographic and magnetoencephalographic signals.

Eye movements and blinks contaminate electroencephalographic (EEG) and magnetoencephalographic (MEG) activity. As the eye moves, the corneo-retinal dipole (CRD) and eyelid introduce potential/field changes in the M/EEG activity. These eye artifacts can affect a brain-computer interface and thereby impinge on neurofeedback quality. Here, we introduce the sparse generalized eye artifact subspace subtraction (SGEYESUB) algorithm that can correct these eye artifacts offline and in real time. We provide an open source reference implementation of the algorithm and the paradigm to obtain calibration data. Once the algorithm is fitted to calibration data (approx. 5 â€‹min), the eye artifact correction reduces to a matrix multiplication. We compared SGEYESUB with 4 state-of-the-art algorithms using M/EEG activity of 69 participants. SGEYESUB achieved the best trade-off between correcting the eye artifacts and preserving brain activity. Residual correlations between the corrected M/EEG channels and the eye artifacts were below 0.1. Error-related and movement-related cortical potentials were attenuated by less than 0.5 â€‹µV. Our results furthermore demonstrate that CRD and eyelid-related artifacts can be assumed to be stationary for at least 1-1.5 â€‹h, validating the feasibility of our approach in offline and online eye artifact correction.

Using machine learning to reveal the population vector from EEG signals.

OBJECTIVE: Since the discovery of the population vector that directly relates neural spiking activity with arm movement direction, it has become feasible to control robotic arms and neuroprostheses using invasively recorded brain signals. For non-invasive approaches, a direct relation between human brain signals and arm movement direction is yet to be established. APPROACH: Here, we investigated electroencephalographic (EEG) signals in temporal and spectral domains in a continuous, circular arm movement task. Using machine learning methods that respect the linear mixture of brain activity within EEG signals, we show that directional information is represented in the temporal domain in amplitude modulations of the same frequency as the arm movement, and in the spectral domain in power modulations of the 20-24 Hz frequency band. MAIN RESULTS: In the temporal domain, the directional information was mainly expressed in primary sensorimotor cortex (SM1) and posterior parietal cortex (PPC) contralateral to the moving arm, while in the spectral domain SM1 and PPC of both hemispheres predicted arm movement direction. The different cortical representations suggest distinct neural representations in both domains. SIGNIFICANCE: This direct relation between neural activity and arm movement direction in both domains demonstrates the potential of machine learning to reveal neuroscientific insights about the dynamics of human arm movements.

HEAR to remove pops and drifts: the high-variance electrode artifact removal (HEAR) algorithm.

A high fraction of artifact-free signals is highly desirable in functional neuroimaging and brain-computer interfacing (BCI). We present the high-variance electrode artifact removal (HEAR) algorithm to remove transient electrode pop and drift (PD) artifacts from electroencephalographic (EEG) signals. Transient PD artifacts reflect impedance variations at the electrode scalp interface that are caused by ion concentration changes. HEAR and its online version (oHEAR) are open-source and publicly available. Both outperformed state of the art offline and online transient, high-variance artifact correction algorithms for simulated EEG signals. (o)HEAR attenuated PD artifacts by approx. 25 dB, and at the same time maintained a high SNR during PD artifact-free periods. For real-world EEG data, (o)HEAR reduced the fraction of outlier trials by half and maintained the waveform of a movement related cortical potential during a center-out reaching task. In the case of BCI training, using oHEAR can improve the reliability of the feedback a user receives through reducing a potential negative impact of PD artifacts.

Tuning characteristics of low-frequency EEG to positions and velocities in visuomotor and oculomotor tracking tasks.

Movement decoders exploit the tuning of neural activity to various movement parameters with the ultimate goal of controlling end-effector action. Invasive approaches, typically relying on spiking activity, have demonstrated feasibility. Results of recent functional neuroimaging studies suggest that information about movement parameters is even accessible non-invasively in the form of low-frequency brain signals. However, their spatiotemporal tuning characteristics to single movement parameters are still unclear. Here, we extend the current understanding of low-frequency electroencephalography (EEG) tuning to position and velocity signals. We recorded EEG from 15 healthy participants while they performed visuomotor and oculomotor pursuit tracking tasks. Linear decoders, fitted to EEG signals in the frequency range of the tracking movements, predicted positions and velocities with moderate correlations (0.2-0.4; above chance level) in both tasks. Predictive activity in terms of decoder patterns was significant in superior parietal and parieto-occipital areas in both tasks. By contrasting the two tracking tasks, we found that predictive activity in contralateral primary sensorimotor and premotor areas exhibited significantly larger tuning to end-effector velocity when the visuomotor tracking task was performed.