Multimodal decoding of error processing in a virtual reality flight simulation.

Technological advances in head-mounted displays (HMDs) facilitate the acquisition of physiological data of the user, such as gaze, pupil size, or heart rate. Still, interactions with such systems can be prone to errors, including unintended behavior or unexpected changes in the presented virtual environments. In this study, we investigated if multimodal physiological data can be used to decode error processing, which has been studied, to date, with brain signals only. We examined the feasibility of decoding errors solely with pupil size data and proposed a hybrid decoding approach combining electroencephalographic (EEG) and pupillometric signals. Moreover, we analyzed if hybrid approaches can improve existing EEG-based classification approaches and focused on setups that offer increased usability for practical applications, such as the presented game-like virtual reality flight simulation. Our results indicate that classifiers trained with pupil size data can decode errors above chance. Moreover, hybrid approaches yielded improved performance compared to EEG-based decoders in setups with a reduced number of channels, which is crucial for many out-of-the-lab scenarios. These findings contribute to the development of hybrid brain-computer interfaces, particularly in combination with wearable devices, which allow for easy acquisition of additional physiological data.

Towards unlocking motor control in spinal cord injured by applying an online EEG-based framework to decode motor intention, trajectory and error processing.

Brain-computer interfaces (BCIs) can translate brain signals directly into commands for external devices. Electroencephalography (EEG)-based BCIs mostly rely on the classification of discrete mental states, leading to unintuitive control. The ERC-funded project "Feel Your Reach" aimed to establish a novel framework based on continuous decoding of hand/arm movement intention, for a more natural and intuitive control. Over the years, we investigated various aspects of natural control, however, the individual components had not yet been integrated. Here, we present a first implementation of the framework in a comprehensive online study, combining (i) goal-directed movement intention, (ii) trajectory decoding, and (iii) error processing in a unique closed-loop control paradigm. Testing involved twelve able-bodied volunteers, performing attempted movements, and one spinal cord injured (SCI) participant. Similar movement-related cortical potentials and error potentials to previous studies were revealed, and the attempted movement trajectories were overall reconstructed. Source analysis confirmed the involvement of sensorimotor and posterior parietal areas for goal-directed movement intention and trajectory decoding. The increased experiment complexity and duration led to a decreased performance than each single BCI. Nevertheless, the study contributes to understanding natural motor control, providing insights for more intuitive strategies for individuals with motor impairments.

Neural and Pupillometric Correlates of Error Perception in an Immersive VR Flight Simulation.

Algorithms detecting erroneous events, as used in brain-computer interfaces, usually rely solely on neural correlates of error perception. The increasing availability of wearable displays with built-in pupillometric sensors enables access to additional physiological data, potentially improving error detection. Hence, we measured both electroencephalographic (EEG) and pupillometric signals of 19 participants while performing a navigation task in an immersive virtual reality (VR) setting. We found EEG and pupillometric correlates of error perception and significant differences between distinct error types. Further, we found that actively performing tasks delays error perception. We believe that the results of this work could contribute to improving error detection, which has rarely been studied in the context of immersive VR.

Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome.

Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.

Methods for motion artifact reduction in online brain-computer interface experiments: a systematic review.

Brain-computer interfaces (BCIs) have emerged as a promising technology for enhancing communication between the human brain and external devices. Electroencephalography (EEG) is particularly promising in this regard because it has high temporal resolution and can be easily worn on the head in everyday life. However, motion artifacts caused by muscle activity, fasciculation, cable swings, or magnetic induction pose significant challenges in real-world BCI applications. In this paper, we present a systematic review of methods for motion artifact reduction in online BCI experiments. Using the PRISMA filter method, we conducted a comprehensive literature search on PubMed, focusing on open access publications from 1966 to 2022. We evaluated 2,333 publications based on predefined filtering rules to identify existing methods and pipelines for motion artifact reduction in EEG data. We present a lookup table of all papers that passed the defined filters, all used methods, and pipelines and compare their overall performance and suitability for online BCI experiments. We summarize suitable methods, algorithms, and concepts for motion artifact reduction in online BCI applications, highlight potential research gaps, and discuss existing community consensus. This review aims to provide a comprehensive overview of the current state of the field and guide researchers in selecting appropriate methods for motion artifact reduction in online BCI experiments.

Videoconference fatigue from a neurophysiological perspective: experimental evidence based on electroencephalography (EEG) and electrocardiography (ECG).

In the recent past, many organizations and people have substituted face-to-face meetings with videoconferences. Among others, tools like Zoom, Teams, and Webex have become the "new normal" of human social interaction in many domains (e.g., business, education). However, this radical adoption and extensive use of videoconferencing tools also has a dark side, referred to as videoconference fatigue (VCF). To date only self-report evidence has shown that VCF is a serious issue. However, based on self-reports alone it is hardly possible to provide a comprehensive understanding of a cognitive phenomenon like VCF. Against this background, we examined VCF also from a neurophysiological perspective. Specifically, we collected and analyzed electroencephalography (continuous and event-related) and electrocardiography (heart rate and heart rate variability) data to investigate whether VCF can also be proven on a neurophysiological level. We conducted a laboratory experiment based on a within-subjects design (N = 35). The study context was a university lecture, which was given in a face-to-face and videoconferencing format. In essence, the neurophysiological data-together with questionnaire data that we also collected-show that 50 min videoconferencing, if compared to a face-to-face condition, results in changes in the human nervous system which, based on existing literature, can undoubtedly be interpreted as fatigue. Thus, individuals and organizations must not ignore the fatigue potential of videoconferencing. A major implication of our study is that videoconferencing should be considered as a possible complement to face-to-face interaction, but not as a substitute.

Decoding movement kinematics from EEG using an interpretable convolutional neural network.

Continuous decoding of hand kinematics has been recently explored for the intuitive control of electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs). Deep neural networks (DNNs) are emerging as powerful decoders, for their ability to automatically learn features from lightly pre-processed signals. However, DNNs for kinematics decoding lack in the interpretability of the learned features and are only used to realize within-subject decoders without testing other training approaches potentially beneficial for reducing calibration time, such as transfer learning. Here, we aim to overcome these limitations by using an interpretable convolutional neural network (ICNN) to decode 2-D hand kinematics (position and velocity) from EEG in a pursuit tracking task performed by 13 participants. The ICNN is trained using both within-subject and cross-subject strategies, and also testing the feasibility of transferring the knowledge learned on other subjects on a new one. Moreover, the network eases the interpretation of learned spectral and spatial EEG features. Our ICNN outperformed most of the other state-of-the-art decoders, showing the best trade-off between performance, size, and training time. Furthermore, transfer learning improved kinematics prediction in the low data regime. The network attributed the highest relevance for decoding to the delta-band across all subjects, and to higher frequencies (alpha, beta, low-gamma) for a cluster of them; contralateral central and parieto-occipital sites were the most relevant, reflecting the involvement of sensorimotor, visual and visuo-motor processing. The approach improved the quality of kinematics prediction from the EEG, at the same time allowing interpretation of the most relevant spectral and spatial features.

Transfer Learning in Trajectory Decoding: Sensor or Source Space?

In this study, across-participant and across-session transfer learning was investigated to minimize the calibration time of the brain-computer interface (BCI) system in the context of continuous hand trajectory decoding. We reanalyzed data from a study with 10 able-bodied participants across three sessions. A leave-one-participant-out (LOPO) model was utilized as a starting model. Recursive exponentially weighted partial least squares regression (REW-PLS) was employed to overcome the memory limitation due to the large pool of training data. We considered four scenarios: generalized with no update (Gen), generalized with cumulative update (GenC), and individual models with cumulative (IndC) and non-cumulative (Ind) updates, with each one trained with sensor-space features or source-space features. The decoding performance in generalized models (Gen and GenC) was lower than the chance level. In individual models, the cumulative update (IndC) showed no significant improvement over the non-cumulative model (Ind). The performance showed the decoder's incapability to generalize across participants and sessions in this task. The results suggested that the best correlation could be achieved with the sensor-space individual model, despite additional anatomical information in the source-space features. The decoding pattern showed a more localized pattern around the precuneus over three sessions in Ind models.

Getting off track: Cortical feedback processing network modulated by continuous error signal during target-feedback mismatch.

Performance monitoring and feedback processing - especially in the wake of erroneous outcomes - represent a crucial aspect of everyday life, allowing us to deal with imminent threats in the short term but also promoting necessary behavioral adjustments in the long term to avoid future conflicts. Over the last thirty years, research extensively analyzed the neural correlates of processing discrete error stimuli, unveiling the error-related negativity (ERN) and error positivity (Pe) as two main components of the cognitive response. However, the connection between the ERN/Pe and distinct stages of error processing, ranging from action monitoring to subsequent corrective behavior, remains ambiguous. Furthermore, mundane actions such as steering a vehicle already transgress the scope of discrete erroneous events and demand fine-tuned feedback control, and thus, the processing of continuous error signals - a topic scarcely researched at present. We analyzed two electroencephalography datasets to investigate the processing of continuous erroneous signals during a target tracking task, employing feedback in various levels and modalities. We observed significant differences between correct (slightly delayed) and erroneous feedback conditions in the larger one of the two datasets that we analyzed, both in sensor and source space. Furthermore, we found strong error-induced modulations that appeared consistent across datasets and error conditions, indicating a clear order of engagement of specific brain regions that correspond to individual components of error processing.

Balance perturbation and error processing elicit distinct brain dynamics.

Objective. The maintenance of balance is a complicated process in the human brain, which involves multisensory processing such as somatosensory and visual processing, motor planning and execution. It was shown that a specific cortical activity called perturbation-evoked potential (PEP) appears in the electroencephalogram (EEG) during balance perturbation. PEPs are primarily recognized by the N1 component with a negative peak localized in frontal and central regions. There has been a doubt in balance perturbation studies whether the N1 potential of perturbation is elicited due to error processing in the brain. The objective of this study is to test whether the brain perceives postural instability as a cognitive error by imposing two types of perturbations consisting of erroneous and correct perturbations.Approach. We conducted novel research to incorporate the experiment designs of both error and balance studies. To this end, participants encountered errors during balance perturbations at rare moments in the experiment. We induced errors by imposing perturbations to participants in the wrong directions and an erroneous perturbation was considered as a situation when the participant was exposed to an opposite direction of the expected/informed one. In correct perturbations, participants were tilted to the same direction, as they were informed. We analyzed the two conditions in time, time-frequency, and source domains.Main results. We showed that two error-related neural markers were derived from the EEG responses, including error positivity (Pe), and error-related alpha suppression (ERAS) during erroneous perturbations. Consequently, early neural correlates of perturbation cannot be interpreted as error-related responses. We discovered distinct patterns of conscious error processing; both Pe and ERAS are associated with conscious sensations of error.Significance. Our findings indicated that early cortical responses of balance perturbation are not associated with neural error processing of the brain, and errors induce distinct cortical responses that are distinguishable from brain dynamics of N1 potential.

Using linear parameter varying autoregressive models to measure cross frequency couplings in EEG signals.

For years now, phase-amplitude cross frequency coupling (CFC) has been observed across multiple brain regions under different physiological and pathological conditions. It has been suggested that CFC serves as a mechanism that facilitates communication and information transfer between local and spatially separated neuronal populations. In non-invasive brain computer interfaces (BCI), CFC has not been thoroughly explored. In this work, we propose a CFC estimation method based on Linear Parameter Varying Autoregressive (LPV-AR) models and we assess its performance using both synthetic data and electroencephalographic (EEG) data recorded during attempted arm/hand movements of spinal cord injured (SCI) participants. Our results corroborate the potentiality of CFC as a feature for movement attempt decoding and provide evidence of the superiority of our proposed CFC estimation approach compared to other commonly used techniques.

Measuring Spinal Cord Potentials and Cortico-Spinal Interactions After Wrist Movements Induced by Neuromuscular Electrical Stimulation.

Electroencephalographic (EEG) correlates of movement have been studied extensively over many years. In the present work, we focus on investigating neural correlates that originate from the spine and study their connectivity to corresponding signals from the sensorimotor cortex using multivariate autoregressive (MVAR) models. To study cortico-spinal interactions, we simultaneously measured spinal cord potentials (SCPs) and somatosensory evoked potentials (SEPs) of wrist movements elicited by neuromuscular electrical stimulation. We identified directional connections between spine and cortex during both the extension and flexion of the wrist using only non-invasive recording techniques. Our connectivity estimation results are in alignment with various studies investigating correlates of movement, i.e., we found the contralateral side of the sensorimotor cortex to be the main sink of information as well as the spine to be the main source of it. Both types of movement could also be clearly identified in the time-domain signals.

Feel Your Reach: An EEG-Based Framework to Continuously Detect Goal-Directed Movements and Error Processing to Gate Kinesthetic Feedback Informed Artificial Arm Control.

Establishing the basic knowledge, methodology, and technology for a framework for the continuous decoding of hand/arm movement intention was the aim of the ERC-funded project "Feel Your Reach". In this work, we review the studies and methods we performed and implemented in the last 6 years, which build the basis for enabling severely paralyzed people to non-invasively control a robotic arm in real-time from electroencephalogram (EEG). In detail, we investigated goal-directed movement detection, decoding of executed and attempted movement trajectories, grasping correlates, error processing, and kinesthetic feedback. Although we have tested some of our approaches already with the target populations, we still need to transfer the "Feel Your Reach" framework to people with cervical spinal cord injury and evaluate the decoders' performance while participants attempt to perform upper-limb movements. While on the one hand, we made major progress towards this ambitious goal, we also critically discuss current limitations.

Continuous 2D trajectory decoding from attempted movement: across-session performance in able-bodied and feasibility in a spinal cord injured participant.

Objective. In people with a cervical spinal cord injury (SCI) or degenerative diseases leading to limited motor function, restoration of upper limb movement has been a goal of the brain-computer interface field for decades. Recently, research from our group investigated non-invasive and real-time decoding of continuous movement in able-bodied participants from low-frequency brain signals during a target-tracking task. To advance our setup towards motor-impaired end users, we consequently chose a new paradigm based on attempted movement.Approach. Here, we present the results of two studies. During the first study, data of ten able-bodied participants completing a target-tracking/shape-tracing task on-screen were investigated in terms of improvements in decoding performance due to user training. In a second study, a spinal cord injured participant underwent the same tasks. To investigate the merit of employing attempted movement in end users with SCI, data of the spinal cord injured participant were recorded twice; once within an observation-only condition, and once while simultaneously attempting movement.Main results. We observed mean correlations well above chance level for continuous motor decoding based on attempted movement in able-bodied participants. Additionally, no global improvement over three sessions within five days, both in sensor and in source space, could be observed across all participants and movement parameters. In the participant with SCI, decoding performance well above chance was found.Significance. No presence of a learning effect in continuous attempted movement decoding in able-bodied participants could be observed. In contrast, non-significantly varying decoding patterns may promote the use of source space decoding in terms of generalized decoders utilizing transfer learning. Furthermore, above-chance correlations for attempted movement decoding ranging between those of observation only and executed movement were seen in one spinal cord injured participant, suggesting attempted movement decoding as a possible link between feasibility studies in able-bodied and actual applications in motor impaired end users.

Applying Dimensionality Reduction Techniques in Source-Space Electroencephalography via Template and Magnetic Resonance Imaging-Derived Head Models to Continuously Decode Hand Trajectories.

Several studies showed evidence supporting the possibility of hand trajectory decoding from low-frequency electroencephalography (EEG). However, the decoding in the source space via source localization is scarcely investigated. In this study, we tried to tackle the problem of collinearity due to the higher number of signals in the source space by two folds: first, we selected signals in predefined regions of interest (ROIs); second, we applied dimensionality reduction techniques to each ROI. The dimensionality reduction techniques were computing the mean (Mean), principal component analysis (PCA), and locality preserving projections (LPP). We also investigated the effect of decoding between utilizing a template head model and a subject-specific head model during the source localization. The results indicated that applying source-space decoding with PCA yielded slightly higher correlations and signal-to-noise (SNR) ratios than the sensor-space approach. We also observed slightly higher correlations and SNRs when applying the subject-specific head model than the template head model. However, the statistical tests revealed no significant differences between the source-space and sensor-space approaches and no significant differences between subject-specific and template head models. The decoder with Mean and PCA utilizes information mainly from precuneus and cuneus to decode the velocity kinematics similarly in the subject-specific and template head models.

Toward passive BCI: asynchronous decoding of neural responses to direction- and angle-specific perturbations during a simulated cockpit scenario.

Neuroimaging studies have provided proof that loss of balance evokes specific neural transient wave complexes in electroencephalography (EEG), called perturbation evoked potentials (PEPs). Online decoding of balance perturbations from ongoing EEG signals can establish the possibility of implementing passive brain-computer interfaces (pBCIs) as a part of aviation/driving assistant systems. In this study, we investigated the feasibility of identifying the existence and expression of perturbations in four different conditions by using EEG signals. Fifteen healthy participants experienced four various postural changes while they sat in a glider cockpit. Sudden perturbations were exposed by a robot connected to a glider and moved to the right and left directions with tilting angles of 5 and 10 degrees. Perturbations occurred in an oddball paradigm in which participants were not aware of the time and expression of the perturbations. We employed a hierarchical approach to separate the perturbation and rest, and then discriminate the expression of perturbations. The performance of the BCI system was evaluated by using classification accuracy and F1 score. Asynchronously, we achieved average accuracies of 89.83 and 73.64% and average F1 scores of 0.93 and 0.60 for binary and multiclass classification, respectively. These results manifest the practicality of pBCI for the detection of balance disturbances in a realistic situation.

Corrigendum: Mental State Detection Using Riemannian Geometry on Electroencephalogram Brain Signals.

[This corrects the article DOI: 10.3389/fnhum.2021.746081.].

Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier.

The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.