Advancing passive BCIs: a feasibility study of two temporal derivative features and effect size-based feature selection in continuous online EEG-based machine error detection.

The emerging integration of Brain-Computer Interfaces (BCIs) in human-robot collaboration holds promise for dynamic adaptive interaction. The use of electroencephalogram (EEG)-measured error-related potentials (ErrPs) for online error detection in assistive devices offers a practical method for improving the reliability of such devices. However, continuous online error detection faces challenges such as developing efficient and lightweight classification techniques for quick predictions, reducing false alarms from artifacts, and dealing with the non-stationarity of EEG signals. Further research is essential to address the complexities of continuous classification in online sessions. With this study, we demonstrated a comprehensive approach for continuous online EEG-based machine error detection, which emerged as the winner of a competition at the 32nd International Joint Conference on Artificial Intelligence. The competition consisted of two stages: an offline stage for model development using pre-recorded, labeled EEG data, and an online stage 3 months after the offline stage, where these models were tested live on continuously streamed EEG data to detect errors in orthosis movements in real time. Our approach incorporates two temporal-derivative features with an effect size-based feature selection technique for model training, together with a lightweight noise filtering method for online sessions without recalibration of the model. The model trained in the offline stage not only resulted in a high average cross-validation accuracy of 89.9% across all participants, but also demonstrated remarkable performance during the online session 3 months after the initial data collection without further calibration, maintaining a low overall false alarm rate of 1.7% and swift response capabilities. Our research makes two significant contributions to the field. Firstly, it demonstrates the feasibility of integrating two temporal derivative features with an effect size-based feature selection strategy, particularly in online EEG-based BCIs. Secondly, our work introduces an innovative approach designed for continuous online error prediction, which includes a straightforward noise rejection technique to reduce false alarms. This study serves as a feasibility investigation into a methodology for seamless error detection that promises to transform practical applications in the domain of neuroadaptive technology and human-robot interaction.

Assessment of mental workload across cognitive tasks using a passive brain-computer interface based on mean negative theta-band amplitudes.

Brain-computer interfaces (BCI) can provide real-time and continuous assessments of mental workload in different scenarios, which can subsequently be used to optimize human-computer interaction. However, assessment of mental workload is complicated by the task-dependent nature of the underlying neural signals. Thus, classifiers trained on data from one task do not generalize well to other tasks. Previous attempts at classifying mental workload across different cognitive tasks have therefore only been partially successful. Here we introduce a novel algorithm to extract frontal theta oscillations from electroencephalographic (EEG) recordings of brain activity and show that it can be used to detect mental workload across different cognitive tasks. We use a published data set that investigated subject dependent task transfer, based on Filter Bank Common Spatial Patterns. After testing, our approach enables a binary classification of mental workload with performances of 92.00 and 92.35%, respectively for either low or high workload vs. an initial no workload condition, with significantly better results than those of the previous approach. It, nevertheless, does not perform beyond chance level when comparing high vs. low workload conditions. Also, when an independent component analysis was done first with the data (and before any additional preprocessing procedure), even though we achieved more stable classification results above chance level across all tasks, it did not perform better than the previous approach. These mixed results illustrate that while the proposed algorithm cannot replace previous general-purpose classification methods, it may outperform state-of-the-art algorithms in specific (workload) comparisons.

Investigating the Single Trial Detectability of Cognitive Face Processing by a Passive Brain-Computer Interface.

An automated recognition of faces enables machines to visually identify a person and to gain access to non-verbal communication, including mimicry. Different approaches in lab settings or controlled realistic environments provided evidence that automated face detection and recognition can work in principle, although applications in complex real-world scenarios pose a different kind of problem that could not be solved yet. Specifically, in autonomous driving-it would be beneficial if the car could identify non-verbal communication of pedestrians or other drivers, as it is a common way of communication in daily traffic. Automated identification from observation whether pedestrians or other drivers communicate through subtle cues in mimicry is an unsolved problem so far, as intent and other cognitive factors are hard to derive from observation. In contrast, communicating persons usually have clear understanding whether they communicate or not, and such information is represented in their mindsets. This work investigates whether the mental processing of faces can be identified through means of a Passive Brain-Computer Interface (pBCI). This then could be used to support the cars' autonomous interpretation of facial mimicry of pedestrians to identify non-verbal communication. Furthermore, the attentive driver can be utilized as a sensor to improve the context awareness of the car in partly automated driving. This work presents a laboratory study in which a pBCI is calibrated to detect responses of the fusiform gyrus in the electroencephalogram (EEG), reflecting face recognition. Participants were shown pictures from three different categories: faces, abstracts, and houses evoking different responses used to calibrate the pBCI. The resulting classifier could distinguish responses to faces from that evoked by other stimuli with accuracy above 70%, in a single trial. Further analysis of the classification approach and the underlying data identified activation patterns in the EEG that corresponds to face recognition in the fusiform gyrus. The resulting pBCI approach is promising as it shows better-than-random accuracy and is based on relevant and intended brain responses. Future research has to investigate whether it can be transferred from the laboratory to the real world and how it can be implemented into artificial intelligences, as used in autonomous driving.

Tracing Pilots' Situation Assessment by Neuroadaptive Cognitive Modeling.

This study presents the integration of a passive brain-computer interface (pBCI) and cognitive modeling as a method to trace pilots' perception and processing of auditory alerts and messages during operations. Missing alerts on the flight deck can result in out-of-the-loop problems that can lead to accidents. By tracing pilots' perception and responses to alerts, cognitive assistance can be provided based on individual needs to ensure they maintain adequate situation awareness. Data from 24 participating aircrew in a simulated flight study that included multiple alerts and air traffic control messages in single pilot setup are presented. A classifier was trained to identify pilots' neurophysiological reactions to alerts and messages from participants' electroencephalogram (EEG). A neuroadaptive ACT-R model using EEG data was compared to a conventional normative model regarding accuracy in representing individual pilots. Results show that passive BCI can distinguish between alerts that are processed by the pilot as task-relevant or irrelevant in the cockpit based on the recorded EEG. The neuroadaptive model's integration of this data resulted in significantly higher performance of 87% overall accuracy in representing individual pilots' responses to alerts and messages compared to 72% accuracy of a normative model that did not consider EEG data. We conclude that neuroadaptive technology allows for implicit measurement and tracing of pilots' perception and processing of alerts on the flight deck. Careful handling of uncertainties inherent to passive BCI and cognitive modeling shows how the representation of pilot cognitive states can be improved iteratively for providing assistance.

A Neuroadaptive Cognitive Model for Dealing With Uncertainty in Tracing Pilots' Cognitive State.

A model-based approach for cognitive assistance is proposed to keep track of pilots' changing demands in dynamic situations. Based on model-tracing with flight deck interactions and EEG recordings, the model is able to represent individual pilots' behavior in response to flight deck alerts. As a first application of the concept, an ACT-R cognitive model is created using data from an empirical flight simulator study on neurophysiological signals of missed acoustic alerts. Results show that uncertainty of individual behavior representation can be significantly reduced by combining cognitive modeling with EEG data. Implications for cognitive assistance in aviation are discussed.

Cognitive and affective probing: a tutorial and review of active learning for neuroadaptive technology.

OBJECTIVE: The interpretation of neurophysiological measurements has a decades-long history, culminating in current real-time brain-computer interfacing (BCI) applications for both patient and healthy populations. Over the course of this history, one focus has been on the investigation of cortical responses to specific stimuli. Such responses can be informative with respect to the human user's mental state at the time of presentation. An ability to decode neurophysiological responses to stimuli in real time becomes particularly powerful when combined with a simultaneous ability to autonomously produce such stimuli. This allows a computer to gather stimulus-response samples and iteratively produce new stimuli based on the information gathered from previous samples, thus acquiring more, and more specific, information. This information can even be obtained without the explicit, voluntary involvement of the user. APPROACH: We define cognitive and affective probing, referring to an application of active learning where repeated sampling is done by eliciting implicit brain responses. In this tutorial, we provide a definition of this method that unifies different past and current implementations based on common aspects. We then discuss a number of aspects that differentiate various possible implementations of cognitive probing. MAIN RESULTS: We argue that a key element is the user model, which serves as both information storage and basis for subsequent probes. Cognitive probing can be used to continuously and autonomously update this model, refining the probes, and obtaining increasingly detailed or accurate information from the resulting brain activity. In contrast to a number of potential advantages of the method, cognitive probing may also pose a threat to informed consent, our privacy of thought, and our ability to assign responsibility to actions mediated by the system. SIGNIFICANCE: This tutorial provides guidelines to both implement, and critically discuss potential ethical implications of, novel cognitive probing applications and research endeavours.

Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation.

The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.

SEREEGA: Simulating event-related EEG activity.

BACKGROUND: Electroencephalography (EEG) is a popular method to monitor brain activity, but it is difficult to evaluate EEG-based analysis methods because no ground-truth brain activity is available for comparison. Therefore, in order to test and evaluate such methods, researchers often use simulated EEG data instead of actual EEG recordings. Simulated data can be used, among other things, to assess or compare signal processing and machine learning algorithms, to model EEG variabilities, and to design source reconstruction methods. NEW METHOD: We present SEREEGA, Simulating Event-Related EEG Activity. SEREEGA is a free and open-source MATLAB-based toolbox dedicated to the generation of simulated epochs of EEG data. It is modular and extensible, at initial release supporting five different publicly available head models and capable of simulating multiple different types of signals mimicking brain activity. This paper presents the architecture and general workflow of this toolbox, as well as a simulated data set demonstrating some of its functions. The toolbox is available at https://github.com/lrkrol/SEREEGA. RESULTS: The simulated data allows established analysis pipelines and classification methods to be applied and is capable of producing realistic results. COMPARISON WITH EXISTING METHODS: Most simulated EEG is coded from scratch. The few open-source methods in existence focus on specific applications or signal types, such as connectivity. SEREEGA unifies the majority of past simulation methods reported in the literature into one toolbox. CONCLUSION: SEREEGA is a general-purpose toolbox to simulate ground-truth EEG data.

Workshops of the Sixth International Brain-Computer Interface Meeting: brain-computer interfaces past, present, and future.

The Sixth International Brain-Computer Interface (BCI) Meeting was held 30 May-3 June 2016 at the Asilomar Conference Grounds, Pacific Grove, California, USA. The conference included 28 workshops covering topics in BCI and brain-machine interface research. Topics included BCI for specific populations or applications, advancing BCI research through use of specific signals or technological advances, and translational and commercial issues to bring both implanted and non-invasive BCIs to market. BCI research is growing and expanding in the breadth of its applications, the depth of knowledge it can produce, and the practical benefit it can provide both for those with physical impairments and the general public. Here we provide summaries of each workshop, illustrating the breadth and depth of BCI research and highlighting important issues and calls for action to support future research and development.

Affective Aspects of Perceived Loss of Control and Potential Implications for Brain-Computer Interfaces.

Most brain-computer interfaces (BCIs) focus on detecting single aspects of user states (e.g., motor imagery) in the electroencephalogram (EEG) in order to use these aspects as control input for external systems. This communication can be effective, but unaccounted mental processes can interfere with signals used for classification and thereby introduce changes in the signal properties which could potentially impede BCI classification performance. To improve BCI performance, we propose deploying an approach that potentially allows to describe different mental states that could influence BCI performance. To test this approach, we analyzed neural signatures of potential affective states in data collected in a paradigm where the complex user state of perceived loss of control (LOC) was induced. In this article, source localization methods were used to identify brain dynamics with source located outside but affecting the signal of interest originating from the primary motor areas, pointing to interfering processes in the brain during natural human-machine interaction. In particular, we found affective correlates which were related to perceived LOC. We conclude that additional context information about the ongoing user state might help to improve the applicability of BCIs to real-world scenarios.

Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving.

We tested the applicability and signal quality of a 16 channel dry electroencephalography (EEG) system in a laboratory environment and in a car under controlled, realistic conditions. The aim of our investigation was an estimation how well a passive Brain-Computer Interface (pBCI) can work in an autonomous driving scenario. The evaluation considered speed and accuracy of self-applicability by an untrained person, quality of recorded EEG data, shifts of electrode positions on the head after driving-related movements, usability, and complexity of the system as such and wearing comfort over time. An experiment was conducted inside and outside of a stationary vehicle with running engine, air-conditioning, and muted radio. Signal quality was sufficient for standard EEG analysis in the time and frequency domain as well as for the use in pBCIs. While the influence of vehicle-induced interferences to data quality was insignificant, driving-related movements led to strong shifts in electrode positions. In general, the EEG system used allowed for a fast self-applicability of cap and electrodes. The assessed usability of the system was still acceptable while the wearing comfort decreased strongly over time due to friction and pressure to the head. From these results we conclude that the evaluated system should provide the essential requirements for an application in an autonomous driving context. Nevertheless, further refinement is suggested to reduce shifts of the system due to body movements and increase the headset's usability and wearing comfort.

Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity.

The effectiveness of today's human-machine interaction is limited by a communication bottleneck as operators are required to translate high-level concepts into a machine-mandated sequence of instructions. In contrast, we demonstrate effective, goal-oriented control of a computer system without any form of explicit communication from the human operator. Instead, the system generated the necessary input itself, based on real-time analysis of brain activity. Specific brain responses were evoked by violating the operators' expectations to varying degrees. The evoked brain activity demonstrated detectable differences reflecting congruency with or deviations from the operators' expectations. Real-time analysis of this activity was used to build a user model of those expectations, thus representing the optimal (expected) state as perceived by the operator. Based on this model, which was continuously updated, the computer automatically adapted itself to the expectations of its operator. Further analyses showed this evoked activity to originate from the medial prefrontal cortex and to exhibit a linear correspondence to the degree of expectation violation. These findings extend our understanding of human predictive coding and provide evidence that the information used to generate the user model is task-specific and reflects goal congruency. This paper demonstrates a form of interaction without any explicit input by the operator, enabling computer systems to become neuroadaptive, that is, to automatically adapt to specific aspects of their operator's mindset. Neuroadaptive technology significantly widens the communication bottleneck and has the potential to fundamentally change the way we interact with technology.

Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls.

Estimating cognitive or affective state from neurophysiological signals and designing applications that make use of this information requires expertise in many disciplines such as neurophysiology, machine learning, experimental psychology, and human factors. This makes it difficult to perform research that is strong in all its aspects as well as to judge a study or application on its merits. On the occasion of the special topic "Using neurophysiological signals that reflect cognitive or affective state" we here summarize often occurring pitfalls and recommendations on how to avoid them, both for authors (researchers) and readers. They relate to defining the state of interest, the neurophysiological processes that are expected to be involved in the state of interest, confounding factors, inadvertently "cheating" with classification analyses, insight on what underlies successful state estimation, and finally, the added value of neurophysiological measures in the context of an application. We hope that this paper will support the community in producing high quality studies and well-validated, useful applications.

Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach.

According to Cognitive Load Theory (CLT), one of the crucial factors for successful learning is the type and amount of working-memory load (WML) learners experience while studying instructional materials. Optimal learning conditions are characterized by providing challenges for learners without inducing cognitive over- or underload. Thus, presenting instruction in a way that WML is constantly held within an optimal range with regard to learners' working-memory capacity might be a good method to provide these optimal conditions. The current paper elaborates how digital learning environments, which achieve this goal can be developed by combining approaches from Cognitive Psychology, Neuroscience, and Computer Science. One of the biggest obstacles that needs to be overcome is the lack of an unobtrusive method of continuously assessing learners' WML in real-time. We propose to solve this problem by applying passive Brain-Computer Interface (BCI) approaches to realistic learning scenarios in digital environments. In this paper we discuss the methodological and theoretical prospects and pitfalls of this approach based on results from the literature and from our own research. We present a strategy on how several inherent challenges of applying BCIs to WML and learning can be met by refining the psychological constructs behind WML, by exploring their neural signatures, by using these insights for sophisticated task designs, and by optimizing algorithms for analyzing electroencephalography (EEG) data. Based on this strategy we applied machine-learning algorithms for cross-task classifications of different levels of WML to tasks that involve studying realistic instructional materials. We obtained very promising results that yield several recommendations for future work.

Predicting motor learning performance from Electroencephalographic data.

BACKGROUND: Research on the neurophysiological correlates of visuomotor integration and learning (VMIL) has largely focused on identifying learning-induced activity changes in cortical areas during motor execution. While such studies have generated valuable insights into the neural basis of VMIL, little is known about the processes that represent the current state of VMIL independently of motor execution. Here, we present empirical evidence that a subject's performance in a 3D reaching task can be predicted on a trial-to-trial basis from pre-trial electroencephalographic (EEG) data. This evidence provides novel insights into the brain states that support successful VMIL. METHODS: Six healthy subjects, attached to a seven degrees-of-freedom (DoF) robot with their right arm, practiced 3D reaching movements in a virtual space, while an EEG recorded their brain's electromagnetic field. A random forest ensemble classifier was used to predict the next trial's performance, as measured by the time needed to reach the goal, from pre-trial data using a leave-one-subject-out cross-validation procedure. RESULTS: The learned models successfully generalized to novel subjects. An analysis of the brain regions, on which the models based their predictions, revealed areas matching prevalent motor learning models. In these brain areas, the α/µ frequency band (8-14 Hz) was found to be most relevant for performance prediction. CONCLUSIONS: VMIL induces changes in cortical processes that extend beyond motor execution, indicating a more complex role of these processes than previously assumed. Our results further suggest that the capability of subjects to modulate their α/µ bandpower in brain regions associated with motor learning may be related to performance in VMIL. Accordingly, training subjects in α/µ-modulation, e.g., by means of a brain-computer interface (BCI), may have a beneficial impact on VMIL.

A Dry EEG-System for Scientific Research and Brain-Computer Interfaces.

Although it ranks among the oldest tools in neuroscientific research, electroencephalography (EEG) still forms the method of choice in a wide variety of clinical and research applications. In the context of brain-computer interfacing (BCI), EEG recently has become a tool to enhance human-machine interaction. EEG could be employed in a wider range of environments, especially for the use of BCI systems in a clinical context or at the homes of patients. However, the application of EEG in these contexts is impeded by the cumbersome preparation of the electrodes with conductive gel that is necessary to lower the impedance between electrodes and scalp. Dry electrodes could provide a solution to this barrier and allow for EEG applications outside the laboratory. In addition, dry electrodes may reduce the time needed for neurological exams in clinical practice. This study evaluates a prototype of a three-channel dry electrode EEG system, comparing it to state-of-the-art conventional EEG electrodes. Two experimental paradigms were used: first, event-related potentials (ERP) were investigated with a variant of the oddball paradigm. Second, features of the frequency domain were compared by a paradigm inducing occipital alpha. Furthermore, both paradigms were used to evaluate BCI classification accuracies of both EEG systems. Amplitude and temporal structure of ERPs as well as features in the frequency domain did not differ significantly between the EEG systems. BCI classification accuracies were equally high in both systems when the frequency domain was considered. With respect to the oddball classification accuracy, there were slight differences between the wet and dry electrode systems. We conclude that the tested dry electrodes were capable to detect EEG signals with good quality and that these signals can be used for research or BCI applications. Easy to handle electrodes may help to foster the use of EEG among a wider range of potential users.

Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general.

Cognitive monitoring is an approach utilizing realtime brain signal decoding (RBSD) for gaining information on the ongoing cognitive user state. In recent decades this approach has brought valuable insight into the cognition of an interacting human. Automated RBSD can be used to set up a brain-computer interface (BCI) providing a novel input modality for technical systems solely based on brain activity. In BCIs the user usually sends voluntary and directed commands to control the connected computer system or to communicate through it. In this paper we propose an extension of this approach by fusing BCI technology with cognitive monitoring, providing valuable information about the users' intentions, situational interpretations and emotional states to the technical system. We call this approach passive BCI. In the following we give an overview of studies which utilize passive BCI, as well as other novel types of applications resulting from BCI technology. We especially focus on applications for healthy users, and the specific requirements and demands of this user group. Since the presented approach of combining cognitive monitoring with BCI technology is very similar to the concept of BCIs itself we propose a unifying categorization of BCI-based applications, including the novel approach of passive BCI.

A public data hub for benchmarking common brain-computer interface algorithms.

Methods of statistical machine learning have recently proven to be very useful in contemporary brain-computer interface (BCI) research based on the discrimination of electroencephalogram (EEG) patterns. Because of this, many research groups develop new algorithms for both feature extraction and classification. However, until now, no large-scale comparison of these algorithms has been accomplished due to the fact that little EEG data is publicly available. Therefore, we at Team PhyPA recorded 32-channel EEGs, electromyograms and electrooculograms of 36 participants during a simple finger movement task. The data are published on our website www.phypa.org and are freely available for downloading. We encourage BCI researchers to test their algorithms on these data and share their results. This work also presents exemplary benchmarking procedures of common feature extraction methods for slow cortical potentials and event-related desynchronization as well as for classification algorithms based on these features.