Tool-use training in augmented reality: plasticity of forearm body schema does not predict sense of ownership or agency in older adults.

In young adults (YA) who practised controlling a virtual tool in augmented reality (AR), the emergence of a sense of body ownership over the tool was associated with the integration of the virtual tool into the body schema (BS). Agency emerged independent of BS plasticity. Here we aimed to replicate these findings in older adults (OA). Although they are still able to learn new motor tasks, brain plasticity and learning capacity are reduced in OA. We predicted that OA would be able to gain control over the virtual tool indicated by the emergence of agency but would show less BS plasticity as compared to YA. Still, an association between BS plasticity and body ownership was expected. OA were trained in AR to control a virtual gripper to enclose and touch a virtual object. In the visuo-tactile (VT) but not the vision-only (V) condition, vibro-tactile feedback was applied through a CyberTouch II glove when the tool touched the object. BS plasticity was assessed with a tactile distance judgement task where participants judged distances between two tactile stimuli applied to their right forearm. Participants further rated their perceived ownership and agency after training. As expected, agency emerged during the use of the tool. However, results did not indicate any changes in the BS of the forearm after virtual tool-use training. Also, an association between BS plasticity and the emergence of body ownership could not be confirmed for OA. Similar to YA, the practice effect was stronger in the visuo-tactile feedback condition compared with the vision-only condition. We conclude that a sense of agency may strongly relate to improvement in tool-use in OA independent of alterations in the BS, while ownership did not emerge due to a lack of BS plasticity.

Emergence of sense of body ownership but not agency during virtual tool-use training is associated with an altered body schema.

In this study we examined if training with a virtual tool in augmented reality (AR) affects the emergence of ownership and agency over the tool and whether this relates to changes in body schema (BS). 34 young adults learned controlling a virtual gripper to grasp a virtual object. In the visuo-tactile (VT) but not the vision-only (V) condition, vibro-tactile feedback was applied to the palm, thumb and index fingers through a CyberTouch II glove when the tool touched the object. Changes in the forearm BS were assessed with a tactile distance judgement task (TDJ) where participants judged distances between two tactile stimuli applied to their right forearm either in proximodistal or mediolateral orientation. Participants further rated their perceived ownership and agency after training. TDJ estimation errors were reduced after training for proximodistal orientations, suggesting that stimuli oriented along the arm axis were perceived as closer together. Higher ratings for ownership were associated with increasing performance level and more BS plasticity, i.e., stronger reduction in TDJ estimation error, and after training in the VT as compared to the V feedback condition, respectively. Agency over the tool was achieved independent of BS plasticity. We conclude that the emergence of a sense of ownership but not agency depends on performance level and the integration of the virtual tool into the arm representation.

AtAwAR Translate: Attention-Aware Language Translation Application in Augmented Reality for Mobile Phones.

As lightweight, low-cost EEG headsets emerge, the feasibility of consumer-oriented brain-computer interfaces (BCI) increases. The combination of portable smartphones and easy-to-use EEG dry electrode headbands offers intriguing new applications and methods of human-computer interaction. In previous research, augmented reality (AR) scenarios have been identified to profit from additional user state information-such as that provided by a BCI. In this work, we implemented a system that integrates user attentional state awareness into a smartphone application for an AR written language translator. The attentional state of the user is classified in terms of internally and externally directed attention by using the Muse 2 electroencephalography headband with four frontal electrodes. The classification results are used to adapt the behavior of the translation app, which uses the smartphone's camera to display translated text as augmented reality elements. We present the first mobile BCI system that uses a smartphone and a low-cost EEG device with few electrodes to provide attention awareness to an AR application. Our case study with 12 participants did not fully support the assumption that the BCI improves usability. However, we are able to show that the classification accuracy and ease of setup are promising paths toward mobile consumer-oriented BCI usage. For future studies, other use cases, applications, and adaptations will be tested for this setup to explore the usability.

Combining Implicit and Explicit Feature Extraction for Eye Tracking: Attention Classification Using a Heterogeneous Input.

Statistical measurements of eye movement-specific properties, such as fixations, saccades, blinks, or pupil dilation, are frequently utilized as input features for machine learning algorithms applied to eye tracking recordings. These characteristics are intended to be interpretable aspects of eye gazing behavior. However, prior research has demonstrated that when trained on implicit representations of raw eye tracking data, neural networks outperform these traditional techniques. To leverage the strengths and information of both feature sets, we integrated implicit and explicit eye tracking features in one classification approach in this work. A neural network was adapted to process the heterogeneous input and predict the internally and externally directed attention of 154 participants. We compared the accuracies reached by the implicit and combined features for different window lengths and evaluated the approaches in terms of person- and task-independence. The results indicate that combining implicit and explicit feature extraction techniques for eye tracking data improves classification results for attentional state detection significantly. The attentional state was correctly classified during new tasks with an accuracy better than chance, and person-independent classification even outperformed person-dependently trained classifiers for some settings. For future experiments and applications that require eye tracking data classification, we suggest to consider implicit data representation in addition to interpretable explicit features.

I-CARE-An Interaction System for the Individual Activation of People with Dementia.

I-CARE is a hand-held activation system that allows professional and informal caregivers to cognitively and socially activate people with dementia in joint activation sessions without special training or expertise. I-CARE consists of an easy-to-use tablet application that presents activation content and a server-based backend system that securely manages the contents and events of activation sessions. It tracks various sources of explicit and implicit feedback from user interactions and different sensors to estimate which content is successful in activating individual users. Over the course of use, I-CARE's recommendation system learns about the individual needs and resources of its users and automatically personalizes the activation content. In addition, information about past sessions can be retrieved such that activations seamlessly build on previous sessions while eligible stakeholders are informed about the current state of care and daily form of their protegees. In addition, caregivers can connect with supervisors and professionals through the I-CARE remote calling feature, to get activation sessions tracked in real time via audio and video support. In this way, I-CARE provides technical support for a decentralized and spontaneous formation of ad hoc activation groups and fosters tight engagement of the social network and caring community. By these means, I-CARE promotes new care infrastructures in the community and the neighborhood as well as relieves professional and informal caregivers.

Imaging Time Series of Eye Tracking Data to Classify Attentional States.

It has been shown that conclusions about the human mental state can be drawn from eye gaze behavior by several previous studies. For this reason, eye tracking recordings are suitable as input data for attentional state classifiers. In current state-of-the-art studies, the extracted eye tracking feature set usually consists of descriptive statistics about specific eye movement characteristics (i.e., fixations, saccades, blinks, vergence, and pupil dilation). We suggest an Imaging Time Series approach for eye tracking data followed by classification using a convolutional neural net to improve the classification accuracy. We compared multiple algorithms that used the one-dimensional statistical summary feature set as input with two different implementations of the newly suggested method for three different data sets that target different aspects of attention. The results show that our two-dimensional image features with the convolutional neural net outperform the classical classifiers for most analyses, especially regarding generalization over participants and tasks. We conclude that current attentional state classifiers that are based on eye tracking can be optimized by adjusting the feature set while requiring less feature engineering and our future work will focus on a more detailed and suited investigation of this approach for other scenarios and data sets.

How Reliably Do Eye Parameters Indicate Internal Versus External Attentional Focus?

Eye behavior is increasingly used as an indicator of internal versus external focus of attention both in research and application. However, available findings are partly inconsistent, which might be attributed to the different nature of the employed types of internal and external cognition tasks. The present study, therefore, investigated how consistently different eye parameters respond to internal versus external attentional focus across three task modalities: numerical, verbal, and visuo-spatial. Three eye parameters robustly differentiated between internal and external attentional focus across all tasks. Blinks, pupil diameter variance, and fixation disparity variance were consistently increased during internally directed attention. We also observed substantial attentional focus effects on other parameters (pupil diameter, fixation disparity, saccades, and microsaccades), but they were moderated by task type. Single-trial analysis of our data using machine learning techniques further confirmed our results: Classifying the focus of attention by means of eye tracking works well across participants, but generalizing across tasks proves to be challenging. Based on the effects of task type on eye parameters, we discuss what eye parameters are best suited as indicators of internal versus external attentional focus in different settings.

EEG-Based Classification of Internally- and Externally-Directed Attention in an Augmented Reality Paradigm.

One problem faced in the design of Augmented Reality (AR) applications is the interference of virtually displayed objects in the user's visual field, with the current attentional focus of the user. Newly generated content can disrupt internal thought processes. If we can detect such internally-directed attention periods, the interruption could either be avoided or even used intentionally. In this work, we designed a special alignment task in AR with two conditions: one with externally-directed attention and one with internally-directed attention. Apart from the direction of attention, the two tasks were identical. During the experiment, we performed a 16-channel EEG recording, which was then used for a binary classification task. Based on selected band power features, we trained a Linear Discriminant Analysis classifier to predict the label for a 13-s window of each trial. Parameter selection, as well as the training of the classifier, were done in a person-dependent manner in a 5-fold cross-validation on the training data. We achieved an average score of approximately 85.37% accuracy on the test data (± 11.27%, range = [66.7%, 100%], 6 participants > 90%, 3 participants = 100%). Our results show that it is possible to discriminate the two states with simple machine learning mechanisms. The analysis of additionally collected data dispels doubts that we classified the difference in movement speed or task load. We conclude that a real-time assessment of internal and external attention in an AR setting in general will be possible.

Decoding Mental Workload in Virtual Environments: A fNIRS Study using an Immersive n-back Task.

Virtual Reality (VR) has emerged as a novel paradigm for immersive applications in training, entertainment, rehabilitation, and other domains. In this paper, we investigate the automatic classification of mental workload from brain activity measured through functional near-infrared spectroscopy (fNIRS) in VR. We present results from a study which implements the established n-back task in an immersive visual scene, including physical interaction. Our results show that user workload can be detected from fNIRS signals in immersive VR tasks both person-dependently and -adaptively.

Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation.

The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.

Mechanisms within the Parietal Cortex Correlate with the Benefits of Random Practice in Motor Adaptation.

The motor learning literature shows an increased retest or transfer performance after practicing under unstable (random) conditions. This random practice effect (also known as contextual interference effect) is frequently investigated on the behavioral level and discussed in the context of mechanisms of the dorsolateral prefrontal cortex and increased cognitive efforts during movement planning. However, there is a lack of studies examining the random practice effect in motor adaptation tasks and, in general, the underlying neural processes of the random practice effect are not fully understood. We tested 24 right-handed human subjects performing a reaching task using a robotic manipulandum. Subjects learned to adapt either to a blocked or a random schedule of different force field perturbations while subjects' electroencephalography (EEG) was recorded. The behavioral results showed a distinct random practice effect in terms of a more stabilized retest performance of the random compared to the blocked practicing group. Further analyses showed that this effect correlates with changes in the alpha band power in electrodes over parietal areas. We conclude that the random practice effect in this study is facilitated by mechanisms within the parietal cortex during movement execution which might reflect online feedback mechanisms.

Increased gamma band power during movement planning coincides with motor memory retrieval.

The retrieval of motor memory requires a previous memory encoding and subsequent consolidation of the specific motor memory. Previous work showed that motor memory seems to rely on different memory components (e.g., implicit, explicit). However, it is still unknown if explicit components contribute to the retrieval of motor memories formed by dynamic adaptation tasks and which neural correlates are linked to memory retrieval. We investigated the lower and higher gamma bands of subjects' electroencephalography during encoding and retrieval of a dynamic adaptation task. A total of 24 subjects were randomly assigned to a treatment and control group. Both groups adapted to a force field A on day 1 and were re-exposed to the same force field A on day 3 of the experiment. On day 2, treatment group learned an interfering force field B whereas control group had a day rest. Kinematic analyses showed that control group improved their initial motor performance from day 1 to day 3 but treatment group did not. This behavioral result coincided with an increased higher gamma band power in the electrodes over prefrontal areas on the initial trials of day 3 for control but not treatment group. Intriguingly, this effect vanished with the subsequent re-adaptation on day 3. We suggest that improved re-test performance in a dynamic motor adaptation task is contributed by explicit memory and that gamma bands in the electrodes over the prefrontal cortex are linked to these explicit components. Furthermore, we suggest that the contribution of explicit memory vanishes with the subsequent re-adaptation while task automaticity increases.

Hybrid fNIRS-EEG based classification of auditory and visual perception processes.

For multimodal Human-Computer Interaction (HCI), it is very useful to identify the modalities on which the user is currently processing information. This would enable a system to select complementary output modalities to reduce the user's workload. In this paper, we develop a hybrid Brain-Computer Interface (BCI) which uses Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) to discriminate and detect visual and auditory stimulus processing. We describe the experimental setup we used for collection of our data corpus with 12 subjects. On this data, we performed cross-validation evaluation, of which we report accuracy for different classification conditions. The results show that the subject-dependent systems achieved a classification accuracy of 97.8% for discriminating visual and auditory perception processes from each other and a classification accuracy of up to 94.8% for detecting modality-specific processes independently of other cognitive activity. The same classification conditions could also be discriminated in a subject-independent fashion with accuracy of up to 94.6 and 86.7%, respectively. We also look at the contributions of the two signal types and show that the fusion of classifiers using different features significantly increases accuracy.

Adaptive cognitive technical systems.

Adaptive cognitive technical systems are capable of sensing the internal state of its user and of adapting its behavior appropriately to those measurements to improve the usability of the system. One important example of such user state is the user's mental workload level. This paper gives an introduction to the topic of workload recognition and adaptation. It reviews the literature on recognition of workload from physiological signals and on how those user state estimates are employed to improve human-machine interaction.

Subject-to-subject transfer for CSP based BCIs: feature space transformation and decision-level fusion.

Modern Brain Computer Interfaces (BCIs) usually require a calibration session to train a machine learning system before each usage. In general, such trained systems are highly specialized to the subject's characteristic activation patterns and cannot be used for other sessions or subjects. This paper presents a feature space transformation that transforms features generated using subject-specific spatial filters into a subject-independent feature space. The transformation can be estimated from little adaptation data of the subject. Furthermore, we combine three different Common Spatial Pattern based feature extraction approaches using decision-level fusion, which enables BCI use when little calibration data is available, but also outperformed the subject-dependent reference approaches for larger amounts of training data.

Classification of mental tasks in the prefrontal cortex using fNIRS.

Functional near infrared spectroscopy (fNIRS) is rapidly gaining interest in both the Neuroscience, as well as the Brain-Computer-Interface (BCI) community. Despite these efforts, most single-trial analysis of fNIRS data is focused on motor-imagery, or mental arithmetics. In this study, we investigate the suitability of different mental tasks, namely mental arithmetics, word generation and mental rotation for fNIRS based BCIs. We provide the first systematic comparison of classification accuracies achieved in a sample study. Data was collected from 10 subjects performing these three tasks.

Mental workload during n-back task-quantified in the prefrontal cortex using fNIRS.

When interacting with technical systems, users experience mental workload. Particularly in multitasking scenarios (e.g., interacting with the car navigation system while driving) it is desired to not distract the users from their primary task. For such purposes, human-machine interfaces (HCIs) are desirable which continuously monitor the users' workload and dynamically adapt the behavior of the interface to the measured workload. While memory tasks have been shown to elicit hemodynamic responses in the brain when averaging over multiple trials, a robust single trial classification is a crucial prerequisite for the purpose of dynamically adapting HCIs to the workload of its user. The prefrontal cortex (PFC) plays an important role in the processing of memory and the associated workload. In this study of 10 subjects, we used functional Near-Infrared Spectroscopy (fNIRS), a non-invasive imaging modality, to sample workload activity in the PFC. The results show up to 78% accuracy for single-trial discrimination of three levels of workload from each other. We use an n-back task (n ∈ {1, 2, 3}) to induce different levels of workload, forcing subjects to continuously remember the last one, two, or three of rapidly changing items. Our experimental results show that measuring hemodynamic responses in the PFC with fNIRS, can be used to robustly quantify and classify mental workload. Single trial analysis is still a young field that suffers from a general lack of standards. To increase comparability of fNIRS methods and results, the data corpus for this study is made available online.

Speaking mode recognition from functional Near Infrared Spectroscopy.

Speech is our most natural form of communication and even though functional Near Infrared Spectroscopy (fNIRS) is an increasingly popular modality for Brain Computer Interfaces (BCIs), there are, to the best of our knowledge, no previous studies on speech related tasks in fNIRS-based BCI. We conducted experiments on 5 subjects producing audible, silently uttered and imagined speech or do not produce any speech. For each of these speaking modes, we recorded fNIRS signals from the subjects performing these tasks and distinguish segments containing speech from those not containing speech, solely based on the fNIRS signals. Accuracies between 69% and 88% were achieved using support vector machines and a Mutual Information based Best Individual Feature approach. We are also able to discriminate the three speaking modes with 61% classification accuracy. We thereby demonstrate that speech is a very promising paradigm for fNIRS based BCI, as classification accuracies compare very favorably to those achieved in motor imagery BCIs with fNIRS.