Coordinating human-robot collaboration by EEG-based human intention prediction and vigilance control.

In human-robot collaboration scenarios with shared workspaces, a highly desired performance boost is offset by high requirements for human safety, limiting speed and torque of the robot drives to levels which cannot harm the human body. Especially for complex tasks with flexible human behavior, it becomes vital to maintain safe working distances and coordinate tasks efficiently. An established approach in this regard is reactive servo in response to the current human pose. However, such an approach does not exploit expectations of the human's behavior and can therefore fail to react to fast human motions in time. To adapt the robot's behavior as soon as possible, predicting human intention early becomes a factor which is vital but hard to achieve. Here, we employ a recently developed type of brain-computer interface (BCI) which can detect the focus of the human's overt attention as a predictor for impending action. In contrast to other types of BCI, direct projection of stimuli onto the workspace facilitates a seamless integration in workflows. Moreover, we demonstrate how the signal-to-noise ratio of the brain response can be used to adjust the velocity of the robot movements to the vigilance or alertness level of the human. Analyzing this adaptive system with respect to performance and safety margins in a physical robot experiment, we found the proposed method could improve both collaboration efficiency and safety distance.

Training the spatially-coded SSVEP BCI on the fly.

BACKGROUND: The spatially-coded SSVEP BCI employs the retinotopic map in the human visual pathway to infer the gaze direction of the operator relative to a flicker stimulus inducing steady-state visual evoked potentials (SSVEPs) in the brain. It has been shown that with this method, up to 16 channels can be encoded using only a single flicker stimulus. Another advantage over conventional frequency-coded SSVEP BCIs, in which channels are encoded by different combinations of frequencies and phases, is that the operator does not have to gaze directly at flickering lights. This can reduce visual fatigue and improve user comfort. Whereas the frequency of the SSVEP response is well predictable, which has enabled the development of frequency-coded SSVEP BCIs which do not require training data, the spatial distribution of the SSVEP response over the scalp differs much more between different people. This requires collecting a substantial amount of training data before the spatially-coded BCI could be put into operation. NEW METHOD: In this study we address this issue by combining the spatially-coded BCI with a feedback channel which the operator uses to flag classification errors, and which allows the system to accumulate valid training data while the BCI is used to solve a spatial navigation task. RESULTS: Starting from the minimal number of samples required by the classification method, the approach achieved an average accuracy of 69 ± 15 %, corresponding to an ITR of 31 ± 17 bits/min, in solving the task for the first time. This accuracy improved to 87 ± 9 % (ITR: 54 ± 14 bits/min) after completing the task 2 more times. Further we show that participants with a stable SSVEP topography over repeated stimulation enable the BCI to achieve higher accuracies. COMPARISON WITH EXISTING METHODS: Compared to a similar system with separate training and application phases, the time to achieve the same output is reduced by more than 50 %. CONCLUSIONS: Evaluating the approach in 17 participants suggests that the performance of the spatially-coded BCI with a minimal set of training samples is sufficient to be operational, and that performance keeps improving in the course of its application.

Instant classification for the spatially-coded BCI.

The spatially-coded SSVEP BCI exploits changes in the topography of the steady-state visual evoked response to visual flicker stimulation in the extrafoveal field of view. In contrast to frequency-coded SSVEP BCIs, the operator does not gaze into any flickering lights; therefore, this paradigm can reduce visual fatigue. Other advantages include high classification accuracies and a simplified stimulation setup. Previous studies of the paradigm used stimulation intervals of a fixed duration. For frequency-coded SSVEP BCIs, it has been shown that dynamically adjusting the trial duration can increase the system's information transfer rate (ITR). We therefore investigated whether a similar increase could be achieved for spatially-coded BCIs by applying dynamic stopping methods. To this end we introduced a new stopping criterion which combines the likelihood of the classification result and its stability across larger data windows. Whereas the BCI achieved an average ITR of 28.4±6.4 bits/min with fixed intervals, dynamic intervals increased the performance to 81.1±44.4 bits/min. Users were able to maintain performance up to 60 minutes of continuous operation. We suggest that the dynamic response time might have worked as a kind of temporal feedback which allowed operators to optimize their brain signals and compensate fatigue.

Neuronal Oscillatory Signatures of Joint Attention and Intersubjectivity in Arrhythmic Coaction.

Hyper-brain studies analyze the brain activity of two or more individuals during some form of interaction. Several studies found signs of inter-subject brain activity coordination, such as power and phase synchronization or information flow. This hyper-brain coordination is frequently studied in paradigms which induce rhythms or even synchronization, e.g., by mirroring movements, turn-based activity in card or economic games, or joint music making. It is therefore interesting to figure out in how far coordinated brain activity may be induced by a rhythmicity in the task and/or the sensory feedback that the partners receive. We therefore studied the EEG brain activity of dyads in a task that required the smooth pursuit of a target and did not involve any extrinsic rhythms. Partners controlled orthogonal axes of the two-dimensional motion of an object that had to be kept on the target. Using several methods for analyzing hyper-brain coupling, we could not detect signs of coordinated brain activity. However, we found several brain regions in which the frequency-specific activity significantly correlated with the objective task performance, the subjective experience thereof, and of the collaboration. Activity in these regions has been linked to motor control, sensorimotor integration, executive control and emotional processing. Our results suggest that neural correlates of intersubjectivity encompass large parts of brain areas that are considered to be involved in sensorimotor control without necessarily coordinating their activity across agents.

Socializing Sensorimotor Contingencies.

The aim of this review is to highlight the idea of grounding social cognition in sensorimotor interactions shared across agents. We discuss an action-oriented account that emerges from a broader interpretation of the concept of sensorimotor contingencies. We suggest that dynamic informational and sensorimotor coupling across agents can mediate the deployment of action-effect contingencies in social contexts. We propose this concept of socializing sensorimotor contingencies (socSMCs) as a shared framework of analysis for processes within and across brains and bodies, and their physical and social environments. In doing so, we integrate insights from different fields, including neuroscience, psychology, and research on human-robot interaction. We review studies on dynamic embodied interaction and highlight empirical findings that suggest an important role of sensorimotor and informational entrainment in social contexts. Furthermore, we discuss links to closely related concepts, such as enactivism, models of coordination dynamics and others, and clarify differences to approaches that focus on mentalizing and high-level cognitive representations. Moreover, we consider conceptual implications of rethinking cognition as social sensorimotor coupling. The insight that social cognitive phenomena like joint attention, mutual trust or empathy rely heavily on the informational and sensorimotor coupling between agents may provide novel remedies for people with disturbed social cognition and for situations of disturbed social interaction. Furthermore, our proposal has potential applications in the field of human-robot interaction where socSMCs principles might lead to more natural and intuitive interfaces for human users.

A Spatially-Coded Visual Brain-Computer Interface for Flexible Visual Spatial Information Decoding.

Conventional visual BCIs, in which control channels are tagged with stimulation patterns to elicit distinguishable brain patterns, has made impressive progress in terms of the information transfer rates (ITRs). However, less development has been seen with respect to user experience and complexity of the technical setup. The requirement to tag each of targets by a unique stimulus substantially limits the flexibility of conventional visual BCI systems. A method for decoding the targets in the environment flexibly was therefore proposed in the present study. A BCI speller with thirteen symbols drawn on paper was developed. The symbols were interspersed with four flickers with distinct frequencies, but the user did not have to gaze at flickers. Rather, subjects could spell a sequence by looking at the symbols on the paper. In a cue-guided spelling task, the average offline and online accuracies reached 89.3± 7.3% and 90.3± 6.9% for 13 subjects, corresponding to ITRs of 43.0± 7.4 bit/min and 43.8± 6.8 bit/min. In an additional free-spelling task for seven out of thirteen subjects, an accuracy of 92.3± 3.1% and an ITR of 45.6± 3.3 bit/min were achieved. Analysis of a simulated online system showed the possibility to reach an average ITR of 105.8 bit/min by reducing the epoch duration from 4 to 1 second. Reliable BCI control is possible by gazing at targets in the environment instead of dedicated stimuli which encode control channels. The proposed method can drastically reduce the technical effort for visual BCIs and thereby advance their applications outside the laboratory.

Non-rhythmic temporal prediction involves phase resets of low-frequency delta oscillations.

The phase of neural oscillatory signals aligns to the predicted onset of upcoming stimulation. Whether such phase alignments represent phase resets of underlying neural oscillations or just rhythmically evoked activity, and whether they can be observed in a rhythm-free visual context, however, remains unclear. Here, we recorded the magnetoencephalogram while participants were engaged in a temporal prediction task, judging the visual or tactile reappearance of a uniformly moving stimulus. The prediction conditions were contrasted with a control condition to dissociate phase adjustments of neural oscillations from stimulus-driven activity. We observed stronger delta band inter-trial phase consistency (ITPC) in a network of sensory, parietal and frontal brain areas, but no power increase reflecting stimulus-driven or prediction-related evoked activity. Delta ITPC further correlated with prediction performance in the cerebellum and visual cortex. Our results provide evidence that phase alignments of low-frequency neural oscillations underlie temporal predictions in a non-rhythmic visual and crossmodal context.

Subjective Evaluation of Performance in a Collaborative Task Is Better Predicted From Autonomic Response Than From True Achievements.

Whereas the fundamental role of the body in social cognition seems to be generally accepted, elucidating the bodily mechanisms associated with non-verbal communication and cooperation between two or more persons is still a challenging endeavor. In this article we propose a fresh approach for investigating the function of the autonomic nervous system that is reflected in parameters of heart rate variability, respiration, and electrodermal activity in a social setting. We analyzed autonomic parameters of dyads solving a target-tracking task together with the partner or individually. A machine classifier was trained to predict the subjects' rating of performance and collaboration either from tracking error data or from the set of autonomic parameters. When subjects collaborated, this classifier could predict the subjective performance ratings better from the autonomic response than from the objective performance of the subjects. However, when they solved the task individually, predictability from autonomic parameters dropped to the level of objective performance, indicating that subjects were more rational in rating their performance in this condition. Moreover, the model captured general knowledge about the population that allows it to predict the performance ratings of an unseen subject significantly better than chance. Our results suggest that, in particular in situations that require collaboration with others, evaluation of performance is shaped by the bodily processes that are quantified by autonomic parameters. Therefore, subjective performance assessments appear to be modulated not only by the output of a rational or discriminative system that tracks the objective performance but to a significant extent also by interoceptive processes.

A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling.

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.

An Oscillator Ensemble Model of Sequence Learning.

Learning and memorizing sequences of events is an important function of the human brain and the basis for forming expectations and making predictions. Learning is facilitated by repeating a sequence several times, causing rhythmic appearance of the individual sequence elements. This observation invites to consider the resulting multitude of rhythms as a spectral "fingerprint" which characterizes the respective sequence. Here we explore the implications of this perspective by developing a neurobiologically plausible computational model which captures this "fingerprint" by attuning an ensemble of neural oscillators. In our model, this attuning process is based on a number of oscillatory phenomena that have been observed in electrophysiological recordings of brain activity like synchronization, phase locking, and reset as well as cross-frequency coupling. We compare the learning properties of the model with behavioral results from a study in human participants and observe good agreement of the errors for different levels of complexity of the sequence to be memorized. Finally, we suggest an extension of the model for processing sequences that extend over several sensory modalities.

A Single-Stimulus, Multitarget BCI Based on Retinotopic Mapping of Motion-Onset VEPs.

OBJECTIVE: We present a new type of brain-computer interface (BCI) that utilizes the retinotopic mapping of motion-onset visual evoked potentials (mVEP) to accomplish four control channels using a single motion stimulus. METHODS: Participants selected a BCI command by fixating one of four target locations around a centrally presented visual motion stimulus. A template-matching method was employed to recognize the users' intention by decoding the position of the motion stimulus in the peripheral visual field, and classification performances were evaluated in an offline manner. The motion stimulus eccentricity between the targets and the visual motion stimulus varied among 5.1°, 6.7°, 9.8°, and 13.0°. RESULTS: Distinct N200 spatial patterns were elicited when participants directed attention overtly to the target locations. A four-class classification accuracy of 72.2 ± 5.05% was achieved with a distance of 5.1° visual angle between the targets and the visual motion stimulus. The classification accuracies decreased with increasing motion stimulus eccentricities but remained separable well above the chance level at 13.0° (47.3 ± 8.54%). CONCLUSION: Our results support the feasibility of a single-stimulus, multitarget mVEP BCI. SIGNIFICANCE: The proposed system can simplify the visual stimulation of mVEP BCIs, improve user experience and pave the way for simple yet efficient BCI communication.

Effect- and Performance-Based Auditory Feedback on Interpersonal Coordination.

When two individuals interact in a collaborative task, such as carrying a sofa or a table, usually spatiotemporal coordination of individual motor behavior will emerge. In many cases, interpersonal coordination can arise independently of verbal communication, based on the observation of the partners' movements and/or the object's movements. In this study, we investigate how social coupling between two individuals can emerge in a collaborative task under different modes of perceptual information. A visual reference condition was compared with three different conditions with new types of additional auditory feedback provided in real time: effect-based auditory feedback, performance-based auditory feedback, and combined effect/performance-based auditory feedback. We have developed a new paradigm in which the actions of both participants continuously result in a seamlessly merged effect on an object simulated by a tablet computer application. Here, participants should temporally synchronize their movements with a 90° phase difference and precisely adjust the finger dynamics in order to keep the object (a ball) accurately rotating on a given circular trajectory on the tablet. Results demonstrate that interpersonal coordination in a joint task can be altered by different kinds of additional auditory information in various ways.

Application of a single-flicker online SSVEP BCI for spatial navigation.

A promising approach for brain-computer interfaces (BCIs) employs the steady-state visual evoked potential (SSVEP) for extracting control information. Main advantages of these SSVEP BCIs are a simple and low-cost setup, little effort to adjust the system parameters to the user and comparatively high information transfer rates (ITR). However, traditional frequency-coded SSVEP BCIs require the user to gaze directly at the selected flicker stimulus, which is liable to cause fatigue or even photic epileptic seizures. The spatially coded SSVEP BCI we present in this article addresses this issue. It uses a single flicker stimulus that appears always in the extrafoveal field of view, yet it allows the user to control four control channels. We demonstrate the embedding of this novel SSVEP stimulation paradigm in the user interface of an online BCI for navigating a 2-dimensional computer game. Offline analysis of the training data reveals an average classification accuracy of 96.9±1.64%, corresponding to an information transfer rate of 30.1±1.8 bits/min. In online mode, the average classification accuracy reached 87.9±11.4%, which resulted in an ITR of 23.8±6.75 bits/min. We did not observe a strong relation between a subject's offline and online performance. Analysis of the online performance over time shows that users can reliably control the new BCI paradigm with stable performance over at least 30 minutes of continuous operation.

Utilizing Retinotopic Mapping for a Multi-Target SSVEP BCI With a Single Flicker Frequency.

In brain-computer interfaces (BCIs) that use the steady-state visual evoked response (SSVEP), the user selects a control command by directing attention overtly or covertly to one out of several flicker stimuli. The different control channels are encoded in the frequency, phase, or time domain of the flicker signals. Here, we present a new type of SSVEP BCI, which uses only a single flicker stimulus and yet affords controlling multiple channels. The approach rests on the observation that the relative position between the stimulus and the foci of overt attention result in distinct topographies of the SSVEP response on the scalp. By classifying these topographies, the computer can determine at which position the user is gazing. Offline data analysis in a study on 12 healthy volunteers revealed that 9 targets can be recognized with about 95±3% accuracy, corresponding to an information transfer rate (ITR) of 40.8 ± 3.3 b/min on average. We explored how the classification accuracy is affected by the number of control channels, the trial length, and the number of EEG channels. Our findings suggest that the EEG data from five channels over parieto-occipital brain areas are sufficient for reliably classifying the topographies and that there is a large potential to improve the ITR by optimizing the trial length. The robust performance and the simple stimulation setup suggest that this approach is a prime candidate for applications on desktop and tablet computers.

Maximizing Information Transfer in SSVEP-Based Brain-Computer Interfaces.

Compared to the different brain signals used in brain-computer interface (BCI) paradigms, the s teady-state visually evoked potential (SSVEP) features a high signal to noise ratio, enabling reliable and fast classification of neural activity patterns without extensive training requirements. In this paper, we present methods to further increase the information transfer rates (ITRs) of SSVEP-based BCIs. Starting with stimulus parameter optimizations methods, we develop an improved approach for the use of Canonical correlation analysis and analyze properties of the SSVEP when the user fixates a target and during transitions between targets. These transitions show a negative effect on the system's ITR which we trace back to delays and dead times of the SSVEP. Using two classifier types adapted to continuous and transient SSVEPs and two control modes (fast feedback and fast input), we present a simulated online BCI implementation which addresses the challenges introduced by transient SSVEPs. The resulting system reaches an average ITR of 181 Bits/min and peak ITR values of up to 295 Bits/min for individual users.

Where's the action? The pragmatic turn in cognitive science.

In cognitive science, we are currently witnessing a 'pragmatic turn', away from the traditional representation-centered framework towards a paradigm that focuses on understanding cognition as 'enactive', as skillful activity that involves ongoing interaction with the external world. The key premise of this view is that cognition should not be understood as providing models of the world, but as subserving action and being grounded in sensorimotor coupling. Accordingly, cognitive processes and their underlying neural activity patterns should be studied primarily with respect to their role in action generation. We suggest that such an action-oriented paradigm is not only conceptually viable, but already supported by much experimental evidence. Numerous findings either overtly demonstrate the action-relatedness of cognition or can be re-interpreted in this new framework. We argue that new vistas on the functional relevance and the presumed 'representational' nature of neural processes are likely to emerge from this paradigm.

An independent brain-computer interface using covert non-spatial visual selective attention.

In this paper, a novel independent brain-computer interface (BCI) system based on covert non-spatial visual selective attention of two superimposed illusory surfaces is described. Perception of two superimposed surfaces was induced by two sets of dots with different colors rotating in opposite directions. The surfaces flickered at different frequencies and elicited distinguishable steady-state visual evoked potentials (SSVEPs) over parietal and occipital areas of the brain. By selectively attending to one of the two surfaces, the SSVEP amplitude at the corresponding frequency was enhanced. An online BCI system utilizing the attentional modulation of SSVEP was implemented and a 3-day online training program with healthy subjects was carried out. The study was conducted with Chinese subjects at Tsinghua University, and German subjects at University Medical Center Hamburg-Eppendorf (UKE) using identical stimulation software and equivalent technical setup. A general improvement of control accuracy with training was observed in 8 out of 18 subjects. An averaged online classification accuracy of 72.6 +/- 16.1% was achieved on the last training day. The system renders SSVEP-based BCI paradigms possible for paralyzed patients with substantial head or ocular motor impairments by employing covert attention shifts instead of changing gaze direction.

An independent brain-computer interface based on covert shifts of non-spatial visual attention.

Modulation of steady-state visual evoked potential (SSVEP) by directing gaze to targets flickering at different frequencies has been utilized in many brain-computer interface (BCI) studies. However, this paradigm may not work with patients suffering from complete locked-in syndrome or other severe motor disabilities that do not allow conscious control of gaze direction. In this paper, we present a novel, independent BCI paradigm based on covert shift of non-spatial visual selective attention. Subjects viewed a display consisting of two spatially overlapping sets of randomly positioned dots. The two dot sets differed in color, motion and flickering frequency. Two types of motion, rotation and linear motion, were investigated. Both, the SSVEP amplitude and phase response were modulated by selectively attending to one of the two dot sets. Offline analysis revealed a predicted online classification accuracy of 69.3+/-10.2% for the rotating dots, and 80.7+/-10.4% for the linearly moving dots.

Temporal dynamics of access to consciousness in the attentional blink.

Presentation of two targets in close temporal succession often results in an impairment of conscious perception for the second stimulus. Previous studies have identified several electrophysiological correlates for this so-called 'attentional blink'. Components of the event-related potential (ERP) such as the N2 and the P3, but also oscillatory brain signals have been shown to distinguish between detected and missed stimuli, and thus, conscious perception. Here we investigate oscillatory responses that specifically relate to conscious stimulus processing together with potential ERP predictors. Our results show that successful target detection is associated with enhanced coherence in the low beta frequency range, but a decrease in alpha coherence before and during target presentation. In addition, we find an inverse relation between the P3 amplitudes associated with the first and second target. We conclude that the resources allocated to first and second target processing are directly mirrored by the P3 component and, moreover, that brain states before and during stimulus presentation, as reflected by oscillatory brain activity, strongly determine the access to consciousness. Thus, becoming aware of a stimulus seems to depend on the dynamic interaction between a number of widely distributed neural processes, rather than on the modulation of one single process or component.

Order in spontaneous behavior.

Brains are usually described as input/output systems: they transform sensory input into motor output. However, the motor output of brains (behavior) is notoriously variable, even under identical sensory conditions. The question of whether this behavioral variability merely reflects residual deviations due to extrinsic random noise in such otherwise deterministic systems or an intrinsic, adaptive indeterminacy trait is central for the basic understanding of brain function. Instead of random noise, we find a fractal order (resembling Lévy flights) in the temporal structure of spontaneous flight maneuvers in tethered Drosophila fruit flies. Lévy-like probabilistic behavior patterns are evolutionarily conserved, suggesting a general neural mechanism underlying spontaneous behavior. Drosophila can produce these patterns endogenously, without any external cues. The fly's behavior is controlled by brain circuits which operate as a nonlinear system with unstable dynamics far from equilibrium. These findings suggest that both general models of brain function and autonomous agents ought to include biologically relevant nonlinear, endogenous behavior-initiating mechanisms if they strive to realistically simulate biological brains or out-compete other agents.