Cross-Subject EEG Feedback for Implicit Image Generation.

Generative models are powerful tools for producing novel information by learning from example data. However, the current approaches require explicit manual input to steer generative models to match human goals. Furthermore, how these models would integrate implicit, diverse feedback and goals of multiple users remains largely unexplored. Here, we present a first-of-its-kind system that produces novel images of faces by inferring human goals directly from cross-subject brain signals while study subjects are looking at example images. We report on an experiment where brain responses to images of faces were recorded using electroencephalography in 30 subjects, focusing on specific salient visual features (VFs). Preferences toward VFs were decoded from subjects' brain responses and used as implicit feedback for a generative adversarial network (GAN), which generated new images of faces. The results from a follow-up user study evaluating the presence of the target salient VFs show that the images generated from brain feedback represent the goal of the study subjects and are comparable to images generated with manual feedback. The methodology provides a stepping stone toward humans-in-the-loop image generation.

Space, time and number: common coding mechanisms and interactions between domains.

Space, time and number are key dimensions that underlie how we perceive, identify and act within the environment. They are interconnected in our behaviour and brain. In this study, we examined interdependencies between these dimensions. To this end, left- and right-handed participants performed an object collision task that required space-time processing and arithmetic tests that involved number processing. Handedness of the participants influenced collision detection with left-handers being more accurate than right-handers, which is in line with the premise that hand preference guides individual differences as a result of sensorimotor experiences and distinct interhemispheric integration patterns. The data further showed that successful collision detection was a predictor for arithmetic achievement, at least in right-handers. These findings suggest that handedness plays a mediating role in binding information processing across domains, likely due to selective connectivity properties within the sensorimotor system that is guided by hemispheric lateralisation patterns.

Time to imagine moving: Simulated motor activity affects time perception.

Sensing the passage of time is important for countless daily tasks, yet time perception is easily influenced by perception, cognition, and emotion. Mechanistic accounts of time perception have traditionally regarded time perception as part of central cognition. Since proprioception, action execution, and sensorimotor contingencies also affect time perception, perception-action integration theories suggest motor processes are central to the experience of the passage of time. We investigated whether sensory information and motor activity may interactively affect the perception of the passage of time. Two prospective timing tasks involved timing a visual stimulus display conveying optical flow at increasing or decreasing velocity. While doing the timing tasks, participants were instructed to imagine themselves moving at increasing or decreasing speed, independently of the optical flow. In the direct-estimation task, the duration of the visual display was explicitly judged in seconds while in the motor-timing task, participants were asked to keep a constant pace of tapping. The direct-estimation task showed imagining accelerating movement resulted in relative overestimation of time, or time dilation, while decelerating movement elicited relative underestimation, or time compression. In the motor-timing task, imagined accelerating movement also accelerated tapping speed, replicating the time-dilation effect. The experiments show imagined movement affects time perception, suggesting a causal role of simulated motor activity. We argue that imagined movements and optical flow are integrated by temporal unfolding of sensorimotor contingencies. Consequently, as physical time is relative to spatial motion, so too is perception of time relative to imaginary motion.

I'll make you an offer you can refuse: How socially sensitive is the MFN?

Medial frontal negativity (MFN) is an event-related potential thought to originate in the anterior cingulate cortex. It is evoked by outcomes being worse than expected, such as when presented with unfair economic proposals during the Ultimatum Game (UG). This could mean the MFN indexes a social-emotional response, as commonly suggested in accounts that relate it to a violation of a social norm of fairness. To examine the link between MFN and norm violation, we designed an EEG experiment with participants acting as representatives in an UG. Participants responded either as themselves, or as representatives of two charities. Of these, a norm-compatible charity conformed to the participant's values, while the norm-incompatible charity contrasted to them. The behavioral results showed that norm-incompatible representation reversed behavior, with almost all fair offers being declined. The MFN, however, was unaffected by the norm representation, with unfair offers consistently evoking MFNs across conditions. We furthermore replicated the curious finding that unexpectedly generous offers evoke as much MFN as unfair offers. Thus, the MFN is not nearly as sensitive to higher-order social-emotional processes as commonly assumed. Instead, the perceived inequality that drives the MFN likely reflects a rational, probabilistic process.

Motor Timing and Covariation with Time Perception: Investigating the Role of Handedness.

Time is a fundamental dimension of our behavior and enables us to guide our actions and to experience time such as predicting collisions or listening to music. In this study, we investigate the regulation and covariation of motor timing and time perception functions in left- and right-handers who are characterized by distinct brain processing mechanisms for cognitive-motor control. To this purpose, we use a combination of tasks that assess the timed responses during movements and the perception of time intervals. The results showed a positive association across left- and right-handers between movement-driven timing and perceived interval duration when adopting a preferred tempo, suggesting cross-domain coupling between both abilities when an intrinsic timescale is present. Handedness guided motor timing during externally-driven conditions that required cognitive intervention, which specifies the relevance of action expertise for the performance of timed-based motor activities. Overall, our results reveal that individual variation across domain-general and domain-specific levels of organization plays a steering role in how one predicts, perceives and experiences time, which accordingly impacts on cognition and behavior.

Manipulating Bodily Presence Affects Cross-Modal Spatial Attention: A Virtual-Reality-Based ERP Study.

Earlier studies have revealed cross-modal visuo-tactile interactions in endogenous spatial attention. The current research used event-related potentials (ERPs) and virtual reality (VR) to identify how the visual cues of the perceiver's body affect visuo-tactile interaction in endogenous spatial attention and at what point in time the effect takes place. A bimodal oddball task with lateralized tactile and visual stimuli was presented in two VR conditions, one with and one without visible hands, and one VR-free control with hands in view. Participants were required to silently count one type of stimulus and ignore all other stimuli presented in irrelevant modality or location. The presence of hands was found to modulate early and late components of somatosensory and visual evoked potentials. For sensory-perceptual stages, the presence of virtual or real hands was found to amplify attention-related negativity on the somatosensory N140 and cross-modal interaction in somatosensory and visual P200. For postperceptual stages, an amplified N200 component was obtained in somatosensory and visual evoked potentials, indicating increased response inhibition in response to non-target stimuli. The effect of somatosensory, but not visual, N200 enhanced when the virtual hands were present. The findings suggest that bodily presence affects sustained cross-modal spatial attention between vision and touch and that this effect is specifically present in ERPs related to early- and late-sensory processing, as well as response inhibition, but do not affect later attention and memory-related P3 activity. Finally, the experiments provide commeasurable scenarios for the estimation of the signal and noise ratio to quantify effects related to the use of a head mounted display (HMD). However, despite valid a-priori reasons for fearing signal interference due to a HMD, we observed no significant drop in the robustness of our ERP measurements.

Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals.

Finding relevant information from large document collections such as the World Wide Web is a common task in our daily lives. Estimation of a user's interest or search intention is necessary to recommend and retrieve relevant information from these collections. We introduce a brain-information interface used for recommending information by relevance inferred directly from brain signals. In experiments, participants were asked to read Wikipedia documents about a selection of topics while their EEG was recorded. Based on the prediction of word relevance, the individual's search intent was modeled and successfully used for retrieving new relevant documents from the whole English Wikipedia corpus. The results show that the users' interests toward digital content can be modeled from the brain signals evoked by reading. The introduced brain-relevance paradigm enables the recommendation of information without any explicit user interaction and may be applied across diverse information-intensive applications.

Manual dexterity: Functional lateralisation patterns and motor efficiency.

Manual tasks are an important goal-directed ability. In this EEG work, we studied how handedness affects the hemispheric lateralisation patterns during performance of visually-driven movements with either hand. The neural correlates were assessed by means of EEG coherence whereas behavioural output was measured by motor error. The EEG data indicated that left- and right-handers showed distinct recruitment patterns. These involved local interactions between brain regions as well as more widespread associations between brain systems. Despite these differences, brain-behaviour correlations highlighted that motor efficiency depended on left-sided brain regions across groups. These results suggest that skilled hand motor control relies on different neural patterns as a function of handedness whereas behavioural efficiency is linked with the left hemisphere. In conclusion, the present findings add to our understanding about principles of lateralised organisation as a function of handedness.

Not My Problem: Vicarious Conflict Adaptation with Human and Virtual Co-actors.

The Simon effect refers to an incompatibility between stimulus and response locations resulting in a conflict situation and, consequently, slower responses. Like other conflict effects, it is commonly reduced after repetitions, suggesting an executive control ability, which flexibly rewires cognitive processing and adapts to conflict. Interestingly, conflict is not necessarily individually defined: the Social Simon effect refers to a scenario where two people who share a task show a conflict effect where a single person does not. Recent studies showed these observations might converge into what could be called vicarious conflict adaptation, with evidence indicating that observing someone else's conflict may subsequently reduce one's own. While plausible, there is reason for doubt: both the social aspect of the Simon Effect, and the degree to which executive control accounts for the conflict adaptation effect, have become foci of debate in recent studies. Here, we present two experiments that were designed to test the social dimension of the effect by varying the social relationship between the actor and the co-actor. In Experiment 1, participants performed a conflict task with a virtual co-actor, while the actor-observer relationship was manipulated as a function of the similarity between response modalities. In Experiment 2, the same task was performed both with a virtual and with a human co-actor, while heart-rate measurements were taken to measure the impact of observed conflict on autonomous activity. While both experiments replicated the interpersonal conflict adaptation effects, neither showed evidence of the critical social dimension. We consider the findings as demonstrating that vicarious conflict adaptation does not rely on the social relationship between the actor and co-actor.

The self in conflict: actors and agency in the mediated sequential Simon task.

Executive control refers to the ability to withstand interference in order to achieve task goals. The effect of conflict adaptation describes that after experiencing interference, subsequent conflict effects are weaker. However, changes in the source of conflict have been found to disrupt conflict adaptation. Previous studies indicated that this specificity is determined by the degree to which one source causes episodic retrieval of a previous source. A virtual reality version of the Simon task was employed to investigate whether changes in a visual representation of the self would similarly affect conflict adaptation. Participants engaged in a mediated Simon task via 3D "avatar" models that either mirrored the participants' movements, or were presented statically. A retrieval cue was implemented as the identity of the avatar: switching it from a male to a female avatar was expected to disrupt the conflict adaptation effect (CAE). The results show that only in static conditions did the CAE depend on the avatar identity, while in dynamic conditions, changes did not cause disruption. We also explored the effect of conflict and adaptation on the degree of movement made with the task-irrelevant hand and replicated the reaction time pattern. The findings add to earlier studies of source-specific conflict adaptation by showing that a visual representation of the self in action can provide a cue that determines episodic retrieval. Furthermore, the novel paradigm is made openly available to the scientific community and is described in its significance for studies of social cognition, cognitive psychology, and human-computer interaction.

Hemispheric asymmetries and the control of motor sequences.

Sequencing of finger positions reflects a prototype of skilled behaviour. In order to perform sequencing, cognitive control supports the requirements and postural transitions. In this electroencephalography (EEG) study, we evaluate the effects of hand dominance and assess the neural correlates of unimanual and bimanual sequencing in left- and right-handers. The behavioural measurements provided an index of response planning (response time to first key press) and response execution (time between successive key presses, taps/s and percentage of correct responses), whereas the neural dynamics was determined by means of EEG coherence, expressing the functional connectivity between brain areas. Correlations between brain activity and behaviour were calculated for exploring the neural correlates that are functionally relevant for sequencing. Brain-behavioural correlations during response planning and execution revealed the significance of circuitry in the left hemisphere, underlining its significant role in the organisation of goal-directed behaviour. This lateralisation profile was independent of intrinsic constraints (hand dominance) and extrinsic demands (task requirements), suggesting essential higher-order computations in the left hemisphere. Overall, the observations highlight that the left hemisphere is specialised for sequential motor organisation in left- and right-handers, suggesting an endogenous hemispheric asymmetry for compound actions and the representation of skill; processes that can be separated from those that are involved in hand dominance.

The meaning of the virtual Midas touch: an ERP study in economic decision making.

The Midas touch refers to the altruistic effects of a brief touch. Though these effects have often been replicated, they remain poorly understood. We investigate the psychophysiology of the effect using remotely transmitted, precisely timed, tactile messages in an economic decision-making game called Ultimatum. Participants were more likely to accept offers after receiving a remotely transmitted touch. Furthermore, we found distinct effects of touch on event-related potentials evoked by (a) feedback regarding accepted and rejected offers, (b) decision cues related to proposals, and (c) the haptic and auditory cues themselves. In each case, a late positive effect of touch was observed and related to the P3. Given the role of the P3 in memory-related functions, the results indicate an indirect relationship between touch and generosity that relies on memory. This hypothesis was further tested and confirmed in the positive effects of touch on later proposals.

Sequential modulations of the Simon effect depend on episodic retrieval.

Sequential modulations of conflict effects, like the reduction of the Simon effect after incompatible trials, have been taken to reflect the operation of a proactive control mechanism commonly called conflict monitoring. However, such modulations are often contaminated by episodic effects like priming and stimulus-response feature integration. It has previously been observed that if the episodic representation of a conflicting trial is altered by rotating the stimulus framing 180(∘) around its axis, the subsequent "conflict adaptation" pattern is eliminated. In Experiment 1, we replicate the findings and provide the basic episodic interpretation. In Experiment 2, we extend the framework to include rotations of 90(∘), and verify that the episodic effects generalize to scenarios of neutral compatibility. Finally, in Experiment 3, we add complete, 360(∘) rotations, and show that the episodic manipulation by itself does not eliminate the conflict adaptation patterns - as long as conditions favor episodic retrieval. The experiments are argued to demonstrate that an episodic account of the conflict adaptation effect can most parsimoniously account for the behavioral effects without relying on higher order cognition. Accordingly, we conclude that conflict adaptation can be understood either as critically depending on episodic retrieval, or alternatively reflecting only episodic retrieval itself.

Developmental changes in motor control: insights from bimanual coordination.

Manual dexterity is known to gradually progress with developmental age. In this study, we evaluate the performance of unimanual and bimanual actions under perturbed and unperturbed conditions in children between 4 and 10 years of age. Behavior was assessed by means of trajectory measurements and degree of bimanual coupling. The results showed that the younger children performed less successfully than the older children in the unimanual and bimanual tasks, with a plateau occurring around the age of 8 years. The SWAN rating scale of inattention and hyperactivity/impulsivity correlated with bimanual behavior at this particular age, suggesting that these traits are predictors of intricate skill performance during a critical developmental phase when significant refinement of control mechanisms occurs. The data furthermore revealed that a rebalancing of the between-hand performance asymmetries enabled superior bimanual coordination patterns in the older children. This suggests that progress in bimanual behavior relies on essential changes in unimanual processing and points to a dynamic interplay of circuitry. Overall, the data highlight a progressive change and integration of control systems due to developmental age with behavioral performance being guided by the existing constraints.

Keep your opponents close: social context affects EEG and fEMG linkage in a turn-based computer game.

In daily life, we often copy the gestures and expressions of those we communicate with, but recent evidence shows that such mimicry has a physiological counterpart: interaction elicits linkage, which is a concordance between the biological signals of those involved. To find out how the type of social interaction affects linkage, pairs of participants played a turn-based computer game in which the level of competition was systematically varied between cooperation and competition. Linkage in the beta and gamma frequency bands was observed in the EEG, especially when the participants played directly against each other. Emotional expression, measured using facial EMG, reflected this pattern, with the most competitive condition showing enhanced linkage over the facial muscle-regions involved in smiling. These effects were found to be related to self-reported social presence: linkage in positive emotional expression was associated with self-reported shared negative feelings. The observed effects confirmed the hypothesis that the social context affected the degree to which participants had similar reactions to their environment and consequently showed similar patterns of brain activity. We discuss the functional resemblance between linkage, as an indicator of a shared physiology and affect, and the well-known mirror neuron system, and how they relate to social functions like empathy.

Cognitive control of response inhibition and switching: hemispheric lateralization and hand preference.

Changing environmental constraints often make already prepared responses unnecessary or inappropriate. Under such circumstances, cognitive control enables to suppress the response or switch to alternative behavior. Here, we examine the neural dynamics of both functions in left- and right-handers who performed two variants of a paradigm that required either inhibition of the response or switching the response between hands. The EEG coherence data showed strengthening of interregional coupling in the alpha band (8-12 Hz) following the target cues with an essential contribution of fronto-medial circuitry and a specific involvement of parietal areas in response switching. Brain-behavioral correlations revealed the functional significance of left-sided regions for successful response inhibition and switching, underlining the significant role of the left hemisphere for the organization of goal-directed activities. This lateralization pattern was observed for both left- and right-handers and suggests dominance of higher-order aspects of action planning in the left hemisphere.

Bimanual control processes and the role of handedness.

OBJECTIVE: Bimanual movements are a fundamental motor skill. Whereas substantial research is available from right-handers, much less is known from left-handers. Accordingly, in the present electroencephalography (EEG) study we evaluated bimanual behavior in left- versus right-handers. METHOD: Thirteen left-handers and 13 right-handers took part in the experiment. Cortical dynamics were evaluated by means of EEG coherence in the beta frequency band (14-28 Hz), and behavioral performance was measured using motor error. RESULTS: The EEG data revealed that right-handers showed a left-sided lateralization pattern whereas left-handers demonstrated a bilateral organization pattern during symmetrical actions. Asymmetry of the bimanual task demands modified the hemispheric profile for both groups and resulted in an additional involvement of the motor-nondominant hemisphere. Brain-behavioral correlations underlined that response planning strongly relied on the left hemisphere irrespective of handedness whereas the motor-dominant hemisphere drove response execution. CONCLUSION: The findings suggest that skilled motor planning may develop preferentially in the left hemisphere.

Subliminal priming and effects of hand dominance.

In the masked priming paradigm, motor responses to targets are influenced by previously presented subliminal primes, and are guided by facilitatory and inhibitory mechanisms that depend on prime-target compatibility/duration. In this study, we evaluate subliminal-driven priming in right- and left-handers during unimanual as well as bimanual tasks. The data from the unimanual tasks confirmed that prime-target compatibility affects performance as a function of prime-target duration. In a bimanual setting, the preferred hand benefitted from facilitation in both handedness groups whereas the non-preferred hand showed a positive priming effect only in left-handers. This denotes that left-handers are more susceptible to response activation of either hand. In addition, inhibitory priming had a stronger effect on the non-preferred than preferred hand, independent of handedness group. Overall, the findings suggest that subliminal-driven mechanisms that assist adaptive motor behavior are sensitive not only to extrinsic (task-related) factors such as prime-target compatibility but also to intrinsic (performer-related) factors such as hand dominance. The data further provide support for handedness-specific effects in motor functions and underline a significant role of hand dominance in the control of bimanual actions.

Compatibility-sequence effects in the Simon task reflect episodic retrieval but not conflict adaptation: evidence from LRP and N2.

Behavioral and psychophysiological studies on the Simon effect have demonstrated that stimuli automatically activate spatially corresponding responses, even if their location is irrelevant to the task. Interestingly, this Simon effect is attenuated after stimulus-response incompatible trials (Gratton effect or compatibility-sequence effect), a pattern that has often been attributed to online conflict adaptation, even though an account in terms of episodic binding and retrieval is just as plausible. Here we show that the compatibility-sequence effect can be eliminated and partly reversed by rotating the boxes in which stimuli are presented in between two given trials, a manipulation that is likely to affect episodic representation but not online control. Sequential modulations of electrophysiological indicators of automatic response priming were also eliminated (N2) or even reversed in sign (LRP), suggesting that these effects are due to episodic retrieval of stimulus-response bindings but not, or to only a negligible degree, to online adaptation.