GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database.

Since thoroughly validated naturalistic affective German speech stimulus databases are rare, we present here a novel validated database of speech sequences assembled with the purpose of emotion induction. The database comprises 37 audio speech sequences with a total duration of 92 minutes for the induction of positive, neutral, and negative emotion: comedian shows intending to elicit humorous and amusing feelings, weather forecasts, and arguments between couples and relatives from movies or television series. Multiple continuous and discrete ratings are used to validate the database to capture the time course and variabilities of valence and arousal. We analyse and quantify how well the audio sequences fulfil quality criteria of differentiation, salience/strength, and generalizability across participants. Hence, we provide a validated speech database of naturalistic scenarios suitable to investigate emotion processing and its time course with German-speaking participants. Information on using the stimulus database for research purposes can be found at the OSF project repository GAUDIE: https://osf.io/xyr6j/ .

Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification.

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Oscillatory neurofeedback networks and poststroke rehabilitative potential in severely impaired stroke patients.

Motor restoration after severe stroke is often limited. However, some of the severely impaired stroke patients may still have a rehabilitative potential. Biomarkers that identify these patients are sparse. Eighteen severely impaired chronic stroke patients with a lack of volitional finger extension participated in an EEG study. During sixty-six trials of kinesthetic motor imagery, a brain-machine interface turned event-related beta-band desynchronization of the ipsilesional sensorimotor cortex into opening of the paralyzed hand by a robotic orthosis. A subgroup of eight patients participated in a subsequent four-week rehabilitation training. Changes of the movement extent were captured with sensors which objectively quantified even discrete improvements of wrist movement. Albeit with the same motor impairment level, patients could be differentiated into two groups, i.e., with and without task-related increase of bilateral cortico-cortical phase synchronization between frontal/premotor and parietal areas. This fronto-parietal integration (FPI) was associated with a significantly higher volitional beta modulation range in the ipsilesional sensorimotor cortex. Following the four-week training, patients with FPI showed significantly higher improvement in wrist movement than those without FPI. Moreover, only the former group improved significantly in the upper extremity Fugl-Meyer-Assessment score. Neurofeedback-related long-range oscillatory coherence may differentiate severely impaired stroke patients with regard to their rehabilitative potential, a finding that needs to be confirmed in larger patient cohorts.

Workload-dependent hemispheric asymmetries during the emotion-cognition interaction: a close-to-naturalistic fNIRS study.

Introduction: We investigated brain activation patterns of interacting emotional distractions and cognitive processes in a close-to-naturalistic functional near-infrared spectroscopy (fNIRS) study. Methods: Eighteen participants engaged in a monitoring-control task, mimicking common air traffic controller requirements. The scenario entailed experiencing both low and high workload, while concurrently being exposed to emotional speech distractions of positive, negative, and neutral valence. Results: Our investigation identified hemispheric asymmetries in prefrontal cortex (PFC) activity during the presentation of negative and positive emotional speech distractions at different workload levels. Thereby, in particular, activation in the left inferior frontal gyrus (IFG) and orbitofrontal cortex (OFC) seems to play a crucial role. Brain activation patterns revealed a cross-over interaction indicating workload-dependent left hemispheric inhibition processes during negative distractions and high workload. For positive emotional distractions under low workload, we observed left-hemispheric PFC recruitment potentially associated with speech-related processes. Furthermore, we found a workload-independent negativity bias for neutral distractions, showing brain activation patterns similar to those of negative distractions. Discussion: In conclusion, lateralized hemispheric processing, regulating emotional speech distractions and integrating emotional and cognitive processes, is influenced by workload levels and stimulus characteristics. These findings advance our understanding of the factors modulating hemispheric asymmetries during the processing and inhibition of emotional distractions, as well as the interplay between emotion and cognition. Moreover, they emphasize the significance of exploring emotion-cognition interactions in more naturalistic settings to gain a deeper understanding of their implications in real-world application scenarios (e.g., working and learning environments).

Combining brain-computer interfaces with deep reinforcement learning for robot training: a feasibility study in a simulation environment.

Deep reinforcement learning (RL) is used as a strategy to teach robot agents how to autonomously learn complex tasks. While sparsity is a natural way to define a reward in realistic robot scenarios, it provides poor learning signals for the agent, thus making the design of good reward functions challenging. To overcome this challenge learning from human feedback through an implicit brain-computer interface (BCI) is used. We combined a BCI with deep RL for robot training in a 3-D physical realistic simulation environment. In a first study, we compared the feasibility of different electroencephalography (EEG) systems (wet- vs. dry-based electrodes) and its application for automatic classification of perceived errors during a robot task with different machine learning models. In a second study, we compared the performance of the BCI-based deep RL training to feedback explicitly given by participants. Our findings from the first study indicate the use of a high-quality dry-based EEG-system can provide a robust and fast method for automatically assessing robot behavior using a sophisticated convolutional neural network machine learning model. The results of our second study prove that the implicit BCI-based deep RL version in combination with the dry EEG-system can significantly accelerate the learning process in a realistic 3-D robot simulation environment. Performance of the BCI-based trained deep RL model was even comparable to that achieved by the approach with explicit human feedback. Our findings emphasize the usage of BCI-based deep RL methods as a valid alternative in those human-robot applications where no access to cognitive demanding explicit human feedback is available.

Brain Oscillation Entrainment by Perceptible and Non-perceptible Rhythmic Light Stimulation.

Objective and Background: Decades of research in the field of steady-state visual evoked potentials (SSVEPs) have revealed great potential of rhythmic light stimulation for brain-computer interfaces. Additionally, rhythmic light stimulation provides a non-invasive method for entrainment of oscillatory activity in the brain. Especially effective protocols enabling non-perceptible rhythmic stimulation and, thereby, reducing eye fatigue and user discomfort are favorable. Here, we investigate effects of (1) perceptible and (2) non-perceptible rhythmic light stimulation as well as attention-based effects of the stimulation by asking participants to focus (a) on the stimulation source directly in an overt attention condition or (b) on a cross-hair below the stimulation source in a covert attention condition. Method: SSVEPs at 10 Hz were evoked with a light-emitting diode (LED) driven by frequency-modulated signals and amplitudes of the current intensity either below or above a previously estimated individual threshold. Furthermore, we explored the effect of attention by asking participants to fixate on the LED directly in the overt attention condition and indirectly attend it in the covert attention condition. By measuring electroencephalography, we analyzed differences between conditions regarding the detection of reliable SSVEPs via the signal-to-noise ratio (SNR) and functional connectivity in occipito-frontal(-central) regions. Results: We could observe SSVEPs at 10 Hz for the perceptible and non-perceptible rhythmic light stimulation not only in the overt but also in the covert attention condition. The SNR and SSVEP amplitudes did not differ between the conditions and SNR values were in all except one participant above significance thresholds suggested by previous literature indicating reliable SSVEP responses. No difference between the conditions could be observed in the functional connectivity in occipito-frontal(-central) regions. Conclusion: The finding of robust SSVEPs even for non-intrusive rhythmic stimulation protocols below an individual perceptibility threshold and without direct fixation on the stimulation source reveals strong potential as a safe stimulation method for oscillatory entrainment in naturalistic applications.

Oscillatory EEG Signatures of Affective Processes during Interaction with Adaptive Computer Systems.

Affect monitoring is being discussed as a novel strategy to make adaptive systems more user-oriented. Basic knowledge about oscillatory processes and functional connectivity underlying affect during naturalistic human-computer interactions (HCI) is, however, scarce. This study assessed local oscillatory power entrainment and distributed functional connectivity in a close-to-naturalistic HCI-paradigm. Sixteen participants interacted with a simulated assistance system which deliberately evoked positive (supporting goal-achievement) and negative (impeding goal-achievement) affective reactions. Electroencephalography (EEG) was used to examine the reactivity of the cortical system during the interaction by studying both event-related (de-)synchronization (ERD/ERS) and event-related functional coupling of cortical networks towards system-initiated assistance. Significantly higher α-band and ß-band ERD in centro-parietal and parieto-occipital regions and ß-band ERD in bi-lateral fronto-central regions were observed during impeding system behavior. Supportive system behavior activated significantly higher γ-band ERS in bi-hemispheric parietal-occipital regions. This was accompanied by functional coupling of remote ß-band and γ-band activity in the medial frontal, left fronto-central and parietal regions, respectively. Our findings identify oscillatory signatures of positive and negative affective processes as reactions to system-initiated assistance. The findings contribute to the development of EEG-based neuroadaptive assistance loops by suggesting a non-obtrusive method for monitoring affect in HCI.

Different oscillatory entrainment of cortical networks during motor imagery and neurofeedback in right and left handers.

Volitional modulation and neurofeedback of sensorimotor oscillatory activity is currently being evaluated as a strategy to facilitate motor restoration following stroke. Knowledge on the interplay between this regional brain self-regulation, distributed network entrainment and handedness is, however, limited. In a randomized cross-over design, twenty-one healthy subjects (twelve right-handers [RH], nine left-handers [LH]) performed kinesthetic motor imagery of left (48 trials) and right finger extension (48 trials). A brain-machine interface turned event-related desynchronization in the beta frequency-band (16-22 Hz) during motor imagery into passive hand opening by a robotic orthosis. Thereby, every participant subsequently activated either the dominant (DH) or non-dominant hemisphere (NDH) to control contralateral hand opening. The task-related cortical networks were studied with electroencephalography. The magnitude of the induced oscillatory modulation range in the sensorimotor cortex was independent of both handedness (RH, LH) and hemispheric specialization (DH, NDH). However, the regional beta-band modulation was associated with different alpha-band networks in RH and LH: RH presented a stronger inter-hemispheric connectivity, while LH revealed a stronger intra-hemispheric interaction. Notably, these distinct network entrainments were independent of hemispheric specialization. In healthy subjects, sensorimotor beta-band activity can be robustly modulated by motor imagery and proprioceptive feedback in both hemispheres independent of handedness. However, right and left handers show different oscillatory entrainment of cortical alpha-band networks during neurofeedback. This finding may inform neurofeedback interventions in future to align them more precisely with the underlying physiology.

What is the optimal task difficulty for reinforcement learning of brain self-regulation?

OBJECTIVE: The balance between action and reward during neurofeedback may influence reinforcement learning of brain self-regulation. METHODS: Eleven healthy volunteers participated in three runs of motor imagery-based brain-machine interface feedback where a robot passively opened the hand contingent to ß-band modulation. For each run, the ß-desynchronization threshold to initiate the hand robot movement increased in difficulty (low, moderate, and demanding). In this context, the incentive to learn was estimated by the change of reward per action, operationalized as the change in reward duration per movement onset. RESULTS: Variance analysis revealed a significant interaction between threshold difficulty and the relationship between reward duration and number of movement onsets (p<0.001), indicating a negative learning incentive for low difficulty, but a positive learning incentive for moderate and demanding runs. Exploration of different thresholds in the same data set indicated that the learning incentive peaked at higher thresholds than the threshold which resulted in maximum classification accuracy. CONCLUSION: Specificity is more important than sensitivity of neurofeedback for reinforcement learning of brain self-regulation. SIGNIFICANCE: Learning efficiency requires adequate challenge by neurofeedback interventions.

Brain state-dependent robotic reaching movement with a multi-joint arm exoskeleton: combining brain-machine interfacing and robotic rehabilitation.

While robot-assisted arm and hand training after stroke allows for intensive task-oriented practice, it has provided only limited additional benefit over dose-matched physiotherapy up to now. These rehabilitation devices are possibly too supportive during the exercises. Neurophysiological signals might be one way of avoiding slacking and providing robotic support only when the brain is particularly responsive to peripheral input. We tested the feasibility of three-dimensional robotic assistance for reaching movements with a multi-joint exoskeleton during motor imagery (MI)-related desynchronization of sensorimotor oscillations in the ß-band. We also registered task-related network changes of cortical functional connectivity by electroencephalography via the imaginary part of the coherence function. Healthy subjects and stroke survivors showed similar patterns-but different aptitudes-of controlling the robotic movement. All participants in this pilot study with nine healthy subjects and two stroke patients achieved their maximum performance during the early stages of the task. Robotic control was significantly higher and less variable when proprioceptive feedback was provided in addition to visual feedback, i.e., when the orthosis was actually attached to the subject's arm during the task. A distributed cortical network of task-related coherent activity in the θ-band showed significant differences between healthy subjects and stroke patients as well as between early and late periods of the task. Brain-robot interfaces (BRIs) may successfully link three-dimensional robotic training to the participants' efforts and allow for task-oriented practice of activities of daily living with a physiologically controlled multi-joint exoskeleton. Changes of cortical physiology during the task might also help to make subject-specific adjustments of task difficulty and guide adjunct interventions to facilitate motor learning for functional restoration, a proposal that warrants further investigation in a larger cohort of stroke patients.

Self-regulation of circumscribed brain activity modulates spatially selective and frequency specific connectivity of distributed resting state networks.

The mechanisms of learning involved in brain self-regulation have still to be unveiled to exploit the full potential of this methodology for therapeutic interventions. This skill of volitionally changing brain activity presumably resembles motor skill learning which in turn is accompanied by plastic changes modulating resting state networks. Along these lines, we hypothesized that brain regulation and neurofeedback would similarly modify intrinsic networks at rest while presenting a distinct spatio-temporal pattern. High-resolution electroencephalography preceded and followed a single neurofeedback training intervention of modulating circumscribed sensorimotor low ß-activity by kinesthetic motor imagery in eleven healthy participants. The participants were kept in the deliberative phase of skill acquisition with high demands for learning self-regulation through stepwise increases of task difficulty. By applying the corrected imaginary part of the coherency function, we observed increased functional connectivity of both the primary motor and the primary somatosensory cortex with their respective contralateral homologous cortices in the low ß-frequency band which was self-regulated during feedback. At the same time, the primary motor cortex-but none of the surrounding cortical areas-showed connectivity to contralateral supplementary motor and dorsal premotor areas in the high ß-band. Simultaneously, the neurofeedback target displayed a specific increase of functional connectivity with an ipsilateral fronto-parietal network in the α-band while presenting a de-coupling with contralateral primary and secondary sensorimotor areas in the very same frequency band. Brain self-regulation modifies resting state connections spatially selective to the neurofeedback target of the dominant hemisphere. These are anatomically distinct with regard to the cortico-cortical connectivity pattern and are functionally specific with regard to the time domain of coherent activity consistent with a Hebbian-like sharpening concept.

Oscillatory entrainment of the motor cortical network during motor imagery is modulated by the feedback modality.

Neurofeedback of self-regulated brain activity in circumscribed cortical regions is used as a novel strategy to facilitate functional restoration following stroke. Basic knowledge about its impact on motor system oscillations and functional connectivity is however scarce. Specifically, a direct comparison between different feedback modalities and their neural signatures is missing. We assessed a neurofeedback training intervention of modulating ß-activity in circumscribed sensorimotor regions by kinesthetic motor imagery (MI). Right-handed healthy participants received two different feedback modalities contingent to their MI-associated brain activity in a cross-over design: (I) visual feedback with a brain-computer interface (BCI) and (II) proprioceptive feedback with a brain-robot interface (BRI) orthosis attached to the right hand. High-density electroencephalography was used to examine the reactivity of the cortical motor system during the training session of each task by studying both local oscillatory power entrainment and distributed functional connectivity. Both feedback modalities activated a distributed functional connectivity network of coherent oscillations. A significantly higher skill and lower variability of self-controlled sensorimotor ß-band modulation could, however, be achieved in the BRI condition. This gain in controlling regional motor oscillations was accompanied by functional coupling of remote ß-band and θ-band activity in bilateral fronto-central regions and left parieto-occipital regions, respectively. The functional coupling of coherent θ-band oscillations correlated moreover with the skill of regional ß-modulation thus revealing a motor learning related network. Our findings indicate that proprioceptive feedback is more suitable than visual feedback to entrain the motor network architecture during the interplay between motor imagery and feedback processing thus resulting in better volitional control of regional brain activity.

Bridging the gap between motor imagery and motor execution with a brain-robot interface.

According to electrophysiological studies motor imagery and motor execution are associated with perturbations of brain oscillations over spatially similar cortical areas. By contrast, neuroimaging and lesion studies suggest that at least partially distinct cortical networks are involved in motor imagery and execution. We sought to further disentangle this relationship by studying the role of brain-robot interfaces in the context of motor imagery and motor execution networks. Twenty right-handed subjects performed several behavioral tasks as indicators for imagery and execution of movements of the left hand, i.e. kinesthetic imagery, visual imagery, visuomotor integration and tonic contraction. In addition, subjects performed motor imagery supported by haptic/proprioceptive feedback from a brain-robot-interface. Principal component analysis was applied to assess the relationship of these indicators. The respective cortical resting state networks in the α-range were investigated by electroencephalography using the phase slope index. We detected two distinct abilities and cortical networks underlying motor control: a motor imagery network connecting the left parietal and motor areas with the right prefrontal cortex and a motor execution network characterized by transmission from the left to right motor areas. We found that a brain-robot-interface might offer a way to bridge the gap between these networks, opening thereby a backdoor to the motor execution system. This knowledge might promote patient screening and may lead to novel treatment strategies, e.g. for the rehabilitation of hemiparesis after stroke.

Lateralized alpha-band cortical networks regulate volitional modulation of beta-band sensorimotor oscillations.

Sensorimotor rhythms (SMRs) are oscillatory brain activities in the α- and ß-bands across the sensorimotor regions of the brain. Each frequency band has its own specific function. The α-band oscillations are closely related to intrinsic cortical networks, whereas oscillations in the ß-band are relevant for the information transfer between the cortex and periphery, as well as for visual and proprioceptive feedback. This study aimed to investigate the interaction between these two frequency bands, under the premise that the regional modulation of ß-band power is linked to a cortical network in the α-band. We therefore designed a procedure to maximize the modulation of ß-band activity over the sensorimotor cortex by combining kinesthetic motor-imagery with closed-loop haptic feedback. The cortical network activity during this procedure was estimated via the phase slope index in electroencephalographic recordings. Analysis of effective connectivity within the α-band network revealed an information flow between the precentral (premotor and primary motor), postcentral (primary somatosensory) and parietal cortical areas. The range of ß-modulation was connected to a reduction of an ipsilateral sensorimotor and parietal α-network and, consequently, to a lateralization of this network to the contralateral side. These results showed that regional sensorimotor oscillatory activity in the ß-band was regulated by cortical coupling of distant areas in the α-band.

Estimating the modulatory effects of nanoparticles on neuronal circuits using computational upscaling.

BACKGROUND: Beside the promising application potential of nanotechnologies in engineering, the use of nanomaterials in medicine is growing. New therapies employing innovative nanocarrier systems to increase specificity and efficacy of drug delivery schemes are already in clinical trials. However the influence of the nanoparticles themselves is still unknown in medical applications, especially for complex interactions in neural systems. The aim of this study was to investigate in vitro effects of coated silver nanoparticles (cAgNP) on the excitability of single neuronal cells and to integrate those findings into an in silico model to predict possible effects on neuronal circuits. METHODS: We first performed patch clamp measurements to investigate the effects of nanosized silver particles, surrounded by an organic coating, on excitability of single cells. We then determined which parameters were altered by exposure to those nanoparticles using the Hodgkin-Huxley model of the sodium current. As a third step, we integrated those findings into a well-defined neuronal circuit of thalamocortical interactions to predict possible changes in network signaling due to the applied cAgNP, in silico. RESULTS: We observed rapid suppression of sodium currents after exposure to cAgNP in our in vitro recordings. In numerical simulations of sodium currents we identified the parameters likely affected by cAgNP. We then examined the effects of such changes on the activity of networks. In silico network modeling indicated effects of local cAgNP application on firing patterns in all neurons in the circuit. CONCLUSION: Our sodium current simulation shows that suppression of sodium currents by cAgNP results primarily by a reduction in the amplitude of the current. The network simulation shows that locally cAgNP-induced changes result in changes in network activity in the entire network, indicating that local application of cAgNP may influence the activity throughout the network.