GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database.

Since thoroughly validated naturalistic affective German speech stimulus databases are rare, we present here a novel validated database of speech sequences assembled with the purpose of emotion induction. The database comprises 37 audio speech sequences with a total duration of 92 minutes for the induction of positive, neutral, and negative emotion: comedian shows intending to elicit humorous and amusing feelings, weather forecasts, and arguments between couples and relatives from movies or television series. Multiple continuous and discrete ratings are used to validate the database to capture the time course and variabilities of valence and arousal. We analyse and quantify how well the audio sequences fulfil quality criteria of differentiation, salience/strength, and generalizability across participants. Hence, we provide a validated speech database of naturalistic scenarios suitable to investigate emotion processing and its time course with German-speaking participants. Information on using the stimulus database for research purposes can be found at the OSF project repository GAUDIE: https://osf.io/xyr6j/ .

Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification.

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Combining brain-computer interfaces with deep reinforcement learning for robot training: a feasibility study in a simulation environment.

Deep reinforcement learning (RL) is used as a strategy to teach robot agents how to autonomously learn complex tasks. While sparsity is a natural way to define a reward in realistic robot scenarios, it provides poor learning signals for the agent, thus making the design of good reward functions challenging. To overcome this challenge learning from human feedback through an implicit brain-computer interface (BCI) is used. We combined a BCI with deep RL for robot training in a 3-D physical realistic simulation environment. In a first study, we compared the feasibility of different electroencephalography (EEG) systems (wet- vs. dry-based electrodes) and its application for automatic classification of perceived errors during a robot task with different machine learning models. In a second study, we compared the performance of the BCI-based deep RL training to feedback explicitly given by participants. Our findings from the first study indicate the use of a high-quality dry-based EEG-system can provide a robust and fast method for automatically assessing robot behavior using a sophisticated convolutional neural network machine learning model. The results of our second study prove that the implicit BCI-based deep RL version in combination with the dry EEG-system can significantly accelerate the learning process in a realistic 3-D robot simulation environment. Performance of the BCI-based trained deep RL model was even comparable to that achieved by the approach with explicit human feedback. Our findings emphasize the usage of BCI-based deep RL methods as a valid alternative in those human-robot applications where no access to cognitive demanding explicit human feedback is available.

Workload-dependent hemispheric asymmetries during the emotion-cognition interaction: a close-to-naturalistic fNIRS study.

Introduction: We investigated brain activation patterns of interacting emotional distractions and cognitive processes in a close-to-naturalistic functional near-infrared spectroscopy (fNIRS) study. Methods: Eighteen participants engaged in a monitoring-control task, mimicking common air traffic controller requirements. The scenario entailed experiencing both low and high workload, while concurrently being exposed to emotional speech distractions of positive, negative, and neutral valence. Results: Our investigation identified hemispheric asymmetries in prefrontal cortex (PFC) activity during the presentation of negative and positive emotional speech distractions at different workload levels. Thereby, in particular, activation in the left inferior frontal gyrus (IFG) and orbitofrontal cortex (OFC) seems to play a crucial role. Brain activation patterns revealed a cross-over interaction indicating workload-dependent left hemispheric inhibition processes during negative distractions and high workload. For positive emotional distractions under low workload, we observed left-hemispheric PFC recruitment potentially associated with speech-related processes. Furthermore, we found a workload-independent negativity bias for neutral distractions, showing brain activation patterns similar to those of negative distractions. Discussion: In conclusion, lateralized hemispheric processing, regulating emotional speech distractions and integrating emotional and cognitive processes, is influenced by workload levels and stimulus characteristics. These findings advance our understanding of the factors modulating hemispheric asymmetries during the processing and inhibition of emotional distractions, as well as the interplay between emotion and cognition. Moreover, they emphasize the significance of exploring emotion-cognition interactions in more naturalistic settings to gain a deeper understanding of their implications in real-world application scenarios (e.g., working and learning environments).

Effects of the COVID-19 Pandemic on Psychological Well-Being and Mental Health Based on a German Online Survey.

Objective and Background: To contain the COVID-19 pandemic, public health actions have changed the everyday life with an inevitable impact on individuals and their social life. Since intact (socio-)psychological functioning and mental health are protective factors contributing to the immune system and preventing diseases, it is crucial to identify individuals with increased vulnerability. Methods: We conducted a German online survey from April until August 2020 investigating health-related, social, behavioral, and psychological effects of the COVID-19 pandemic. One hundred and seventy three adults participating in the survey were analyzed (39.9% male, age: M = 44.81±13.31). We explored effects on mental health by (a) clustering participants in two clusters and (b) analyzing the clusters using correlations and regression models. Results: Participants belonged either to a cluster characterized by higher general well-being or to a more concerned cluster depending on their responses. The correlation analysis revealed a significant negative relation between age and well-being with younger participants revealing higher depression scores in the concerned cluster. Furthermore, multiple regression models revealed that the number of risk factors only has a significant influence on psychological well-being in the concerned but not in the comfortable cluster. Conclusion: We found that especially participants at (a) younger age and (b) greater risk of a severe course of disease reported reduced mental well-being and seemed to be weakened in their psychological protective factors in our sample. These insights allow to provide tailored recommendations for preventive and immediate actions to promote psychological well-being and reduce stress.

Brain Oscillation Entrainment by Perceptible and Non-perceptible Rhythmic Light Stimulation.

Objective and Background: Decades of research in the field of steady-state visual evoked potentials (SSVEPs) have revealed great potential of rhythmic light stimulation for brain-computer interfaces. Additionally, rhythmic light stimulation provides a non-invasive method for entrainment of oscillatory activity in the brain. Especially effective protocols enabling non-perceptible rhythmic stimulation and, thereby, reducing eye fatigue and user discomfort are favorable. Here, we investigate effects of (1) perceptible and (2) non-perceptible rhythmic light stimulation as well as attention-based effects of the stimulation by asking participants to focus (a) on the stimulation source directly in an overt attention condition or (b) on a cross-hair below the stimulation source in a covert attention condition. Method: SSVEPs at 10 Hz were evoked with a light-emitting diode (LED) driven by frequency-modulated signals and amplitudes of the current intensity either below or above a previously estimated individual threshold. Furthermore, we explored the effect of attention by asking participants to fixate on the LED directly in the overt attention condition and indirectly attend it in the covert attention condition. By measuring electroencephalography, we analyzed differences between conditions regarding the detection of reliable SSVEPs via the signal-to-noise ratio (SNR) and functional connectivity in occipito-frontal(-central) regions. Results: We could observe SSVEPs at 10 Hz for the perceptible and non-perceptible rhythmic light stimulation not only in the overt but also in the covert attention condition. The SNR and SSVEP amplitudes did not differ between the conditions and SNR values were in all except one participant above significance thresholds suggested by previous literature indicating reliable SSVEP responses. No difference between the conditions could be observed in the functional connectivity in occipito-frontal(-central) regions. Conclusion: The finding of robust SSVEPs even for non-intrusive rhythmic stimulation protocols below an individual perceptibility threshold and without direct fixation on the stimulation source reveals strong potential as a safe stimulation method for oscillatory entrainment in naturalistic applications.

Oscillatory EEG Signatures of Affective Processes during Interaction with Adaptive Computer Systems.

Affect monitoring is being discussed as a novel strategy to make adaptive systems more user-oriented. Basic knowledge about oscillatory processes and functional connectivity underlying affect during naturalistic human-computer interactions (HCI) is, however, scarce. This study assessed local oscillatory power entrainment and distributed functional connectivity in a close-to-naturalistic HCI-paradigm. Sixteen participants interacted with a simulated assistance system which deliberately evoked positive (supporting goal-achievement) and negative (impeding goal-achievement) affective reactions. Electroencephalography (EEG) was used to examine the reactivity of the cortical system during the interaction by studying both event-related (de-)synchronization (ERD/ERS) and event-related functional coupling of cortical networks towards system-initiated assistance. Significantly higher α-band and ß-band ERD in centro-parietal and parieto-occipital regions and ß-band ERD in bi-lateral fronto-central regions were observed during impeding system behavior. Supportive system behavior activated significantly higher γ-band ERS in bi-hemispheric parietal-occipital regions. This was accompanied by functional coupling of remote ß-band and γ-band activity in the medial frontal, left fronto-central and parietal regions, respectively. Our findings identify oscillatory signatures of positive and negative affective processes as reactions to system-initiated assistance. The findings contribute to the development of EEG-based neuroadaptive assistance loops by suggesting a non-obtrusive method for monitoring affect in HCI.