Detection of arousal and valence from facial expressions and physiological responses evoked by different types of stressors.

Automatically detecting mental state such as stress from video images of the face could support evaluating stress responses in applicants for high risk jobs or contribute to timely stress detection in challenging operational settings (e.g., aircrew, command center operators). Challenges in automatically estimating mental state include the generalization of models across contexts and across participants. We here aim to create robust models by training them using data from different contexts and including physiological features. Fifty-one participants were exposed to different types of stressors (cognitive, social evaluative and startle) and baseline variants of the stressors. Video, electrocardiogram (ECG), electrodermal activity (EDA) and self-reports (arousal and valence) were recorded. Logistic regression models aimed to classify between high and low arousal and valence across participants, where "high" and "low" were defined relative to the center of the rating scale. Accuracy scores of different models were evaluated: models trained and tested within a specific context (either a baseline or stressor variant of a task), intermediate context (baseline and stressor variant of a task), or general context (all conditions together). Furthermore, for these different model variants, only the video data was included, only the physiological data, or both video and physiological data. We found that all (video, physiological and video-physio) models could successfully distinguish between high- and low-rated arousal and valence, though performance tended to be better for (1) arousal than valence, (2) specific context than intermediate and general contexts, (3) video-physio data than video or physiological data alone. Automatic feature selection resulted in inclusion of 3-20 features, where the models based on video-physio data usually included features from video, ECG and EDA. Still, performance of video-only models approached the performance of video-physio models. Arousal and valence ratings by three experienced human observers scores based on part of the video data did not match with self-reports. In sum, we showed that it is possible to automatically monitor arousal and valence even in relatively general contexts and better than humans can (in the given circumstances), and that non-contact video images of faces capture an important part of the information, which has practical advantages.

Using Galvanic Vestibular Stimulation to Induce Post-Roll Illusion in a Fixed-Base Flight Simulator.

INTRODUCTION: The illusions of head motion induced by galvanic vestibular stimulation (GVS) can be used to compromise flight performance of pilots in fixed-base simulators. However, the stimuli used in the majority of studies fail to mimic disorientation in realistic flight because they are independent from the simulated aircraft motion. This study investigated the potential of bilateral-bipolar GVS coupled to aircraft roll in a fixed-base simulator to mimic vestibular spatial disorientation illusions, specifically the "post-roll illusion" observed during flight.METHODS: There were 14 nonpilot subjects exposed to roll stimuli in a flight simulator operating in a fixed-base mode. GVS was delivered via carbon rubber electrodes on the mastoid processes. The electrical stimulus was driven by the high-pass filtered aircraft roll rate to mimic the semicircular canals' physiological response. The post-roll test scenarios excluded outside visual cues or instruments and required subjects to actively maintain a constant bank angle after an abrupt stop following a passive prolonged roll maneuver. The anticipated outcome was an overshot in roll elicited by the GVS signal.RESULTS: The responses across subjects showed large variability, with less than a third aligning with the post-roll illusion. Subjective ratings suggest that the high-pass filtered GVS stimuli were mild and did not induce a clear sense of roll direction. However, uncontrolled head movements during stimulation might have obscured the intended effects of GVS-evoked illusory head movements.CONCLUSION: The mild and transient GVS stimuli used in this study, together with the uncontrolled head movements, did not convincingly mimic the post-roll illusion.Houben MMJ, Stuldreher IV, Forbes PA, Groen EL. Using galvanic vestibular stimulation to induce post-roll illusion in a fixed-base flight simulator. Aerosp Med Hum Perform. 2024; 95(2):84-92.

The Interplay of Peer Victimization and Parasympathetic Nervous System Activity on Acute Inflammatory Stress Responses in Adolescence.

This study examined the extent to which adolescent peer victimization predicted acute inflammatory responses to stress, and whether both resting parasympathetic nervous system (PNS) activity and PNS stress reactivity moderated this association. 83 adolescents (Mage = 14.89, SDage = 0.52, 48% female) reported their history of peer victimization and were exposed to a standardized social stress task before and after which dried blood spot samples were collected to assay inflammatory markers. Inflammatory responses to the stress task were assessed with a latent inflammatory change factor using the cytokines interleukin-6 (IL-6), interleukin-10 (IL-10), and tumor necrosis factor-α (TNF-α). PNS functioning, indexed by high-frequency heart rate variability, was measured at rest and during the stressor. Contrary to hypotheses, analyses revealed no direct relation between peer victimization and acute inflammatory responses, and resting PNS activity did not moderate this association. However, peer victimization predicted stronger inflammatory responses among adolescents with weaker PNS reactivity to the stress task (b = 0.63, p = .02). This association was not observed among adolescents with stronger PNS reactivity, for whom a negative but non-significant trend was found. Weaker PNS reactivity may thus indicate victimized adolescents' vulnerability for acute inflammatory responses, whereas stronger PNS reactivity may indicate adolescents' resilience to a social stressor.

Robustness of Physiological Synchrony in Wearable Electrodermal Activity and Heart Rate as a Measure of Attentional Engagement to Movie Clips.

Individuals that pay attention to narrative stimuli show synchronized heart rate (HR) and electrodermal activity (EDA) responses. The degree to which this physiological synchrony occurs is related to attentional engagement. Factors that can influence attention, such as instructions, salience of the narrative stimulus and characteristics of the individual, affect physiological synchrony. The demonstrability of synchrony depends on the amount of data used in the analysis. We investigated how demonstrability of physiological synchrony varies with varying group size and stimulus duration. Thirty participants watched six 10 min movie clips while their HR and EDA were monitored using wearable sensors (Movisens EdaMove 4 and Wahoo Tickr, respectively). We calculated inter-subject correlations as a measure of synchrony. Group size and stimulus duration were varied by using data from subsets of the participants and movie clips in the analysis. We found that for HR, higher synchrony correlated significantly with the number of answers correct for questions about the movie, confirming that physiological synchrony is associated with attention. For both HR and EDA, with increasing amounts of data used, the percentage of participants with significant synchrony increased. Importantly, we found that it did not matter how the amount of data was increased. Increasing the group size or increasing the stimulus duration led to the same results. Initial comparisons with results from other studies suggest that our results do not only apply to our specific set of stimuli and participants. All in all, the current work can act as a guideline for future research, indicating the amount of data minimally needed for robust analysis of synchrony based on inter-subject correlations.

Electrodermal activity as an index of food neophobia outside the lab.

Introduction: Understanding how food neophobia affects food experience may help to shift toward sustainable diets. Previous research suggests that individuals with higher food neophobia are more aroused and attentive when observing food-related stimuli. The present study examined whether electrodermal activity (EDA), as index of arousal, relates to food neophobia outside the lab when exposed to a single piece of food. Methods: The EDA of 153 participants was analyzed as part of a larger experiment conducted at a festival. Participants completed the 10-item Food Neophobia Scale. Subsequently, they saw three lids covering three foods: a hotdog labeled as "meat", a hotdog labeled as "100% plant-based", and tofu labeled as "100% plant-based". Participants lifted the lids consecutively and the area-under-the-curve (AUC) of the skin conductance response (SCR) was captured between 20 s before and 20 s after each food reveal. Results: We found a significant positive correlation between food neophobia and AUC of SCR during presentation of the first and second hotdog and a trend for tofu. These correlations remained significant even when only including the SCR data prior to the food reveal (i.e., an anticipatory response). Discussion: The association between food neophobia and EDA indicates that food neophobic individuals are more aroused upon the presentation of food. We show for the first time that the anticipation of being presented with food already increased arousal for food neophobic individuals. These findings also indicate that EDA can be meaningfully determined using wearables outside the lab, in a relatively uncontrolled setting for single-trial analysis.

Physiological synchrony in electrodermal activity predicts decreased vigilant attention induced by sleep deprivation.

Introduction: When multiple individuals are presented with narrative movie or audio clips, their electrodermal activity (EDA) and heart rate show significant similarities. Higher levels of such inter-subject physiological synchrony are related with higher levels of attention toward the narrative, as for instance expressed by more correctly answered questions about the narrative. We here investigate whether physiological synchrony in EDA and heart rate during watching of movie clips predicts performance on a subsequent vigilant attention task among participants exposed to a night of total sleep deprivation. Methods: We recorded EDA and heart rate of 54 participants during a night of total sleep deprivation. Every hour from 22:00 to 07:00 participants watched a 10-min movie clip during which we computed inter-subject physiological synchrony. Afterwards, they answered questions about the movie and performed the psychomotor vigilance task (PVT) to capture attentional performance. Results: We replicated findings that inter-subject correlations in EDA and heart rate predicted the number of correct answers on questions about the movie clips. Furthermore, we found that inter-subject correlations in EDA, but not in heart rate, predicted PVT performance. Individuals' mean EDA and heart rate also predicted their PVT performance. For EDA, inter-subject correlations explained more variance of PVT performance than individuals' mean EDA. Discussion: Together, these findings confirm the association between physiological synchrony and attention. Physiological synchrony in EDA does not only capture the attentional processing during the time that it is determined, but also proves valuable for capturing more general changes in the attentional state of monitored individuals.

Measuring Implicit Approach-Avoidance Tendencies towards Food Using a Mobile Phone outside the Lab.

Implicit ('unconscious') approach-avoidance tendencies towards stimuli can be measured using the Approach Avoidance Task (AAT). We recently expanded a toolbox for analyzing the raw data of a novel, mobile version of the AAT (mAAT), that asks participants to move their phone towards their face (pull) or away (push) in response to images presented on the phone. We here tested the mAAT reaction time and the mAAT distance in a study with 71 Dutch participants that were recruited online and performed an experiment without coming to the laboratory. The participants used both the mAAT and (explicit) rating scales to respond to photographic images of food. As hypothesized, the rated wanting, rated valence and mAAT reaction time indicated a preference for palatable over unpalatable food, and for Dutch over Asian food. Additionally, as expected, arousal was rated higher for unpalatable than for palatable food, and higher for Dutch than for Asian food. The mAAT distance indicated that the unpalatable food images were moved across larger distances, regardless of the movement direction (pull or push), compared to the palatable food images; and the Dutch food images were moved across larger distances than the Asian food images. We conclude that the mAAT can be used to implicitly probe approach-avoidance motivation for complex images in the food domain. The new measure of mAAT distance may be used as an implicit measure of arousal. The ratings and the mAAT measures do not reflect the exact same information and may complement each other. Implicit measures, such as mAAT variables, are particularly valuable when response biases that can occur when using explicit ratings are expected.

Unsupervised Clustering of Individuals Sharing Selective Attentional Focus Using Physiological Synchrony.

Research on brain signals as indicators of a certain attentional state is moving from laboratory environments to everyday settings. Uncovering the attentional focus of individuals in such settings is challenging because there is usually limited information about real-world events, as well as a lack of data from the real-world context at hand that is correctly labeled with respect to individuals' attentional state. In most approaches, such data is needed to train attention monitoring models. We here investigate whether unsupervised clustering can be combined with physiological synchrony in the electroencephalogram (EEG), electrodermal activity (EDA), and heart rate to automatically identify groups of individuals sharing attentional focus without using knowledge of the sensory stimuli or attentional focus of any of the individuals. We used data from an experiment in which 26 participants listened to an audiobook interspersed with emotional sounds and beeps. Thirteen participants were instructed to focus on the narrative of the audiobook and 13 participants were instructed to focus on the interspersed emotional sounds and beeps. We used a broad range of commonly applied dimensionality reduction ordination techniques-further referred to as mappings-in combination with unsupervised clustering algorithms to identify the two groups of individuals sharing attentional focus based on physiological synchrony. Analyses were performed using the three modalities EEG, EDA, and heart rate separately, and using all possible combinations of these modalities. The best unimodal results were obtained when applying clustering algorithms on physiological synchrony data in EEG, yielding a maximum clustering accuracy of 85%. Even though the use of EDA or heart rate by itself did not lead to accuracies significantly higher than chance level, combining EEG with these measures in a multimodal approach generally resulted in higher classification accuracies than when using only EEG. Additionally, classification results of multimodal data were found to be more consistent across algorithms than unimodal data, making algorithm choice less important. Our finding that unsupervised classification into attentional groups is possible is important to support studies on attentional engagement in everyday settings.

Physiological Synchrony in EEG, Electrodermal Activity and Heart Rate Detects Attentionally Relevant Events in Time.

Interpersonal physiological synchrony (PS), or the similarity of physiological signals between individuals over time, may be used to detect attentionally engaging moments in time. We here investigated whether PS in the electroencephalogram (EEG), electrodermal activity (EDA), heart rate and a multimodal metric signals the occurrence of attentionally relevant events in time in two groups of participants. Both groups were presented with the same auditory stimulus, but were instructed to attend either to the narrative of an audiobook (audiobook-attending: AA group) or to interspersed emotional sounds and beeps (stimulus-attending: SA group). We hypothesized that emotional sounds could be detected in both groups as they are expected to draw attention involuntarily, in a bottom-up fashion. Indeed, we found this to be the case for PS in EDA or the multimodal metric. Beeps, that are expected to be only relevant due to specific "top-down" attentional instructions, could indeed only be detected using PS among SA participants, for EDA, EEG and the multimodal metric. We further hypothesized that moments in the audiobook accompanied by high PS in either EEG, EDA, heart rate or the multimodal metric for AA participants would be rated as more engaging by an independent group of participants compared to moments corresponding to low PS. This hypothesis was not supported. Our results show that PS can support the detection of attentionally engaging events over time. Currently, the relation between PS and engagement is only established for well-defined, interspersed stimuli, whereas the relation between PS and a more abstract self-reported metric of engagement over time has not been established. As the relation between PS and engagement is dependent on event type and physiological measure, we suggest to choose a measure matching with the stimulus of interest. When the stimulus type is unknown, a multimodal metric is most robust.

A Usability Study of Physiological Measurement in School Using Wearable Sensors.

Measuring psychophysiological signals of adolescents using unobtrusive wearable sensors may contribute to understanding the development of emotional disorders. This study investigated the feasibility of measuring high quality physiological data and examined the validity of signal processing in a school setting. Among 86 adolescents, a total of more than 410 h of electrodermal activity (EDA) data were recorded using a wrist-worn sensor with gelled electrodes and over 370 h of heart rate data were recorded using a chest-strap sensor. The results support the feasibility of monitoring physiological signals at school. We describe specific challenges and provide recommendations for signal analysis, including dealing with invalid signals due to loose sensors, and quantization noise that can be caused by limitations in analog-to-digital conversion in wearable devices and be mistaken as physiological responses. Importantly, our results show that using toolboxes for automatic signal preprocessing, decomposition, and artifact detection with default parameters while neglecting differences between devices and measurement contexts yield misleading results. Time courses of students' physiological signals throughout the course of a class were found to be clearer after applying our proposed preprocessing steps.

Physiological synchrony in EEG, electrodermal activity and heart rate reflects shared selective auditory attention.

OBJECTIVE: Concurrent changes in physiological signals across multiple listeners (physiological synchrony-PS), as caused by shared affective or cognitive processes, may be a suitable marker of selective attentional focus. We aimed to identify the selective attention of participants based on PS with individuals sharing attention with respect to different stimulus aspects. APPROACH: We determined PS in electroencephalography (EEG), electrodermal activity (EDA) and electrocardiographic inter-beat interval (IBI) of participants who all heard the exact same audio track, but were instructed to either attend to the audiobook or to interspersed auditory events such as affective sounds and beeps that attending participants needed to keep track of. MAIN RESULTS: PS in all three measures reflected the selective attentional focus of participants. In EEG and EDA, PS was higher for participants when linked to participants with the same attentional instructions than when linked to participants instructed to focus on different stimulus aspects, but in IBI this effect did not reach significance. Comparing PS between a participant and members from the same or the different attentional group allowed for the correct identification of the participant's attentional instruction in 96%, 73% and 73% of the cases, for EEG, EDA and IBI, respectively, all well above chance level. PS with respect to the attentional groups also predicted performance on post-audio questions about the groups' stimulus content. SIGNIFICANCE: Our results show that selective attention of participants can be monitored using PS, not only in EEG, but also in EDA and IBI. These results are promising for real-world applications, where wearables measuring peripheral signals like EDA and IBI may be preferred over EEG sensors.