Robustness of Physiological Synchrony in Wearable Electrodermal Activity and Heart Rate as a Measure of Attentional Engagement to Movie Clips.

Individuals that pay attention to narrative stimuli show synchronized heart rate (HR) and electrodermal activity (EDA) responses. The degree to which this physiological synchrony occurs is related to attentional engagement. Factors that can influence attention, such as instructions, salience of the narrative stimulus and characteristics of the individual, affect physiological synchrony. The demonstrability of synchrony depends on the amount of data used in the analysis. We investigated how demonstrability of physiological synchrony varies with varying group size and stimulus duration. Thirty participants watched six 10 min movie clips while their HR and EDA were monitored using wearable sensors (Movisens EdaMove 4 and Wahoo Tickr, respectively). We calculated inter-subject correlations as a measure of synchrony. Group size and stimulus duration were varied by using data from subsets of the participants and movie clips in the analysis. We found that for HR, higher synchrony correlated significantly with the number of answers correct for questions about the movie, confirming that physiological synchrony is associated with attention. For both HR and EDA, with increasing amounts of data used, the percentage of participants with significant synchrony increased. Importantly, we found that it did not matter how the amount of data was increased. Increasing the group size or increasing the stimulus duration led to the same results. Initial comparisons with results from other studies suggest that our results do not only apply to our specific set of stimuli and participants. All in all, the current work can act as a guideline for future research, indicating the amount of data minimally needed for robust analysis of synchrony based on inter-subject correlations.

Technology to Automatically Record Eating Behavior in Real Life: A Systematic Review.

To monitor adherence to diets and to design and evaluate nutritional interventions, it is essential to obtain objective knowledge about eating behavior. In most research, measures of eating behavior are based on self-reporting, such as 24-h recalls, food records (food diaries) and food frequency questionnaires. Self-reporting is prone to inaccuracies due to inaccurate and subjective recall and other biases. Recording behavior using nonobtrusive technology in daily life would overcome this. Here, we provide an up-to-date systematic overview encompassing all (close-to) publicly or commercially available technologies to automatically record eating behavior in real-life settings. A total of 1328 studies were screened and, after applying defined inclusion and exclusion criteria, 122 studies were included for in-depth evaluation. Technologies in these studies were categorized by what type of eating behavior they measure and which type of sensor technology they use. In general, we found that relatively simple sensors are often used. Depending on the purpose, these are mainly motion sensors, microphones, weight sensors and photo cameras. While several of these technologies are commercially available, there is still a lack of publicly available algorithms that are needed to process and interpret the resulting data. We argue that future work should focus on developing robust algorithms and validating these technologies in real-life settings. Combining technologies (e.g., prompting individuals for self-reports at sensed, opportune moments) is a promising route toward ecologically valid studies of eating behavior.

Eye tracking: empirical foundations for a minimal reporting guideline.

In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research (Section "An empirically based minimal reporting guideline").

Early Detection of Exposure to Toxic Chemicals Using Continuously Recorded Multi-Sensor Physiology.

Early detection of exposure to a toxic chemical, e.g., in a military context, can be life-saving. We propose to use machine learning techniques and multiple continuously measured physiological signals to detect exposure, and to identify the chemical agent. Such detection and identification could be used to alert individuals to take appropriate medical counter measures in time. As a first step, we evaluated whether exposure to an opioid (fentanyl) or a nerve agent (VX) could be detected in freely moving guinea pigs using features from respiration, electrocardiography (ECG) and electroencephalography (EEG), where machine learning models were trained and tested on different sets (across subject classification). Results showed this to be possible with close to perfect accuracy, where respiratory features were most relevant. Exposure detection accuracy rose steeply to over 95% correct during the first five minutes after exposure. Additional models were trained to correctly classify an exposed state as being induced either by fentanyl or VX. This was possible with an accuracy of almost 95%, where EEG features proved to be most relevant. Exposure detection models that were trained on subsets of animals generalized to subsets of animals that were exposed to other dosages of different chemicals. While future work is required to validate the principle in other species and to assess the robustness of the approach under different, realistic circumstances, our results indicate that utilizing different continuously measured physiological signals for early detection and identification of toxic agents is promising.

A Usability Study of Physiological Measurement in School Using Wearable Sensors.

Measuring psychophysiological signals of adolescents using unobtrusive wearable sensors may contribute to understanding the development of emotional disorders. This study investigated the feasibility of measuring high quality physiological data and examined the validity of signal processing in a school setting. Among 86 adolescents, a total of more than 410 h of electrodermal activity (EDA) data were recorded using a wrist-worn sensor with gelled electrodes and over 370 h of heart rate data were recorded using a chest-strap sensor. The results support the feasibility of monitoring physiological signals at school. We describe specific challenges and provide recommendations for signal analysis, including dealing with invalid signals due to loose sensors, and quantization noise that can be caused by limitations in analog-to-digital conversion in wearable devices and be mistaken as physiological responses. Importantly, our results show that using toolboxes for automatic signal preprocessing, decomposition, and artifact detection with default parameters while neglecting differences between devices and measurement contexts yield misleading results. Time courses of students' physiological signals throughout the course of a class were found to be clearer after applying our proposed preprocessing steps.

Explicit and Implicit Responses to Tasting Drinks Associated with Different Tasting Experiences.

Probing food experience or liking through verbal ratings has its shortcomings. We compare explicit ratings to a range of (neuro)physiological and behavioral measures with respect to their performance in distinguishing drinks associated with different emotional experience. Seventy participants tasted and rated the valence and arousal of eight regular drinks and a "ground truth" high-arousal, low-valence vinegar solution. The discriminative power for distinguishing between the vinegar solution and the regular drinks was highest for sip size, followed by valence ratings, arousal ratings, heart rate, skin conductance level, facial expression of "disgust," pupil diameter, and Electroencephalogram (EEG) frontal alpha asymmetry. Within the regular drinks, a positive correlation was found between rated arousal and heart rate, and a negative correlation between rated arousal and Heart Rate Variability (HRV). Most physiological measures showed consistent temporal patterns over time following the announcement of the drink and taking a sip. This was consistent over all nine drinks, but the peaks were substantially higher for the vinegar solution than for the regular drinks, likely caused by emotion. Our results indicate that implicit variables have the potential to differentiate between drinks associated with different emotional experiences. In addition, this study gives us insight into the physiological temporal response patterns associated with taking a sip.

Reproducible machine learning research in mental workload classification using EEG.

This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.

Detection of arousal and valence from facial expressions and physiological responses evoked by different types of stressors.

Automatically detecting mental state such as stress from video images of the face could support evaluating stress responses in applicants for high risk jobs or contribute to timely stress detection in challenging operational settings (e.g., aircrew, command center operators). Challenges in automatically estimating mental state include the generalization of models across contexts and across participants. We here aim to create robust models by training them using data from different contexts and including physiological features. Fifty-one participants were exposed to different types of stressors (cognitive, social evaluative and startle) and baseline variants of the stressors. Video, electrocardiogram (ECG), electrodermal activity (EDA) and self-reports (arousal and valence) were recorded. Logistic regression models aimed to classify between high and low arousal and valence across participants, where "high" and "low" were defined relative to the center of the rating scale. Accuracy scores of different models were evaluated: models trained and tested within a specific context (either a baseline or stressor variant of a task), intermediate context (baseline and stressor variant of a task), or general context (all conditions together). Furthermore, for these different model variants, only the video data was included, only the physiological data, or both video and physiological data. We found that all (video, physiological and video-physio) models could successfully distinguish between high- and low-rated arousal and valence, though performance tended to be better for (1) arousal than valence, (2) specific context than intermediate and general contexts, (3) video-physio data than video or physiological data alone. Automatic feature selection resulted in inclusion of 3-20 features, where the models based on video-physio data usually included features from video, ECG and EDA. Still, performance of video-only models approached the performance of video-physio models. Arousal and valence ratings by three experienced human observers scores based on part of the video data did not match with self-reports. In sum, we showed that it is possible to automatically monitor arousal and valence even in relatively general contexts and better than humans can (in the given circumstances), and that non-contact video images of faces capture an important part of the information, which has practical advantages.

Seven robust and easy to obtain biomarkers to measure acute stress.

With the purpose of identifying a sensitive, robust, and easy-to-measure set of biomarkers to assess stress reactivity, we here study a large set of relatively easy to obtain markers reflecting subjective, autonomic nervous system (ANS), endocrine, and inflammatory responses to acute social stress (n = 101). A subset of the participants was exposed to another social stressor the next day (n = 48) while being measured in the same way. Acute social stress was induced following standardized procedures. The markers investigated were self-reported positive and negative affect, heart rate, electrodermal activity, salivary cortisol, and ten inflammatory markers both in capillary plasma and salivary samples, including IL-22 which has not been studied in response to acute stress in humans before. Robust effects (significant effect in the same direction for both days) were found for self-reported negative affect, heart rate, electrodermal activity, plasma IL-5, plasma IL-22, salivary IL-8 and salivary IL-10. Of these seven markers, the participants' IL-22 responses on the first day were positively correlated to those on the second day. We found no correlations between salivary and capillary plasma stress responses for any of the ten cytokines and somewhat unexpectedly, cytokine responses in saliva seemed more pronounced and more in line with previous literature than cytokines in capillary plasma. In sum, seven robust and easy to obtain biomarkers to measure acute stress response were identified and should be used in future stress research to detect and examine stress reactivity. This includes IL-22 in plasma as a promising novel marker.

Distinguishing between target and nontarget fixations in a visual search task using fixation-related potentials.

The P300 event-related potential (ERP) can be used to infer whether an observer is looking at a target or not. Common practice in P300 experiments and applications is that observers are asked to fixate their eyes while stimuli are presented. We investigated the possibility to differentiate between single target and nontarget fixations in a target search task involving eye movements by using EEG epochs synchronized to fixation onset (fixation-related potentials: FRPs). Participants systematically scanned search displays consisting of six small Landolt Cs in search of Cs with a particular orientation. After each search display, they indicated whether and where target Cs had been presented. As expected, an FRP component consistent with the P300 reliably distinguished between target and nontarget fixations. It was possible to classify single FRPs into target and nontarget FRPs above chance (on average 62% correct, where 50% would be chance). These results are the first step to practical applications such as covertly monitoring observers' interests and supporting search tasks.

The Interplay of Peer Victimization and Parasympathetic Nervous System Activity on Acute Inflammatory Stress Responses in Adolescence.

This study examined the extent to which adolescent peer victimization predicted acute inflammatory responses to stress, and whether both resting parasympathetic nervous system (PNS) activity and PNS stress reactivity moderated this association. 83 adolescents (Mage = 14.89, SDage = 0.52, 48% female) reported their history of peer victimization and were exposed to a standardized social stress task before and after which dried blood spot samples were collected to assay inflammatory markers. Inflammatory responses to the stress task were assessed with a latent inflammatory change factor using the cytokines interleukin-6 (IL-6), interleukin-10 (IL-10), and tumor necrosis factor-α (TNF-α). PNS functioning, indexed by high-frequency heart rate variability, was measured at rest and during the stressor. Contrary to hypotheses, analyses revealed no direct relation between peer victimization and acute inflammatory responses, and resting PNS activity did not moderate this association. However, peer victimization predicted stronger inflammatory responses among adolescents with weaker PNS reactivity to the stress task (b = 0.63, p = .02). This association was not observed among adolescents with stronger PNS reactivity, for whom a negative but non-significant trend was found. Weaker PNS reactivity may thus indicate victimized adolescents' vulnerability for acute inflammatory responses, whereas stronger PNS reactivity may indicate adolescents' resilience to a social stressor.

Understanding the combined effects of sleep deprivation and acute social stress on cognitive performance using a comprehensive approach.

Background: Sleep deprivation (SD) and acute social stress are common, often unavoidable, and frequently co-occurring stressors in high-risk professions. Both stressors are known to acutely induce inflammatory responses and an increasing body of literature suggests this may lead to cognitive impairment. This study examined the combined effects of total SD and acute social stress on cognitive performance and took a comprehensive approach to explore their (shared) underlying mechanism leading to cognitive decline. Method: We recorded cognitive performance on a response inhibition task and a multitask and monitored a range of inflammatory, psychophysiological and self-reported markers in 101 participants, both before and after one night of either sleep (control group: N = 48) or SD (N = 53), and both before and after a social stressor (Trier Social Stress Test). Results: SD decreased cognitive performance. The social stress test also results in cognitive performance decline in the control group on the response inhibition task, but improved rather than decreased performance of sleep deprived participants on both tasks. The subjective ratings of mental effort also reflect this antagonistic interaction, indicating that the social stressor when sleep-deprived also reduced mental effort. In the inflammatory and physiological measures, this pattern was only reflected by IL-22 in blood. SD reduced blood IL-22 concentrations, and the social stress reduced IL-22 in the control group as well, but not in sleep-deprived participants. There were no interactive effects of SD and social stress on any other inflammatory or psychophysiological measures. The effects of the social stress test on autonomic measures and subjective results suggest that increased arousal may have benefited sleep-deprived participants' cognitive performance. Discussion: SD generally decreased cognitive performance and increased required mental effort. By contrast, the isolated effects of a social stressor were not generic, showing a positive effect on cognitive performance when sleep deprived. Our study is the first that studied combined effects of sleep deprivation and acute social stress on cognitive performance and inflammatory markers. It provides a comprehensive overview of effects of these stressors on a range of variables. We did not show unequivocal evidence of an underlying physiological mechanism explaining changes in performance due to (the combination of) sleep deprivation and social stress, but consider IL-22 as a possible cytokine involved in this mechanism and certainly worth following up on in future research.

Physiological synchrony in electrodermal activity predicts decreased vigilant attention induced by sleep deprivation.

Introduction: When multiple individuals are presented with narrative movie or audio clips, their electrodermal activity (EDA) and heart rate show significant similarities. Higher levels of such inter-subject physiological synchrony are related with higher levels of attention toward the narrative, as for instance expressed by more correctly answered questions about the narrative. We here investigate whether physiological synchrony in EDA and heart rate during watching of movie clips predicts performance on a subsequent vigilant attention task among participants exposed to a night of total sleep deprivation. Methods: We recorded EDA and heart rate of 54 participants during a night of total sleep deprivation. Every hour from 22:00 to 07:00 participants watched a 10-min movie clip during which we computed inter-subject physiological synchrony. Afterwards, they answered questions about the movie and performed the psychomotor vigilance task (PVT) to capture attentional performance. Results: We replicated findings that inter-subject correlations in EDA and heart rate predicted the number of correct answers on questions about the movie clips. Furthermore, we found that inter-subject correlations in EDA, but not in heart rate, predicted PVT performance. Individuals' mean EDA and heart rate also predicted their PVT performance. For EDA, inter-subject correlations explained more variance of PVT performance than individuals' mean EDA. Discussion: Together, these findings confirm the association between physiological synchrony and attention. Physiological synchrony in EDA does not only capture the attentional processing during the time that it is determined, but also proves valuable for capturing more general changes in the attentional state of monitored individuals.

Electrodermal activity as an index of food neophobia outside the lab.

Introduction: Understanding how food neophobia affects food experience may help to shift toward sustainable diets. Previous research suggests that individuals with higher food neophobia are more aroused and attentive when observing food-related stimuli. The present study examined whether electrodermal activity (EDA), as index of arousal, relates to food neophobia outside the lab when exposed to a single piece of food. Methods: The EDA of 153 participants was analyzed as part of a larger experiment conducted at a festival. Participants completed the 10-item Food Neophobia Scale. Subsequently, they saw three lids covering three foods: a hotdog labeled as "meat", a hotdog labeled as "100% plant-based", and tofu labeled as "100% plant-based". Participants lifted the lids consecutively and the area-under-the-curve (AUC) of the skin conductance response (SCR) was captured between 20 s before and 20 s after each food reveal. Results: We found a significant positive correlation between food neophobia and AUC of SCR during presentation of the first and second hotdog and a trend for tofu. These correlations remained significant even when only including the SCR data prior to the food reveal (i.e., an anticipatory response). Discussion: The association between food neophobia and EDA indicates that food neophobic individuals are more aroused upon the presentation of food. We show for the first time that the anticipation of being presented with food already increased arousal for food neophobic individuals. These findings also indicate that EDA can be meaningfully determined using wearables outside the lab, in a relatively uncontrolled setting for single-trial analysis.

Control-display mapping in brain-computer interfaces.

Event-related potential (ERP) based brain-computer interfaces (BCIs) employ differences in brain responses to attended and ignored stimuli. When using a tactile ERP-BCI for navigation, mapping is required between navigation directions on a visual display and unambiguously corresponding tactile stimuli (tactors) from a tactile control device: control-display mapping (CDM). We investigated the effect of congruent (both display and control horizontal or both vertical) and incongruent (vertical display, horizontal control) CDMs on task performance, the ERP and potential BCI performance. Ten participants attended to a target (determined via CDM), in a stream of sequentially vibrating tactors. We show that congruent CDM yields best task performance, enhanced the P300 and results in increased estimated BCI performance. This suggests a reduced availability of attentional resources when operating an ERP-BCI with incongruent CDM. Additionally, we found an enhanced N2 for incongruent CDM, which indicates a conflict between visual display and tactile control orientations. PRACTITIONER SUMMARY: Incongruency in control-display mapping reduces task performance. In this study, brain responses, task and system performance are related to (in)congruent mapping of command options and the corresponding stimuli in a brain-computer interface (BCI). Directional congruency reduces task errors, increases available attentional resources, improves BCI performance and thus facilitates human-computer interaction.

Linking Categorical and Dimensional Approaches to Assess Food-Related Emotions.

Reflecting the two main prevailing and opposing views on the nature of emotions, emotional responses to food and beverages are typically measured using either (a) a categorical (lexicon-based) approach where users select or rate the terms that best express their food-related feelings or (b) a dimensional approach where they rate perceived food items along the dimensions of valence and arousal. Relating these two approaches is problematic since a response in terms of valence and arousal is not easily expressed in terms of emotions (like happy or disgusted). In this study, we linked the dimensional approach to a categorical approach by establishing mapping between a set of 25 emotion terms (EsSense25) and the valence-arousal space (via the EmojiGrid graphical response tool), using a set of 20 food images. In two 'matching' tasks, the participants first imagined how the food shown in a given image would make them feel and then reported either the emotional terms or the combination of valence and arousal that best described their feelings. In two labeling tasks, the participants first imagined experiencing a given emotion term and then they selected either the foods (images) that appeared capable to elicit that feeling or reported the combination of valence and arousal that best reflected that feeling. By combining (1) the mapping between the emotion terms and the food images with (2) the mapping of the food images to the valence-arousal space, we established (3) an indirect (via the images) mapping of the emotion terms to the valence-arousal space. The results show that the mapping between terms and images was reliable and that the linkages have straightforward and meaningful interpretations. The valence and arousal values that were assigned to the emotion terms through indirect mapping to the valence-arousal space were typically less extreme than those that were assigned through direct mapping.

Serial Dependence of Emotion Within and Between Stimulus Sensory Modalities.

How we perceive the world is not solely determined by what we sense at a given moment in time, but also by what we processed recently. Here we investigated whether such serial dependencies for emotional stimuli transfer from one modality to another. Participants were presented a random sequence of emotional sounds and images and instructed to rate the valence and arousal of each stimulus (Experiment 1). For both ratings, we conducted an intertrial analysis, based on whether the rating on the previous trial was low or high. We found a positive serial dependence for valence and arousal regardless of the stimulus modality on two consecutive trials. In Experiment 2, we examined whether passively perceiving a stimulus is sufficient to induce a serial dependence. In Experiment 2, participants were instructed to rate the stimuli only on active trials and not on passive trials. The participants were informed that the active and passive trials were presented in alternating order, so that they were able to prepare for the task. We conducted an intertrial analysis on active trials, based on whether the rating on the previous passive trial (determined in Experiment 1) was low or high. For both ratings, we again observed positive serial dependencies regardless of the stimulus modality. We conclude that the emotional experience triggered by one stimulus affects the emotional experience for a subsequent stimulus regardless of their sensory modalities, that this occurs in a bottom-up fashion, and that this can be explained by residual activation in the emotional network in the brain.

Unsupervised Clustering of Individuals Sharing Selective Attentional Focus Using Physiological Synchrony.

Research on brain signals as indicators of a certain attentional state is moving from laboratory environments to everyday settings. Uncovering the attentional focus of individuals in such settings is challenging because there is usually limited information about real-world events, as well as a lack of data from the real-world context at hand that is correctly labeled with respect to individuals' attentional state. In most approaches, such data is needed to train attention monitoring models. We here investigate whether unsupervised clustering can be combined with physiological synchrony in the electroencephalogram (EEG), electrodermal activity (EDA), and heart rate to automatically identify groups of individuals sharing attentional focus without using knowledge of the sensory stimuli or attentional focus of any of the individuals. We used data from an experiment in which 26 participants listened to an audiobook interspersed with emotional sounds and beeps. Thirteen participants were instructed to focus on the narrative of the audiobook and 13 participants were instructed to focus on the interspersed emotional sounds and beeps. We used a broad range of commonly applied dimensionality reduction ordination techniques-further referred to as mappings-in combination with unsupervised clustering algorithms to identify the two groups of individuals sharing attentional focus based on physiological synchrony. Analyses were performed using the three modalities EEG, EDA, and heart rate separately, and using all possible combinations of these modalities. The best unimodal results were obtained when applying clustering algorithms on physiological synchrony data in EEG, yielding a maximum clustering accuracy of 85%. Even though the use of EDA or heart rate by itself did not lead to accuracies significantly higher than chance level, combining EEG with these measures in a multimodal approach generally resulted in higher classification accuracies than when using only EEG. Additionally, classification results of multimodal data were found to be more consistent across algorithms than unimodal data, making algorithm choice less important. Our finding that unsupervised classification into attentional groups is possible is important to support studies on attentional engagement in everyday settings.

Measuring Implicit Approach-Avoidance Tendencies towards Food Using a Mobile Phone outside the Lab.

Implicit ('unconscious') approach-avoidance tendencies towards stimuli can be measured using the Approach Avoidance Task (AAT). We recently expanded a toolbox for analyzing the raw data of a novel, mobile version of the AAT (mAAT), that asks participants to move their phone towards their face (pull) or away (push) in response to images presented on the phone. We here tested the mAAT reaction time and the mAAT distance in a study with 71 Dutch participants that were recruited online and performed an experiment without coming to the laboratory. The participants used both the mAAT and (explicit) rating scales to respond to photographic images of food. As hypothesized, the rated wanting, rated valence and mAAT reaction time indicated a preference for palatable over unpalatable food, and for Dutch over Asian food. Additionally, as expected, arousal was rated higher for unpalatable than for palatable food, and higher for Dutch than for Asian food. The mAAT distance indicated that the unpalatable food images were moved across larger distances, regardless of the movement direction (pull or push), compared to the palatable food images; and the Dutch food images were moved across larger distances than the Asian food images. We conclude that the mAAT can be used to implicitly probe approach-avoidance motivation for complex images in the food domain. The new measure of mAAT distance may be used as an implicit measure of arousal. The ratings and the mAAT measures do not reflect the exact same information and may complement each other. Implicit measures, such as mAAT variables, are particularly valuable when response biases that can occur when using explicit ratings are expected.