Detecting and Predicting Pilot Mental Workload Using Heart Rate Variability: A Systematic Review.

Measuring pilot mental workload (MWL) is crucial for enhancing aviation safety. However, MWL is a multi-dimensional construct that could be affected by multiple factors. Particularly, in the context of a more automated cockpit setting, the traditional methods of assessing pilot MWL may face challenges. Heart rate variability (HRV) has emerged as a potential tool for detecting pilot MWL during real-flight operations. This review aims to investigate the relationship between HRV and pilot MWL and to assess the performance of machine-learning-based MWL detection systems using HRV parameters. A total of 29 relevant papers were extracted from three databases for review based on rigorous eligibility criteria. We observed significant variability across the reviewed studies, including study designs and measurement methods, as well as machine-learning techniques. Inconsistent results were observed regarding the differences in HRV measures between pilots under varying levels of MWL. Furthermore, for studies that developed HRV-based MWL detection systems, we examined the diverse model settings and discovered that several advanced techniques could be used to address specific challenges. This review serves as a practical guide for researchers and practitioners who are interested in employing HRV indicators for evaluating MWL and wish to incorporate cutting-edge techniques into their MWL measurement approaches.

Implementation of a Real-Time Brain-to-Brain Synchrony Estimation Algorithm for Neuroeducation Applications.

This study centers on creating a real-time algorithm to estimate brain-to-brain synchronization during social interactions, specifically in collaborative and competitive scenarios. This type of algorithm can provide useful information in the educational context, for instance, during teacher-student or student-student interactions. Positioned within the context of neuroeducation and hyperscanning, this research addresses the need for biomarkers as metrics for feedback, a missing element in current teaching methods. Implementing the bispectrum technique with multiprocessing functions in Python, the algorithm effectively processes electroencephalography signals and estimates brain-to-brain synchronization between pairs of subjects during (competitive and collaborative) activities that imply specific cognitive processes. Noteworthy differences, such as higher bispectrum values in collaborative tasks compared to competitive ones, emerge with reliability, showing a total of 33.75% of significant results validated through a statistical test. While acknowledging progress, this study identifies areas of opportunity, including embedded operations, wider testing, and improved result visualization. Beyond academia, the algorithm's utility extends to classrooms, industries, and any setting involving human interactions. Moreover, the presented algorithm is shared openly, to facilitate implementations by other researchers, and is easily adjustable to other electroencephalography devices. This research not only bridges a technological gap but also contributes insights into the importance of interactions in educational contexts.

Assessment of Drivers' Mental Workload by Multimodal Measures during Auditory-Based Dual-Task Driving Scenarios.

Assessing drivers' mental workload is crucial for reducing road accidents. This study examined drivers' mental workload in a simulated auditory-based dual-task driving scenario, with driving tasks as the main task, and auditory-based N-back tasks as the secondary task. A total of three levels of mental workload (i.e., low, medium, high) were manipulated by varying the difficulty levels of the secondary task (i.e., no presence of secondary task, 1-back, 2-back). Multimodal measures, including a set of subjective measures, physiological measures, and behavioral performance measures, were collected during the experiment. The results showed that an increase in task difficulty led to increased subjective ratings of mental workload and a decrease in task performance for the secondary N-back tasks. Significant differences were observed across the different levels of mental workload in multimodal physiological measures, such as delta waves in EEG signals, fixation distance in eye movement signals, time- and frequency-domain measures in ECG signals, and skin conductance in EDA signals. In addition, four driving performance measures related to vehicle velocity and the deviation of pedal input and vehicle position also showed sensitivity to the changes in drivers' mental workload. The findings from this study can contribute to a comprehensive understanding of effective measures for mental workload assessment in driving scenarios and to the development of smart driving systems for the accurate recognition of drivers' mental states.

Enhancing Stress Detection: A Comprehensive Approach through rPPG Analysis and Deep Learning Techniques.

Stress has emerged as a major concern in modern society, significantly impacting human health and well-being. Statistical evidence underscores the extensive social influence of stress, especially in terms of work-related stress and associated healthcare costs. This paper addresses the critical need for accurate stress detection, emphasising its far-reaching effects on health and social dynamics. Focusing on remote stress monitoring, it proposes an efficient deep learning approach for stress detection from facial videos. In contrast to the research on wearable devices, this paper proposes novel Hybrid Deep Learning (DL) networks for stress detection based on remote photoplethysmography (rPPG), employing (Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), 1D Convolutional Neural Network (1D-CNN)) models with hyperparameter optimisation and augmentation techniques to enhance performance. The proposed approach yields a substantial improvement in accuracy and efficiency in stress detection, achieving up to 95.83% accuracy with the UBFC-Phys dataset while maintaining excellent computational efficiency. The experimental results demonstrate the effectiveness of the proposed Hybrid DL models for rPPG-based-stress detection.

Direct machine learning reconstruction of respiratory variation waveforms from resting state fMRI data in a pediatric population.

In many functional magnetic resonance imaging (fMRI) studies, respiratory signals are unavailable or do not have acceptable quality due to issues with subject compliance, equipment failure or signal error. In large databases, such as the Human Connectome Projects, over half of the respiratory recordings may be unusable. As a result, the direct removal of low frequency respiratory variations from the blood oxygen level-dependent (BOLD) signal time series is not possible. This study proposes a deep learning-based method for reconstruction of respiratory variation (RV) waveforms directly from BOLD fMRI data in pediatric participants (aged 5 to 21 years old), and does not require any respiratory measurement device. To do this, the Lifespan Human Connectome Project in Development (HCP-D) dataset, which includes respiratory measurements, was used to both train a convolutional neural network (CNN) and evaluate its performance. Results show that a CNN can capture informative features from the BOLD signal time course and reconstruct accurate RV timeseries, especially when the subject has a prominent respiratory event. This work advances the use of direct estimation of physiological parameters from fMRI, which will eventually lead to reduced complexity and decrease the burden on participants because they may not be required to wear a respiratory bellows.

Brain-Computer Interfaces in Visualized Medicine.

The brain-computer interface (BCI), also known as a brain-machine interface (BMI), has attracted extensive attention in biomedical applications. More importantly, BCI technologies have substantially revolutionized early predictions, diagnostic techniques, and rehabilitation strategies addressing acute diseases because of BCI's innovations and clinical translations. Therefore, in this chapter, a comprehensive description of the basic concepts of BCI will be exhibited, and various visualization techniques employed in BCI's medical applications will be discussed.

Fused CNN-LSTM deep learning emotion recognition model using electroencephalography signals.

Introduction: The traditional machine learning-based emotion recognition models have shown effective performance for classifying Electroencephalography (EEG) based emotions.Methods: The different machine learning algorithms outperform the various EEG based emotion models for valence and arousal. But the downside is to devote numerous efforts to designing features from the given noisy signals which are also a very time-consuming process. The Deep Learning analysis overcomes the hand-engineered feature extraction and selection problems.Results: In this study, the Database of Emotion analysis using Physiological signals (DEAP) has been visualized to classify High-Arousal- Low-Arousal (HALA), High-Valence-Low-Valence (HVLV), familiarity, Dominance and Liking emotions. The fusion of deep learning models, namely CNN and LSTM-RNN seems to perform better for the analysis of emotions using EEG signals. The average accuracies analyzed by the fused deep learning classification model for DEAP are 97.39%, 97.41%, 98.21%, 97.68%, and 97.89% for HALA, HVLV, familiarity, dominance and liking respectively. The model has been evaluated over the SJTU Emotion EEG Dataset (SEED) dataset too for the detection of positive and negative emotions, which results with an average accuracy of 93.74%.Conclusion: The results show that the developed model can classify the inner emotions of different EEG based emotion databases.

Wearable Health Devices for Diagnosis Support: Evolution and Future Tendencies.

The use of wearable devices has increased substantially in recent years. This, together with the rise of telemedicine, has led to the use of these types of devices in the healthcare field. In this work, we carried out a detailed study on the use of these devices (regarding the general trends); we analyzed the research works and devices marketed in the last 10 years. This analysis extracted relevant information on the general trend of use, as well as more specific aspects, such as the use of sensors, communication technologies, and diseases. A comparison was made between the commercial and research aspects linked to wearables in the healthcare field, and upcoming trends were analyzed.

Exploring the Applicability of Physiological Monitoring to Manage Physical Fatigue in Firefighters.

Physical fatigue reduces productivity and quality of work while increasing the risk of injuries and accidents among safety-sensitive professionals. To prevent its adverse effects, researchers are developing automated assessment methods that, despite being highly accurate, require a comprehensive understanding of underlying mechanisms and variables' contributions to determine their real-life applicability. This work aims to evaluate the performance variations of a previously developed four-level physical fatigue model when alternating its inputs to have a comprehensive view of the impact of each physiological variable on the model's functioning. Data from heart rate, breathing rate, core temperature and personal characteristics from 24 firefighters during an incremental running protocol were used to develop the physical fatigue model based on an XGBoosted tree classifier. The model was trained 11 times with different input combinations resulting from alternating four groups of features. Performance measures from each case showed that heart rate is the most relevant signal for estimating physical fatigue. Breathing rate and core temperature enhanced the model when combined with heart rate but showed poor performance individually. Overall, this study highlights the advantage of using more than one physiological measure for improving physical fatigue modelling. The findings can contribute to variables and sensor selection in occupational applications and as the foundation for further field research.

Stress Detection Using Frequency Spectrum Analysis of Wrist-Measured Electrodermal Activity.

The article deals with the detection of stress using the electrodermal activity (EDA) signal measured at the wrist. We present an approach for feature extraction from EDA. The approach uses frequency spectrum analysis in multiple frequency bands. We evaluate the proposed approach using the 4 Hz EDA signal measured at the wrist in the publicly available Wearable Stress and Affect Detection (WESAD) dataset. Seven existing approaches to stress detection using EDA signals measured by wrist-worn sensors are analysed and the reported results are compared with ours. The proposed approach represents an improvement in accuracy over the other techniques studied. Moreover, we focus on time to detection (TTD) and show that our approach is able to outperform competing techniques, with fewer data points. The proposed feature extraction is computationally inexpensive, thus the presented approach is suitable for use in real-world wearable applications where both short response times and high detection performance are important. We report both binary (stress vs. no stress) as well as three-class (baseline/stress/amusement) results.

Deep Learning for Detecting Multi-Level Driver Fatigue Using Physiological Signals: A Comprehensive Approach.

A large share of traffic accidents is related to driver fatigue. In recent years, many studies have been organized in order to diagnose and warn drivers. In this research, a new approach was presented in order to detect multi-level driver fatigue. A multi-level driver tiredness diagnostic database based on physiological signals including ECG, EEG, EMG, and respiratory effort was developed for this aim. The EEG signal was used for processing and other recorded signals were used to confirm the driver's fatigue so that fatigue was not confirmed based on self-report questionnaires. A customized architecture based on adversarial generative networks and convolutional neural networks (end-to-end) was utilized to select/extract features and classify different levels of fatigue. In the customized architecture, with the objective of eliminating uncertainty, type 2 fuzzy sets were used instead of activation functions such as Relu and Leaky Relu, and the performance of each was investigated. The final accuracy obtained in the three scenarios considered, two-level, three-level, and five-level, were 96.8%, 95.1%, and 89.1%, respectively. Given the suggested model's optimal performance, which can identify five various levels of driver fatigue with high accuracy, it can be employed in practical applications of driver fatigue to warn drivers.

Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification.

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Food Choices after Cognitive Load: An Affective Computing Approach.

Psychology and nutritional science research has highlighted the impact of negative emotions and cognitive load on calorie consumption behaviour using subjective questionnaires. Isolated studies in other domains objectively assess cognitive load without considering its effects on eating behaviour. This study aims to explore the potential for developing an integrated eating behaviour assistant system that incorporates cognitive load factors. Two experimental sessions were conducted using custom-developed experimentation software to induce different stimuli. During these sessions, we collected 30 h of physiological, food consumption, and affective states questionnaires data to automatically detect cognitive load and analyse its effect on food choice. Utilising grid search optimisation and leave-one-subject-out cross-validation, a support vector machine model achieved a mean classification accuracy of 85.12% for the two cognitive load tasks using eight relevant features. Statistical analysis was performed on calorie consumption and questionnaire data. Furthermore, 75% of the subjects with higher negative affect significantly increased consumption of specific foods after high-cognitive-load tasks. These findings offer insights into the intricate relationship between cognitive load, affective states, and food choice, paving the way for an eating behaviour assistant system to manage food choices during cognitive load. Future research should enhance system capabilities and explore real-world applications.

Behavior and Task Classification Using Wearable Sensor Data: A Study across Different Ages.

In this paper, we face the problem of task classification starting from physiological signals acquired using wearable sensors with experiments in a controlled environment, designed to consider two different age populations: young adults and older adults. Two different scenarios are considered. In the first one, subjects are involved in different cognitive load tasks, while in the second one, space varying conditions are considered, and subjects interact with the environment, changing the walking conditions and avoiding collision with obstacles. Here, we demonstrate that it is possible not only to define classifiers that rely on physiological signals to predict tasks that imply different cognitive loads, but it is also possible to classify both the population group age and the performed task. The whole workflow of data collection and analysis, starting from the experimental protocol, data acquisition, signal denoising, normalization with respect to subject variability, feature extraction and classification is described here. The dataset collected with the experiments together with the codes to extract the features of the physiological signals are made available for the research community.

Explainable Artificial Intelligence (XAI) in Pain Research: Understanding the Role of Electrodermal Activity for Automated Pain Recognition.

Artificial intelligence and especially deep learning methods have achieved outstanding results for various applications in the past few years. Pain recognition is one of them, as various models have been proposed to replace the previous gold standard with an automated and objective assessment. While the accuracy of such models could be increased incrementally, the understandability and transparency of these systems have not been the main focus of the research community thus far. Thus, in this work, several outcomes and insights of explainable artificial intelligence applied to the electrodermal activity sensor data of the PainMonit and BioVid Heat Pain Database are presented. For this purpose, the importance of hand-crafted features is evaluated using recursive feature elimination based on impurity scores in Random Forest (RF) models. Additionally, Gradient-weighted class activation mapping is applied to highlight the most impactful features learned by deep learning models. Our studies highlight the following insights: (1) Very simple hand-crafted features can yield comparative performances to deep learning models for pain recognition, especially when properly selected with recursive feature elimination. Thus, the use of complex neural networks should be questioned in pain recognition, especially considering their computational costs; and (2) both traditional feature engineering and deep feature learning approaches rely on simple characteristics of the input time-series data to make their decision in the context of automated pain recognition.

Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals.

The use of machine learning (ML) techniques in affective computing applications focuses on improving the user experience in emotion recognition. The collection of input data (e.g., physiological signals), together with expert annotations are part of the established standard supervised learning methodology used to train human emotion recognition models. However, these models generally require large amounts of labeled data, which is expensive and impractical in the healthcare context, in which data annotation requires even more expert knowledge. To address this problem, this paper explores the use of the self-supervised learning (SSL) paradigm in the development of emotion recognition methods. This approach makes it possible to learn representations directly from unlabeled signals and subsequently use them to classify affective states. This paper presents the key concepts of emotions and how SSL methods can be applied to recognize affective states. We experimentally analyze and compare self-supervised and fully supervised training of a convolutional neural network designed to recognize emotions. The experimental results using three emotion datasets demonstrate that self-supervised representations can learn widely useful features that improve data efficiency, are widely transferable, are competitive when compared to their fully supervised counterparts, and do not require the data to be labeled for learning.

An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals.

Emotion charting using multimodal signals has gained great demand for stroke-affected patients, for psychiatrists while examining patients, and for neuromarketing applications. Multimodal signals for emotion charting include electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, and galvanic skin response (GSR) signals. EEG, ECG, and GSR are also known as physiological signals, which can be used for identification of human emotions. Due to the unbiased nature of physiological signals, this field has become a great motivation in recent research as physiological signals are generated autonomously from human central nervous system. Researchers have developed multiple methods for the classification of these signals for emotion detection. However, due to the non-linear nature of these signals and the inclusion of noise, while recording, accurate classification of physiological signals is a challenge for emotion charting. Valence and arousal are two important states for emotion detection; therefore, this paper presents a novel ensemble learning method based on deep learning for the classification of four different emotional states including high valence and high arousal (HVHA), low valence and low arousal (LVLA), high valence and low arousal (HVLA) and low valence high arousal (LVHA). In the proposed method, multimodal signals (EEG, ECG, and GSR) are preprocessed using bandpass filtering and independent components analysis (ICA) for noise removal in EEG signals followed by discrete wavelet transform for time domain to frequency domain conversion. Discrete wavelet transform results in spectrograms of the physiological signal and then features are extracted using stacked autoencoders from those spectrograms. A feature vector is obtained from the bottleneck layer of the autoencoder and is fed to three classifiers SVM (support vector machine), RF (random forest), and LSTM (long short-term memory) followed by majority voting as ensemble classification. The proposed system is trained and tested on the AMIGOS dataset with k-fold cross-validation. The proposed system obtained the highest accuracy of 94.5% and shows improved results of the proposed method compared with other state-of-the-art methods.

Machine Learning Approach to Model Physical Fatigue during Incremental Exercise among Firefighters.

Physical fatigue is a serious threat to the health and safety of firefighters. Its effects include decreased cognitive abilities and a heightened risk of accidents. Subjective scales and, recently, on-body sensors have been used to monitor physical fatigue among firefighters and safety-sensitive professionals. Considering the capabilities (e.g., noninvasiveness and continuous monitoring) and limitations (e.g., assessed fatiguing tasks and models validation procedures) of current approaches, this study aimed to develop a physical fatigue prediction model combining cardiorespiratory and thermoregulatory measures and machine learning algorithms within a firefighters' sample. Sensory data from heart rate, breathing rate and core temperature were recorded from 24 participants during an incremental running protocol. Various supervised machine learning algorithms were examined using 21 features extracted from the physiological variables and participants' characteristics to estimate four physical fatigue conditions: low, moderate, heavy and severe. Results showed that the XGBoosted Trees algorithm achieved the best outcomes with an average accuracy of 82% and accuracies of 93% and 86% for recognising the low and severe levels. Furthermore, this study evaluated different methods to assess the models' performance, concluding that the group cross-validation method presents the most practical results. Overall, this study highlights the advantages of using multiple physiological measures for enhancing physical fatigue modelling. It proposes a promising health and safety management tool and lays the foundation for future studies in field conditions.

Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition.

Emotion recognition is a key attribute for realizing advances in human-computer interaction, especially when using non-intrusive physiological sensors, such as electroencephalograph (EEG) and electrocardiograph. Although functional connectivity of EEG has been utilized for emotion recognition, the graph theory analysis of EEG connectivity patterns has not been adequately explored. The exploitation of brain network characteristics could provide valuable information regarding emotions, while the combination of EEG and peripheral physiological signals can reveal correlation patterns of human internal state. In this work, a graph theoretical analysis of EEG functional connectivity patterns along with fusion between EEG and peripheral physiological signals for emotion recognition has been proposed. After extracting functional connectivity from EEG signals, both global and local graph theory features are extracted. Those features are concatenated with statistical features from peripheral physiological signals and fed to different classifiers and a Convolutional Neural Network (CNN) for emotion recognition. The average accuracy on the DEAP dataset using CNN was 55.62% and 57.38% for subject-independent valence and arousal classification, respectively, and 83.94% and 83.87% for subject-dependent classification. Those scores went up to 75.44% and 78.77% for subject-independent classification and 88.27% and 90.84% for subject-dependent classification using a feature selection algorithm, exceeding the current state-of-the-art results.

SenseHunger: Machine Learning Approach to Hunger Detection Using Wearable Sensors.

The perception of hunger and satiety is of great importance to maintaining a healthy body weight and avoiding chronic diseases such as obesity, underweight, or deficiency syndromes due to malnutrition. There are a number of disease patterns, characterized by a chronic loss of this perception. To our best knowledge, hunger and satiety cannot be classified using non-invasive measurements. Aiming to develop an objective classification system, this paper presents a multimodal sensory system using associated signal processing and pattern recognition methods for hunger and satiety detection based on non-invasive monitoring. We used an Empatica E4 smartwatch, a RespiBan wearable device, and JINS MEME smart glasses to capture physiological signals from five healthy normal weight subjects inactively sitting on a chair in a state of hunger and satiety. After pre-processing the signals, we compared different feature extraction approaches, either based on manual feature engineering or deep feature learning. Comparative experiments were carried out to determine the most appropriate sensor channel, device, and classifier to reliably discriminate between hunger and satiety states. Our experiments showed that the most discriminative features come from three specific sensor modalities: Electrodermal Activity (EDA), infrared Thermopile (Tmp), and Blood Volume Pulse (BVP).