Multimodal Emotion Recognition based on EEG and EOG Signals evoked by the Video-odor Stimuli.

Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals' emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What's more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition.

Collecting emotional physiological signals is significant in building affective Human-Computer Interactions (HCI). However, how to evoke subjects' emotions efficiently in EEG-related emotional experiments is still a challenge. In this work, we developed a novel experimental paradigm that allows odors dynamically participate in different stages of video-evoked emotions, to investigate the efficiency of olfactory-enhanced videos in inducing subjects' emotions; According to the period that the odors participated in, the stimuli were divided into four patterns, i.e., the olfactory-enhanced video in early/later stimulus periods (OVEP/OVLP), and the traditional videos in early/later stimulus periods (TVEP/TVLP). The differential entropy (DE) feature and four classifiers were employed to test the efficiency of emotion recognition. The best average accuracies of the OVEP, OVLP, TVEP, and TVLP were 50.54%, 51.49%, 40.22%, and 57.55%, respectively. The experimental results indicated that the OVEP significantly outperformed the TVEP on classification performance, while there was no significant difference between the OVLP and TVLP. Besides, olfactory-enhanced videos achieved higher efficiency in evoking negative emotions than traditional videos. Moreover, we found that the neural patterns in response to emotions under different stimulus methods were stable, and for Fp1, FP2, and F7, there existed significant differences in whether adopt the odors.

LSTM-based real-time signal quality assessment for blood volume pulse analysis.

Remote photoplethysmogram (rPPG) is a low-cost method to extract blood volume pulse (BVP). Some crucial vital signs, such as heart rate (HR) and respiratory rate (RR) etc. can be achieved from BVP for clinical medicine and healthcare application. As compared to the conventional PPG methods, rPPG is more promising because of its non-contacted measurement. However, both BVP detection methods, especially rPPG, are susceptible to motion and illumination artifacts, which lead to inaccurate estimation of vital signs. Signal quality assessment (SQA) is a method to measure the quality of BVP signals and ensure the credibility of estimated physiological parameters. But the existing SQA methods are not suitable for real-time processing. In this paper, we proposed an end-to-end BVP signal quality evaluation method based on a long short-term memory network (LSTM-SQA). Two LSTM-SQA models were trained using the BVP signals obtained with PPG and rPPG techniques so that the quality of BVP signals derived from these two methods can be evaluated, respectively. As there is no publicly available rPPG dataset with quality annotations, we designed a training sample generation method with blind source separation, by which two kinds of training datasets respective to PPG and rPPG were built. Each dataset consists of 38400 high and low-quality BVP segments. The achieved models were verified on three public datasets (IIP-HCI dataset, UBFC-Phys dataset, and LGI-PPGI dataset). The experimental results show that the proposed LSTM-SQA models can effectively predict the quality of the BVP signal in real-time.

Classifying early and late mild cognitive impairment stages of Alzheimer's disease by fusing default mode networks extracted with multiple seeds.

BACKGROUND: The default mode network (DMN) in resting state has been increasingly used in disease diagnosis since it was found in 2001. Prior work has mainly focused on extracting a single DMN with various techniques. However, by using seeding-based analysis with more than one desirable seed, we can obtain multiple DMNs, which are likely to have complementary information, and thus are more promising for disease diagnosis. In the study, we used 18 early mild cognitive impairment (EMCI) participants and 18 late mild cognitive impairment (LMCI) participants of Alzheimer's disease (AD). First, we used seeding-based analysis with four seeds to extract four DMNs for each subject. Then, we conducted fusion analysis for all different combinations of the four DMNs. Finally, we carried out nonlinear support vector machine classification based on the mixing coefficients from the fusion analysis. RESULTS: We found that (1) the four DMNs corresponding to the four different seeds indeed capture different functional regions of each subject; (2) Maps of the four DMNs in the most different joint source from fusion analysis are centered at the regions of the corresponding seeds; (3) Classification results reveal the effectiveness of using multiple seeds to extract DMNs. When using a single seed, the regions of posterior cingulate cortex (PCC) extractions of EMCI and LMCI show the largest difference. For multiple-seed cases, the regions of PCC extraction and right lateral parietal cortex (RLP) extraction provide complementary information for each other in fusion, which improves the classification accuracy. Furthermore, the regions of left lateral parietal cortex (LLP) extraction and RLP extraction also have complementary effect in fusion. In summary, AD diagnosis can be improved by exploiting complementary information of DMNs extracted with multiple seeds. CONCLUSIONS: In this study, we applied fusion analysis to the DMNs extracted by using different seeds for exploiting the complementary information hidden among the separately extracted DMNs, and the results supported our expectation that using the complementary information can improve classification accuracy.