Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing.

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

Development and Calibration of an Eye-Tracking Fixation Identification Algorithm for Immersive Virtual Reality.

Fixation identification is an essential task in the extraction of relevant information from gaze patterns; various algorithms are used in the identification process. However, the thresholds used in the algorithms greatly affect their sensitivity. Moreover, the application of these algorithm to eye-tracking technologies integrated into head-mounted displays, where the subject's head position is unrestricted, is still an open issue. Therefore, the adaptation of eye-tracking algorithms and their thresholds to immersive virtual reality frameworks needs to be validated. This study presents the development of a dispersion-threshold identification algorithm applied to data obtained from an eye-tracking system integrated into a head-mounted display. Rules-based criteria are proposed to calibrate the thresholds of the algorithm through different features, such as number of fixations and the percentage of points which belong to a fixation. The results show that distance-dispersion thresholds between 1-1.6° and time windows between 0.25-0.4 s are the acceptable range parameters, with 1° and 0.25 s being the optimum. The work presents a calibrated algorithm to be applied in future experiments with eye-tracking integrated into head-mounted displays and guidelines for calibrating fixation identification algorithms.

Home-exercise Childhood Obesity Intervention: A Randomized Clinical Trial Comparing Print Versus Web-based (Move It) Platforms.

PURPOSE: To compare the impact of adhering to a Mediterranean diet plus mixed physical exercise program (Move-It) implemented by means of printed instructions or via a web-platform (with or without e-mail support) on body composition, physical fitness, and blood pressure. DESIGN AND METHODS: Randomized clinical trial. Fifty-two overweight or obese Spanish children and adolescents were randomly assigned to the print-based (n = 18), Move-It (n = 18), or Move-It plus support (n = 16) intervention groups. Two-way mixed ANOVA tests were used to compare any changes between the groups in terms of percentage body fat, physical fitness (VO2peak), handgrip strength, and systolic and diastolic blood pressure. The measurements were taken before and after a three-month mixed-exercise (aerobic and resistance) and Mediterranean-diet program which was either implemented by means of printed instructions or via a web-platform (with or without e-mail support). RESULTS: No statistical differences were found between groups. However, the results highlighted significant improvements in body fat percentage metrics over time for all three groups (print-based: -1.8%, 95%CI -3.3% to -0.3%; Move-It: -1.8%, 95%CI -3.3% to -0.3%; Move-It plus support: -2.0%, 95%CI -3.7% to -0.4%, P < 0.05). We also observed a tendency towards improvement in the VO2peak, handgrip strength, and blood pressure variable values 10 min after the exercise-stress test in these three groups. CONCLUSIONS: The program improved the body composition, regardless of the way it is implemented. PRACTICE IMPLICATIONS: A mixed physical exercise program lasting for three months, combined with a Mediterranean diet, improves the body composition of children and adolescents with overweight/obesity.

Competitive active video games: Physiological and psychological responses in children and adolescents.

BACKGROUND: Recent strategies to reduce sedentary behaviour in children include replacing sedentary screen time for active video games. Active video game studies have focused principally on the metabolic consumption of a single player, with physiological and psychological responses of opponent-based multiplayer games to be further evaluated. OBJECTIVE: To determine whether adding a competitive component to playing active video games impacts physiological and psychological responses in players. METHODS: Sixty-two healthy Caucasian children and adolescents, nine to 14 years years of age, completed three conditions (8 min each) in random order: treadmill walking, and single and opponent-based Kinect active video games. Affect, arousal, rate of perceived exertion, heart rate and percentage of heart rate reserve were measured for each participant and condition. RESULTS: Kinect conditions revealed significantly higher heart rate, percentage of heart rate reserve, rate of perceived exertion and arousal when compared with treadmill walking (P<0.001). Opponent-based condition revealed lower values for the rate of perceived exertion (P=0.02) and higher affect (P=0.022) when compared with single play. CONCLUSION: Competitive active video games improved children's psychological responses (affect and rate of perceived exertion) compared with single play, providing a solution that may contribute toward improved adherence to physical activity.

HISTORIQUE: Parmi les récentes stratégies en vue de réduire les comportements sédentaires chez les enfants, soulignons le remplacement du temps d'écran sédentaire par des jeux vidéo actifs. Les études sur ce type de jeux ont surtout porté sur la consommation métabolique en mode solo. Les réponses physiologiques et psychologiques de jeux en mode multijoueur comportant des adversaires n'ont pas encore été évaluées. OBJECTIF: Déterminer si l'ajout d'un élément compétitif aux jeux vidéo actifs influe sur les réponses physiologiques et psychologiques chez les joueurs. MÉTHODOLOGIE: Soixante-deux enfants et adolescents blancs en bonne santé de neuf à 14 ans ont effectué trois conditionnements de huit minutes chacun, dans un ordre aléatoire : marche sur tapis roulant et jeux vidéo actifs Kinect en mode solo et multijoueur (Microsoft Corporation, États-Unis). Les chercheurs ont mesuré l'affect, l'éveil, le taux d'effort perçu, la fréquence cardiaque et le pourcentage de réserve de fréquence cardiaque de chaque participant et de chaque conditionnement. RÉSULTATS: Le conditionnement Kinect a suscité une augmentation significative de la fréquence cardiaque, du pourcentage de réserve de fréquence cardiaque, d'effort perçu et d'éveil par rapport à la marche sur tapis roulant (P<0,001). Le conditionnement contre un adversaire révélait des valeurs plus faibles de taux d'effort perçu (P=0,02) et un affect plus élevé (P=0,022) que le jeu en mode solo. CONCLUSION: Les jeux vidéo actifs compétitifs donnent de meilleures réponses psychologiques chez les enfants (affect et taux d'efforts perçu) que le jeu en mode solo. Cette solution pourrait contribuer à une meilleure adhérence à l'activité physique.

Reliability and validity of TIPS wireless ECG prototypes.

The aims of the present study are to examine the reliability and validity of the Heart Rate signal registered using two self-made wireless ECG systems, R-Tips and TipsShirt, and to compare them with another commercial ECG device typically used in psychophysiology studies. An ECG simulator was used to artificially generate signals corresponding to different cardiac frequencies. Results of the reliability study showed that the signal acquisition, signal processing and signal transmission were reliable and valid for R-Tips and TipsShirt. Consequently, these wireless ECG prototypes could be used for studies where the freedom of movements of the participants is fundamental without any loss of quality in the registered signals.

GameTeen: new tools for evaluating and training emotional regulation strategies.

The aim of this paper is to describe GameTeen, a novel instrument for the assessment and training of Emotional Regulation (ER) strategies in adolescent population. These new tools are based on the use of 3D serious games that can be played under different settings. The evolution of ER strategies will be monitored in two ways depending on the setting where the tool is presented. Firstly, in the laboratory, physiological signals and facial expressions of participants will be recorded. Secondly, in real life settings, ecological momentary assessment tools will be used to obtain answers from the subjects using their mobile phone. The goal is to obtain more attractive and reliable tools to evaluate and train ER strategies.

Recognizing Personality Traits Using Consumer Behavior Patterns in a Virtual Retail Store.

Virtual reality (VR) is a useful tool to study consumer behavior while they are immersed in a realistic scenario. Among several other factors, personality traits have been shown to have a substantial influence on purchasing behavior. The primary objective of this study was to classify consumers based on the Big Five personality domains using their behavior while performing different tasks in a virtual shop. The personality recognition was ascertained using behavioral measures received from VR hardware, including eye-tracking, navigation, posture and interaction. Responses from 60 participants were collected while performing free and directed search tasks in a virtual hypermarket. A set of behavioral features was processed, and the personality domains were recognized using a statistical supervised machine learning classifier algorithm via a support vector machine. The results suggest that the open-mindedness personality type can be classified using eye gaze patterns, while extraversion is related to posture and interactions. However, a combination of signals must be exhibited to detect conscientiousness and negative emotionality. The combination of all measures and tasks provides better classification accuracy for all personality domains. The study indicates that a consumer's personality can be recognized using the behavioral sensors included in commercial VR devices during a purchase in a virtual retail store.

Heart rate variability analysis for the assessment of immersive emotional arousal using virtual reality: Comparing real and virtual scenarios.

Many affective computing studies have developed automatic emotion recognition models, mostly using emotional images, audio and videos. In recent years, virtual reality (VR) has been also used as a method to elicit emotions in laboratory environments. However, there is still a need to analyse the validity of VR in order to extrapolate the results it produces and to assess the similarities and differences in physiological responses provoked by real and virtual environments. We investigated the cardiovascular oscillations of 60 participants during a free exploration of a real museum and its virtualisation viewed through a head-mounted display. The differences between the heart rate variability features in the high and low arousal stimuli conditions were analysed through statistical hypothesis testing; and automatic arousal recognition models were developed across the real and the virtual conditions using a support vector machine algorithm with recursive feature selection. The subjects' self-assessments suggested that both museums elicited low and high arousal levels. In addition, the real museum showed differences in terms of cardiovascular responses, differences in vagal activity, while arousal recognition reached 72.92% accuracy. However, we did not find the same arousal-based autonomic nervous system change pattern during the virtual museum exploration. The results showed that, while the direct virtualisation of a real environment might be self-reported as evoking psychological arousal, it does not necessarily evoke the same cardiovascular changes as a real arousing elicitation. These contribute to the understanding of the use of VR in emotion recognition research; future research is needed to study arousal and emotion elicitation in immersive VR.

Do You See What I See? Effectiveness of 360-Degree vs. 2D Video Ads Using a Neuroscience Approach.

This study compares cognitive and emotional responses to 360-degree vs. static (2D) videos in terms of visual attention, brand recognition, engagement of the prefrontal cortex, and emotions. Hypotheses are proposed based on the interactivity literature, cognitive overload, advertising response model and motivation, opportunity, and ability theoretical frameworks, and tested using neurophysiological tools: electroencephalography, eye-tracking, electrodermal activity, and facial coding. The results revealed that gaze view depends on ad content, visual attention paid being lower in 360-degree FMCG ads than in 2D ads. Brand logo recognition is lower in 360-degree ads than in 2D video ads. Overall, 360-degree ads for durable products increase positive emotions, which carries the risk of non-exposure to some of the ad content. In testing four ads for durable goods and fast-moving consumer goods (FMCG) this research explains the mechanism through which 360-degree video ads outperform standard versions.

An ontology for Cognitive Behavioral Therapy. Application to Obesity.

In the last few years the use of medical and biomedical ontologies has increased considerably. It is very common to find applications and semantic webs using this kind of ontologies and a large number of papers have been written explaining why ontologies are useful in these fields. In this paper the use of ontologies for psychology and its benefits is discussed and a first ontology for obesity treatment is presented.

A new protocol test for physical activity research in obese children (etiobe project).

A new protocol is presented to validate TIPS (portable physiological monitoring device designed by I3BH that can get respiration, ecg and activity of the patient) for physical habits detection. Physiological and activity parameters and data from questionnaires have been acquired from a group of obese & non-obese children (n=20). Children completed activities from sedentary level to vigorous level. Preliminary results show variability on the response of children's effort and feasibility of TIPS platform as an ambulatory tool.

Virtual Reality in Marketing: A Framework, Review, and Research Agenda.

Marketing scholars and practitioners are showing increasing interest in Extended Reality (XR) technologies (XRs), such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), as very promising technological tools for producing satisfactory consumer experiences that mirror those experienced in physical stores. However, most of the studies published to date lack a certain measure of methodological rigor in their characterization of XR technologies and in the assessment techniques used to characterize the consumer experience, which limits the generalization of the results. We argue that it is necessary to define a rigorous methodological framework for the use of XRs in marketing. This article reviews the literature on XRs in marketing, and provides a conceptual framework to organize this disparate body of work.

Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum.

Virtual reality is a powerful tool in human behaviour research. However, few studies compare its capacity to evoke the same emotional responses as in real scenarios. This study investigates psycho-physiological patterns evoked during the free exploration of an art museum and the museum virtualized through a 3D immersive virtual environment (IVE). An exploratory study involving 60 participants was performed, recording electroencephalographic and electrocardiographic signals using wearable devices. The real vs. virtual psychological comparison was performed using self-assessment emotional response tests, whereas the physiological comparison was performed through Support Vector Machine algorithms, endowed with an effective feature selection procedure for a set of state-of-the-art metrics quantifying cardiovascular and brain linear and nonlinear dynamics. We included an initial calibration phase, using standardized 2D and 360° emotional stimuli, to increase the accuracy of the model. The self-assessments of the physical and virtual museum support the use of IVEs in emotion research. The 2-class (high/low) system accuracy was 71.52% and 77.08% along the arousal and valence dimension, respectively, in the physical museum, and 75.00% and 71.08% in the virtual museum. The previously presented 360° stimuli contributed to increasing the accuracy in the virtual museum. Also, the real vs. virtual classifier accuracy was 95.27%, using only EEG mean phase coherency features, which demonstrates the high involvement of brain synchronization in emotional virtual reality processes. These findings provide an important contribution at a methodological level and to scientific knowledge, which will effectively guide future emotion elicitation and recognition systems using virtual reality.

Virtual Reality as a New Approach for Risk Taking Assessment.

Understanding how people behave when facing hazardous situations, how intrinsic and extrinsic factors influence the risk taking (RT) decision making process and to what extent it is possible to modify their reactions externally, are questions that have long interested academics and society in general. In the spheres, among others, of Occupational Safety and Health (OSH), the military, finance and sociology, this topic has multidisciplinary implications because we all constantly face RT situations. Researchers have hitherto assessed RT profiles by conducting questionnaires prior to and after the presentation of stimuli; however, this can lead to the production of biased, non-realistic, RT profiles. This is due to the reflexive nature of choosing an answer in a questionnaire, which is remote from the reactive, emotional and impulsive decision making processes inherent to real, risky situations. One way to address this question is to exploit VR capabilities to generate immersive environments that recreate realistic seeming but simulated hazardous situations. We propose VR as the next-generation tool to study RT processes, taking advantage of the big four families of metrics which can provide objective assessment methods with high ecological validity: the real-world risks approach (high presence VR environments triggering real-world reactions), embodied interactions (more natural interactions eliciting more natural behaviors), stealth assessment (unnoticed real-time assessments offering efficient behavioral metrics) and physiological real-time measurement (physiological signals avoiding subjective bias). Additionally, VR can provide an invaluable tool, after the assessment phase, to train in skills related to RT due to its transferability to real-world situations.

Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors.

Affective Computing has emerged as an important field of study that aims to develop systems that can automatically recognize emotions. Up to the present, elicitation has been carried out with non-immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for affective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were designed to elicit four possible arousal-valence combinations, as described in each quadrant of the Circumplex Model of Affects. An experiment involving the recording of the electroencephalography (EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features was extracted from these signals using various state-of-the-art metrics that quantify brain and cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classifier to predict the subject's arousal and valence perception. The model's accuracy was 75.00% along the arousal dimension and 71.21% along the valence dimension. Our findings validate the use of Immersive Virtual Environments to elicit and automatically recognize different emotional states from neural and cardiac dynamics; this development could have novel applications in fields as diverse as Architecture, Health, Education and Videogames.

Consumer Neuroscience-Based Metrics Predict Recall, Liking and Viewing Rates in Online Advertising.

The purpose of the present study is to investigate whether the effectiveness of a new ad on digital channels (YouTube) can be predicted by using neural networks and neuroscience-based metrics (brain response, heart rate variability and eye tracking). Neurophysiological records from 35 participants were exposed to 8 relevant TV Super Bowl commercials. Correlations between neurophysiological-based metrics, ad recall, ad liking, the ACE metrix score and the number of views on YouTube during a year were investigated. Our findings suggest a significant correlation between neuroscience metrics and self-reported of ad effectiveness and the direct number of views on the YouTube channel. In addition, and using an artificial neural network based on neuroscience metrics, the model classifies (82.9% of average accuracy) and estimate the number of online views (mean error of 0.199). The results highlight the validity of neuromarketing-based techniques for predicting the success of advertising responses. Practitioners can consider the proposed methodology at the design stages of advertising content, thus enhancing advertising effectiveness. The study pioneers the use of neurophysiological methods in predicting advertising success in a digital context. This is the first article that has examined whether these measures could actually be used for predicting views for advertising on YouTube.

Using Virtual Reality to Distract Overweight Children from Bodily Sensations During Exercise.

This study analyzes the potential of virtual reality (VR) to enhance attentional distraction in overweight children as they experience bodily sensations during exercise. It has been suggested that one reason why obese children stop exercising is the perception of bodily sensations. In a counterbalanced design, a total of 109 children (33 overweight, 10-15 years old) were asked to walk twice for 6 minutes on a treadmill under one of two conditions: (a) traditional condition (TC)-focusing their attention on their physical feelings and sensations or (b) distraction condition (DC)-focusing their attention on a virtual environment. Attentional focus during exercise, bad-good feeling states (pre- and postexperimental), perceived exertion (3 minutes and post), heart rate, and enjoyment were assessed. Results indicated that overweight children focused on internal information under the TC, but they significantly shifted their attention to regard the external environment in the DC. This attentional distraction effect of VR was more intense in overweight than in normal-weight children. No differences between groups were found when examining changes in feeling states and perceived exertion. VR increased enjoyment during exercise, and children preferred exercise using virtual environments. VR is useful to promote distraction and may help overweight and obese children to enjoy exercise.

Cardiovascular fitness in youth: association with obesity and metabolic abnormalities.

UNLABELLED: Therapies currently implemented for obesity are focused on nutritional aspects and on physical activity. In order to make physical activity a positive therapy instead of triggering disabilities it is relevant to accurately assess cardiovascular fitness. OBJECTIVE: To assess the cardiovascular fitness by measuring the peak oxygen consumption and to asses their relationship with classical cardiometabolic parameters. METHODS: A modified Balke protocol was applied to one hundred and twenty-six Caucasians (60% males), ranging between 9 and 16 years old, who underwent an assessment of obesity. The non-obese group consisted of healthy age and sex matched subjects who were invited to participate from the general population. RESULTS: Significant differences in consumption of oxygen peak between non-obese and obese individuals were observed. In contrast, no significant differences existed between the categories of obesity. Furthermore in obese subjects consumption of oxygen peak was inversely correlated with parameters of cardiometabolic risk, particularly insulin and HOMA index. In addition, two predictive equations of consumption of oxygen peak, with an R2 of 0.74 and 0.84, respectively, have been developed. CONCLUSION: The consumption of oxygen peak is a relevant clinical parameter that should be included in the routine clinical assessment of obese subjects. Therefore, it is crucial to make exercise tests more affordable which can be achieved by employing predictive equations.

Las terapias que se implantan actualmente para la obesidad se centran en los aspectos nutricionales y sobre la actividad física. Con el fin de hacer que la actividad física sea una terapia positiva en vez de un desencadenador de discapacidades, es relevante evaluar de forma precisa el entrenamiento cardiovascular. Objetivo: evaluar el entrenamiento cardiovascular midiendo el consumo máximo de oxígeno y evaluar su relación con los parámetros cardiometabólicos clásicos. Métodos: se aplicó el protocolo modificado de Balke a 126 individuos caucásicos (60 % de varones), con edades entre 9 y 16 años, que se sometieron a una evaluación de obesidad. El grupo de no obesos consistía de individuos sanos, de la población general, emparejados por edad y sexo y a los que se les invitó a participar. Resultados: se observaron diferencias significativas en el consumo máximo de oxígeno entre los indiviudos obesos y no obesos. Por contra, no existían diferencias significativas entre las categorías de obesidad. Además, en los sujetos obesos, el consumo máximo de oxígeno se correlacionó de forma inversa con los parámetros de riesgo cardiometabólico, particularmente con la insulina y el índice HOMA. Además, se han desarrollado dos ecuaciones predictivas del consumo máximo de oxígeno con una R2 de 0,74 y de 0,84, respectivamente. Conclusión: el consumo máximo de oxígeno es un parámetro clínico relevante que debería incluirse en la evaluación clínica rutinaria de los sujetos obesos. Por lo tanto, es crucial hacer que las pruebas de esfuerzo sean más asequibles, que puedan alcanzarse empleando las ecuaciones predictivas.