Emotional and cognitive influences in air traffic controller tasks: An investigation using a virtual environment?

Air traffic controllers are required to perform complex tasks which require attention and high precision. This study investigates how the difficulty of such tasks influences emotional states, cognitive workload and task performance. We use quantitative and qualitative measurements, including the recording of pupil dilation and changes in affect using questionnaires. Participants were required to perform a number of air traffic control tasks using the immersive human accessible Virtual Reality space in the "eXperience Induction Machine". Based on the data collected, we developed and validated a model which integrates personality, workload and affective theories. Our results indicate that the difficulty of an air traffic control task has a direct influence on cognitive workload as well as on the self-reported mood; whereas both mood and workload seem to change independently. In addition, we show that personality, in particular neuroticism, affects both mood and performance of the participants.

The Affective Slider: A Digital Self-Assessment Scale for the Measurement of Human Emotions.

Self-assessment methods are broadly employed in emotion research for the collection of subjective affective ratings. The Self-Assessment Manikin (SAM), a pictorial scale developed in the eighties for the measurement of pleasure, arousal, and dominance, is still among the most popular self-reporting tools, despite having been conceived upon design principles which are today obsolete. By leveraging on state-of-the-art user interfaces and metacommunicative pictorial representations, we developed the Affective Slider (AS), a digital self-reporting tool composed of two slider controls for the quick assessment of pleasure and arousal. To empirically validate the AS, we conducted a systematic comparison between AS and SAM in a task involving the emotional assessment of a series of images taken from the International Affective Picture System (IAPS), a database composed of pictures representing a wide range of semantic categories often used as a benchmark in psychological studies. Our results show that the AS is equivalent to SAM in the self-assessment of pleasure and arousal, with two added advantages: the AS does not require written instructions and it can be easily reproduced in latest-generation digital devices, including smartphones and tablets. Moreover, we compared new and normative IAPS ratings and found a general drop in reported arousal of pictorial stimuli. Not only do our results demonstrate that legacy scales for the self-report of affect can be replaced with new measurement tools developed in accordance to modern design principles, but also that standardized sets of stimuli which are widely adopted in research on human emotion are not as effective as they were in the past due to a general desensitization towards highly arousing content.

Network dynamics with BrainX(3): a large-scale simulation of the human brain network with real-time interaction.

BrainX(3) is a large-scale simulation of human brain activity with real-time interaction, rendered in 3D in a virtual reality environment, which combines computational power with human intuition for the exploration and analysis of complex dynamical networks. We ground this simulation on structural connectivity obtained from diffusion spectrum imaging data and model it on neuronal population dynamics. Users can interact with BrainX(3) in real-time by perturbing brain regions with transient stimulations to observe reverberating network activity, simulate lesion dynamics or implement network analysis functions from a library of graph theoretic measures. BrainX(3) can thus be used as a novel immersive platform for exploration and analysis of dynamical activity patterns in brain networks, both at rest or in a task-related state, for discovery of signaling pathways associated to brain function and/or dysfunction and as a tool for virtual neurosurgery. Our results demonstrate these functionalities and shed insight on the dynamics of the resting-state attractor. Specifically, we found that a noisy network seems to favor a low firing attractor state. We also found that the dynamics of a noisy network is less resilient to lesions. Our simulations on TMS perturbations show that even though TMS inhibits most of the network, it also sparsely excites a few regions. This is presumably due to anti-correlations in the dynamics and suggests that even a lesioned network can show sparsely distributed increased activity compared to healthy resting-state, over specific brain areas.

Fast mental states decoding in mixed reality.

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Inference of human affective states from psychophysiological measurements extracted under ecologically valid conditions.

Compared to standard laboratory protocols, the measurement of psychophysiological signals in real world experiments poses technical and methodological challenges due to external factors that cannot be directly controlled. To address this problem, we propose a hybrid approach based on an immersive and human accessible space called the eXperience Induction Machine (XIM), that incorporates the advantages of a laboratory within a life-like setting. The XIM integrates unobtrusive wearable sensors for the acquisition of psychophysiological signals suitable for ambulatory emotion research. In this paper, we present results from two different studies conducted to validate the XIM as a general-purpose sensing infrastructure for the study of human affective states under ecologically valid conditions. In the first investigation, we recorded and classified signals from subjects exposed to pictorial stimuli corresponding to a range of arousal levels, while they were free to walk and gesticulate. In the second study, we designed an experiment that follows the classical conditioning paradigm, a well-known procedure in the behavioral sciences, with the additional feature that participants were free to move in the physical space, as opposed to similar studies measuring physiological signals in constrained laboratory settings. Our results indicate that, by using our sensing infrastructure, it is indeed possible to infer human event-elicited affective states through measurements of psychophysiological signals under ecological conditions.

The social perceptual salience effect.

Affective processes appraise the salience of external stimuli preparing the agent for action. So far, the relationship between stimuli, affect, and action has been mainly studied in highly controlled laboratory conditions. In order to find the generalization of this relationship to social interaction, we assess the influence of the salience of social stimuli on human interaction. We constructed reality ball game in a mixed reality space where pairs of people collaborated in order to compete with an opposing team. We coupled the players with team members with varying social salience by using both physical and virtual representations of remote players (i.e., avatars). We observe that, irrespective of the team composition, winners and losers display significantly different inter- and intrateam spatial behaviors. We show that subjects regulate their interpersonal distance to both virtual and physical team members in similar ways, but in proportion to the vividness of the stimulus. As an independent validation of this social salience effect, we show that this behavioral effect is also displayed in physiological correlates of arousal. In addition, we found a strong correlation between performance, physiology, and the subjective reports of the subjects. Our results show that proxemics is consistent with affective responses, confirming the existence of a social salience effect. This provides further support for the so-called law of apparent reality, and it generalizes it to the social realm, where it can be used to design more efficient social artifacts.