Modeling inter-individual differences in ambulatory-based multimodal signals via metric learning: a case study of personalized well-being estimation of healthcare workers.

Introduction: Intelligent ambulatory tracking can assist in the automatic detection of psychological and emotional states relevant to the mental health changes of professionals with high-stakes job responsibilities, such as healthcare workers. However, well-known differences in the variability of ambulatory data across individuals challenge many existing automated approaches seeking to learn a generalizable means of well-being estimation. This paper proposes a novel metric learning technique that improves the accuracy and generalizability of automated well-being estimation by reducing inter-individual variability while preserving the variability pertaining to the behavioral construct. Methods: The metric learning technique implemented in this paper entails learning a transformed multimodal feature space from pairwise similarity information between (dis)similar samples per participant via a Siamese neural network. Improved accuracy via personalization is further achieved by considering the trait characteristics of each individual as additional input to the metric learning models, as well as individual trait base cluster criteria to group participants followed by training a metric learning model for each group. Results: The outcomes of the proposed models demonstrate significant improvement over the other inter-individual variability reduction and deep neural baseline methods for stress, anxiety, positive affect, and negative affect. Discussion: This study lays the foundation for accurate estimation of psychological and emotional states in realistic and ambulatory environments leading to early diagnosis of mental health changes and enabling just-in-time adaptive interventions.

TILES-2019: A longitudinal physiologic and behavioral data set of medical residents in an intensive care unit.

The TILES-2019 data set consists of behavioral and physiological data gathered from 57 medical residents (i.e., trainees) working in an intensive care unit (ICU) in the United States. The data set allows for the exploration of longitudinal changes in well-being, teamwork, and job performance in a demanding environment, as residents worked in the ICU for three weeks. Residents wore a Fitbit, a Bluetooth-based proximity sensor, and an audio-feature recorder. They completed daily surveys and interviews at the beginning and end of their rotation. In addition, we collected data from environmental sensors (i.e., Internet-of-Things Bluetooth data hubs) and obtained hospital records (e.g., patient census) and residents' job evaluations. This data set may be may be of interest to researchers interested in workplace stress, group dynamics, social support, the physical and psychological effects of witnessing patient deaths, predicting survey data from sensors, and privacy-aware and privacy-preserving machine learning. Notably, a small subset of the data was collected during the first wave of the COVID-19 pandemic.

A multimodal analysis of physical activity, sleep, and work shift in nurses with wearable sensor data.

Night shift workers are often associated with circadian misalignment and physical discomfort, which may lead to burnout and decreased work performance. Moreover, the irregular work hours can lead to significant negative health outcomes such as poor eating habits, smoking, and being sedentary more often. This paper uses commercial wearable sensors to explore correlates and differences in the level of physical activity, sleep, and circadian misalignment indicators among day shift nurses and night shift nurses. We identify which self-reported assessments of affect, life satisfaction, and sleep quality, are associated with physiological and behavioral signals captured by wearable sensors. The results using data collected from 113 nurses in a large hospital setting, over a period of 10 weeks, indicate that night shift nurses are more sedentary, and report lower levels of life satisfaction than day-shift nurses. Moreover, night shift nurses report poorer sleep quality, which may be correlated with challenges in their attempts to fall asleep on off-days.

Dynamical systems modeling of day-to-day signal-based patterns of emotional self-regulation and stress spillover in highly-demanding health professions.

As hospital workers face a growing number of patients and have to meet increasingly rigorous standards of care, their ability to successfully modulate their emotional reactions and flexibly handle stress presents a significant challenge. This paper examines a multimodal signal-driven way to quantify emotion self-regulation and stress spillover through a dynamical systems model (DSM). The proposed DSM models day-to-day changes of emotional arousal, captured through speech, physiology, and daily activity measures, and its interplay with daily stress. The parameters of the DSM quantify the degree of self-regulation and stress spillover, and are associated with work performance and cognitive ability in a multimodal dataset of 130 full-time hospital workers recorded over a 10-week period. Linear regression experiments indicate the effectiveness of the proposed features to reliably estimate individuals' work performance and cognitive ability, providing significantly higher Pearson's correlations compared to aggregate measures of emotional arousal. Results from this study demonstrate the importance of quantifying oscillatory behaviors from longitudinal ambulatory signals and can potentially deepen our understanding of emotion self-regulation and stress spillover using signal-driven measurements, which complement self-reports and provide estimates of the psychological constructs of interest in a fine-grained time resolution.

TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers.

We present a novel longitudinal multimodal corpus of physiological and behavioral data collected from direct clinical providers in a hospital workplace. We designed the study to investigate the use of off-the-shelf wearable and environmental sensors to understand individual-specific constructs such as job performance, interpersonal interaction, and well-being of hospital workers over time in their natural day-to-day job settings. We collected behavioral and physiological data from n = 212 participants through Internet-of-Things Bluetooth data hubs, wearable sensors (including a wristband, a biometrics-tracking garment, a smartphone, and an audio-feature recorder), together with a battery of surveys to assess personality traits, behavioral states, job performance, and well-being over time. Besides the default use of the data set, we envision several novel research opportunities and potential applications, including multi-modal and multi-task behavioral modeling, authentication through biometrics, and privacy-aware and privacy-preserving machine learning.

Multimodal Human and Environmental Sensing for Longitudinal Behavioral Studies in Naturalistic Settings: Framework for Sensor Selection, Deployment, and Management.

BACKGROUND: Recent advances in mobile technologies for sensing human biosignals are empowering researchers to collect real-world data outside of the laboratory, in natural settings where participants can perform their daily activities with minimal disruption. These new sensing opportunities usher a host of challenges and constraints for both researchers and participants. OBJECTIVE: This viewpoint paper aims to provide a comprehensive guide to aid research teams in the selection and management of sensors before beginning and while conducting human behavior studies in the wild. The guide aims to help researchers achieve satisfactory participant compliance and minimize the number of unexpected procedural outcomes. METHODS: This paper presents a collection of challenges, consideration criteria, and potential solutions for enabling researchers to select and manage appropriate sensors for their research studies. It explains a general data collection framework suitable for use with modern consumer sensors, enabling researchers to address many of the described challenges. In addition, it provides a description of the criteria affecting sensor selection, management, and integration that researchers should consider before beginning human behavior studies involving sensors. On the basis of a survey conducted in mid-2018, this paper further illustrates an organized snapshot of consumer-grade human sensing technologies that can be used for human behavior research in natural settings. RESULTS: The research team applied the collection of methods and criteria to a case study aimed at predicting the well-being of nurses and other staff in a hospital. Average daily compliance for sensor usage measured by the presence of data exceeding half the total possible hours each day was about 65%, yielding over 355,000 hours of usable sensor data across 212 participants. A total of 6 notable unexpected events occurred during the data collection period, all of which had minimal impact on the research project. CONCLUSIONS: The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.

An Evaluation of EEG-based Metrics for Engagement Assessment of Distance Learners.

Maintaining students' cognitive engagement in educational settings is crucial to their performance, though quantifying this mental state in real-time for distance learners has not been studied extensively in natural distance learning environments. We record electroencephalographic (EEG) data of students watching online lecture videos and use it to predict engagement rated by human annotators. An evaluation of prior EEG-based engagement metrics that utilize power spectral density (PSD) features is presented. We examine the predictive power of various supervised machine learning approaches with both subject-independent and individualized models when using simple PSD feature functions. Our results show that engagement metrics with few power band variables, including those proposed in prior research, do not produce predictions consistent with human observations. We quantify the performance disparity between cross-subject and per-subject models and demonstrate that individual differences in EEG patterns necessitate a more complex metric for educational engagement assessment in natural distance learning environments.