Comparison of clinical geneticist and computer visual attention in assessing genetic conditions.

Artificial intelligence (AI) for facial diagnostics is increasingly used in the genetics clinic to evaluate patients with potential genetic conditions. Current approaches focus on one type of AI called Deep Learning (DL). While DL- based facial diagnostic platforms have a high accuracy rate for many conditions, less is understood about how this technology assesses and classifies (categorizes) images, and how this compares to humans. To compare human and computer attention, we performed eye-tracking analyses of geneticist clinicians (n = 22) and non-clinicians (n = 22) who viewed images of people with 10 different genetic conditions, as well as images of unaffected individuals. We calculated the Intersection-over-Union (IoU) and Kullback-Leibler divergence (KL) to compare the visual attentions of the two participant groups, and then the clinician group against the saliency maps of our deep learning classifier. We found that human visual attention differs greatly from DL model's saliency results. Averaging over all the test images, IoU and KL metric for the successful (accurate) clinician visual attentions versus the saliency maps were 0.15 and 11.15, respectively. Individuals also tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians (IoU and KL of clinicians versus non-clinicians were 0.47 and 2.73, respectively). This study shows that humans (at different levels of expertise) and a computer vision model examine images differently. Understanding these differences can improve the design and use of AI tools, and lead to more meaningful interactions between clinicians and AI technologies.

Approximating facial expression effects on diagnostic accuracy via generative AI in medical genetics.

Artificial intelligence (AI) is increasingly used in genomics research and practice, and generative AI has garnered significant recent attention. In clinical applications of generative AI, aspects of the underlying datasets can impact results, and confounders should be studied and mitigated. One example involves the facial expressions of people with genetic conditions. Stereotypically, Williams (WS) and Angelman (AS) syndromes are associated with a "happy" demeanor, including a smiling expression. Clinical geneticists may be more likely to identify these conditions in images of smiling individuals. To study the impact of facial expression, we analyzed publicly available facial images of approximately 3500 individuals with genetic conditions. Using a deep learning (DL) image classifier, we found that WS and AS images with non-smiling expressions had significantly lower prediction probabilities for the correct syndrome labels than those with smiling expressions. This was not seen for 22q11.2 deletion and Noonan syndromes, which are not associated with a smiling expression. To further explore the effect of facial expressions, we computationally altered the facial expressions for these images. We trained HyperStyle, a GAN-inversion technique compatible with StyleGAN2, to determine the vector representations of our images. Then, following the concept of InterfaceGAN, we edited these vectors to recreate the original images in a phenotypically accurate way but with a different facial expression. Through online surveys and an eye-tracking experiment, we examined how altered facial expressions affect the performance of human experts. We overall found that facial expression is associated with diagnostic accuracy variably in different genetic conditions.

Transcranial direct current stimulation as an additional treatment to selective serotonin reuptake inhibitors in adults with major depressive disorder in Germany (DepressionDC): a triple-blind, randomised, sham-controlled, multicentre trial.

BACKGROUND: Transcranial direct current stimulation (tDCS) has been proposed as a feasible treatment for major depressive disorder (MDD). However, meta-analytic evidence is heterogenous and data from multicentre trials are scarce. We aimed to assess the efficacy of tDCS versus sham stimulation as an additional treatment to a stable dose of selective serotonin reuptake inhibitors (SSRIs) in adults with MDD. METHODS: The DepressionDC trial was triple-blind, randomised, and sham-controlled and conducted at eight hospitals in Germany. Patients being treated at a participating hospital aged 18-65 years were eligible if they had a diagnosis of MDD, a score of at least 15 on the Hamilton Depression Rating Scale (21-item version), no response to at least one antidepressant trial in their current depressive episode, and treatment with an SSRI at a stable dose for at least 4 weeks before inclusion; the SSRI was continued at the same dose during stimulation. Patients were allocated (1:1) by fixed-blocked randomisation to receive either 30 min of 2 mA bifrontal tDCS every weekday for 4 weeks, then two tDCS sessions per week for 2 weeks, or sham stimulation at the same intervals. Randomisation was stratified by site and baseline Montgomery-Åsberg Depression Rating Scale (MADRS) score (ie, <31 or ≥31). Participants, raters, and operators were masked to treatment assignment. The primary outcome was change on the MADRS at week 6, analysed in the intention-to-treat population. Safety was assessed in all patients who received at least one treatment session. The trial was registered with ClinicalTrials.gov (NCT02530164). FINDINGS: Between Jan 19, 2016, and June 15, 2020, 3601 individuals were assessed for eligibility. 160 patients were included and randomly assigned to receive either active tDCS (n=83) or sham tDCS (n=77). Six patients withdrew consent and four patients were found to have been wrongly included, so data from 150 patients were analysed (89 [59%] were female and 61 [41%] were male). No intergroup difference was found in mean improvement on the MADRS at week 6 between the active tDCS group (n=77; -8·2, SD 7·2) and the sham tDCS group (n=73; -8·0, 9·3; difference 0·3 [95% CI -2·4 to 2·9]). Significantly more participants had one or more mild adverse events in the active tDCS group (50 [60%] of 83) than in the sham tDCS group (33 [43%] of 77; p=0·028). INTERPRETATION: Active tDCS was not superior to sham stimulation during a 6-week period. Our trial does not support the efficacy of tDCS as an additional treatment to SSRIs in adults with MDD. FUNDING: German Federal Ministry of Education and Research.

Decoding emotions: Exploring the validity of sentiment analysis in psychotherapy.

OBJECTIVE: Given the importance of emotions in psychotherapy, valid measures are essential for research and practice. As emotions are expressed at different levels, multimodal measurements are needed for a nuanced assessment. Natural Language Processing (NLP) could augment the measurement of emotions. The study explores the validity of sentiment analysis in psychotherapy transcripts. METHOD: We used a transformer-based NLP algorithm to analyze sentiments in 85 transcripts from 35 patients. Construct and criterion validity were evaluated using self- and therapist reports and process and outcome measures via correlational, multitrait-multimethod, and multilevel analyses. RESULTS: The results provide indications in support of the sentiments' validity. For example, sentiments were significantly related to self- and therapist reports of emotions in the same session. Sentiments correlated significantly with in-session processes (e.g., coping experiences), and an increase in positive sentiments throughout therapy predicted better outcomes after treatment termination. DISCUSSION: Sentiment analysis could serve as a valid approach to assessing the emotional tone of psychotherapy sessions and may contribute to the multimodal measurement of emotions. Future research could combine sentiment analysis with automatic emotion recognition in facial expressions and vocal cues via the Nonverbal Behavior Analyzer (NOVA). Limitations (e.g., exploratory study with numerous tests) and opportunities are discussed.

COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition.

Automatically recognising apparent emotions from face and voice is hard, in part because of various sources of uncertainty, including in the input data and the labels used in a machine learning framework. This paper introduces an uncertainty-aware multimodal fusion approach that quantifies modality-wise aleatoric or data uncertainty towards emotion prediction. We propose a novel fusion framework, in which latent distributions over unimodal temporal context are learned by constraining their variance. These variance constraints, Calibration and Ordinal Ranking, are designed such that the variance estimated for a modality can represent how informative the temporal context of that modality is w.r.t. emotion recognition. When well-calibrated, modality-wise uncertainty scores indicate how much their corresponding predictions are likely to differ from the ground truth labels. Well-ranked uncertainty scores allow the ordinal ranking of different frames across different modalities. To jointly impose both these constraints, we propose a softmax distributional matching loss. Our evaluation on AVEC 2019 CES, CMU-MOSEI, and IEMOCAP datasets shows that the proposed multimodal fusion method not only improves the generalisation performance of emotion recognition models and their predictive uncertainty estimates, but also makes the models robust to novel noise patterns encountered at test time.

Flow in human-robot collaboration-multimodal analysis and perceived challenge detection in industrial scenarios.

Introduction: Flow state, the optimal experience resulting from the equilibrium between perceived challenge and skill level, has been extensively studied in various domains. However, its occurrence in industrial settings has remained relatively unexplored. Notably, the literature predominantly focuses on Flow within mentally demanding tasks, which differ significantly from industrial tasks. Consequently, our understanding of emotional and physiological responses to varying challenge levels, specifically in the context of industry-like tasks, remains limited. Methods: To bridge this gap, we investigate how facial emotion estimation (valence, arousal) and Heart Rate Variability (HRV) features vary with the perceived challenge levels during industrial assembly tasks. Our study involves an assembly scenario that simulates an industrial human-robot collaboration task with three distinct challenge levels. As part of our study, we collected video, electrocardiogram (ECG), and NASA-TLX questionnaire data from 37 participants. Results: Our results demonstrate a significant difference in mean arousal and heart rate between the low-challenge (Boredom) condition and the other conditions. We also found a noticeable trend-level difference in mean heart rate between the adaptive (Flow) and high-challenge (Anxiety) conditions. Similar differences were also observed in a few other temporal HRV features like Mean NN and Triangular index. Considering the characteristics of typical industrial assembly tasks, we aim to facilitate Flow by detecting and balancing the perceived challenge levels. Leveraging our analysis results, we developed an HRV-based machine learning model for discerning perceived challenge levels, distinguishing between low and higher-challenge conditions. Discussion: This work deepens our understanding of emotional and physiological responses to perceived challenge levels in industrial contexts and provides valuable insights for the design of adaptive work environments.

Gaze detection as a social cue to initiate natural human-robot collaboration in an assembly task.

Introduction: In this work we explore a potential approach to improve human-robot collaboration experience by adapting cobot behavior based on natural cues from the operator. Methods: Inspired by the literature on human-human interactions, we conducted a wizard-of-oz study to examine whether a gaze towards the cobot can serve as a trigger for initiating joint activities in collaborative sessions. In this study, 37 participants engaged in an assembly task while their gaze behavior was analyzed. We employed a gaze-based attention recognition model to identify when the participants look at the cobot. Results: Our results indicate that in most cases (83.74%), the joint activity is preceded by a gaze towards the cobot. Furthermore, during the entire assembly cycle, the participants tend to look at the cobot mostly around the time of the joint activity. Given the above results, a fully integrated system triggering joint action only when the gaze is directed towards the cobot was piloted with 10 volunteers, of which one characterized by high-functioning Autism Spectrum Disorder. Even though they had never interacted with the robot and did not know about the gaze-based triggering system, most of them successfully collaborated with the cobot and reported a smooth and natural interaction experience. Discussion: To the best of our knowledge, this is the first study to analyze the natural gaze behavior of participants working on a joint activity with a robot during a collaborative assembly task and to attempt the full integration of an automated gaze-based triggering system.

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario.

Introduction: Since the COVID-19 pandemic, working environments and private lives have changed dramatically. Digital technologies and media have become more and more important and have found their way into nearly all private and work environments. Communication situations have been largely relocated to virtual spaces. One of these scenarios is digital job interviews. Job interviews are usually-also in the non-digital world-perceived as stressful and associated with biological stress responses. We here present and evaluate a newly developed laboratory stressor that is based on a digital job interview-scenario. Methods: N = 45 healthy people participated in the study (64.4% female; mean age: 23.2 ± 3.6 years; mean body mass index = 22.8 ± 4.0 kg/m2). Salivary alpha-amylase (sAA) and cortisol were assessed as measures for biological stress responses. Furthermore, perceived stress was rated at the time points of the saliva samplings. The job interviews lasted between 20 and 25 min. All materials, including instructions for the experimenter (i.e., the job interviewer) and the data set used for statistical analysis, as well as a multimodal data set, which includes further measures, are publicly available. Results: Typical subjective and biological stress-response patterns were found, with peak sAA and perceived stress levels observed immediately after the job interviews and peak cortisol concentrations 5 min afterwards. Female participants experienced the scenario as more stressful than male participants. Cortisol peaks were higher for participants who experienced the situation as a threat in comparison to participants who experienced it as a challenge. Associations between the strength of the stress response with further person characteristics and psychological variables such as BMI, age, coping styles, and personality were not found. Discussion: Overall, our method is well-suited to induce biological and perceived stress, mostly independent of person characteristics and psychological variables. The setting is naturalistic and easily implementable in standardized laboratory settings.

Relieving the burden of intensive labeling for stress monitoring in the wild by using semi-supervised learning.

Stress, a natural process affecting individuals' wellbeing, has a profound impact on overall quality of life. Researchers from diverse fields employ various technologies and methodologies to investigate it and alleviate the negative effects of this phenomenon. Wearable devices, such as smart bands, capture physiological data, including heart rate variability, motions, and electrodermal activity, enabling stress level monitoring through machine learning models. However, labeling data for model accuracy assessment poses a significant challenge in stress-related research due to incomplete or inaccurate labels provided by individuals in their daily lives. To address this labeling predicament, our study proposes implementing Semi-Supervised Learning (SSL) models. Through comparisons with deep learning-based supervised models and clustering-based unsupervised models, we evaluate the performance of our SSL models. Our experiments show that our SSL models achieve 77% accuracy with a classifier trained on an augmented dataset prepared using the label propagation (LP) algorithm. Additionally, our deep autoencoder network achieves 76% accuracy. These results highlight the superiority of SSL models over unsupervised learning techniques and their comparable performance to supervised learning models, even with limited labeled data. By relieving the burden of labeling in daily life stress recognition, our study advances stress-related research, recognizing stress as a natural process rather than a disease. This facilitates the development of more efficient and accurate stress monitoring methods in the wild.

Few-Shot Meta-Learning for Recognizing Facial Phenotypes of Genetic Disorders.

Computer vision has useful applications in precision medicine and recognizing facial phenotypes of genetic disorders is one of them. Many genetic disorders are known to affect faces' visual appearance and geometry. Automated classification and similarity retrieval aid physicians in decision-making to diagnose possible genetic conditions as early as possible. Previous work has addressed the problem as a classification problem; however, the sparse label distribution, having few labeled samples, and huge class imbalances across categories make representation learning and generalization harder. In this study, we used a facial recognition model trained on a large corpus of healthy individuals as a pre-task and transferred it to facial phenotype recognition. Furthermore, we created simple baselines of few-shot meta-learning methods to improve our base feature descriptor. Our quantitative results on GestaltMatcher Database (GMDB) show that our CNN baseline surpasses previous works, including GestaltMatcher, and few-shot meta-learning strategies improve retrieval performance in frequent and rare classes.