Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data.

Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches-Audio Branch, Video Branch, and Text Branch-each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks-reading and interviewing-implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

Facial mimicry interference reduces working memory accuracy for facial emotion expressions.

Facial mimicry, the tendency to imitate facial expressions of other individuals, has been shown to play a critical role in the processing of emotion expressions. At the same time, there is evidence suggesting that its role might change when the cognitive demands of the situation increase. In such situations, understanding another person is dependent on working memory. However, whether facial mimicry influences working memory representations for facial emotion expressions is not fully understood. In the present study, we experimentally interfered with facial mimicry by using established behavioral procedures, and investigated how this interference influenced working memory recall for facial emotion expressions. Healthy, young adults (N = 36) performed an emotion expression n-back paradigm with two levels of working memory load, low (1-back) and high (2-back), and three levels of mimicry interference: high, low, and no interference. Results showed that, after controlling for block order and individual differences in the perceived valence and arousal of the stimuli, the high level of mimicry interference impaired accuracy when working memory load was low (1-back) but, unexpectedly, not when load was high (2-back). Working memory load had a detrimental effect on performance in all three mimicry conditions. We conclude that facial mimicry might support working memory for emotion expressions when task load is low, but that the supporting effect possibly is reduced when the task becomes more cognitively challenging.

Divergent effects of chronic continuous and intermittent social defeat stress on emotional behaviors: Impact on phosphorylated CREB and BDNF protein levels in the rat hippocampus.

Chronic psychosocial stress stands as a significant heterogeneous risk factor for psychiatric disorders. The brain's physiological response to such stress varies based on the frequency and intensity of stress episodes. However, whether stress episodes divergently could affect hippocampal cyclic AMP response element-binding protein (CREB)-brain-derived neurotrophic factor (BDNF) signaling remains unclear, a key regulator of psychiatric symptoms. We aimed to assess how two distinct patterns of social defeat stress exposure impact anxiety- and depression-like behaviors, fear, and hippocampal CREB-BDNF signaling in adult male rats. To explore this, adult male Sprague-Dawley rats were subjected to psychosocial stress using a Resident/Intruder paradigm for ten consecutive days (continuous social defeat stress: [CS]) or ten social defeat stress over the course of 21 days (intermittent social defeat stress [IS]). Behavioral tests (including novelty-suppressed feeding test, forced swimming test, and contextually conditioned fear) were conducted. Protein expression levels of phosphorylated CREB and BDNF in the dorsal and ventral hippocampi were examined. CS led to heightened anxiety-like behavior, fear, and increased levels of phosphorylated CREB in both the dorsal and ventral hippocampi. Conversely, IS resulted in increased anxiety-like behavior and behavioral despair alongside decreased levels of phosphorylated CREB and BDNF, particularly in the dorsal hippocampus. These findings indicate that chronic psychosocial stress divergently affects hippocampal CREB-BDNF signaling and emotional regulation depending on the stress episode. Such insights could enhance our understanding of the molecular basis of the heterogeneity of psychiatric disorders and facilitate the development of innovative treatment approaches to patients with psychiatric disorders.

Social good reappraisal as a novel and effective emotion regulation strategy.

Society asks individuals, such as front-line medical and emergency personnel or social media moderators, to help others under highly negative emotional circumstances, and those individuals need to regulate their emotions for their own well being. A well-studied form of emotion regulation is reappraisal, the use of cognitive processes used to reinterpret initial emotional responses to negative events. Distancing (pretending that a situation is distant in time or space) is well documented to be an effective form of emotion regulation, but it may not be applicable in social contexts where individuals must engage with distressing events to help others. Here, for the first time, we asked whether a novel reappraisal strategy focused on Social Good-imagining that an aversive event is also an opportunity to prevent harm to others-can be an effective form of reappraisal. In a pre-registered experiment, participants were randomly assigned to Distancing or Social Good conditions as they viewed neutral or highly aversive images and then reported their subjective emotional states with or without reappraisal. Both Distancing and Social Good reappraisals led to significantly less negative affect. Distancing yielded a stronger effect, but importantly, participants reported both Distancing and Social Good as equally easy to employ and both were effective across multiple demographic and personality characteristics, indicating the broad value of both as effective forms of reappraisal. Across both reappraisal conditions, effective reappraisal increased with age and positive affect. These findings indicate that Social Good is an effective reappraisal strategy and raise the possibility that it could be particularly valuable in contexts in which emotionally demanding tasks are completed on behalf of the good for other people.

Image-based facial emotion recognition using convolutional neural network on emognition dataset.

Detecting emotions from facial images is difficult because facial expressions can vary significantly. Previous research on using deep learning models to classify emotions from facial images has been carried out on various datasets that contain a limited range of expressions. This study expands the use of deep learning for facial emotion recognition (FER) based on Emognition dataset that includes ten target emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, sadness, and neutral. A series of data preprocessing was carried out to convert video data into images and augment the data. This study proposes Convolutional Neural Network (CNN) models built through two approaches, which are transfer learning (fine-tuned) with pre-trained models of Inception-V3 and MobileNet-V2 and building from scratch using the Taguchi method to find robust combination of hyperparameters setting. The proposed model demonstrated favorable performance over a series of experimental processes with an accuracy and an average F1-score of 96% and 0.95, respectively, on the test data.

Predatory Odor Exposure as a Potential Paradigm for Studying Emotional Modulation of Memory Consolidation-The Role of the Noradrenergic Transmission in the Basolateral Amygdala.

The pivotal role of the basolateral amygdala (BLA) in the emotional modulation of hippocampal plasticity and memory consolidation is well-established. Specifically, multiple studies have demonstrated that the activation of the noradrenergic (NA) system within the BLA governs these modulatory effects. However, most current evidence has been obtained by direct infusion of synthetic NA or beta-adrenergic agonists. In the present study, we aimed to investigate the effect of endogenous NA release in the BLA, induced by a natural aversive stimulus (coyote urine), on memory consolidation for a low-arousing, hippocampal-dependent task. Our experiments combined a weak object location task (OLT) version with subsequent mild predator odor exposure (POE). To investigate the role of endogenous NA in the BLA in memory modulation, a subset of the animals (Wistar rats) was treated with the non-selective beta-blocker propranolol at the end of the behavioral procedures. Hippocampal tissue was collected 90 min after drug infusion or after the OLT test, which was performed 24 h later. We used the obtained samples to estimate the levels of phosphorylated CREB (pCREB) and activity-regulated cytoskeleton-associated protein (Arc)-two molecular markers of experience-dependent changes in neuronal activity. The result suggests that POE has the potential to become a valuable behavioral paradigm for studying the interaction between BLA and the hippocampus in memory prioritization and selectivity.

Tactile emoticons: Conveying social emotions and intentions with manual and robotic tactile feedback during social media communications.

Touch offers important non-verbal possibilities for socioaffective communication. Yet most digital communications lack capabilities regarding exchanging affective tactile messages (tactile emoticons). Additionally, previous studies on tactile emoticons have not capitalised on knowledge about the affective effects of certain mechanoreceptors in the human skin, e.g., the C-Tactile (CT) system. Here, we examined whether gentle manual stroking delivered in velocities known to optimally activate the CT system (defined as 'tactile emoticons'), during lab-simulated social media communications could convey increased feelings of social support and other prosocial intentions compared to (1) either stroking touch at CT sub-optimal velocities, or (2) standard visual emoticons. Participants (N = 36) felt more social intent with CT-optimal compared to sub-optimal velocities, or visual emoticons. In a second, preregistered study (N = 52), we investigated whether combining visual emoticons with tactile emoticons, this time delivered at CT-optimal velocities by a soft robotic device, could enhance the perception of prosocial intentions and affect participants' physiological measures (e.g., skin conductance rate) in comparison to visual emoticons alone. Visuotactile emoticons conveyed more social intent overall and in anxious participants affected physiological measures more than visual emoticons. The results suggest that emotional social media communications can be meaningfully enhanced by tactile emoticons.

Brain representations of affective valence and intensity in sustained pleasure and pain.

Pleasure and pain are two fundamental, intertwined aspects of human emotions. Pleasurable sensations can reduce subjective feelings of pain and vice versa, and we often perceive the termination of pain as pleasant and the absence of pleasure as unpleasant. This implies the existence of brain systems that integrate them into modality-general representations of affective experiences. Here, we examined representations of affective valence and intensity in an functional MRI (fMRI) study (n = 58) of sustained pleasure and pain. We found that the distinct subpopulations of voxels within the ventromedial and lateral prefrontal cortices, the orbitofrontal cortex, the anterior insula, and the amygdala were involved in decoding affective valence versus intensity. Affective valence and intensity predictive models showed significant decoding performance in an independent test dataset (n = 62). These models were differentially connected to distinct large-scale brain networks-the intensity model to the ventral attention network and the valence model to the limbic and default mode networks. Overall, this study identified the brain representations of affective valence and intensity across pleasure and pain, promoting a systems-level understanding of human affective experiences.

Modulation of response times in early-stage Parkinson's disease during emotional processing of embodied and non-embodied stimuli.

Valence (positive and negative) and content (embodied vs non-embodied) characteristics of visual stimuli have been shown to influence motor readiness, as tested with response time paradigms. Both embodiment and emotional processing are affected in Parkinson's disease (PD) due to basal ganglia dysfunction. Here we aimed to investigate, using a two-choice response time paradigm, motor readiness when processing embodied (emotional body language [EBL] and emotional facial expressions [FACS]) vs non-embodied (emotional scenes [IAPS]) stimuli with neutral, happy, and fearful content. We enrolled twenty-five patients with early-stage PD and twenty-five age matched healthy participants. Motor response during emotional processing was assessed by measuring response times (RTs) in a home-based, forced two-choice discrimination task where participants were asked to discriminate the emotional stimulus from the neutral one. Rating of valence and arousal was also performed. A clinical and neuropsychological evaluation was performed on PD patients. Results showed that RTs for PD patients were longer for all conditions compared to HC and that RTs were generally longer in both groups for EBL compared to FACS and IAPS, with the sole exception retrieved for PD, where in discriminating fearful stimuli, RTs for EBL were longer compared to FACS but not to IAPS. Furthermore, in PD only, when discriminating fearful respect to neutral stimuli, RTs were shorter when discriminating FACS compared to IAPS. This study shows that PD patients were faster in discriminating fearful embodied stimuli, allowing us to speculate on mechanisms involving an alternative, compensatory, emotional motor pathway for PD patients undergoing fear processing.

Age-related differences in the evaluation of highly arousing language.

Emotional properties of words can profoundly affect their processing, depending on both the valence (pleasantness) and the degree of arousal (excitation) that the word elicits. Words that are strongly emotionally arousing (such as taboo words) can interfere with subsequent language processing (White & Abrams, 2021). However, little is known about whether or how aging affects the processing of highly arousing language. The present study provides a characterization of how adults across the lifespan evaluate highly arousing language with a simple rating task that included taboo words, which have previously been used to examine lexical interference caused by arousal, and humorous words, which are also highly arousing without being negatively valenced. While arousal ratings were strongly positively correlated with both tabooness and humor ratings for young adults, these relationships weakened with age and overall arousal ratings were lower for middle-aged and older adults compared to young adults. Age effects cannot be readily accounted for by age-related differences in psychosocial variables such as self-reported profanity avoidance or religiosity. The effect of age on arousal should be considered in the design of studies examining age-related changes in emotional language processing. Furthermore, age differences in arousal should be considered as a potential mechanism in studies exploring emotional language processing across adulthood. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Do pictures' emotional valence and arousal affect younger and older adults' narratives?

Emotional content, specifically negative valence, can differentially influence speech production in younger and older adults' autobiographical narratives, which have been interpreted as reflecting age differences in emotion regulation. However, age differences in emotional reactivity are another possible explanation, as younger and older adults frequently differ in their affective responses to negative and positive pictures. The present experiment investigated whether a picture's valence (pleasantness) and arousal (intensity) influenced older adults' production of narratives about those pictures. Thirty younger and 30 older participants produced narratives about pictures that varied in valence (positive, negative, and neutral) and arousal (high, low). Narratives were recorded via Zoom, transcribed, and analyzed with Linguistic Inquiry and Word Count-22 to get measures of emotional word use, disfluencies, and linguistic distance. Results showed that negative valence increased age differences in speech production independent of picture arousal: Relative to younger adults, older adults used more positive words, fewer negative words, and had more silent pauses when telling narratives about negative pictures. In contrast, high arousal decreased age differences such that older adults used fewer positive words in narratives about positive pictures and more linguistically distant words evidenced by fewer present-tense verbs, relative to narratives about low-arousal pictures. Contrary to an explanation of enhanced regulation or control over emotions in older adulthood, these findings support the idea that older adults' speech production is influenced by their reactivity or affective response to emotional stimuli even when the task is not to communicate one's emotions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Within the sound of trouble: Do humans use pitch to correctly assess emotional arousal across species?

Comments on an article by Jay W. Schwartz , Kayleigh H. Pierson, and Alexander K. Reece (see record 2024-19488-001). In this issue, Schwartz et al. (2024) tackle the pitch rule in humans by testing to what extent we use pitch alone to judge emotional arousal across closely and distantly related animal species. The findings of Schwartz et al. open a number of intriguing possibilities for future research: Notably important additional steps would include to further investigate the accuracy of the pitch rule across closely and distantly related species. Upon this, in order to study the evolutionary ancestry of the pitch rule, it will be necessary to study its applicability across nonhumans. Particularly interesting would be the inclusion of subject species that have been found to eavesdrop on heterospecific alarm calls. Previous research (see Hoeschele, 2017 for a review) as well as present findings on human ratings of macaque versus cricket calls also suggest that we should additionally focus on sound features that compliment emotional arousal rating beyond pitch such as spectral information. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Ingestible pills reveal gastric correlates of emotions.

Although it is generally held that gastrointestinal (GI) signals are related to emotions, direct evidence for such a link is currently lacking. One of the reasons why the internal milieu of the GI system is poorly investigated is because visceral organs are difficult to access and monitor. To directly measure the influence of endoluminal markers of GI activity on the emotional experience, we asked a group of healthy male participants to ingest a pill that measured pH, pressure, and temperature of their GI tract while they watched video clips that consistently induced disgust, fear, happiness, sadness, or a control neutral state. In addition to the objective physiological markers of GI activity, subjective ratings of perceived emotions and visceral (i.e. gastric, respiratory and cardiac) sensations were recorded, as well as changes in heart rate (HR), heart rate variability (HRV) and spontaneous eyes blinks as non-gastric behavioral and autonomic markers of the emotional experience. We found that when participants observed fearful and disgusting video clips, they reported to perceive not only cardiac and respiratory sensations but also gastric sensations, such as nausea. Moreover, we found that there was a clear relation between the physiology of the stomach and the perceived emotions. Specifically, when disgusting video clips were displayed, the more acidic the pH, the more participants reported feelings of disgust and fear; the less acidic the pH, the more they reported happiness. Complementing the results found in the deep gastric realm, we found that disgusting stimuli induced a significant increase in HRV compared to the neutral scenarios, and together with fearful video clips a decrease in HR. Our findings suggest that gastric signals contribute to unique emotional states and that ingestible pills may open new avenues for exploring the deep-body physiology of emotions.

Mother-infant interaction characteristics associate with infant falling reactivity and child peer problems at pre-school age.

This longitudinal study investigated the associations between mother-infant interaction characteristics at 9 months of age, maternal mental health, infant temperament in the first year postpartum, and child behaviour at 3 years of age. The infants (N = 54, 22 females) mainly had White British ethnic backgrounds (85.7%). Results showed that i) mother-infant dyadic affective mutuality positively correlated with infant falling reactivity, suggesting that better infant regulatory skills are associated with the dyad's ability to share and understand each other's emotions; and ii) maternal respect for infant autonomy predicted fewer child peer problems at 3 years of age, suggesting that maternal respect for the validity of the infant's individuality promotes better social and emotional development in early childhood.

Recognition of facial expressions of emotions in tactile drawings by blind children, children with low vision and sighted children.

In the context of blindness, studies on the recognition of facial expressions of emotions by touch are essential to define the compensatory touch abilities and to create adapted tools on emotions. This study is the first to examine the effect of visual experience in the recognition of tactile drawings of facial expressions of emotions by children with different visual experiences. To this end, we compared the recognition rates of tactile drawings of emotions between blind children, children with low vision and sighted children aged 6-12 years. Results revealed no effect of visual experience on recognition rates. However, an effect of emotions and an interaction effect between emotions and visual experience were found. Indeed, while all children had a low average recognition rate, the drawings of fear, anger and disgust were particularly poorly recognized. Moreover, sighted children were significantly better at recognizing the drawings of surprise and sadness than the blind children who only showed high recognition rates for joy. The results of this study support the importance of developing emotion tools that can be understood by children with different visual experiences.

TPRO-NET: an EEG-based emotion recognition method reflecting subtle changes in emotion.

Emotion recognition based on Electroencephalogram (EEG) has been applied in various fields, including human-computer interaction and healthcare. However, for the popular Valence-Arousal-Dominance emotion model, researchers often classify the dimensions into high and low categories, which cannot reflect subtle changes in emotion. Furthermore, there are issues with the design of EEG features and the efficiency of transformer. To address these issues, we have designed TPRO-NET, a neural network that takes differential entropy and enhanced differential entropy features as input and outputs emotion categories through convolutional layers and improved transformer encoders. For our experiments, we categorized the emotions in the DEAP dataset into 8 classes and those in the DREAMER dataset into 5 classes. On the DEAP and the DREAMER datasets, TPRO-NET achieved average accuracy rates of 97.63%/97.47%/97.88% and 98.18%/98.37%/98.40%, respectively, on the Valence/Arousal/Dominance dimension for the subject-dependent experiments. Compared to other advanced methods, TPRO-NET demonstrates superior performance.

Expressive suppression mediates the relationship between sleep quality and generalized anxiety symptomology.

Anxiety disorders are the most prevalent worldwide mental health disorder, resulting in high societal costs. Emotion regulation and sleep quality are associated with the development of psychopathologies including anxiety. However, it is unknown whether habitual emotion regulation strategy use can mediate the influence of sleep quality on anxiety symptomology. An opportunity sample in a healthy population completed the Pittsburgh Sleep Quality Index to provide a measure of sleep quality, the Emotion Regulation Questionnaire to assess habitual use of emotion regulation strategies, and the Generalized Anxiety Disorder Scale to record anxiety symptomology. Data were analysed using correlation and regression-based mediation analyses. Improved sleep quality was predictive of reduced habitual use of expressive suppression and reduced anxiety symptomology. Additionally, increased use of expressive suppression was predictive of greater anxiety symptomology. Cognitive reappraisal was not associated with sleep quality or anxiety severity. Further, novel findings using mediation analyses show that expressive suppression partially mediated the relationship between sleep quality and anxiety. Whilst longitudinal and experimental research are needed to establish causality, these findings suggest that simultaneously targeting improvements in sleep quality and the use of specific emotion regulation strategies, including expressive suppression, may improve the efficacy of interventions focussed on reducing anxiety-related symptomology.

How breaking a sweat affects mood: The mediating role of self-efficacy between physical exercise and emotion regulation ability.

OBJECTIVE: This study investigates the association between physical exercise and emotion regulation abilities among college students, introducing self-efficacy as a mediating variable to analyze the pathway mechanism through which physical exercise affects emotion regulation abilities. METHODS: A cross-sectional study design was employed, utilizing a stratified random sampling method to survey three colleges in Jiangsu Province, China. Physical Activity Rating Scale, Physical Activity Self-efficacy Scale, and Emotional Intelligence Scale were used to measure the college student population. Regression analysis and mediation tests assessed whether self-efficacy mediates the relationship between physical exercise and college students' emotion regulation abilities. A total of 5,430 valid questionnaires were collected. RESULTS: The distribution of college students' physical activities was 77.0% for low, 13.1% for medium, and 9.3% for high levels. Physical activities were significantly and positively correlated with self-efficacy and emotional management abilities (r = 0.298,0.105;P<0.01), and self-efficacy was significantly and positively correlated with emotional management abilities (r = 0.322, P<0.01). Situational motivation and subjective support under self-efficacy were 0.08 and 0.255, respectively, and the adjusted R2 was 0.107. Self-efficacy played a fully mediating role between physical activities and emotional management abilities, with a total effect value of 0.032. The values of the direct and indirect effects were 0.003 and 0.029, accounting for 8.95% and 90.74% of the total effect, respectively. CONCLUSION: The physical exercise behavior of college students is primarily characterized by low intensity. Physical exercise among college students can positively predict their ability to regulate emotions. Self-efficacy fully mediates the relationship between physical exercise and emotion regulation ability among college students. College students can indirectly influence their ability to regulate emotions through physical exercise and self-efficacy.

Beyond faces: the contribution of the amygdala to visual processing in the macaque brain.

The amygdala is present in a diverse range of vertebrate species, such as lizards, rodents, and primates; however, its structure and connectivity differs across species. The increased connections to visual sensory areas in primate species suggests that understanding the visual selectivity of the amygdala in detail is critical to revealing the principles underlying its function in primate cognition. Therefore, we designed a high-resolution, contrast-agent enhanced, event-related fMRI experiment, and scanned 3 adult rhesus macaques, while they viewed 96 naturalistic stimuli. Half of these stimuli were social (defined by the presence of a conspecific), the other half were nonsocial. We also nested manipulations of emotional valence (positive, neutral, and negative) and visual category (faces, nonfaces, animate, and inanimate) within the stimulus set. The results reveal widespread effects of emotional valence, with the amygdala responding more on average to inanimate objects and animals than faces, bodies, or social agents in this experimental context. These findings suggest that the amygdala makes a contribution to primate vision that goes beyond an auxiliary role in face or social perception. Furthermore, the results highlight the importance of stimulus selection and experimental design when probing the function of the amygdala and other visually responsive brain regions.

Grip strength, working memory, and emotion perception in middle-aged males.

This study examined the association between grip strength and emotional working memory in middle-aged adults. Seventy-six males aged 40-60years (mean=48.5years, SD=5.4) participated in this cross-sectional study. They completed a muscular fitness assessment using a maximum grip strength test and emotional n-back tasks under two emotion conditions (fearful and neutral facial pictures) and two working memory loads (1-back and 2-back tasks). Hierarchical regression analyses indicated that greater muscular fitness was associated with superior working memory performance in the fearful condition in both the 1-back and 2-back tasks, after controlling for confounders. Greater muscular fitness was also associated with superior working memory performance in the neutral condition when the working memory load was high (2-back task) but not low (1-back task). These findings suggest a positive association between muscular fitness and emotional working memory and highlight the importance of maintaining muscular fitness for physical and cognitive-emotional well-being in middle-aged adults.