Automated measurement of infant and mother Duchenne facial expressions in the Face-to-Face/Still-Face.

Although still-face effects are well-studied, little is known about the degree to which the Face-to-Face/Still-Face (FFSF) is associated with the production of intense affective displays. Duchenne smiling expresses more intense positive affect than non-Duchenne smiling, while Duchenne cry-faces express more intense negative affect than non-Duchenne cry-faces. Forty 4-month-old infants and their mothers completed the FFSF, and key affect-indexing facial Action Units (AUs) were coded by expert Facial Action Coding System coders for the first 30 s of each FFSF episode. Computer vision software, automated facial affect recognition (AFAR), identified AUs for the entire 2-min episodes. Expert coding and AFAR produced similar infant and mother Duchenne and non-Duchenne FFSF effects, highlighting the convergent validity of automated measurement. Substantive AFAR analyses indicated that both infant Duchenne and non-Duchenne smiling declined from the FF to the SF, but only Duchenne smiling increased from the SF to the RE. In similar fashion, the magnitude of mother Duchenne smiling changes over the FFSF were 2-4 times greater than non-Duchenne smiling changes. Duchenne expressions appear to be a sensitive index of intense infant and mother affective valence that are accessible to automated measurement and may be a target for future FFSF research.

Intensive Longitudinal Assessment of Adolescents to Predict Suicidal Thoughts and Behaviors.

OBJECTIVE: Suicide is a leading cause of death among adolescents. However, there are no clinical tools to detect proximal risk for suicide. METHOD: Participants included 13- to 18-year-old adolescents (N = 103) reporting a current depressive, anxiety, and/or substance use disorder who owned a smartphone; 62% reported current suicidal ideation, with 25% indicating a past-year attempt. At baseline, participants were administered clinical interviews to assess lifetime disorders and suicidal thoughts and behaviors (STBs). Self-reports assessing symptoms and suicide risk factors also were obtained. In addition, the Effortless Assessment of Risk States (EARS) app was installed on adolescent smartphones to acquire daily mood and weekly suicidal ideation severity during the 6-month follow-up period. Adolescents completed STB and psychiatric service use interviews at the 1-, 3-, and 6-month follow-up assessments. RESULTS: K-means clustering based on aggregates of weekly suicidal ideation scores resulted in a 3-group solution reflecting high-risk (n = 26), medium-risk (n = 47), and low-risk (n = 30) groups. Of the high-risk group, 58% reported suicidal events (ie, suicide attempts, psychiatric hospitalizations, emergency department visits, ideation severity requiring an intervention) during the 6-month follow-up period. For participants in the high-risk and medium-risk groups (n = 73), mood disturbances in the preceding 7 days predicted clinically significant ideation, with a 1-SD decrease in mood doubling participants' likelihood of reporting clinically significant ideation on a given week. CONCLUSION: Intensive longitudinal assessment through use of personal smartphones offers a feasible method to assess variability in adolescents' emotional experiences and suicide risk. Translating these tools into clinical practice may help to reduce the needless loss of life among adolescents.

Manipulating facial musculature with functional electrical stimulation as an intervention for major depressive disorder: a focused search of literature for a proposal.

BACKGROUND: Major Depressive Disorder (MDD) is associated with interoceptive deficits expressed throughout the body, particularly the facial musculature. According to the facial feedback hypothesis, afferent feedback from the facial muscles suffices to alter the emotional experience. Thus, manipulating the facial muscles could provide a new "mind-body" intervention for MDD. This article provides a conceptual overview of functional electrical stimulation (FES), a novel neuromodulation-based treatment modality that can be potentially used in the treatment of disorders of disrupted brain connectivity, such as MDD. METHODS: A focused literature search was performed for clinical studies of FES as a modulatory treatment for mood symptoms. The literature is reviewed in a narrative format, integrating theories of emotion, facial expression, and MDD. RESULTS: A rich body of literature on FES supports the notion that peripheral muscle manipulation in patients with stroke or spinal cord injury may enhance central neuroplasticity, restoring lost sensorimotor function. These neuroplastic effects suggest that FES may be a promising innovative intervention for psychiatric disorders of disrupted brain connectivity, such as MDD. Recent pilot data on repetitive FES applied to the facial muscles in healthy participants and patients with MDD show early promise, suggesting that FES may attenuate the negative interoceptive bias associated with MDD by enhancing positive facial feedback. Neurobiologically, the amygdala and nodes of the emotion-to-motor transformation loop may serve as potential neural targets for facial FES in MDD, as they integrate proprioceptive and interoceptive inputs from muscles of facial expression and fine-tune their motor output in line with socio-emotional context. CONCLUSIONS: Manipulating facial muscles may represent a mechanistically novel treatment strategy for MDD and other disorders of disrupted brain connectivity that is worthy of investigation in phase II/III trials.

Randomized trial of brief interpersonal psychotherapy and cognitive behavioral therapy for depression delivered both in-person and by telehealth.

BACKGROUND: Expert consensus guidelines recommend Cognitive Behavioral Therapy (CBT) and Interpersonal Psychotherapy (IPT), interventions that were historically delivered face-to-face, as first-line treatments for Major Depressive Disorder (MDD). Despite the ubiquity of telehealth following the COVID-19 pandemic, little is known about differential outcomes with CBT versus IPT delivered in-person (IP) or via telehealth (TH) or whether working alliance is affected. METHODS: Adults meeting DSM-5 criteria for MDD were randomly assigned to either 8 sessions of IPT or CBT (group). Mid-trial, COVID-19 forced a change of therapy delivery from IP to TH (study phase). We compared changes in Hamilton Rating Scale for Depression (HRSD-17) and Working Alliance Inventory (WAI) scores for individuals by group and phase: CBT-IP (n = 24), CBT-TH (n = 11), IPT-IP (n = 25) and IPT-TH (n = 17). RESULTS: HRSD-17 scores declined significantly from pre to post treatment (pre: M = 17.7, SD = 4.4 vs. post: M = 11.7, SD = 5.9; p < .001; d = 1.45) without significant group or phase effects. WAI scores did not differ by group or phase. Number of completed therapy sessions was greater for TH (M = 7.8, SD = 1.2) relative to IP (M = 7.2, SD = 1.6) (Mann-Whitney U = 387.50, z = -2.24, p = .025). LIMITATIONS: Participants were not randomly assigned to IP versus TH. Sample size is small. CONCLUSIONS: This study provides preliminary evidence supporting the efficacy of both brief IPT and CBT, delivered by either TH or IP, for depression. It showed that working alliance is preserved in TH, and delivery via TH may improve therapy adherence. Prospective, randomized controlled trials are needed to definitively test efficacy of brief IPT and CBT delivered via TH versus IP.

Interpretation of Depression Detection Models via Feature Selection Methods.

Given the prevalence of depression worldwide and its major impact on society, several studies employed artificial intelligence modelling to automatically detect and assess depression. However, interpretation of these models and cues are rarely discussed in detail in the AI community, but have received increased attention lately. In this study, we aim to analyse the commonly selected features using a proposed framework of several feature selection methods and their effect on the classification results, which will provide an interpretation of the depression detection model. The developed framework aggregates and selects the most promising features for modelling depression detection from 38 feature selection algorithms of different categories. Using three real-world depression datasets, 902 behavioural cues were extracted from speech behaviour, speech prosody, eye movement and head pose. To verify the generalisability of the proposed framework, we applied the entire process to depression datasets individually and when combined. The results from the proposed framework showed that speech behaviour features (e.g. pauses) are the most distinctive features of the depression detection model. From the speech prosody modality, the strongest feature groups were F0, HNR, formants, and MFCC, while for the eye activity modality they were left-right eye movement and gaze direction, and for the head modality it was yaw head movement. Modelling depression detection using the selected features (even though there are only 9 features) outperformed using all features in all the individual and combined datasets. Our feature selection framework did not only provide an interpretation of the model, but was also able to produce a higher accuracy of depression detection with a small number of features in varied datasets. This could help to reduce the processing time needed to extract features and creating the model.

Infant AFAR: Automated facial action recognition in infants.

Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.

Deep Brain Stimulation for Depression Informed by Intracranial Recordings.

The success of deep brain stimulation (DBS) for treating Parkinson's disease has led to its application to several other disorders, including treatment-resistant depression. Results with DBS for treatment-resistant depression have been heterogeneous, with inconsistencies largely driven by incomplete understanding of the brain networks regulating mood, especially on an individual basis. We report results from the first subject treated with DBS for treatment-resistant depression using an approach that incorporates intracranial recordings to personalize understanding of network behavior and its response to stimulation. These recordings enabled calculation of individually optimized DBS stimulation parameters using a novel inverse solution approach. In the ensuing double-blind, randomized phase incorporating these bespoke parameter sets, DBS led to remission of symptoms and dramatic improvement in quality of life. Results from this initial case demonstrate the feasibility of this personalized platform, which may be used to improve surgical neuromodulation for a vast array of neurologic and psychiatric disorders.

Toward Causal Understanding of Therapist-Client Relationships: A Study of Language Modality and Social Entrainment.

The relationship between a therapist and their client is one of the most critical determinants of successful therapy. The working alliance is a multifaceted concept capturing the collaborative aspect of the therapist-client relationship; a strong working alliance has been extensively linked to many positive therapeutic outcomes. Although therapy sessions are decidedly multimodal interactions, the language modality is of particular interest given its recognized relationship to similar dyadic concepts such as rapport, cooperation, and affiliation. Specifically, in this work we study language entrainment, which measures how much the therapist and client adapt toward each other's use of language over time. Despite the growing body of work in this area, however, relatively few studies examine causal relationships between human behavior and these relationship metrics: does an individual's perception of their partner affect how they speak, or does how they speak affect their perception? We explore these questions in this work through the use of structural equation modeling (SEM) techniques, which allow for both multilevel and temporal modeling of the relationship between the quality of the therapist-client working alliance and the participants' language entrainment. In our first experiment, we demonstrate that these techniques perform well in comparison to other common machine learning models, with the added benefits of interpretability and causal analysis. In our second analysis, we interpret the learned models to examine the relationship between working alliance and language entrainment and address our exploratory research questions. The results reveal that a therapist's language entrainment can have a significant impact on the client's perception of the working alliance, and that the client's language entrainment is a strong indicator of their perception of the working alliance. We discuss the implications of these results and consider several directions for future work in multimodality.

Long-term ecological assessment of intracranial electrophysiology synchronized to behavioral markers in obsessive-compulsive disorder.

Detection of neural signatures related to pathological behavioral states could enable adaptive deep brain stimulation (DBS), a potential strategy for improving efficacy of DBS for neurological and psychiatric disorders. This approach requires identifying neural biomarkers of relevant behavioral states, a task best performed in ecologically valid environments. Here, in human participants with obsessive-compulsive disorder (OCD) implanted with recording-capable DBS devices, we synchronized chronic ventral striatum local field potentials with relevant, disease-specific behaviors. We captured over 1,000 h of local field potentials in the clinic and at home during unstructured activity, as well as during DBS and exposure therapy. The wide range of symptom severity over which the data were captured allowed us to identify candidate neural biomarkers of OCD symptom intensity. This work demonstrates the feasibility and utility of capturing chronic intracranial electrophysiology during daily symptom fluctuations to enable neural biomarker identification, a prerequisite for future development of adaptive DBS for OCD and other psychiatric disorders.

Reconsidering the Duchenne Smile: Formalizing and Testing Hypotheses about Eye Constriction and Positive Emotion.

The common view of emotional expressions is that certain configurations of facial-muscle movements reliably reveal certain categories of emotion. The principal exemplar of this view is the Duchenne smile, a configuration of facial-muscle movements (i.e., smiling with eye constriction) that has been argued to reliably reveal genuine positive emotion. In this paper, we formalized a list of hypotheses that have been proposed regarding the Duchenne smile, briefly reviewed the literature weighing on these hypotheses, identified limitations and unanswered questions, and conducted two empirical studies to begin addressing these limitations and answering these questions. Both studies analyzed a database of 751 smiles observed while 136 participants completed experimental tasks designed to elicit amusement, embarrassment, fear, and physical pain. Study 1 focused on participants' self-reported positive emotion and Study 2 focused on how third-party observers would perceive videos of these smiles. Most of the hypotheses that have been proposed about the Duchenne smile were either contradicted by or only weakly supported by our data. Eye constriction did provide some information about experienced positive emotion, but this information was lacking in specificity, already provided by other smile characteristics, and highly dependent on context. Eye constriction provided more information about perceived positive emotion, including some unique information over other smile characteristics, but context was also important here as well. Overall, our results suggest that accurately inferring positive emotion from a smile requires more sophisticated methods than simply looking for the presence/absence (or even the intensity) of eye constriction.

A Novel Framework for Network-Targeted Neuropsychiatric Deep Brain Stimulation.

Deep brain stimulation (DBS) has emerged as a promising therapy for neuropsychiatric illnesses, including depression and obsessive-compulsive disorder, but has shown inconsistent results in prior clinical trials. We propose a shift away from the empirical paradigm for developing new DBS applications, traditionally based on testing brain targets with conventional stimulation paradigms. Instead, we propose a multimodal approach centered on an individualized intracranial investigation adapted from the epilepsy monitoring experience, which integrates comprehensive behavioral assessment, such as the Research Domain Criteria proposed by the National Institutes of Mental Health. In this paradigm-shifting approach, we combine readouts obtained from neurophysiology, behavioral assessments, and self-report during broad exploration of stimulation parameters and behavioral tasks to inform the selection of ideal DBS parameters. Such an approach not only provides a foundational understanding of dysfunctional circuits underlying symptom domains in neuropsychiatric conditions but also aims to identify generalizable principles that can ultimately enable individualization and optimization of therapy without intracranial monitoring.

A Person- and Time-Varying Vector Autoregressive Model to Capture Interactive Infant-Mother Head Movement Dynamics.

Head movement is an important but often overlooked component of emotion and social interaction. Examination of regularity and differences in head movements of infant-mother dyads over time and across dyads can shed light on whether and how mothers and infants alter their dynamics over the course of an interaction to adapt to each others. One way to study these emergent differences in dynamics is to allow parameters that govern the patterns of interactions to change over time, and according to person- and dyad-specific characteristics. Using two estimation approaches to implement variations of a vector-autoregressive model with time-varying coefficients, we investigated the dynamics of automatically-tracked head movements in mothers and infants during the Face-Face/Still-Face Procedure (SFP) with 24 infant-mother dyads. The first approach requires specification of a confirmatory model for the time-varying parameters as part of a state-space model, whereas the second approach handles the time-varying parameters in a semi-parametric ("mostly" model-free) fashion within a generalized additive modeling framework. Results suggested that infant-mother head movement dynamics varied in time both within and across episodes of the SFP, and varied based on infants' subsequently-assessed attachment security. Code for implementing the time-varying vector-autoregressive model using two R packages, dynr and mgcv, is provided.

Human-Guided Modality Informativeness for Affective States.

This paper studies the hypothesis that not all modalities are always needed to predict affective states. We explore this hypothesis in the context of recognizing three affective states that have shown a relation to a future onset of depression: positive, aggressive, and dysphoric. In particular, we investigate three important modalities for face-to-face conversations: vision, language, and acoustic modality. We first perform a human study to better understand which subset of modalities people find informative, when recognizing three affective states. As a second contribution, we explore how these human annotations can guide automatic affect recognition systems to be more interpretable while not degrading their predictive performance. Our studies show that humans can reliably annotate modality informativeness. Further, we observe that guided models significantly improve interpretability, i.e., they attend to modalities similarly to how humans rate the modality informativeness, while at the same time showing a slight increase in predictive performance.

Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions.

Critical obstacles in training classifiers to detect facial actions are the limited sizes of annotated video databases and the relatively low frequencies of occurrence of many actions. To address these problems, we propose an approach that makes use of facial expression generation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. To evaluate this approach, a deep neural network was trained on two separate datasets: One network was trained on video of synthesized facial expressions generated from FERA17; the other network was trained on unaltered video from the same database. Both networks used the same train and validation partitions and were tested on the test partition of actual video from FERA17. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.

Crossing Domains for AU Coding: Perspectives, Approaches, and Measures.

Facial action unit (AU) detectors have performed well when trained and tested within the same domain. How well do AU detectors transfer to domains in which they have not been trained? We review literature on cross-domain transfer and conduct experiments to address limitations of prior research. We evaluate generalizability in four publicly available databases. EB+ (an expanded version of BP4D+), Sayette GFT, DISFA and UNBC Shoulder Pain (SP). The databases differ in observational scenarios, context, participant diversity, range of head pose, video resolution, and AU base rates. In most cases performance decreased with change in domain, often to below the threshold needed for behavioral research. However, exceptions were noted. Deep and shallow approaches generally performed similarly and average results were slightly better for deep model compared to shallow one. Occlusion sensitivity maps revealed that local specificity was greater for AU detection within than cross domains. The findings suggest that more varied domains and deep learning approaches may be better suited for generalizability and suggest the need for more attention to characteristics that vary between domains. Until further improvement is realized, caution is warranted when applying AU classifiers from one domain to another.

Automated Measurement of Head Movement Synchrony during Dyadic Depression Severity Interviews.

With few exceptions, most research in automated assessment of depression has considered only the patient's behavior to the exclusion of the therapist's behavior. We investigated the interpersonal coordination (synchrony) of head movement during patient-therapist clinical interviews. Our sample consisted of patients diagnosed with major depressive disorder. They were recorded in clinical interviews (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. For each session, patient and therapist 3D head movement was tracked from 2D videos. Head angles in the horizontal (pitch) and vertical (yaw) axes were used to measure head movement. Interpersonal coordination of head movement between patients and therapists was measured using windowed cross-correlation. Patterns of coordination in head movement were investigated using the peak picking algorithm. Changes in head movement coordination over the course of treatment were measured using a hierarchical linear model (HLM). The results indicated a strong effect for patient-therapist head movement synchrony. Within-dyad variability in head movement coordination was found to be higher than between-dyad variability, meaning that differences over time in a dyad were higher as compared to the differences between dyads. Head movement synchrony did not change over the course of treatment with change in depression severity. To the best of our knowledge, this study is the first attempt to analyze the mutual influence of patient-therapist head movement in relation to depression severity.

Cross-domain AU Detection: Domains, Learning Approaches, and Measures.

Facial action unit (AU) detectors have performed well when trained and tested within the same domain. Do AU detectors transfer to new domains in which they have not been trained? To answer this question, we review literature on cross-domain transfer and conduct experiments to address limitations of prior research. We evaluate both deep and shallow approaches to AU detection (CNN and SVM, respectively) in two large, well-annotated, publicly available databases, Expanded BP4D+ and GFT. The databases differ in observational scenarios, participant characteristics, range of head pose, video resolution, and AU base rates. For both approaches and databases, performance decreased with change in domain, often to below the threshold needed for behavioral research. Decreases were not uniform, however. They were more pronounced for GFT than for Expanded BP4D+ and for shallow relative to deep learning. These findings suggest that more varied domains and deep learning approaches may be better suited for promoting generalizability. Until further improvement is realized, caution is warranted when applying AU classifiers from one domain to another.