Monitoring Berlin Heart EXCOR by computer vision: A preliminary implementation and evaluation.

BACKGROUND: EXCOR Pediatric is one of the most commonly used ventricular assist devices (VAD) for small children; it requires visual inspection of the diaphragm movement to assess its operating status. Although this visual inspection can only be performed by trained medical professionals, it can also be attempted by the recent advances in computer vision technology. METHODS: Movement of the diaphragm in the operating EXCOR VAD was recorded as movies and annotated frame-by-frame in three classes according to the state of the diaphragm: "fill," "mid," and "empty." Three models, MobileNetV3, EfficientNetV2, and MobileViT, were trained using the frames, and their performance was evaluated based on the accuracy and area under the receiver operating characteristic curve (AROC). RESULTS: A total of 152 movies were available from two participants. Only the 10 mL pumps were used. Ninety-eight movies were used for annotation and frame extraction, and 7949 frames per class were included in the analysis. The macro-average accuracies of the three models were 0.88, 0.91, and 0.93, and the AROC were 0.99, 0.98, and 0.99 for MobileNetV3, EfficientNetV2, and MobileViT, respectively. CONCLUSION: Image recognition models based on lightweight deep neural networks could detect the diaphragm state of EXCOR VAD with sufficient accuracy, although there were limited variations in the dataset. This suggests the potential of computer vision for the automated monitoring of the EXCOR diaphragm position.

Development and validation of a facial expression database based on the dimensional and categorical model of emotions.

The present study describes the development and validation of a facial expression database comprising five different horizontal face angles in dynamic and static presentations. The database includes twelve expression types portrayed by eight Japanese models. This database was inspired by the dimensional and categorical model of emotions: surprise, fear, sadness, anger with open mouth, anger with closed mouth, disgust with open mouth, disgust with closed mouth, excitement, happiness, relaxation, sleepiness, and neutral (static only). The expressions were validated using emotion classification and Affect Grid rating tasks [Russell, Weiss, & Mendelsohn, 1989. Affect Grid: A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 57(3), 493-502]. The results indicate that most of the expressions were recognised as the intended emotions and could systematically represent affective valence and arousal. Furthermore, face angle and facial motion information influenced emotion classification and valence and arousal ratings. Our database will be available online at the following URL. https://www.dh.aist.go.jp/database/face2017/ .

Neural basis of decision making guided by emotional outcomes.

Emotional events resulting from a choice influence an individual's subsequent decision making. Although the relationship between emotion and decision making has been widely discussed, previous studies have mainly investigated decision outcomes that can easily be mapped to reward and punishment, including monetary gain/loss, gustatory stimuli, and pain. These studies regard emotion as a modulator of decision making that can be made rationally in the absence of emotions. In our daily lives, however, we often encounter various emotional events that affect decisions by themselves, and mapping the events to a reward or punishment is often not straightforward. In this study, we investigated the neural substrates of how such emotional decision outcomes affect subsequent decision making. By using functional magnetic resonance imaging (fMRI), we measured brain activities of humans during a stochastic decision-making task in which various emotional pictures were presented as decision outcomes. We found that pleasant pictures differentially activated the midbrain, fusiform gyrus, and parahippocampal gyrus, whereas unpleasant pictures differentially activated the ventral striatum, compared with neutral pictures. We assumed that the emotional decision outcomes affect the subsequent decision by updating the value of the options, a process modeled by reinforcement learning models, and that the brain regions representing the prediction error that drives the reinforcement learning are involved in guiding subsequent decisions. We found that some regions of the striatum and the insula were separately correlated with the prediction error for either pleasant pictures or unpleasant pictures, whereas the precuneus was correlated with prediction errors for both pleasant and unpleasant pictures.

Clinical significance and normal reference value of serum creatinine to cystatin C ratio in children.

BACKGROUND & AIMS: Sarcopenia is a serious problem in adults and children. However, limited modalities are available for diagnosing pediatric sarcopenia. The serum creatinine to cystatin C ratio (Cre/CysC ratio) is a promising method for muscle quantification, although its clinical significance in the pediatric population is unknown. This study aimed to evaluate the relationship between the Cre/CysC ratio and physical performance. METHODS: This was a single-center retrospective study. Patients aged <15 years who had visited the University of Tokyo Hospital for measurements of serum creatinine and cystatin C levels, body height, and body weight were included. The patients were assigned according to their age (<2 or ≥2 years), and the relationship between the Cre/CysC ratio and physical performance at the time of measurement was analyzed. RESULTS: We included 266 patients, revealing a significant relationship between Cre/CysC ratio and physical performance in children aged ≥2 years (p < 0.001) but not in children aged <2 years (p = 0.42). The repeater-operator curve analysis of Cre/CysC to predict bedridden status showed good performance (the area under the curve was 0.82 (95% CI, 0.75-0.89)) and the cut-off value 0.44 had good accuracy (sensitivity 0.87, specificity 0.61). CONCLUSIONS: The Cre/CysC ratio was a significant marker of impaired physical performance, and a Cre/CysC ratio <0.44 accurately predicted bedridden status in children aged >2 years.

Categorical and dimensional perceptions in decoding emotional facial expressions.

We investigated whether categorical perception and dimensional perception can co-occur while decoding emotional facial expressions. In Experiment 1, facial continua with endpoints consisting of four basic emotions (i.e., happiness-fear and anger-disgust) were created by a morphing technique. Participants rated each facial stimulus using a categorical strategy and a dimensional strategy. The results show that the happiness-fear continuum was divided into two clusters based on valence, even when using the dimensional strategy. Moreover, the faces were arrayed in order of the physical changes within each cluster. In Experiment 2, we found a category boundary within other continua (i.e., surprise-sadness and excitement-disgust) with regard to the arousal and valence dimensions. These findings indicate that categorical perception and dimensional perception co-occurred when emotional facial expressions were rated using a dimensional strategy, suggesting a hybrid theory of categorical and dimensional accounts.

[Recognition of dynamic facial expressions in peripheral and central vision].

The present study investigated the effects of dynamic information on the recognition of emotional facial expressions across the visual field (i.e., central or peripheral vision). Facial stimuli with three pleasant expressions (excited, happy, and relaxed) and three unpleasant expressions (fearful, angry, and sad) were selected on the basis of valence and activation. The facial stimuli were presented dynamically or statically at either the central or peripheral visual field. Participants evaluated the emotional state of the target facial expression using a forced-choice task (N=34) and an Affect Grid (Russell, Weiss, & Mendelsohn, 1989) (N =39) requiring categorical and dimensional judgments about facial expressions. The results of the forced-choice task showed that only dynamic angry faces in peripheral vision had better recognition than the equivalent faces in the static condition. The results of the Affect Grid indicated that only the pleasant expressions presented in the peripheral field were significantly rated as more strongly pleasant. These findings suggest that an effect of dynamic information is more salient in peripheral vision than in central vision for recognizing certain facial expressions.

Trait Respect Is Linked to Reduced Gray Matter Volume in the Anterior Temporal Lobe.

Respect is a positive other-oriented social emotion upon the recognition of excellence in others. We previously reported that respect-related brain activity in the left anterior temporal lobe (ATL). Since brain activity and structure are often involved in common cognitive functions, we investigated the morphological properties of the left ATL using voxel-based morphometry analysis. We found an association of trait respect with reduced gray matter volume (GMV) in part of the left ATL. Moreover, since the ATL is involved in general conceptual knowledge, we investigated the relationships between other social emotions with similar properties as respect and the GMV of the left ATL. We observed an association of reduced GMV with empathic concern, which is an other-oriented and affective aspect of trait empathy. Our findings indicated an association of the left ATL with other-oriented and affective aspect of social emotions.

Respect and admiration differentially activate the anterior temporal lobe.

Admiration and respect are positive social emotions often experienced when recognizing excellent behavior in another person. Although both strongly rely on appraisal of behavior, admiration focuses on the admirable behavior of a person, while respect focuses on the person as a whole. The evaluation and interpretation of the social behavior of another person are dependent on semantic memory. Social semantic knowledge is represented in the anterior temporal lobe (ATL), and ATL activity is modulated by conceptual details of semantic knowledge. As respect requires evaluation of not only excellent behavior but also of the person as a whole, we hypothesized that the ATL is differentially activated by admiration and respect. To test our hypothesis, we conducted functional magnetic resonance imaging experiments. We presented participants with vignettes describing admirable behavior of fictitious characters and asked them to imagine and report how they would normally feel when encountering the situation described in the vignettes, i.e., admiration or respect and its intensity. A part of the left ATL was more strongly modulated by the intensity of respect than of admiration. Although admiration and respect are often considered to be closely related, our results indicate that the neural substrates underlying these emotions are different.

Enhanced facial EMG activity in response to dynamic facial expressions.

The suggestion that dynamic facial expressions of emotion induce more evident facial mimicry than static ones remains controversial. We investigated this issue by recording EMG from the corrugator supercilii and zygomatic major. Dynamic and static facial expressions of anger and happiness were presented. Dynamic presentations of angry expressions induced stronger EMG activity from the corrugator supercilii than static presentations, while dynamic presentations of happy expressions induced stronger EMG activity from the zygomatic major compared to static presentations. These results indicate that dynamic facial expressions induce facial EMG activity interpretable as facial mimicry more evidently than static expressions.

Untrustworthiness inhibits congruent facial reactions to happy faces.

This study investigated how trustworthiness modulates facial reactions to emotional faces, specifically emotional mimicry. In order to manipulate the trustworthiness of others, we used the trust game. The participants (N=37) perceived the partners as trustworthy or untrustworthy based on whether the partners tended to behave fairly or unfairly in the game with a within-subjects design. Facial electromyograms were measured while viewing the emotional faces displayed by the partners. The results demonstrated that the zygomaticus major activity in response to happy faces was greater for the trustworthy partners than for the untrustworthy partners. The results suggest that emotional mimicry of happiness is more inhibited for those who are non-affiliative and unfavorable, such as the untrustworthy partners. On the other hand, there was no effect of trustworthiness on congruent facial reactions to sad and angry faces. However, the zygomaticus major activity for angry faces was smaller for the trustworthy partners than for the untrustworthy partners, suggesting that positive expressions to angry faces of the trustworthy partners are inhibited. Trustworthiness constructed by social exchange modulates not only emotional mimicry for happy faces but also incongruent facial reactions to angry faces.

Context sensitivity in the detection of changes in facial emotion.

In social contexts, reading subtle changes in others' facial expressions is a crucial communication skill. To measure this ability, we developed an expression-change detection task, wherein a series of pictures of changes in an individual's facial expressions within contextual scenes were presented. The results demonstrated that the detection of subtle changes was highly sensitive to contextual stimuli. That is, participants identified the direction of facial-expression changes more accurately and more quickly when they were 'appropriate'-consistent with the valence of the contextual stimulus change-than when they were 'inappropriate'. Moreover, individual differences in sensitivity to contextual stimuli were correlated with scores on the Toronto Alexithymia Scale, a commonly used measure of alexithymia tendencies. These results suggest that the current behavioural task might facilitate investigations of the role of context in human social cognition.

Modulation of Emotional Category Induced by Temporal Factors in Emotion Recognition.

Categorical perception (CP), the perceptual experience whereby continuous sensory phenomena are perceived as distinct and separate percepts, is one of the most characteristic features of information processing in human cognition. CP is considered as the result of the integration of the top-down processing including background knowledge and verbal labeling and the bottom-up processing such as physical characteristics of the sensory signal. However, the underlying mechanisms governing the integration remain unclear. To address this issue, we focused on the temporal characteristics of CP of facial expression. In the current study, we investigated the contributions of temporal factors in CP processes, using facial expression recognition tasks as an example of CP. Participants completed an identification task and a discrimination task, well-established tasks for evaluating CP of facial expressions, with variable temporal parameters, that is, duration of stimulus presentation and delay time (interval between stimulus and response). The results demonstrated that the emotionally ambiguous stimuli are categorized more distinctively with the extension of delay length, not of stimulus duration. In contrast, the category boundary for facial expressions shifted toward "happy" with extention in stimulus duration, not in delay length. This dissociation between the impact of stimulus duration and delay suggests that there are two processes contributing to CP of facial emotion; one process may reflect the internal processing associated with the length of the delay period including verbal labeling of the stimuli, and the other process may reflect the temporal summation of stimulus inputs, associated with stimulus duration. These findings suggest that paying more attention to temporal factors in CP could be useful for further study of the mechanisms underlying CP.

Individual differences in heart rate variability are associated with the avoidance of negative emotional events.

Although the emotional outcome of a choice generally affects subsequent decisions, humans can inhibit the influence of emotion. Heart rate variability (HRV) has emerged as an objective measure of individual differences in the capacity for inhibitory control. In the present study, we investigated how individual differences in HRV at rest are associated with the emotional effects of the outcome of a choice on subsequent decision making using a decision-making task in which emotional pictures appeared as decision outcomes. We used a reinforcement learning model to characterize the observed behaviors according to several parameters, namely, the learning rate and the motivational value of positive and negative pictures. Consequently, we found that individuals with a lower resting HRV exhibited a greater negative motivational value in response to negative pictures, suggesting that these individuals tend to avoid negative pictures compared with individuals with a higher resting HRV.

Event-related potentials elicited by pre-attentive emotional changes in temporal context.

The ability to detect emotional change in the environment is essential for adaptive behavior. The current study investigated whether event-related potentials (ERPs) can reflect emotional change in a visual sequence. To assess pre-attentive processing, we examined visual mismatch negativity (vMMN): the negative potentials elicited by a deviant (infrequent) stimulus embedded in a sequence of standard (frequent) stimuli. Participants in two experiments pre-attentively viewed visual sequences of Japanese kanji with different emotional connotations while ERPs were recorded. The visual sequence in Experiment 1 consisted of neutral standards and two types of emotional deviants with a strong and weak intensity. Although the results indicated that strongly emotional deviants elicited more occipital negativity than neutral standards, it was unclear whether these negativities were derived from emotional deviation in the sequence or from the emotional significance of the deviants themselves. In Experiment 2, the two identical emotional deviants were presented against different emotional standards. One type of deviants was emotionally incongruent with the standard and the other type of deviants was emotionally congruent with the standard. The results indicated that occipital negativities elicited by deviants resulted from perceptual changes in a visual sequence at a latency of 100-200 ms and from emotional changes at latencies of 200-260 ms. Contrary to the results of the ERP experiment, reaction times to deviants showed no effect of emotional context; negative stimuli were consistently detected more rapidly than were positive stimuli. Taken together, the results suggest that brain signals can reflect emotional change in a temporal context.

Contextual modulation of physiological and psychological responses triggered by emotional stimuli.

A series of emotional events successively occur in temporal context. The present study investigated how physiological and psychological responses are modulated by emotional context. Skin conductance response (SCR), heart rate, corrugator activity, zygomatic activity, and subjective feelings during emotional picture viewing were measured. To create an emotional context, a unpleasant or pleasant picture was preceded by three types of pictures, i.e., unpleasant, pleasant, and neutral pictures, resulting in six pairings. The results showed that viewing an unpleasant picture attenuated pleasant feelings induced by the following pleasant picture. On the other hand, preceding pleasant pictures decreased SCR to the following pictures. The effects of contextual modulation on emotional responses might be due to the informative function of pre-existing feelings; unpleasant feelings signal a threatening environment, whereas pleasant feelings signal a benign environment. With respect to facial muscle activities, viewing a pleasant picture decreased corrugator activity in response to the preceding picture. These findings suggest several types of contextual modulation effects on psychological, autonomic, and somatic responses to emotional stimuli.

Relationships among facial mimicry, emotional experience, and emotion recognition.

BACKGROUND: The relationships between facial mimicry and subsequent psychological processes remain unclear. We hypothesized that the congruent facial muscle activity would elicit emotional experiences and that the experienced emotion would induce emotion recognition. METHODOLOGY/PRINCIPAL FINDINGS: To test this hypothesis, we re-analyzed data collected in two previous studies. We recorded facial electromyography (EMG) from the corrugator supercilii and zygomatic major and obtained ratings on scales of valence and arousal for experienced emotions (Study 1) and for experienced and recognized emotions (Study 2) while participants viewed dynamic and static facial expressions of negative and positive emotions. Path analyses showed that the facial EMG activity consistently predicted the valence ratings for the emotions experienced in response to dynamic facial expressions. The experienced valence ratings in turn predicted the recognized valence ratings in Study 2. CONCLUSION: These results suggest that facial mimicry influences the sharing and recognition of emotional valence in response to others' dynamic facial expressions.

The implicit processing of categorical and dimensional strategies: an fMRI study of facial emotion perception.

OUR UNDERSTANDING OF FACIAL EMOTION PERCEPTION HAS BEEN DOMINATED BY TWO SEEMINGLY OPPOSING THEORIES: the categorical and dimensional theories. However, we have recently demonstrated that hybrid processing involving both categorical and dimensional perception can be induced in an implicit manner (Fujimura etal., 2012). The underlying neural mechanisms of this hybrid processing remain unknown. In this study, we tested the hypothesis that separate neural loci might intrinsically encode categorical and dimensional processing functions that serve as a basis for hybrid processing. We used functional magnetic resonance imaging to measure neural correlates while subjects passively viewed emotional faces and performed tasks that were unrelated to facial emotion processing. Activity in the right fusiform face area (FFA) increased in response to psychologically obvious emotions and decreased in response to ambiguous expressions, demonstrating the role of the FFA in categorical processing. The amygdala, insula and medial prefrontal cortex exhibited evidence of dimensional (linear) processing that correlated with physical changes in the emotional face stimuli. The occipital face area and superior temporal sulcus did not respond to these changes in the presented stimuli. Our results indicated that distinct neural loci process the physical and psychological aspects of facial emotion perception in a region-specific and implicit manner.

Decision-making based on emotional images.

The emotional outcome of a choice affects subsequent decision making. While the relationship between decision making and emotion has attracted attention, studies on emotion and decision making have been independently developed. In this study, we investigated how the emotional valence of pictures, which was stochastically contingent on participants' choices, influenced subsequent decision making. In contrast to traditional value-based decision-making studies that used money or food as a reward, the "reward value" of the decision outcome, which guided the update of value for each choice, is unknown beforehand. To estimate the reward value of emotional pictures from participants' choice data, we used reinforcement learning models that have successfully been used in previous studies for modeling value-based decision making. Consequently, we found that the estimated reward value was asymmetric between positive and negative pictures. The negative reward value of negative pictures (relative to neutral pictures) was larger in magnitude than the positive reward value of positive pictures. This asymmetry was not observed in valence for an individual picture, which was rated by the participants regarding the emotion experienced upon viewing it. These results suggest that there may be a difference between experienced emotion and the effect of the experienced emotion on subsequent behavior. Our experimental and computational paradigm provides a novel way for quantifying how and what aspects of emotional events affect human behavior. The present study is a first step toward relating a large amount of knowledge in emotion science and in taking computational approaches to value-based decision making.

Effects of dynamic information in recognising facial expressions on dimensional and categorical judgments.

We investigated the effects of dynamic information on decoding facial expressions. A dynamic face entailed a change from a neutral to a full-blown expression, whereas a static face included only the full-blown expression. Sixty-eight participants were divided into two groups, the dynamic condition and the static condition. The facial stimuli expressed eight kinds of emotions (excited, happy, calm, sleepy, sad, angry, fearful, and surprised) according to a dimensional perspective. Participants evaluated each facial stimulus using two methods, the Affect Grid (Russell et al, 1989 Personality and Social Psychology 29 497-510) and the forced-choice task, allowing for dimensional and categorical judgment interpretations. For activation ratings in dimensional judgments, the results indicated that dynamic calm faces, low-activation expressions were rated as less activated than static faces. For categorical judgments, dynamic excited, happy, and fearful faces, which are high- and middle-activation expressions, had higher ratings than did those under the static condition. These results suggest that the beneficial effect of dynamic information depends on the emotional properties of facial expressions.

Facial expression arousal level modulates facial mimicry.

We investigated the effect of facial expression arousal level and mode of presentation on facial mimicry. High- and low-arousal facial expressions indicating pleasant and unpleasant emotions were presented both statically and dynamically. Participants' facial electromyographic (EMG) reactions were recorded from the zygomatic major and corrugator supercilii muscles. Stronger zygomatic major muscle activity was evoked for high- compared to low-arousal pleasant expressions. Comparable activity was induced in the corrugator supercilii muscle in response to both high- and low-arousal unpleasant expressions, and this was true for both dynamic and static presentations. These results suggest that the arousal levels of pleasant, but not unpleasant, facial expressions can enhance facial mimicry.