Performance evaluation of automated scoring for the descriptive similarity response task.

We examined whether a machine-learning-based automated scoring system can mimic the human similarity task performance. We trained a bidirectional encoder representations from transformer-model based on the semantic similarity test (SST), which presented participants with a word pair and asked them to write about how the two concepts were similar. In Experiment 1, based on the fivefold cross validation, we showed the model trained on the combination of the responses (N = 1600) and classification criteria (which is the rubric of the SST; N = 616) scored the correct labels with 83% accuracy. In Experiment 2, using the test data obtained from different participants in different timing from Experiment 1, we showed the models trained on the responses alone and the combination of responses and classification criteria scored the correct labels in 80% accuracy. In addition, human-model scoring showed inter-rater reliability of 0.63, which was almost the same as that of human-human scoring (0.67 to 0.72). These results suggest that the machine learning model can reach human-level performance in scoring the Japanese version of the SST.

Emotional characteristic analysis of human gait while real-time movie viewing.

Emotion recognition is useful in many applications such as preventing crime or improving customer satisfaction. Most of current methods are performed using facial features, which require close-up face information. Such information is difficult to capture with normal security cameras. The advantage of using gait and posture over conventional biometrics such as facial features is that gaits and postures can be obtained unobtrusively from faraway, even in a noisy environment. This study aims to investigate and analyze the relationship between human emotions and their gaits or postures. We collected a dataset made from the input of 49 participants for our experiments. Subjects were instructed to walk naturally in a circular walking path, while watching emotion-inducing videos on Microsoft HoloLens 2 smart glasses. An OptiTrack motion-capturing system was used for recording the gaits and postures of participants. The angles between body parts and walking straightness were calculated as features for comparison of body-part movements while walking under different emotions. Results of statistical analyses show that the subjects' arm swings are significantly different among emotions. And the arm swings on one side of the body could reveal subjects' emotions more obviously than those on the other side. Our results suggest that the arm movements together with information of arm side and walking straightness can reveal the subjects' current emotions while walking. That is, emotions of humans are unconsciously expressed by their arm swings, especially by the left arm, when they are walking in a non-straight walking path. We found that arm swings in happy emotion are larger than arm swings in sad emotion. To the best of our knowledge, this study is the first to perform emotion induction by showing emotion-inducing videos to the participants using smart glasses during walking instead of showing videos before walking. This induction method is expected to be more consistent and more realistic than conventional methods. Our study will be useful for implementation of emotion recognition applications in real-world scenarios, since our emotion induction method and the walking direction we used are designed to mimic the real-time emotions of humans as they walk in a non-straight walking direction.

A test of indirect grounding of abstract concepts using multimodal distributional semantics.

How are abstract concepts grounded in perceptual experiences for shaping human conceptual knowledge? Recent studies on abstract concepts emphasizing the role of language have argued that abstract concepts are grounded indirectly in perceptual experiences and language (or words) functions as a bridge between abstract concepts and perceptual experiences. However, this "indirect grounding" view remains largely speculative and has hardly been supported directly by empirical evidence. In this paper, therefore, we test the indirect grounding view by means of multimodal distributional semantics, in which the meaning of a word (i.e., a concept) is represented as the combination of textual and visual vectors. The newly devised multimodal distributional semantic model incorporates the indirect grounding view by computing the visual vector of an abstract word through the visual vectors of concrete words semantically related to that abstract word. An evaluation experiment is conducted in which conceptual representation is predicted from multimodal vectors using a multilayer feed-forward neural network. The analysis of prediction performance demonstrates that the indirect grounding model achieves significantly better performance in predicting human conceptual representation of abstract words than other models that mimic competing views on abstract concepts, especially than the direct grounding model in which the visual vectors of abstract words are computed directly from the images of abstract concepts. This result lends some plausibility to the indirect grounding view as a cognitive mechanism of grounding abstract concepts.

Context-prosody interaction in sarcasm comprehension: A functional magnetic resonance imaging study.

During conversation, sarcasm is perceived as an incongruity between the context, content, and prosody of the utterance. We hypothesized that prosody modifies the context‒content incongruity effect. Thus, we conducted a functional magnetic resonance imaging study with an auditory sarcasm detection task in 22 healthy adult participants. The participants listened to a short conversation according to which they had done either a good or bad deed, about which their conversational partner made a positive comment. When the context was positive (congruent with the content of utterance), positive prosody lessened the sarcasm rating, whereas negative prosody enhanced this rating. When the context was negative, the positive prosody effect disappeared, while negative prosody increased the sarcasm rating. Thus, context‒content incongruity is the primary determinant of sarcasm comprehension; and is modified by prosody in a context-dependent manner. Neuroimaging results showed that the context‒content incongruity effect was notable in the cerebellum and the mentalizing network, representing what was uttered in a particular context. The content‒prosody incongruity effect was observed in the bilateral amygdala, representing the manner of utterance. The interaction between these incongruity effects was found in the bilateral dorsolateral prefrontal cortex, extending to the inferior frontal gyrus and the salience network, including the anterior insular cortex and the caudal part of the dorso-medial prefrontal cortex. These findings indicate that two distinct incongruity detection systems for sarcasm comprehension are integrated in the prefrontal cortices through the salience network.

Exploring What Is Encoded in Distributional Word Vectors: A Neurobiologically Motivated Analysis.

The pervasive use of distributional semantic models or word embeddings for both cognitive modeling and practical application is because of their remarkable ability to represent the meanings of words. However, relatively little effort has been made to explore what types of information are encoded in distributional word vectors. Knowing the internal knowledge embedded in word vectors is important for cognitive modeling using distributional semantic models. Therefore, in this paper, we attempt to identify the knowledge encoded in word vectors by conducting a computational experiment using Binder et al.'s (2016) featural conceptual representations based on neurobiologically motivated attributes. In an experiment, these conceptual vectors are predicted from text-based word vectors using a neural network and linear transformation, and prediction performance is compared among various types of information. The analysis demonstrates that abstract information is generally predicted more accurately by word vectors than perceptual and spatiotemporal information, and specifically, the prediction accuracy of cognitive and social information is higher. Emotional information is also found to be successfully predicted for abstract words. These results indicate that language can be a major source of knowledge about abstract attributes, and they support the recent view that emphasizes the importance of language for abstract concepts. Furthermore, we show that word vectors can capture some types of perceptual and spatiotemporal information about concrete concepts and some relevant word categories. This suggests that language statistics can encode more perceptual knowledge than often expected.

The role of the amygdala in incongruity resolution: the case of humor comprehension.

A dominant theory of humor comprehension suggests that people understand humor by first perceiving some incongruity in an expression and then resolving it. This is called "the incongruity-resolution theory." Experimental studies have investigated the neural basis of humor comprehension, and multiple neural substrates have been proposed; however, the specific substrate for incongruity resolution is still unknown. The reason may be that the resolution phase, despite its importance in humor comprehension, has not been successfully distinguished from the perception phase because both phases occur almost simultaneously. To reveal the substrate, we conducted a functional magnetic resonance study using 51 healthy participants. We used a humor-producing frame of "Given A, I'd say B, because C" so as to focus on the resolution phase independently by suspending humor processing just after the perception phase. This frame allowed us to separate the two phases. Based on our results, incongruity resolution evoked positive emotion and activated the left amygdala, which is known to be related to positive emotion. On the basis of these findings, we argue that the amygdala plays an important role in humor comprehension, considering its functional role in emotional evaluation, particularly the relevance detection for incoming stimuli.

The role of prosody and context in sarcasm comprehension: Behavioral and fMRI evidence.

A hearer's perception of an utterance as sarcastic depends on integration of the heard statement, the discourse context, and the prosody of the utterance, as well as evaluation of the incongruity among these aspects. The effect of prosody in sarcasm comprehension is evident in everyday conversation, but little is known about its underlying mechanism or neural substrates. To elucidate the neural underpinnings of sarcasm comprehension in the auditory modality, we conducted a functional MRI experiment with 21 adult participants. The participants were provided with a short vignette in which a child had done either a good or bad deed, about which a parent made a positive comment. The participants were required to judge the degree of the sarcasm in the parent's positive comment (praise), which was accompanied by either positive or negative affective prosody. The behavioral data revealed that an incongruent combination of utterance and the context (i.e., the parent's positive comment on a bad deed by the child) induced perception of sarcasm. There was a significant interaction between context and prosody: sarcasm perception was enhanced when positive prosody was used in the context of a bad deed or, vice versa, when negative prosody was used in the context of a good deed. The corresponding interaction effect was observed in the rostro-ventral portion of the left inferior frontal gyrus corresponding to Brodmann's Area (BA) 47. Negative prosody incongruent with a positive utterance (praise) activated the bilateral insula extending to the right inferior frontal gyrus, anterior cingulate cortex, and brainstem. Our findings provide evidence that the left inferior frontal gyrus, particularly BA 47, is involved in integration of discourse context and utterance with affective prosody in the comprehension of sarcasm.

A Complex Network Approach to Distributional Semantic Models.

A number of studies on network analysis have focused on language networks based on free word association, which reflects human lexical knowledge, and have demonstrated the small-world and scale-free properties in the word association network. Nevertheless, there have been very few attempts at applying network analysis to distributional semantic models, despite the fact that these models have been studied extensively as computational or cognitive models of human lexical knowledge. In this paper, we analyze three network properties, namely, small-world, scale-free, and hierarchical properties, of semantic networks created by distributional semantic models. We demonstrate that the created networks generally exhibit the same properties as word association networks. In particular, we show that the distribution of the number of connections in these networks follows the truncated power law, which is also observed in an association network. This indicates that distributional semantic models can provide a plausible model of lexical knowledge. Additionally, the observed differences in the network properties of various implementations of distributional semantic models are consistently explained or predicted by considering the intrinsic semantic features of a word-context matrix and the functions of matrix weighting and smoothing. Furthermore, to simulate a semantic network with the observed network properties, we propose a new growing network model based on the model of Steyvers and Tenenbaum. The idea underlying the proposed model is that both preferential and random attachments are required to reflect different types of semantic relations in network growth process. We demonstrate that this model provides a better explanation of network behaviors generated by distributional semantic models.

Discourse Goals Affect the Process and Product of Nominal Metaphor Production.

Although a large number of studies have addressed metaphor comprehension, only a few attempts have so far been made at exploring the process of metaphor production. Therefore, in this paper, we address the problem of how people generate nominal metaphors or identify an apt vehicle for a given topic of nominal metaphors. Specifically, we examine how the process and product of metaphor production differ between two discourse goals of metaphor, namely an explanatory purpose (e.g., to clarify) and a literary purpose (e.g., to aesthetically pleasing). Experiment 1 analyzed the metaphors (or vehicles) generated in the metaphor production task, and demonstrated that people identified more prototypical exemplars of the property to be attributed to the topic as a vehicle for explanatory metaphors than for literary metaphors. In addition, it was found that metaphors generated for the explanatory purpose were more apt and conventional, and had high topic-vehicle similarity than those generated for the literary purpose, while metaphors generated for the literary purpose were more familiar and imageable than those for the explanatory purpose. Experiment 2 used a priming paradigm to assess the online availability of prototypical and less prototypical members of the topic property during metaphor production. The result was that both prototypical and less prototypical members were activated in producing literary metaphors, while neither members were activated in the production of explanatory metaphors. These findings indicate that the process of metaphor production is affected by discourse goals of metaphor; less prototypical members of the category are searched for a vehicle during the production of literary metaphors, and thus literary metaphors are generated with less prototypical vehicles than explanatory metaphors.

Adjective metaphors evoke negative meanings.

Previous metaphor studies have paid much attention to nominal metaphors and predicative metaphors, but little attention has been given to adjective metaphors. Although some studies have focused on adjective metaphors, they only examined differences in the acceptability of various types of adjective metaphors. This paper explores the cognitive effects evoked by adjective metaphors. Three psychological experiments revealed that (1) adjective metaphors, especially those modified by color adjectives, tend to evoke negative effect; (2) although the meanings of metaphors are basically affected by the meanings of their vehicles, when a vehicle has a neutral meaning, negative meanings are evoked most frequently for adjective metaphors compared to nominal and predicative metaphors; (3) negative meanings evoked by adjective metaphors are related to poeticness, and poetic metaphors evoke negative meanings more easily than less poetic metaphors. Our research sheds new light on studies of the use of metaphor, which is one of the most basic human cognitive abilities.

Computational exploration of metaphor comprehension processes using a semantic space model.

Recent metaphor research has revealed that metaphor comprehension involves both categorization and comparison processes. This finding has triggered the following central question: Which property determines the choice between these two processes for metaphor comprehension? Three competing views have been proposed to answer this question: the conventionality view (Bowdle & Gentner, 2005), aptness view (Glucksberg & Haught, 2006b), and interpretive diversity view (Utsumi, 2007); these views, respectively, argue that vehicle conventionality, metaphor aptness, and interpretive diversity determine the choice between the categorization and comparison processes. This article attempts to answer the question regarding which views are plausible by using cognitive modeling and computer simulation based on a semantic space model. In the simulation experiment, categorization and comparison processes are modeled in a semantic space constructed by latent semantic analysis. These two models receive word vectors for the constituent words of a metaphor and compute a vector for the metaphorical meaning. The resulting vectors can be evaluated according to the degree to which they mimic the human interpretation of the same metaphor; the maximum likelihood estimation determines which of the two models better explains the human interpretation. The result of the model selection is then predicted by three metaphor properties (i.e., vehicle conventionality, aptness, and interpretive diversity) to test the three views. The simulation experiment for Japanese metaphors demonstrates that both interpretive diversity and vehicle conventionality affect the choice between the two processes. On the other hand, it is found that metaphor aptness does not affect this choice. This result can be treated as computational evidence supporting the interpretive diversity and conventionality views.

Networked interaction therapy: relieving stress in memory-impaired people and their family members.

To relieve the stress in the lives of memory-impaired people and their family members, we propose the concept of Networked Interaction Therapy that connects them with community support group members via Internet by utilizing Internet communication, image understanding and sensory-interaction media technologies. To study their need and acceptance for our services, we made short video clips to illustrate our concept, and interviewed the care givers. Based on these studies, we also conducted the experiments with the low-fidelity mock-ups.