Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Association Between Systemic Immune Inflammation Index and Diabetes Mellitus in the NHANES 2003-2018 Population.

Objects: This study aimed to explore the association between the Systemic Immune-Inflammation Index (SII) and diabetes mellitus (DM) and to assess its influence on the prognosis of the DM and no-DM groups. Methods: The study used data from the National Health and Nutrition Examination Survey; 9643 participants were included. Logistic regression analysis was employed to evaluate connections between SII and DM. We used the Cox proportional hazards model, restricted cubic spline, and Kaplan-Meier curve to analyze the relationship between SII and mortality. Results: The logistic regression analysis indicated that a significant increase in the likelihood of developing DM with higher SII levels (odds ratio, 1.31; 95% CI, 1.09-1.57, P = .003). The Cox model showed that there is a positive association between increased SII and higher all-cause mortality. The hazard ratios for SII were 1.53 (1.31, 1.78), 1.61 (1.31, 1.98), and 1.41 (1.12, 1.78) in the total, DM and non-DM groups, respectively. We observed a linear correlation between SII and all-cause mortality in DM participants, whereas non-DM participants and the total population showed a nonlinear correlation. Conclusion: Elevated SII levels are linked to an augmented risk of DM. Those with DM and higher SII levels demonstrated an elevated risk of mortality.

Individual differences and motor planning influence self-recognition of actions.

Although humans can recognize their body movements in point-light displays, self-recognition ability varies substantially across action types and participants. Are these variations primarily due to an awareness of visually distinct movement patterns, or to underlying factors related to motoric planning and/or individual differences? To address this question, we conducted a large-scale study in self-action recognition (N = 101). We motion captured whole-body movements of participants who performed 27 different actions across action goals and degree of motor planning. After a long delay period (~ 1 month), participants were tested in a self-recognition task: identifying their point-light action amongst three other point-light actors performing identical actions. We report a self-advantage effect from point-light actions, consistent with prior work in self-action recognition. Further, we found that self-recognition was modulated by the action complexity (associated with the degree of motor planning in performed actions) and individual differences linked to motor imagery and subclinical autism and schizotypy. Using dynamic time warping, we found sparse evidence in support of visual distinctiveness as a primary contributor to self-recognition, though speed distinctiveness negatively influenced self-recognition performance. Together, our results reveal that self-action recognition involves more than an awareness of visually distinct movements, with important implications for how the motor system may be involved.

Cognitive complexity explains processing asymmetry in judgments of similarity versus difference.

Human judgments of similarity and difference are sometimes asymmetrical, with the former being more sensitive than the latter to relational overlap, but the theoretical basis for this asymmetry remains unclear. We test an explanation based on the type of information used to make these judgments (relations versus features) and the comparison process itself (similarity versus difference). We propose that asymmetries arise from two aspects of cognitive complexity that impact judgments of similarity and difference: processing relations between entities is more cognitively demanding than processing features of individual entities, and comparisons assessing difference are more cognitively complex than those assessing similarity. In Experiment 1 we tested this hypothesis for both verbal comparisons between word pairs, and visual comparisons between sets of geometric shapes. Participants were asked to select one of two options that was either more similar to or more different from a standard. On unambiguous trials, one option was unambiguously more similar to the standard; on ambiguous trials, one option was more featurally similar to the standard, whereas the other was more relationally similar. Given the higher cognitive complexity of processing relations and of assessing difference, we predicted that detecting relational difference would be particularly demanding. We found that participants (1) had more difficulty detecting relational difference than they did relational similarity on unambiguous trials, and (2) tended to emphasize relational information more when judging similarity than when judging difference on ambiguous trials. The latter finding was replicated using more complex story stimuli (Experiment 2). We showed that this pattern can be captured by a computational model of comparison that weights relational information more heavily for similarity than for difference judgments.

Reproductive factors and risk of cardiovascular diseases and all-cause and cardiovascular mortality in American women: NHANES 2003-2018.

BACKGROUND: The evidence regarding the association of reproductive factors with cardiovascular diseases (CVDs) is limited. AIMS: To investigate the relationship of reproductive factors with the risk of CVDs, as well as all-cause and cardiovascular mortality. METHODS: This study included 16,404 adults with reproductive factors from the National Health and Nutrition Examination Survey (NHANES) and followed up until 31 December 2019. Logistic models and restricted cubic spline models were used to assess the association of reproductive factors with CVDs. COX proportional hazards models and restricted cubic spline models, with adjustment for potential confounding, were employed to analyze the relation between reproductive factors and cardiovascular and all-cause death. RESULTS: There is a nonlinear relationship between age at menarche and CVDs. Age at menopause ≤ 11(OR 1.36, 95% CI 1.10-1.69) was associated with an increased risk of CVDs compared to ages 12-13 years. Age at Menopause ≤ 44 (OR 1.69, 95% CI 1.40-2.03) was associated with increased CVDs compared to age 35-49 years. Number of pregnancies ≥ 5(OR 1.26, 95% CI 1.02-1.55) was associated with an increased risk of CVDs compared to one pregnancy. In continuous variable COX regression models, a later age at menopause (HR 0.98, 95% CI 0.97-0.99) and a longer reproductive lifespan (HR 0.98, 95% CI 0.97-0.99) were associated with a decreased risk of all-cause death. A later age at menopause (HR 0.98, 95% CI 0.97-0.99) and a longer reproductive lifespan (HR 0.98, 95% CI 0.97-0.99) were associated with a decreased risk of cardiac death. CONCLUSIONS: Female reproductive factors are significant risk factors for CVDs American women.

Patterns of saliency and semantic features distinguish gaze of expert and novice viewers of surveillance footage.

When viewing the actions of others, we not only see patterns of body movements, but we also "see" the intentions and social relations of people. Experienced forensic examiners - Closed Circuit Television (CCTV) operators - have been shown to convey superior performance in identifying and predicting hostile intentions from surveillance footage than novices. However, it remains largely unknown what visual content CCTV operators actively attend to, and whether CCTV operators develop different strategies for active information seeking from what novices do. Here, we conducted computational analysis for the gaze-centered stimuli captured by experienced CCTV operators and novices' eye movements when viewing the same surveillance footage. Low-level image features were extracted by a visual saliency model, whereas object-level semantic features were extracted by a deep convolutional neural network (DCNN), AlexNet, from gaze-centered regions. We found that the looking behavior of CCTV operators differs from novices by actively attending to visual contents with different patterns of saliency and semantic features. Expertise in selectively utilizing informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions.

For deep networks, the whole equals the sum of the parts.

Deep convolutional networks exceed humans in sensitivity to local image properties, but unlike biological vision systems, do not discover and encode abstract relations that capture important properties of objects and events in the world. Coupling network architectures with additional machinery for encoding abstract relations will make deep networks better models of human abilities and more versatile and capable artificial devices.

Two Computational Approaches to Visual Analogy: Task-Specific Models Versus Domain-General Mapping.

Advances in artificial intelligence have raised a basic question about human intelligence: Is human reasoning best emulated by applying task-specific knowledge acquired from a wealth of prior experience, or is it based on the domain-general manipulation and comparison of mental representations? We address this question for the case of visual analogical reasoning. Using realistic images of familiar three-dimensional objects (cars and their parts), we systematically manipulated viewpoints, part relations, and entity properties in visual analogy problems. We compared human performance to that of two recent deep learning models (Siamese Network and Relation Network) that were directly trained to solve these problems and to apply their task-specific knowledge to analogical reasoning. We also developed a new model using part-based comparison (PCM) by applying a domain-general mapping procedure to learned representations of cars and their component parts. Across four-term analogies (Experiment 1) and open-ended analogies (Experiment 2), the domain-general PCM model, but not the task-specific deep learning models, generated performance similar in key aspects to that of human reasoners. These findings provide evidence that human-like analogical reasoning is unlikely to be achieved by applying deep learning with big data to a specific type of analogy problem. Rather, humans do (and machines might) achieve analogical reasoning by learning representations that encode structural information useful for multiple tasks, coupled with efficient computation of relational similarity.

Zero-shot visual reasoning through probabilistic analogical mapping.

Human reasoning is grounded in an ability to identify highly abstract commonalities governing superficially dissimilar visual inputs. Recent efforts to develop algorithms with this capacity have largely focused on approaches that require extensive direct training on visual reasoning tasks, and yield limited generalization to problems with novel content. In contrast, a long tradition of research in cognitive science has focused on elucidating the computational principles underlying human analogical reasoning; however, this work has generally relied on manually constructed representations. Here we present visiPAM (visual Probabilistic Analogical Mapping), a model of visual reasoning that synthesizes these two approaches. VisiPAM employs learned representations derived directly from naturalistic visual inputs, coupled with a similarity-based mapping operation derived from cognitive theories of human reasoning. We show that without any direct training, visiPAM outperforms a state-of-the-art deep learning model on an analogical mapping task. In addition, visiPAM closely matches the pattern of human performance on a novel task involving mapping of 3D objects across disparate categories.

Aesthetic preferences for prototypical movements in human actions.

A commonplace sight is seeing other people walk. Our visual system specializes in processing such actions. Notably, we are not only quick to recognize actions, but also quick to judge how elegantly (or not) people walk. What movements appear appealing, and why do we have such aesthetic experiences? Do aesthetic preferences for body movements arise simply from perceiving others' positive emotions? To answer these questions, we showed observers different point-light walkers who expressed neutral, happy, angry, or sad emotions through their movements and measured the observers' impressions of aesthetic appeal, emotion positivity, and naturalness of these movements. Three experiments were conducted. People showed consensus in aesthetic impressions even after controlling for emotion positivity, finding prototypical walks more aesthetically pleasing than atypical walks. This aesthetic prototype effect could be accounted for by a computational model in which walking actions are treated as a single category (as opposed to multiple emotion categories). The aesthetic impressions were affected both directly by the objective prototypicality of the movements, and indirectly through the mediation of perceived naturalness. These findings extend the boundary of category learning, and hint at possible functions for action aesthetics.

Emergent analogical reasoning in large language models.

The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of Generative Pre-trained Transformer (GPT)-3) on a range of analogical tasks, including a non-visual matrix reasoning task based on the rule structure of Raven's Standard Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings; preliminary tests of GPT-4 indicated even better performance. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.

Relational and lexical similarity in analogical reasoning and recognition memory: Behavioral evidence and computational evaluation.

We examined the role of different types of similarity in both analogical reasoning and recognition memory. On recognition tasks, people more often falsely report having seen a recombined word pair (e.g., flower: garden) if it instantiates the same semantic relation (e.g., is a part of) as a studied word pair (e.g., house: town). This phenomenon, termed relational luring, has been interpreted as evidence that explicit relation representations-known to play a central role in analogical reasoning-also impact episodic memory. We replicate and extend previous studies, showing that relation-based false alarms in recognition memory occur after participants encode word pairs either by making relatedness judgments about individual words presented sequentially, or by evaluating analogies between pairs of word pairs. To test alternative explanations of relational luring, we implemented an established model of recognition memory, the Generalized Context Model (GCM). Within this basic framework, we compared representations of word pairs based on similarities derived either from explicit relations or from lexical semantics (i.e., individual word meanings). In two experiments on recognition memory, best-fitting values of GCM parameters enabled both similarity models (even the model based solely on lexical semantics) to predict relational luring with comparable accuracy. However, the model based on explicit relations proved more robust to parameter variations than that based on lexical similarity. We found this same pattern of modeling results when applying GCM to an independent set of data reported by Popov, Hristova, and Anders (2017). In accord with previous work, we also found that explicit relation representations are necessary for modeling analogical reasoning. Our findings support the possibility that explicit relations, which are central to analogical reasoning, also play an important role in episodic memory.

In situ bidirectional human-robot value alignment.

A prerequisite for social coordination is bidirectional communication between teammates, each playing two roles simultaneously: as receptive listeners and expressive speakers. For robots working with humans in complex situations with multiple goals that differ in importance, failure to fulfill the expectation of either role could undermine group performance due to misalignment of values between humans and robots. Specifically, a robot needs to serve as an effective listener to infer human users' intents from instructions and feedback and as an expressive speaker to explain its decision processes to users. Here, we investigate how to foster effective bidirectional human-robot communications in the context of value alignment-collaborative robots and users form an aligned understanding of the importance of possible task goals. We propose an explainable artificial intelligence (XAI) system in which a group of robots predicts users' values by taking in situ feedback into consideration while communicating their decision processes to users through explanations. To learn from human feedback, our XAI system integrates a cooperative communication model for inferring human values associated with multiple desirable goals. To be interpretable to humans, the system simulates human mental dynamics and predicts optimal explanations using graphical models. We conducted psychological experiments to examine the core components of the proposed computational framework. Our results show that real-time human-robot mutual understanding in complex cooperative tasks is achievable with a learning model based on bidirectional communication. We believe that this interaction framework can shed light on bidirectional value alignment in communicative XAI systems and, more broadly, in future human-machine teaming systems.

Aesthetic preferences for causality in biological movements arise from visual processes.

"People watching" is a ubiquitous component of human activities. An important aspect of such activities is the aesthetic experience that arises naturally from seeing how elegant people move their bodies in performing different actions. What makes some body movements look better than others? We examine how the human visual system gives rise to aesthetic experience from observing actions, using "creatures" generated by spatially scrambling locations of a point-light walker's joints. Observers rated how aesthetically pleasing and lifelike creatures were when the trajectories of joints were generated either from an upright walker (thus exhibiting gravitational acceleration) or an inverted walker (thus defying gravity), and were either congruent to the direction of global body displacements or incongruent (as in the moonwalk). Observers gave both higher aesthetic and animacy ratings for creatures with upright compared to inverted trajectories, and congruent compared to incongruent movements. Moreover, after controlling for animacy, aesthetic preferences for causally plausible movements (those in accord with gravity and body displacement) persisted. This systematicity in aesthetic impressions, even in the absence of explicit recognition of the moving agents, suggests an important role of automatic perceptual mechanisms in determining aesthetic experiences.

Probabilistic analogical mapping with semantic relation networks.

The human ability to flexibly reason using analogies with domain-general content depends on mechanisms for identifying relations between concepts, and for mapping concepts and their relations across analogs. Building on a recent model of how semantic relations can be learned from nonrelational word embeddings, we present a new computational model of mapping between two analogs. The model adopts a Bayesian framework for probabilistic graph matching, operating on semantic relation networks constructed from distributed representations of individual concepts and of relations between concepts. Through comparisons of model predictions with human performance in a novel mapping task requiring integration of multiple relations, as well as in several classic studies, we demonstrate that the model accounts for a broad range of phenomena involving analogical mapping by both adults and children. We also show the potential for extending the model to deal with analog retrieval. Our approach demonstrates that human-like analogical mapping can emerge from comparison mechanisms applied to rich semantic representations of individual concepts and relations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models.

We propose CX-ToM, short for counterfactual explanations with theory-of-mind, a new explainable AI (XAI) framework for explaining decisions made by a deep convolutional neural network (CNN). In contrast to the current methods in XAI that generate explanations as a single shot response, we pose explanation as an iterative communication process, i.e., dialogue between the machine and human user. More concretely, our CX-ToM framework generates a sequence of explanations in a dialogue by mediating the differences between the minds of the machine and human user. To do this, we use Theory of Mind (ToM) which helps us in explicitly modeling the human's intention, the machine's mind as inferred by the human, as well as human's mind as inferred by the machine. Moreover, most state-of-the-art XAI frameworks provide attention (or heat map) based explanations. In our work, we show that these attention-based explanations are not sufficient for increasing human trust in the underlying CNN model. In CX-ToM, we instead use counterfactual explanations called fault-lines which we define as follows: given an input image I for which a CNN classification model M predicts class c pred , a fault-line identifies the minimal semantic-level features (e.g., stripes on zebra), referred to as explainable concepts, that need to be added to or deleted from I to alter the classification category of I by M to another specified class c alt . Extensive experiments verify our hypotheses, demonstrating that our CX-ToM significantly outperforms the state-of-the-art XAI models.

Predicting patterns of similarity among abstract semantic relations.

Although models of word meanings based on distributional semantics have proved effective in predicting human judgments of similarity among individual concepts, it is less clear whether or how such models might be extended to account for judgments of similarity among relations between concepts. Here we combine an individual-differences approach with computational modeling to predict human judgments of similarity among word pairs instantiating a variety of abstract semantic relations (e.g., contrast, cause-effect, part-whole). A measure of cognitive capacity predicted individual differences in the ability to discriminate among distinct relations. The human pattern of relational similarity judgments, both at the group level and for individual participants, was best predicted by a model that takes representations of word meanings based on distributional semantics as its inputs and uses them to learn an explicit representation of relations. These findings indicate that although the meanings of abstract semantic relations are not directly coded in the meanings of individual words, important aspects of relational similarity can be derived from distributional semantics. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Infant perception of causal motion produced by humans and inanimate objects.

Both the movements of people and inanimate objects are intimately bound up with physical causality. Furthermore, in contrast to object movements, causal relationships between limb movements controlled by humans and their body displacements uniquely reflect agency and goal-directed actions in support of social causality. To investigate the development of sensitivity to causal movements, we examined the looking behavior of infants between 9 and 18 months of age when viewing movements of humans and objects. We also investigated whether individual differences in gender and gross motor functions may impact the development of the visual preferences for causal movements. In Experiment 1, infants were presented with walking stimuli showing either normal body translation or a "moonwalk" that reversed the horizontal motion of body translations. In Experiment 2, infants were presented with unperformable actions beyond infants' gross motor functions (i.e., long jump) either with or without ecologically valid body displacement. In Experiment 3, infants were presented with rolling movements of inanimate objects that either complied with or violated physical causality. We found that female infants showed longer looking times to normal walking stimuli than to moonwalk stimuli, but did not differ in their looking time to movements of inanimate objects and unperformable actions. In contrast, male infants did not show sensitivity to causal movement for either category. Additionally, female infants looked longer at social stimuli of human actions than male infants. Under the tested circumstances, our findings indicate that female infants have developed a sensitivity to causal consistency between limb movements and body translations of biological motion, only for actions with previous visual and motor exposures, and demonstrate a preference toward social information.

A unified psychological space for human perception of physical and social events.

One of the great feats of human perception is the generation of quick impressions of both physical and social events based on sparse displays of motion trajectories. Here we aim to provide a unified theory that captures the interconnections between perception of physical and social events. A simulation-based approach is used to generate a variety of animations depicting rich behavioral patterns. Human experiments used these animations to reveal that perception of dynamic stimuli undergoes a gradual transition from physical to social events. A learning-based computational framework is proposed to account for human judgments. The model learns to identify latent forces by inferring a family of potential functions capturing physical laws, and value functions describing the goals of agents. The model projects new animations into a sociophysical space with two psychological dimensions: an intuitive sense of whether physical laws are violated, and an impression of whether an agent possesses intentions to perform goal-directed actions. This derived sociophysical space predicts a meaningful partition between physical and social events, as well as a gradual transition from physical to social perception. The space also predicts human judgments of whether individual objects are lifeless objects in motion, or human agents performing goal-directed actions. These results demonstrate that a theoretical unification based on physical potential functions and goal-related values can account for the human ability to form an immediate impression of physical and social events. This ability provides an important pathway from perception to higher cognition.

Exploring biological motion perception in two-stream convolutional neural networks.

Visual recognition of biological motion recruits form and motion processes supported by both dorsal and ventral pathways. This neural architecture inspired the two-stream convolutional neural network (CNN) model, which includes a spatial CNN to process appearance information in a sequence of image frames, a temporal CNN to process optical flow information, and a fusion network to integrate the features extracted by the two CNNs and make final decisions about action recognition. In five simulations, we compared the CNN model's performance with classical findings in biological motion perception. The CNNs trained with raw RGB action videos showed weak performance in recognizing point-light actions. Additional transfer training with actions shown in other display formats (e.g., skeletal) was necessary for CNNs to recognize point-light actions. The CNN models exhibited largely viewpoint-dependent recognition of actions, with a limited ability to generalize to viewpoints close to the training views. The CNNs predicted the inversion effect in the presence of global body configuration, but failed to predict the inversion effect driven solely by local motion signals. The CNNs provided a qualitative account of some behavioral results observed in human biological motion perception for fine discrimination tasks with noisy inputs, such as point-light actions with disrupted local motion signals, and walking actions with temporally misaligned motion cues. However, these successes are limited by the CNNs' lack of adaptive integration for form and motion processes, and failure to incorporate specialized mechanisms (e.g., a life detector) as well as top-down influences on biological motion perception.