Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Cognitive complexity explains processing asymmetry in judgments of similarity versus difference.

Human judgments of similarity and difference are sometimes asymmetrical, with the former being more sensitive than the latter to relational overlap, but the theoretical basis for this asymmetry remains unclear. We test an explanation based on the type of information used to make these judgments (relations versus features) and the comparison process itself (similarity versus difference). We propose that asymmetries arise from two aspects of cognitive complexity that impact judgments of similarity and difference: processing relations between entities is more cognitively demanding than processing features of individual entities, and comparisons assessing difference are more cognitively complex than those assessing similarity. In Experiment 1 we tested this hypothesis for both verbal comparisons between word pairs, and visual comparisons between sets of geometric shapes. Participants were asked to select one of two options that was either more similar to or more different from a standard. On unambiguous trials, one option was unambiguously more similar to the standard; on ambiguous trials, one option was more featurally similar to the standard, whereas the other was more relationally similar. Given the higher cognitive complexity of processing relations and of assessing difference, we predicted that detecting relational difference would be particularly demanding. We found that participants (1) had more difficulty detecting relational difference than they did relational similarity on unambiguous trials, and (2) tended to emphasize relational information more when judging similarity than when judging difference on ambiguous trials. The latter finding was replicated using more complex story stimuli (Experiment 2). We showed that this pattern can be captured by a computational model of comparison that weights relational information more heavily for similarity than for difference judgments.

Reproductive factors and risk of cardiovascular diseases and all-cause and cardiovascular mortality in American women: NHANES 2003-2018.

BACKGROUND: The evidence regarding the association of reproductive factors with cardiovascular diseases (CVDs) is limited. AIMS: To investigate the relationship of reproductive factors with the risk of CVDs, as well as all-cause and cardiovascular mortality. METHODS: This study included 16,404 adults with reproductive factors from the National Health and Nutrition Examination Survey (NHANES) and followed up until 31 December 2019. Logistic models and restricted cubic spline models were used to assess the association of reproductive factors with CVDs. COX proportional hazards models and restricted cubic spline models, with adjustment for potential confounding, were employed to analyze the relation between reproductive factors and cardiovascular and all-cause death. RESULTS: There is a nonlinear relationship between age at menarche and CVDs. Age at menopause ≤ 11(OR 1.36, 95% CI 1.10-1.69) was associated with an increased risk of CVDs compared to ages 12-13 years. Age at Menopause ≤ 44 (OR 1.69, 95% CI 1.40-2.03) was associated with increased CVDs compared to age 35-49 years. Number of pregnancies ≥ 5(OR 1.26, 95% CI 1.02-1.55) was associated with an increased risk of CVDs compared to one pregnancy. In continuous variable COX regression models, a later age at menopause (HR 0.98, 95% CI 0.97-0.99) and a longer reproductive lifespan (HR 0.98, 95% CI 0.97-0.99) were associated with a decreased risk of all-cause death. A later age at menopause (HR 0.98, 95% CI 0.97-0.99) and a longer reproductive lifespan (HR 0.98, 95% CI 0.97-0.99) were associated with a decreased risk of cardiac death. CONCLUSIONS: Female reproductive factors are significant risk factors for CVDs American women.

Relational and lexical similarity in analogical reasoning and recognition memory: Behavioral evidence and computational evaluation.

We examined the role of different types of similarity in both analogical reasoning and recognition memory. On recognition tasks, people more often falsely report having seen a recombined word pair (e.g., flower: garden) if it instantiates the same semantic relation (e.g., is a part of) as a studied word pair (e.g., house: town). This phenomenon, termed relational luring, has been interpreted as evidence that explicit relation representations-known to play a central role in analogical reasoning-also impact episodic memory. We replicate and extend previous studies, showing that relation-based false alarms in recognition memory occur after participants encode word pairs either by making relatedness judgments about individual words presented sequentially, or by evaluating analogies between pairs of word pairs. To test alternative explanations of relational luring, we implemented an established model of recognition memory, the Generalized Context Model (GCM). Within this basic framework, we compared representations of word pairs based on similarities derived either from explicit relations or from lexical semantics (i.e., individual word meanings). In two experiments on recognition memory, best-fitting values of GCM parameters enabled both similarity models (even the model based solely on lexical semantics) to predict relational luring with comparable accuracy. However, the model based on explicit relations proved more robust to parameter variations than that based on lexical similarity. We found this same pattern of modeling results when applying GCM to an independent set of data reported by Popov, Hristova, and Anders (2017). In accord with previous work, we also found that explicit relation representations are necessary for modeling analogical reasoning. Our findings support the possibility that explicit relations, which are central to analogical reasoning, also play an important role in episodic memory.

For deep networks, the whole equals the sum of the parts.

Deep convolutional networks exceed humans in sensitivity to local image properties, but unlike biological vision systems, do not discover and encode abstract relations that capture important properties of objects and events in the world. Coupling network architectures with additional machinery for encoding abstract relations will make deep networks better models of human abilities and more versatile and capable artificial devices.

Emergence of analogy from relation learning.

By middle childhood, humans are able to learn abstract semantic relations (e.g., antonym, synonym, category membership) and use them to reason by analogy. A deep theoretical challenge is to show how such abstract relations can arise from nonrelational inputs, thereby providing key elements of a protosymbolic representation system. We have developed a computational model that exploits the potential synergy between deep learning from "big data" (to create semantic features for individual words) and supervised learning from "small data" (to create representations of semantic relations between words). Given as inputs labeled pairs of lexical representations extracted by deep learning, the model creates augmented representations by remapping features according to the rank of differences between values for the two words in each pair. These augmented representations aid in coping with the feature alignment problem (e.g., matching those features that make "love-hate" an antonym with the different features that make "rich-poor" an antonym). The model extracts weight distributions that are used to estimate the probabilities that new word pairs instantiate each relation, capturing the pattern of human typicality judgments for a broad range of abstract semantic relations. A measure of relational similarity can be derived and used to solve simple verbal analogies with human-level accuracy. Because each acquired relation has a modular representation, basic symbolic operations are enabled (notably, the converse of any learned relation can be formed without additional training). Abstract semantic relations can be induced by bootstrapping from nonrelational inputs, thereby enabling relational generalization and analogical reasoning.

Distributed Code for Semantic Relations Predicts Neural Similarity during Analogical Reasoning.

The ability to generate and process semantic relations is central to many aspects of human cognition. Theorists have long debated whether such relations are coarsely coded as links in a semantic network or finely coded as distributed patterns over some core set of abstract relations. The form and content of the conceptual and neural representations of semantic relations are yet to be empirically established. Using sequential presentation of verbal analogies, we compared neural activities in making analogy judgments with predictions derived from alternative computational models of relational dissimilarity to adjudicate among rival accounts of how semantic relations are coded and compared in the brain. We found that a frontoparietal network encodes the three relation types included in the design. A computational model based on semantic relations coded as distributed representations over a pool of abstract relations predicted neural activities for individual relations within the left superior parietal cortex and for second-order comparisons of relations within a broader left-lateralized network.

A unified psychological space for human perception of physical and social events.

One of the great feats of human perception is the generation of quick impressions of both physical and social events based on sparse displays of motion trajectories. Here we aim to provide a unified theory that captures the interconnections between perception of physical and social events. A simulation-based approach is used to generate a variety of animations depicting rich behavioral patterns. Human experiments used these animations to reveal that perception of dynamic stimuli undergoes a gradual transition from physical to social events. A learning-based computational framework is proposed to account for human judgments. The model learns to identify latent forces by inferring a family of potential functions capturing physical laws, and value functions describing the goals of agents. The model projects new animations into a sociophysical space with two psychological dimensions: an intuitive sense of whether physical laws are violated, and an impression of whether an agent possesses intentions to perform goal-directed actions. This derived sociophysical space predicts a meaningful partition between physical and social events, as well as a gradual transition from physical to social perception. The space also predicts human judgments of whether individual objects are lifeless objects in motion, or human agents performing goal-directed actions. These results demonstrate that a theoretical unification based on physical potential functions and goal-related values can account for the human ability to form an immediate impression of physical and social events. This ability provides an important pathway from perception to higher cognition.

Verbal analogy problem sets: An inventory of testing materials.

Analogical reasoning is an active topic of investigation across education, artificial intelligence (AI), cognitive psychology, and related fields. In all fields of inquiry, explicit analogy problems provide useful tools for investigating the mechanisms underlying analogical reasoning. Such sets have been developed by researchers working in the fields of educational testing, AI, and cognitive psychology. However, these analogy tests have not been systematically made accessible across all the relevant fields. The present paper aims to remedy this situation by presenting a working inventory of verbal analogy problem sets, intended to capture and organize sets from diverse sources.

Deep convolutional networks do not classify based on global object shape.

Deep convolutional networks (DCNNs) are achieving previously unseen performance in object classification, raising questions about whether DCNNs operate similarly to human vision. In biological vision, shape is arguably the most important cue for recognition. We tested the role of shape information in DCNNs trained to recognize objects. In Experiment 1, we presented a trained DCNN with object silhouettes that preserved overall shape but were filled with surface texture taken from other objects. Shape cues appeared to play some role in the classification of artifacts, but little or none for animals. In Experiments 2-4, DCNNs showed no ability to classify glass figurines or outlines but correctly classified some silhouettes. Aspects of these results led us to hypothesize that DCNNs do not distinguish object's bounding contours from other edges, and that DCNNs access some local shape features, but not global shape. In Experiment 5, we tested this hypothesis with displays that preserved local features but disrupted global shape, and vice versa. With disrupted global shape, which reduced human accuracy to 28%, DCNNs gave the same classification labels as with ordinary shapes. Conversely, local contour changes eliminated accurate DCNN classification but caused no difficulty for human observers. These results provide evidence that DCNNs have access to some local shape information in the form of local edge relations, but they have no access to global object shapes.

Causal Action: A Fundamental Constraint on Perception and Inference About Body Movements.

The human body navigates the environment via locomotory movements that leverage gravity and limb biomechanics to propel the body in a particular direction. This process creates a causal link between limb movements and whole-body translation. However, it is unknown whether humans use this causal relation as a constraint in perception and inference with body movements. In the present study, participants rated actions of other individuals as more natural when limb movements (as a cause) occurred before body displacements (as an effect) than when limb movements temporally lagged behind body displacements. This causal expectation for human body movements not only affected perceptual impressions regarding the naturalness of observed actions but also guided the interpretation of motion cues within a more generalized causal context. We interpret these results within a framework of causality as evidence that the constraint of causal action plays an important role in perception and inference with body movements.

Individual differences in spontaneous analogical transfer.

Research on analogical problem solving has shown that people often fail to spontaneously notice the relevance of a semantically remote source analog when solving a target problem, although they are able to form mappings and derive inferences when given a hint to recall the source. Relatively little work has investigated possible individual differences that predict spontaneous transfer, or how such differences may interact with interventions that facilitate transfer. In this study, fluid intelligence was measured for participants in an analogical problem-solving task, using an abridged version of the Raven's Progressive Matrices (RPM) test. In two experiments, we systematically compared the effect of augmenting verbal descriptions of the source with animations or static diagrams. Solution rates to Duncker's radiation problem were measured across varying source presentation conditions, and participants' understanding of the relevant source material was assessed. The pattern of transfer was best fit by a moderated mediation model: the positive impact of fluid intelligence on spontaneous transfer was mediated by its influence on source comprehension; however, this path was in turn modulated by provision of a supplemental animation via its influence on comprehension of the source. Animated source depictions were most beneficial in facilitating spontaneous transfer for those participants with low scores on the fluid intelligence measure.

Human efficiency in detecting and discriminating biological motion.

Using an "information meter" provided by ideal observer analysis, we measured the efficiency with which human observers processed different walking stimuli against luminance noise and spatial uncertainty to either detect the presence of a walker or to discriminate the walking direction. Human efficiency was examined across four renderings of a human walker: contour, point lights, silhouette, and skeleton. We replicated the previous finding of low discrimination efficiency in biological motion (Gold, Tadin, Cook, & Blake, 2008) and also found low detection efficiency for biological motion. Interestingly, in both detection and discrimination tasks, the skeleton display was among those yielding the highest level of efficiency in processing visual information. This finding suggests that structural information about the relative position of joints, highlighted in the skeleton display, provides a critical component of the internal representation for biological motion.

Neural adaptation in pSTS correlates with perceptual aftereffects to biological motion and with autistic traits.

The adaptive nature of biological motion perception has been documented in behavioral studies, with research showing that prolonged viewing of an action can bias judgments of subsequent actions towards the opposite of its attributes. However, the neural mechanisms underlying action adaptation aftereffects remain unknown. We examined adaptation-induced changes in brain responses to an ambiguous action after adapting to walking or running actions within two bilateral regions of interest: 1) human middle temporal area (hMT+), a lower-level motion-sensitive region of cortex, and 2) posterior superior temporal sulcus (pSTS), a higher-level action-selective area. We found a significant correlation between neural adaptation strength in right pSTS and perceptual aftereffects to biological motion measured behaviorally, but not in hMT+. The magnitude of neural adaptation in right pSTS was also strongly correlated with individual differences in the degree of autistic traits. Participants with more autistic traits exhibited less adaptation-induced modulations of brain responses in right pSTS and correspondingly weaker perceptual aftereffects. These results suggest a direct link between perceptual aftereffects and adaptation of neural populations in right pSTS after prolonged viewing of a biological motion stimulus, and highlight the potential importance of this brain region for understanding differences in social-cognitive processing along the autistic spectrum.

Causal competition based on generic priors.

Although we live in a complex and multi-causal world, learners often lack sufficient data and/or cognitive resources to acquire a fully veridical causal model. The general goal of making precise predictions with energy-efficient representations suggests a generic prior favoring causal models that include a relatively small number of strong causes. Such "sparse and strong" priors make it possible to quickly identify the most potent individual causes, relegating weaker causes to secondary status or eliminating them from consideration altogether. Sparse-and-strong priors predict that competition will be observed between candidate causes of the same polarity (i.e., generative or else preventive) even if they occur independently. For instance, the strength of a moderately strong cause should be underestimated when an uncorrelated strong cause also occurs in the general learning environment, relative to when a weaker cause also occurs. We report three experiments investigating whether independently-occurring causes (either generative or preventive) compete when people make judgments of causal strength. Cue competition was indeed observed for both generative and preventive causes. The data were used to assess alternative computational models of human learning in complex multi-causal situations.

A comparison of form processing involved in the perception of biological and nonbiological movements.

Although there is evidence for specialization in the human brain for processing biological motion per se, few studies have directly examined the specialization of form processing in biological motion perception. The current study was designed to systematically compare form processing in perception of biological (human walkers) to nonbiological (rotating squares) stimuli. Dynamic form-based stimuli were constructed with conflicting form cues (position and orientation), such that the objects were perceived to be moving ambiguously in two directions at once. In Experiment 1, we used the classification image technique to examine how local form cues are integrated across space and time in a bottom-up manner. By comparing with a Bayesian observer model that embodies generic principles of form analysis (e.g., template matching) and integrates form information according to cue reliability, we found that human observers employ domain-general processes to recognize both human actions and nonbiological object movements. Experiments 2 and 3 found differential top-down effects of spatial context on perception of biological and nonbiological forms. When a background does not involve social information, observers are biased to perceive foreground object movements in the direction opposite to surrounding motion. However, when a background involves social cues, such as a crowd of similar objects, perception is biased toward the same direction as the crowd for biological walking stimuli, but not for rotating nonbiological stimuli. The model provided an accurate account of top-down modulations by adjusting the prior probabilities associated with the internal templates, demonstrating the power and flexibility of the Bayesian approach for visual form perception.

Joints and their relations as critical features in action discrimination: evidence from a classification image method.

Classifying an action as a runner or a walker is a seemingly effortless process. However, it is difficult to determine which features are used with hypothesis-driven research, because biological motion stimuli generally consist of about a dozen joints, yielding an enormous number of potential relationships among them. Here, we develop a hypothesis-free approach based on a classification image method, using experimental data from relatively few trials (∼1,000 trials per subject). Employing ambiguous actions morphed between a walker and a runner, we identified three types of features that play important roles in discriminating bipedal locomotion presented in a side view: (a) critical joint feature, supported by the finding that the similarity of the movements of feet and wrists to prototypical movements of these joints were most reliably used across all participants; (b) structural features, indicated by contributions from almost all other joints, potentially through a form-based analysis; and (c) relational features, revealed by statistical correlations between joint contributions, specifically relations between the two feet, and relations between the wrists/elbow and the hips. When the actions were inverted, only critical joint features remained to significantly influence discrimination responses. When actions were presented with continuous depth rotation, critical joint features and relational features associated strongly with responses. Using a double-pass paradigm, we estimated that the internal noise is about twice as large as the external noise, consistent with previous findings. Overall, our novel design revealed a rich set of critical features that are used in action discrimination. The visual system flexibly selects a subset of features depending on viewing conditions.

The discovery and comparison of symbolic magnitudes.

Humans and other primates are able to make relative magnitude comparisons, both with perceptual stimuli and with symbolic inputs that convey magnitude information. Although numerous models of magnitude comparison have been proposed, the basic question of how symbolic magnitudes (e.g., size or intelligence of animals) are derived and represented in memory has received little attention. We argue that symbolic magnitudes often will not correspond directly to elementary features of individual concepts. Rather, magnitudes may be formed in working memory based on computations over more basic features stored in long-term memory. We present a model of how magnitudes can be acquired and compared based on BARTlet, a representationally simpler version of Bayesian Analogy with Relational Transformations (BART; Lu, Chen, & Holyoak, 2012). BARTlet operates on distributions of magnitude variables created by applying dimension-specific weights (learned with the aid of empirical priors derived from pre-categorical comparisons) to more primitive features of objects. The resulting magnitude distributions, formed and maintained in working memory, are sensitive to contextual influences such as the range of stimuli and polarity of the question. By incorporating psychological reference points that control the precision of magnitudes in working memory and applying the tools of signal detection theory, BARTlet is able to account for a wide range of empirical phenomena involving magnitude comparisons, including the symbolic distance effect and the semantic congruity effect. We discuss the role of reference points in cognitive and social decision-making, and implications for the evolution of relational representations.

Individual differences and motor planning influence self-recognition of actions.

Although humans can recognize their body movements in point-light displays, self-recognition ability varies substantially across action types and participants. Are these variations primarily due to an awareness of visually distinct movement patterns, or to underlying factors related to motoric planning and/or individual differences? To address this question, we conducted a large-scale study in self-action recognition (N = 101). We motion captured whole-body movements of participants who performed 27 different actions across action goals and degree of motor planning. After a long delay period (~ 1 month), participants were tested in a self-recognition task: identifying their point-light action amongst three other point-light actors performing identical actions. We report a self-advantage effect from point-light actions, consistent with prior work in self-action recognition. Further, we found that self-recognition was modulated by the action complexity (associated with the degree of motor planning in performed actions) and individual differences linked to motor imagery and subclinical autism and schizotypy. Using dynamic time warping, we found sparse evidence in support of visual distinctiveness as a primary contributor to self-recognition, though speed distinctiveness negatively influenced self-recognition performance. Together, our results reveal that self-action recognition involves more than an awareness of visually distinct movements, with important implications for how the motor system may be involved.

Association Between Systemic Immune Inflammation Index and Diabetes Mellitus in the NHANES 2003-2018 Population.

Objects: This study aimed to explore the association between the Systemic Immune-Inflammation Index (SII) and diabetes mellitus (DM) and to assess its influence on the prognosis of the DM and no-DM groups. Methods: The study used data from the National Health and Nutrition Examination Survey; 9643 participants were included. Logistic regression analysis was employed to evaluate connections between SII and DM. We used the Cox proportional hazards model, restricted cubic spline, and Kaplan-Meier curve to analyze the relationship between SII and mortality. Results: The logistic regression analysis indicated that a significant increase in the likelihood of developing DM with higher SII levels (odds ratio, 1.31; 95% CI, 1.09-1.57, P = .003). The Cox model showed that there is a positive association between increased SII and higher all-cause mortality. The hazard ratios for SII were 1.53 (1.31, 1.78), 1.61 (1.31, 1.98), and 1.41 (1.12, 1.78) in the total, DM and non-DM groups, respectively. We observed a linear correlation between SII and all-cause mortality in DM participants, whereas non-DM participants and the total population showed a nonlinear correlation. Conclusion: Elevated SII levels are linked to an augmented risk of DM. Those with DM and higher SII levels demonstrated an elevated risk of mortality.