Using drawings and deep neural networks to characterize the building blocks of human visual similarity.

Early in life and without special training, human beings discern resemblance between abstract visual stimuli, such as drawings, and the real-world objects they represent. We used this capacity for visual abstraction as a tool for evaluating deep neural networks (DNNs) as models of human visual perception. Contrasting five contemporary DNNs, we evaluated how well each explains human similarity judgments among line drawings of recognizable and novel objects. For object sketches, human judgments were dominated by semantic category information; DNN representations contributed little additional information. In contrast, such features explained significant unique variance perceived similarity of abstract drawings. In both cases, a vision transformer trained to blend representations of images and their natural language descriptions showed the greatest ability to explain human perceptual similarity-an observation consistent with contemporary views of semantic representation and processing in the human mind and brain. Together, the results suggest that the building blocks of visual similarity may arise within systems that learn to use visual information, not for specific classification, but in service of generating semantic representations of objects.

Better than Goodenough? Evaluating new computational techniques for finding diagnostic structure in children's drawings.

In her 1926 book Measurement of Intelligence by Drawings, Florence Goodenough pioneered the quantitative analysis of children's human-figure drawings as a tool for evaluating their cognitive development. This influential work launched a broad enterprise in cognitive evaluation that continues to the present day, with most clinicians and researchers deploying variants of the checklist-based scoring methods that Goodenough invented. Yet recent work leveraging computational innovations in cognitive science suggests that human-figure drawings possess much richer structure than checklist-based approaches can capture. The current study uses these contemporary tools to characterize structure in the images from Goodenough's original work, then assesses whether this structure carries information about demographic and cognitive characteristics of the participants in that early study. The results show that contemporary methods can reliably extract information about participant age, gender, and mental faculties from images produced over 100 years ago, with no expert training and with minimal human effort. Moreover, the new analyses suggest a different relationship between drawing and mental ability than that captured by Goodenough's highly influential approach, with important implications for the use of drawings in cognitive evaluation in the present day.

Human perception and machine vision reveal rich latent structure in human figure drawings.

For over a hundred years, children's drawings have been used to assess children's intellectual, emotional, and physical development, characterizing children on the basis of intuitively derived checklists to identify the presence or absence of features within children's drawings. The current study investigates whether contemporary data science tools, including deep neural network models of vision and crowd-based similarity ratings, can reveal latent structure in human figure drawings beyond that captured by checklists, and whether such structure can aid in understanding aspects of the child's cognitive, perceptual, and motor competencies. We introduce three new metrics derived from innovations in machine vision and crowd-sourcing of human judgments and show that they capture a wealth of information about the participant beyond that expressed by standard measures, including age, gender, motor abilities, personal/social behaviors, and communicative skills. Machine-and human-derived metrics captured somewhat different aspects of structure across drawings, and each were independently useful for predicting some participant characteristics. For example, machine embeddings seemed sensitive to the magnitude of the drawing on the page and stroke density, while human-derived embeddings appeared sensitive to the overall shape and parts of a drawing. Both metrics, however, independently explained variation on some outcome measures. Machine embeddings explained more variation than human embeddings on all subscales of the Ages and Stages Questionnaire (a parent report of developmental milestones) and on measures of grip and pinch strength, while each metric accounted for unique variance in models predicting the participant's gender. This research thus suggests that children's drawings may provide a richer basis for characterizing aspects of cognitive, behavioral, and motor development than previously thought.

Decoding semantic representations in mind and brain.

A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition.

Semantic tiles or hub-and-spokes?

New results from Popham et al. generate 'semantic maps' from spoken narratives and movies that appear remarkably aligned near visual cortex. We consider whether such findings are consistent with the hub-and-spokes view of semantic representation or whether they require a rethinking of the cortical knowledge system.

Semantic diversity is best measured with unscaled vectors: Reply to Cevoli, Watkins and Rastle (2020).

Semantic diversity refers to the degree of semantic variability in the contexts in which a particular word is used. We have previously proposed a method for measuring semantic diversity based on latent semantic analysis (LSA). In a recent paper, Cevoli et al. (2020) attempted to replicate our method and obtained different semantic diversity values. They suggested that this discrepancy occurred because they scaled their LSA vectors by their singular values, while we did not. Using their new results, they argued that semantic diversity is not related to ambiguity in word meaning, as we originally proposed. In this reply, we demonstrate that the use of unscaled vectors provides better fits to human semantic judgements than scaled ones. Thus we argue that our original semantic diversity measure should be preferred over the Cevoli et al. version. We replicate Cevoli et al.'s analysis using the original semantic diversity measure and find (a) our original measure is a better predictor of word recognition latencies than the Cevoli et al. equivalent and (b) that, unlike Cevoli et al.'s measure, our semantic diversity is reliably associated with a measure of polysemy based on dictionary definitions. We conclude that the Hoffman et al. semantic diversity measure is better-suited to capturing the contextual variability among words and that words appearing in a more diverse set of contexts have more variable semantic representations. However, we found that homonyms did not have higher semantic diversity values than non-homonyms, suggesting that the measure does not capture this special case of ambiguity.

Evidence for a deep, distributed and dynamic code for animacy in human ventral anterior temporal cortex.

How does the human brain encode semantic information about objects? This paper reconciles two seemingly contradictory views. The first proposes that local neural populations independently encode semantic features; the second, that semantic representations arise as a dynamic distributed code that changes radically with stimulus processing. Combining simulations with a well-known neural network model of semantic memory, multivariate pattern classification, and human electrocorticography, we find that both views are partially correct: information about the animacy of a depicted stimulus is distributed across ventral temporal cortex in a dynamic code possessing feature-like elements posteriorly but with elements that change rapidly and nonlinearly in anterior regions. This pattern is consistent with the view that anterior temporal lobes serve as a deep cross-modal 'hub' in an interactive semantic network, and more generally suggests that tertiary association cortices may adopt dynamic distributed codes difficult to detect with common brain imaging methods.

Reverse-engineering the cortical architecture for controlled semantic cognition.

We employ a reverse-engineering approach to illuminate the neurocomputational building blocks that combine to support controlled semantic cognition: the storage and context-appropriate use of conceptual knowledge. By systematically varying the structure of a computational model and assessing the functional consequences, we identified the architectural properties that best promote some core functions of the semantic system. Semantic cognition presents a challenging test case, as the brain must achieve two seemingly contradictory functions: abstracting context-invariant conceptual representations across time and modalities, while producing specific context-sensitive behaviours appropriate for the immediate task. These functions were best achieved in models possessing a single, deep multimodal hub with sparse connections from modality-specific regions, and control systems acting on peripheral rather than deep network layers. The reverse-engineered model provides a unifying account of core findings in the cognitive neuroscience of controlled semantic cognition, including evidence from anatomy, neuropsychology and functional brain imaging.

Finding Distributed Needles in Neural Haystacks.

The human cortex encodes information in complex networks that can be anatomically dispersed and variable in their microstructure across individuals. Using simulations with neural network models, we show that contemporary statistical methods for functional brain imaging-including univariate contrast, searchlight multivariate pattern classification, and whole-brain decoding with L1 or L2 regularization-each have critical and complementary blind spots under these conditions. We then introduce the sparse-overlapping-sets (SOS) LASSO-a whole-brain multivariate approach that exploits structured sparsity to find network-distributed information-and show in simulation that it captures the advantages of other approaches while avoiding their limitations. When applied to fMRI data to find neural responses that discriminate visually presented faces from other visual stimuli, each method yields a different result, but existing approaches all support the canonical view that face perception engages localized areas in posterior occipital and temporal regions. In contrast, SOS LASSO uncovers a network spanning all four lobes of the brain. The result cannot reflect spurious selection of out-of-system areas because decoding accuracy remains exceedingly high even when canonical face and place systems are removed from the dataset. When used to discriminate visual scenes from other stimuli, the same approach reveals a localized signal consistent with other methods-illustrating that SOS LASSO can detect both widely distributed and localized representational structure. Thus, structured sparsity can provide an unbiased method for testing claims of functional localization. For faces and possibly other domains, such decoding may reveal representations more widely distributed than previously suspected.SIGNIFICANCE STATEMENT Brain systems represent information as patterns of activation over neural populations connected in networks that can be widely distributed anatomically, variable across individuals, and intermingled with other networks. We show that four widespread statistical approaches to functional brain imaging have critical blind spots in this scenario and use simulations with neural network models to illustrate why. We then introduce a new approach designed specifically to find radically distributed representations in neural networks. In simulation and in fMRI data collected in the well studied domain of face perception, the new approach discovers extensive signal missed by the other methods-suggesting that prior functional imaging work may have significantly underestimated the degree to which neurocognitive representations are distributed and variable across individuals.

The importance of standards for sharing of computational models and data.

The Target Article by Lee et al. (2019) highlights the ways in which ongoing concerns about research reproducibility extend to model-based approaches in cognitive science. Whereas Lee et al. focus primarily on the importance of research practices to improve model robustness, we propose that the transparent sharing of model specifications, including their inputs and outputs, is also essential to improving the reproducibility of model-based analyses. We outline an ongoing effort (within the context of the Brain Imaging Data Structure community) to develop standards for the sharing of the structure of computational models and their outputs.

Human hippocampal replay during rest prioritizes weakly learned information and predicts memory performance.

The hippocampus replays experiences during quiet rest periods, and this replay benefits subsequent memory. A critical open question is how memories are prioritized for this replay. We used functional magnetic resonance imaging (fMRI) pattern analysis to track item-level replay in the hippocampus during an awake rest period after participants studied 15 objects and completed a memory test. Objects that were remembered less well were replayed more during the subsequent rest period, suggesting a prioritization process in which weaker memories-memories most vulnerable to forgetting-are selected for replay. In a second session 12 hours later, more replay of an object during a rest period predicted better subsequent memory for that object. Replay predicted memory improvement across sessions only for participants who slept during that interval. Our results provide evidence that replay in the human hippocampus prioritizes weakly learned information, predicts subsequent memory performance, and relates to memory improvement across a delay with sleep.

Sleep Benefits Memory for Semantic Category Structure While Preserving Exemplar-Specific Information.

Semantic memory encompasses knowledge about both the properties that typify concepts (e.g. robins, like all birds, have wings) as well as the properties that individuate conceptually related items (e.g. robins, in particular, have red breasts). We investigate the impact of sleep on new semantic learning using a property inference task in which both kinds of information are initially acquired equally well. Participants learned about three categories of novel objects possessing some properties that were shared among category exemplars and others that were unique to an exemplar, with exposure frequency varying across categories. In Experiment 1, memory for shared properties improved and memory for unique properties was preserved across a night of sleep, while memory for both feature types declined over a day awake. In Experiment 2, memory for shared properties improved across a nap, but only for the lower-frequency category, suggesting a prioritization of weakly learned information early in a sleep period. The increase was significantly correlated with amount of REM, but was also observed in participants who did not enter REM, suggesting involvement of both REM and NREM sleep. The results provide the first evidence that sleep improves memory for the shared structure of object categories, while simultaneously preserving object-unique information.

A unified model of human semantic knowledge and its disorders.

How is knowledge about the meanings of words and objects represented in the human brain? Current theories embrace two radically different proposals: either distinct cortical systems have evolved to represent different kinds of things, or knowledge for all kinds is encoded within a single domain-general network. Neither view explains the full scope of relevant evidence from neuroimaging and neuropsychology. Here we propose that graded category-specificity emerges in some components of the semantic network through joint effects of learning and network connectivity. We test the proposal by measuring connectivity amongst cortical regions implicated in semantic representation, then simulating healthy and disordered semantic processing in a deep neural network whose architecture mirrors this structure. The resulting neuro-computational model explains the full complement of neuroimaging and patient evidence adduced in support of both domain-specific and domain-general approaches, reconciling long-standing disputes about the nature and origins of this uniquely human cognitive faculty.

The neural and computational bases of semantic cognition.

Semantic cognition refers to our ability to use, manipulate and generalize knowledge that is acquired over the lifespan to support innumerable verbal and non-verbal behaviours. This Review summarizes key findings and issues arising from a decade of research into the neurocognitive and neurocomputational underpinnings of this ability, leading to a new framework that we term controlled semantic cognition (CSC). CSC offers solutions to long-standing queries in philosophy and cognitive science, and yields a convergent framework for understanding the neural and computational bases of healthy semantic cognition and its dysfunction in brain disorders.

Conceptual knowledge representation: A cross-section of current research.

How is conceptual knowledge encoded in the brain? This special issue of Cognitive Neuropsychology takes stock of current efforts to answer this question through a variety of methods and perspectives. Across this work, three questions recur, each fundamental to knowledge representation in the mind and brain. First, what are the elements of conceptual representation? Second, to what extent are conceptual representations embodied in sensory and motor systems? Third, how are conceptual representations shaped by context, especially linguistic context? In this introductory article we provide relevant background on these themes and introduce how they are addressed by our contributing authors.

A Model of Emergent Category-specific Activation in the Posterior Fusiform Gyrus of Sighted and Congenitally Blind Populations.

Theories about the neural bases of semantic knowledge tend between two poles, one proposing that distinct brain regions are innately dedicated to different conceptual domains and the other suggesting that all concepts are encoded within a single network. Category-sensitive functional activations in the fusiform cortex of the congenitally blind have been taken to support the former view but also raise several puzzles. We use neural network models to assess a hypothesis that spans the two poles: The interesting functional activation patterns reflect the base connectivity of a domain-general semantic network. Both similarities and differences between sighted and congenitally blind groups can emerge through learning in a neural network, but only in architectures adopting real anatomical constraints. Surprisingly, the same constraints suggest a novel account of a quite different phenomenon: the dyspraxia observed in patients with semantic impairments from anterior temporal pathology. From this work, we suggest that the cortical semantic network is wired not to encode knowledge of distinct conceptual domains but to promote learning about both conceptual and affordance structure in the environment.

Reprint of: Semantic impairment disrupts perception, memory, and naming of secondary but not primary colours.

To investigate how basic aspects of perception are shaped by acquired knowledge about the world, we assessed colour perception and cognition in patients with semantic dementia (SD), a disorder that progressively erodes conceptual knowledge. We observed a previously undocumented pattern of impairment to colour perception and cognition characterized by: (i) a normal ability to discriminate between only subtly different colours but an impaired ability to group different colours into categories, (ii) normal perception and memory for the colours red, green, and blue but impaired perception and memory for colours lying between these regions of a fully-saturated and luminant spectrum, and (iii) normal naming of polar colours in the opponent-process colour system (red, green, blue, yellow, white, and black) but impaired naming of other basic colours (brown, gray, pink, and orange). The results suggest that fundamental aspects of perception can be shaped by acquired knowledge about the world, but only within limits.

Disorders of representation and control in semantic cognition: Effects of familiarity, typicality, and specificity.

We present a case-series comparison of patients with cross-modal semantic impairments consequent on either (a) bilateral anterior temporal lobe atrophy in semantic dementia (SD) or (b) left-hemisphere fronto-parietal and/or posterior temporal stroke in semantic aphasia (SA). Both groups were assessed on a new test battery designed to measure how performance is influenced by concept familiarity, typicality and specificity. In line with previous findings, performance in SD was strongly modulated by all of these factors, with better performance for more familiar items (regardless of typicality), for more typical items (regardless of familiarity) and for tasks that did not require very specific classification, consistent with the gradual degradation of conceptual knowledge in SD. The SA group showed significant impairments on all tasks but their sensitivity to familiarity, typicality and specificity was more variable and governed by task-specific effects of these factors on controlled semantic processing. The results are discussed with reference to theories about the complementary roles of representation and manipulation of semantic knowledge.

Impact of dialect use on a basic component of learning to read.

Can some black-white differences in reading achievement be traced to differences in language background? Many African American children speak a dialect that differs from the mainstream dialect emphasized in school. We examined how use of alternative dialects affects decoding, an important component of early reading and marker of reading development. Behavioral data show that use of the alternative pronunciations of words in different dialects affects reading aloud in developing readers, with larger effects for children who use more African American English (AAE). Mechanisms underlying this effect were explored with a computational model, investigating factors affecting reading acquisition. The results indicate that the achievement gap may be due in part to differences in task complexity: children whose home and school dialects differ are at greater risk for reading difficulties because tasks such as learning to decode are more complex for them.

Semantic impairment disrupts perception, memory, and naming of secondary but not primary colours.

To investigate how basic aspects of perception are shaped by acquired knowledge about the world, we assessed colour perception and cognition in patients with semantic dementia (SD), a disorder that progressively erodes conceptual knowledge. We observed a previously undocumented pattern of impairment to colour perception and cognition characterized by: (i) a normal ability to discriminate between only subtly different colours but an impaired ability to group different colours into categories, (ii) normal perception and memory for the colours red, green, and blue but impaired perception and memory for colours lying between these regions of a fully-saturated and luminant spectrum, and (iii) normal naming of polar colours in the opponent-process colour system (red, green, blue, yellow, white, and black) but impaired naming of other basic colours (brown, gray, pink, and orange). The results suggest that fundamental aspects of perception can be shaped by acquired knowledge about the world, but only within limits.