High-performing neural network models of visual cortex benefit from high latent dimensionality.

Geometric descriptions of deep neural networks (DNNs) have the potential to uncover core representational principles of computational models in neuroscience. Here we examined the geometry of DNN models of visual cortex by quantifying the latent dimensionality of their natural image representations. A popular view holds that optimal DNNs compress their representations onto low-dimensional subspaces to achieve invariance and robustness, which suggests that better models of visual cortex should have lower dimensional geometries. Surprisingly, we found a strong trend in the opposite direction-neural networks with high-dimensional image subspaces tended to have better generalization performance when predicting cortical responses to held-out stimuli in both monkey electrophysiology and human fMRI data. Moreover, we found that high dimensionality was associated with better performance when learning new categories of stimuli, suggesting that higher dimensional representations are better suited to generalize beyond their training domains. These findings suggest a general principle whereby high-dimensional geometry confers computational benefits to DNN models of visual cortex.

Early Electrophysiological Markers of Navigational Affordances in Scenes.

Scene perception and spatial navigation are interdependent cognitive functions, and there is increasing evidence that cortical areas that process perceptual scene properties also carry information about the potential for navigation in the environment (navigational affordances). However, the temporal stages by which visual information is transformed into navigationally relevant information are not yet known. We hypothesized that navigational affordances are encoded during perceptual processing and therefore should modulate early visually evoked ERPs, especially the scene-selective P2 component. To test this idea, we recorded ERPs from participants while they passively viewed computer-generated room scenes matched in visual complexity. By simply changing the number of doors (0 doors, 1 door, 2 doors, 3 doors), we were able to systematically vary the number of pathways that afford movement in the local environment, while keeping the overall size and shape of the environment constant. We found that rooms with 0 doors evoked a higher P2 response than rooms with three doors, consistent with prior research reporting higher P2 amplitude to closed relative to open scenes. Moreover, we found P2 amplitude scaled linearly with the number of doors in the scenes. Navigability effects on the ERP waveform were also observed in a multivariate analysis, which showed significant decoding of the number of doors and their location at earlier time windows. Together, our results suggest that navigational affordances are represented in the early stages of scene perception. This complements research showing that the occipital place area automatically encodes the structure of navigable space and strengthens the link between scene perception and navigation.

Perceived Distance Alters Memory for Scene Boundaries.

Memory often fills in what is not there. A striking example of this is boundary extension, whereby observers mistakenly recall a view that extends beyond what was seen. However, not all visual memories extend in this way, which suggests that this process depends on specific scene properties. What factors determine when visual memories will include details that go beyond perceptual experience? Here, seven experiments (N = 1,100 adults) explored whether spatial scale-specifically, perceived viewing distance-drives boundary extension. We created fake miniatures by exploiting tilt shift, a photographic effect that selectively reduces perceived distance while preserving other scene properties (e.g., making a distant railway appear like a model train). Fake miniaturization increased boundary extension for otherwise identical scenes: Participants who performed a scene-memory task misremembered fake-miniaturized views as farther away than they actually were. This effect went beyond low-level image changes and generalized to a completely different distance manipulation. Thus, visual memory is modulated by the spatial scale at which the environment is viewed.

Unveiling functions of the visual cortex using task-specific deep neural networks.

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

Coding of navigational affordances in the human visual system.

A central component of spatial navigation is determining where one can and cannot go in the immediate environment. We used fMRI to test the hypothesis that the human visual system solves this problem by automatically identifying the navigational affordances of the local scene. Multivoxel pattern analyses showed that a scene-selective region of dorsal occipitoparietal cortex, known as the occipital place area, represents pathways for movement in scenes in a manner that is tolerant to variability in other visual features. These effects were found in two experiments: One using tightly controlled artificial environments as stimuli, the other using a diverse set of complex, natural scenes. A reconstruction analysis demonstrated that the population codes of the occipital place area could be used to predict the affordances of novel scenes. Taken together, these results reveal a previously unknown mechanism for perceiving the affordance structure of navigable space.

Similarly oriented objects appear more numerous.

Several non-numerical factors influence the numerical estimation of visual arrays, including the spacing of items and whether they are arranged randomly or symmetrically. Here we report a novel numerosity illusion we term the coherence illusion. When items in an array have a coherent orientation (all pointing in the same direction) they seem to be more numerous than when items are oriented randomly. Participants show parametric effects of orientation coherence in three distinct numerical judgment tasks. These findings are not predicted by any current model of numerical estimation. We discuss array entropy as a possible framework for explaining both the coherence illusion and the previously reported regular-random illusion.

Computational mechanisms underlying cortical responses to the affordance properties of visual scenes.

Biologically inspired deep convolutional neural networks (CNNs), trained for computer vision tasks, have been found to predict cortical responses with remarkable accuracy. However, the internal operations of these models remain poorly understood, and the factors that account for their success are unknown. Here we develop a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses. We focused on responses in the occipital place area (OPA), a scene-selective region of dorsal occipitoparietal cortex. In a previous study, we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes; that is, information about where one can and cannot move within the immediate environment. We hypothesized that this affordance information could be extracted using a set of purely feedforward computations. To test this idea, we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification. We found that responses in the CNN to scene images were highly predictive of fMRI responses in the OPA. Moreover the CNN accounted for the portion of OPA variance relating to the navigational affordances of scenes. The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA. We then ran a series of in silico experiments on this model to gain insights into its internal operations. These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations, both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex. These computations also exhibited a strong preference for information in the lower visual field, which is consistent with known retinotopic biases in the OPA. Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment, such as boundary-defining junctions and large extended surfaces. Together, these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations. More broadly, they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithms.

Causal Evidence for a Mechanism of Semantic Integration in the Angular Gyrus as Revealed by High-Definition Transcranial Direct Current Stimulation.

A defining aspect of human cognition is the ability to integrate conceptual information into complex semantic combinations. For example, we can comprehend "plaid" and "jacket" as individual concepts, but we can also effortlessly combine these concepts to form the semantic representation of "plaid jacket." Many neuroanatomic models of semantic memory propose that heteromodal cortical hubs integrate distributed semantic features into coherent representations. However, little work has specifically examined these proposed integrative mechanisms and the causal role of these regions in semantic integration. Here, we test the hypothesis that the angular gyrus (AG) is critical for integrating semantic information by applying high-definition transcranial direct current stimulation (tDCS) to an fMRI-guided region-of-interest in the left AG. We found that anodal stimulation to the left AG modulated semantic integration but had no effect on a letter-string control task. Specifically, anodal stimulation to the left AG resulted in faster comprehension of semantically meaningful combinations like "tiny radish" relative to non-meaningful combinations, such as "fast blueberry," when compared to the effects observed during sham stimulation and stimulation to a right-hemisphere control brain region. Moreover, the size of the effect from brain stimulation correlated with the degree of semantic coherence between the word pairs. These findings demonstrate that the left AG plays a causal role in the integration of lexical-semantic information, and that high-definition tDCS to an associative cortical hub can selectively modulate integrative processes in semantic memory. SIGNIFICANCE STATEMENT: A major goal of neuroscience is to understand the neural basis of behaviors that are fundamental to human intelligence. One essential behavior is the ability to integrate conceptual knowledge from semantic memory, allowing us to construct an almost unlimited number of complex concepts from a limited set of basic constituents (e.g., "leaf" and "wet" can be combined into the more complex representation "wet leaf"). Here, we present a novel approach to studying integrative processes in semantic memory by applying focal brain stimulation to a heteromodal cortical hub implicated in semantic processing. Our findings demonstrate a causal role of the left angular gyrus in lexical-semantic integration and provide motivation for novel therapeutic applications in patients with lexical-semantic deficits.

Converging evidence for the neuroanatomic basis of combinatorial semantics in the angular gyrus.

Human thought and language rely on the brain's ability to combine conceptual information. This fundamental process supports the construction of complex concepts from basic constituents. For example, both "jacket" and "plaid" can be represented as individual concepts, but they can also be integrated to form the more complex representation "plaid jacket." Although this process is central to the expression and comprehension of language, little is known about its neural basis. Here we present evidence for a neuroanatomic model of conceptual combination from three experiments. We predicted that the highly integrative region of heteromodal association cortex in the angular gyrus would be critical for conceptual combination, given its anatomic connectivity and its strong association with semantic memory in functional neuroimaging studies. Consistent with this hypothesis, we found that the process of combining concepts to form meaningful representations specifically modulates neural activity in the angular gyrus of healthy adults, independent of the modality of the semantic content integrated. We also found that individual differences in the structure of the angular gyrus in healthy adults are related to variability in behavioral performance on the conceptual combination task. Finally, in a group of patients with neurodegenerative disease, we found that the degree of atrophy in the angular gyrus is specifically related to impaired performance on combinatorial processing. These converging anatomic findings are consistent with a critical role for the angular gyrus in conceptual combination.

Semantics of the Visual Environment Encoded in Parahippocampal Cortex.

Semantic representations capture the statistics of experience and store this information in memory. A fundamental component of this memory system is knowledge of the visual environment, including knowledge of objects and their associations. Visual semantic information underlies a range of behaviors, from perceptual categorization to cognitive processes such as language and reasoning. Here we examine the neuroanatomic system that encodes visual semantics. Across three experiments, we found converging evidence indicating that knowledge of verbally mediated visual concepts relies on information encoded in a region of the ventral-medial temporal lobe centered on parahippocampal cortex. In an fMRI study, this region was strongly engaged by the processing of concepts relying on visual knowledge but not by concepts relying on other sensory modalities. In a study of patients with the semantic variant of primary progressive aphasia (semantic dementia), atrophy that encompassed this region was associated with a specific impairment in verbally mediated visual semantic knowledge. Finally, in a structural study of healthy adults from the fMRI experiment, gray matter density in this region related to individual variability in the processing of visual concepts. The anatomic location of these findings aligns with recent work linking the ventral-medial temporal lobe with high-level visual representation, contextual associations, and reasoning through imagination. Together, this work suggests a critical role for parahippocampal cortex in linking the visual environment with knowledge systems in the human brain.

A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing.

When a piece of fruit is in a bowl, and the bowl is on a table, we appreciate not only the individual objects and their features, but also the relations containment and support, which abstract away from the particular objects involved. Independent representation of roles (e.g., containers vs. supporters) and "fillers" of those roles (e.g., bowls vs. cups, tables vs. chairs) is a core principle of language and higher-level reasoning. But does such role-filler independence also arise in automatic visual processing? Here, we show that it does, by exploring a surprising error that such independence can produce. In four experiments, participants saw a stream of images containing different objects arranged in force-dynamic relations-e.g., a phone contained in a basket, a marker resting on a garbage can, or a knife sitting in a cup. Participants had to respond to a single target image (e.g., a phone in a basket) within a stream of distractors presented under time constraints. Surprisingly, even though participants completed this task quickly and accurately, they false-alarmed more often to images matching the target's relational category than to those that did not-even when those images involved completely different objects. In other words, participants searching for a phone in a basket were more likely to mistakenly respond to a knife in a cup than to a marker on a garbage can. Follow-up experiments ruled out strategic responses and also controlled for various confounding image features. We suggest that visual processing represents relations abstractly, in ways that separate roles from fillers.

Gray matter density of auditory association cortex relates to knowledge of sound concepts in primary progressive aphasia.

Long-term memory integrates the multimodal information acquired through perception into unified concepts, supporting object recognition, thought, and language. While some theories of human cognition have considered concepts to be abstract symbols, recent functional neuroimaging evidence has supported an alternative theory: that concepts are multimodal representations associated with the sensory and motor systems through which they are acquired. However, few studies have examined the effects of cortical lesions on the sensory and motor associations of concepts. We tested the hypothesis that individuals with disease in auditory association cortex would have difficulty processing concepts with strong sound associations (e.g., thunder). Human participants with the logopenic variant of primary progressive aphasia (lvPPA) performed a recognition task on words with strong associations in three modalities: Sound, Sight, and Manipulation. LvPPA participants had selective difficulty on Sound words relative to other modalities. Structural MRI analysis in lvPPA revealed gray matter atrophy in auditory association cortex, as defined functionally in a separate BOLD fMRI study of healthy adults. Moreover, lvPPA showed reduced gray matter density in the region of auditory association cortex that healthy participants activated when processing the same Sound words in a separate BOLD fMRI experiment. Finally, reduced gray matter density in this region in lvPPA directly correlated with impaired performance on Sound words. These findings support the hypothesis that conceptual memories are represented in the sensory and motor association cortices through which they are acquired.

Heteromodal conceptual processing in the angular gyrus.

Concepts bind together the features commonly associated with objects and events to form networks in long-term semantic memory. These conceptual networks are the basis of human knowledge and underlie perception, imagination, and the ability to communicate about experiences and the contents of the environment. Although it is often assumed that this distributed semantic information is integrated in higher-level heteromodal association cortices, open questions remain about the role and anatomic basis of heteromodal representations in semantic memory. Here we used combined neuroimaging evidence from functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) to characterize the cortical networks underlying concept representation. Using a lexical decision task, we examined the processing of concepts in four semantic categories that varied on their sensory-motor feature associations (sight, sound, manipulation, and abstract). We found that the angular gyrus was activated across all categories regardless of their modality-specific feature associations, consistent with a heteromodal account for the angular gyrus. Exploratory analyses suggested that categories with weighted sensory-motor features additionally recruited modality-specific association cortices. Furthermore, DTI tractography identified white matter tracts connecting these regions of modality-specific functional activation with the angular gyrus. These findings are consistent with a distributed semantic network that includes a heteromodal, integrative component in the angular gyrus in combination with sensory-motor feature representations in modality-specific association cortices.

Category-specific semantic memory: converging evidence from bold fMRI and Alzheimer's disease.

Patients with Alzheimer's disease have category-specific semantic memory difficulty for natural relative to manufactured objects. We assessed the basis for this deficit by asking healthy adults and patients to judge whether pairs of words share a feature (e.g. "banana:lemon-COLOR"). In an fMRI study, healthy adults showed gray matter (GM) activation of temporal-occipital cortex (TOC) where visual-perceptual features may be represented, and prefrontal cortex (PFC) which may contribute to feature selection. Tractography revealed dorsal and ventral stream white matter (WM) projections between PFC and TOC. Patients had greater difficulty with natural than manufactured objects. This was associated with greater overlap between diseased GM areas correlated with natural kinds in patients and fMRI activation in healthy adults for natural kinds. The dorsal WM projection between PFC and TOC in patients correlated only with judgments of natural kinds. Patients thus remained dependent on the same neural network as controls during judgments of natural kinds, despite disease in these areas. For manufactured objects, patients' judgments showed limited correlations with PFC and TOC GM areas activated by controls, and did not correlate with the PFC-TOC dorsal WM tract. Regions outside of the PFC-TOC network thus may help support patients' judgments of manufactured objects. We conclude that a large-scale neural network for semantic memory implicates both feature knowledge representations in modality-specific association cortex and heteromodal regions important for accessing this knowledge, and that patients' relative deficit for natural kinds is due in part to their dependence on this network despite disease in these areas.

Scene context is predictive of unconstrained object similarity judgments.

What makes objects alike in the human mind? Computational approaches for characterizing object similarity have largely focused on the visual forms of objects or their linguistic associations. However, intuitive notions of object similarity may depend heavily on contextual reasoning-that is, objects may be grouped together in the mind if they occur in the context of similar scenes or events. Using large-scale analyses of natural scene statistics and human behavior, we found that a computational model of the associations between objects and their scene contexts is strongly predictive of how humans spontaneously group objects by similarity. Specifically, we learned contextual prototypes for a diverse set of object categories by taking the average response of a convolutional neural network (CNN) to the scene contexts in which the objects typically occurred. In behavioral experiments, we found that contextual prototypes were strongly predictive of human similarity judgments for a large set of objects and rivaled the performance of models based on CNN representations of the objects themselves or word embeddings for their names. Together, our findings reveal the remarkable degree to which the natural statistics of context predict commonsense notions of object similarity.

Hierarchical organization of social action features along the lateral visual pathway.

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.

Scene memories are biased toward high-probability views.

Visual scenes are often remembered as if they were observed from a different viewpoint. Some scenes are remembered as farther than they appeared, and others as closer. These memory distortions-also known as boundary extension and contraction-are strikingly consistent for a given scene, but their cause remains unknown. We tested whether these distortions can be explained by an inferential process that adjusts scene memories toward high-probability views, using viewing depth as a test case. We first carried out a large-scale analysis of depth maps of natural indoor scenes to quantify the statistical probability of views in depth. We then assessed human observers' memory for these scenes at various depths and found that viewpoint judgments were consistently biased toward the modal depth, even when just a few seconds elapsed between viewing and reporting. Thus, scenes closer than the modal depth showed a boundary-extension bias (remembered as farther-away), and scenes farther than the modal depth showed a boundary-contraction bias (remembered as closer). By contrast, scenes at the modal depth did not elicit a consistent bias in either direction. This same pattern of results was observed in a follow-up experiment using tightly controlled stimuli from virtual environments. Together, these findings show that scene memories are biased toward statistically probable views, which may serve to increase the accuracy of noisy or incomplete scene representations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Object representations in the human brain reflect the co-occurrence statistics of vision and language.

A central regularity of visual perception is the co-occurrence of objects in the natural environment. Here we use machine learning and fMRI to test the hypothesis that object co-occurrence statistics are encoded in the human visual system and elicited by the perception of individual objects. We identified low-dimensional representations that capture the latent statistical structure of object co-occurrence in real-world scenes, and we mapped these statistical representations onto voxel-wise fMRI responses during object viewing. We found that cortical responses to single objects were predicted by the statistical ensembles in which they typically occur, and that this link between objects and their visual contexts was made most strongly in parahippocampal cortex, overlapping with the anterior portion of scene-selective parahippocampal place area. In contrast, a language-based statistical model of the co-occurrence of object names in written text predicted responses in neighboring regions of object-selective visual cortex. Together, these findings show that the sensory coding of objects in the human brain reflects the latent statistics of object context in visual and linguistic experience.

The new classification of primary progressive aphasia into semantic, logopenic, or nonfluent/agrammatic variants.

Primary progressive aphasia (PPA), typically resulting from a neurodegenerative disease such as frontotemporal lobar degeneration or Alzheimer's disease, is characterized by a progressive loss of specific language functions with relative sparing of other cognitive domains. Three variants of PPA are now recognized: semantic variant, logopenic variant, and nonfluent/agrammatic variant. We discuss recent work characterizing the neurolinguistic, neuropsychological, imaging and pathologic profiles associated with these variants. Improved reliability of diagnoses will be increasingly important as trials for etiology-specific treatments become available. We also discuss the implications of these syndromes for theories of language function.