Category Trumps Shape as an Organizational Principle of Object Space in the Human Occipitotemporal Cortex.

The organizational principles of the object space represented in the human ventral visual cortex are debated. Here we contrast two prominent proposals that, in addition to an organization in terms of animacy, propose either a representation related to aspect ratio (stubby-spiky) or to the distinction between faces and bodies. We designed a critical test that dissociates the latter two categories from aspect ratio and investigated responses from human fMRI (of either sex) and deep neural networks (BigBiGAN). Representational similarity and decoding analyses showed that the object space in the occipitotemporal cortex and BigBiGAN was partially explained by animacy but not by aspect ratio. Data-driven approaches showed clusters for face and body stimuli and animate-inanimate separation in the representational space of occipitotemporal cortex and BigBiGAN, but no arrangement related to aspect ratio. In sum, the findings go in favor of a model in terms of an animacy representation combined with strong selectivity for faces and bodies.SIGNIFICANCE STATEMENT We contrasted animacy, aspect ratio, and face-body as principal dimensions characterizing object space in the occipitotemporal cortex. This is difficult to test, as typically faces and bodies differ in aspect ratio (faces are mostly stubby and bodies are mostly spiky). To dissociate the face-body distinction from the difference in aspect ratio, we created a new stimulus set in which faces and bodies have a similar and very wide distribution of values along the shape dimension of the aspect ratio. Brain imaging (fMRI) with this new stimulus set showed that, in addition to animacy, the object space is mainly organized by the face-body distinction and selectivity for aspect ratio is minor (despite its wide distribution).

The representational hierarchy in human and artificial visual systems in the presence of object-scene regularities.

Human vision is still largely unexplained. Computer vision made impressive progress on this front, but it is still unclear to which extent artificial neural networks approximate human object vision at the behavioral and neural levels. Here, we investigated whether machine object vision mimics the representational hierarchy of human object vision with an experimental design that allows testing within-domain representations for animals and scenes, as well as across-domain representations reflecting their real-world contextual regularities such as animal-scene pairs that often co-occur in the visual environment. We found that DCNNs trained in object recognition acquire representations, in their late processing stage, that closely capture human conceptual judgements about the co-occurrence of animals and their typical scenes. Likewise, the DCNNs representational hierarchy shows surprising similarities with the representational transformations emerging in domain-specific ventrotemporal areas up to domain-general frontoparietal areas. Despite these remarkable similarities, the underlying information processing differs. The ability of neural networks to learn a human-like high-level conceptual representation of object-scene co-occurrence depends upon the amount of object-scene co-occurrence present in the image set thus highlighting the fundamental role of training history. Further, although mid/high-level DCNN layers represent the category division for animals and scenes as observed in VTC, its information content shows reduced domain-specific representational richness. To conclude, by testing within- and between-domain selectivity while manipulating contextual regularities we reveal unknown similarities and differences in the information processing strategies employed by human and artificial visual systems.

Understanding Human Object Vision: A Picture Is Worth a Thousand Representations.

Objects are the core meaningful elements in our visual environment. Classic theories of object vision focus upon object recognition and are elegant and simple. Some of their proposals still stand, yet the simplicity is gone. Recent evolutions in behavioral paradigms, neuroscientific methods, and computational modeling have allowed vision scientists to uncover the complexity of the multidimensional representational space that underlies object vision. We review these findings and propose that the key to understanding this complexity is to relate object vision to the full repertoire of behavioral goals that underlie human behavior, running far beyond object recognition. There might be no such thing as core object recognition, and if it exists, then its importance is more limited than traditionally thought.

Going after the bigger picture: Using high-capacity models to understand mind and brain.

Deep neural networks (DNNs) provide a unique opportunity to move towards a generic modelling framework in psychology. The high representational capacity of these models combined with the possibility for further extensions has already allowed us to investigate the forest, namely the complex landscape of representations and processes that underlie human cognition, without forgetting about the trees, which include individual psychological phenomena.

Untangling the Animacy Organization of Occipitotemporal Cortex.

Some of the most impressive functional specializations in the human brain are found in the occipitotemporal cortex (OTC), where several areas exhibit selectivity for a small number of visual categories, such as faces and bodies, and spatially cluster based on stimulus animacy. Previous studies suggest this animacy organization reflects the representation of an intuitive taxonomic hierarchy, distinct from the presence of face- and body-selective areas in OTC. Using human functional magnetic resonance imaging, we investigated the independent contribution of these two factors-the face-body division and taxonomic hierarchy-in accounting for the animacy organization of OTC and whether they might also be reflected in the architecture of several deep neural networks that have not been explicitly trained to differentiate taxonomic relations. We found that graded visual selectivity, based on animal resemblance to human faces and bodies, masquerades as an apparent animacy continuum, which suggests that taxonomy is not a separate factor underlying the organization of the ventral visual pathway.SIGNIFICANCE STATEMENT Portions of the visual cortex are specialized to determine whether types of objects are animate in the sense of being capable of self-movement. Two factors have been proposed as accounting for this animacy organization: representations of faces and bodies and an intuitive taxonomic continuum of humans and animals. We performed an experiment to assess the independent contribution of both of these factors. We found that graded visual representations, based on animal resemblance to human faces and bodies, masquerade as an apparent animacy continuum, suggesting that taxonomy is not a separate factor underlying the organization of areas in the visual cortex.

Using deep neural networks to evaluate object vision tasks in rats.

In the last two decades rodents have been on the rise as a dominant model for visual neuroscience. This is particularly true for earlier levels of information processing, but a number of studies have suggested that also higher levels of processing such as invariant object recognition occur in rodents. Here we provide a quantitative and comprehensive assessment of this claim by comparing a wide range of rodent behavioral and neural data with convolutional deep neural networks. These networks have been shown to capture hallmark properties of information processing in primates through a succession of convolutional and fully connected layers. We find that performance on rodent object vision tasks can be captured using low to mid-level convolutional layers only, without any convincing evidence for the need of higher layers known to simulate complex object recognition in primates. Our approach also reveals surprising insights on assumptions made before, for example, that the best performing animals would be the ones using the most abstract representations-which we show to likely be incorrect. Our findings suggest a road ahead for further studies aiming at quantifying and establishing the richness of representations underlying information processing in animal models at large.

How Visual Expertise Changes Representational Geometry: A Behavioral and Neural Perspective.

Many people develop expertise in specific domains of interest, such as chess, microbiology, radiology, and, the case in point in our study: ornithology. It is poorly understood to what extent such expertise alters brain function. Previous neuroimaging studies of expertise have typically focused upon the category level, for example, selectivity for birds versus nonbird stimuli. We present a multivariate fMRI study focusing upon the representational similarity among objects of expertise at the subordinate level. We compare the neural representational spaces of experts and novices to behavioral judgments. At the behavioral level, ornithologists (n = 20) have more fine-grained and task-dependent representations of item similarity that are more consistent among experts compared to control participants. At the neural level, the neural patterns of item similarity are more distinct and consistent in experts than in novices, which is in line with the behavioral results. In addition, these neural patterns in experts show stronger correlations with behavior compared to novices. These findings were prominent in frontal regions, and some effects were also found in occipitotemporal regions. This study illustrates the potential of an analysis of representational geometry to understand to what extent expertise changes neural information processing.

The unreliable influence of multivariate noise normalization on the reliability of neural dissimilarity.

Representational similarity analysis (RSA) is a key element in the multivariate pattern analysis toolkit. The central construct of the method is the representational dissimilarity matrix (RDM), which can be generated for datasets from different modalities (neuroimaging, behavior, and computational models) and directly correlated in order to evaluate their second-order similarity. Given the inherent noisiness of neuroimaging signals it is important to evaluate the reliability of neuroimaging RDMs in order to determine whether these comparisons are meaningful. Recently, multivariate noise normalization (NNM) has been proposed as a widely applicable method for boosting signal estimates for RSA, regardless of choice of dissimilarity metrics, based on evidence that the analysis improves the within-subject reliability of RDMs (Guggenmos et al. 2018; Walther et al. 2016). We revisited this issue with three fMRI datasets and evaluated the impact of NNM on within- and between-subject reliability and RSA effect sizes using multiple dissimilarity metrics. We also assessed its impact across regions of interest from the same dataset, its interaction with spatial smoothing, and compared it to GLMdenoise, which has also been proposed as a method that improves signal estimates for RSA (Charest et al. 2018). We found that across these tests the impact of NNM was highly variable, as also seems to be the case for other analysis choices. Overall, we suggest being conservative before adding steps and complexities to the (pre)processing pipeline for RSA.

Training for object recognition with increasing spatial frequency: A comparison of deep learning with human vision.

The ontogenetic development of human vision and the real-time neural processing of visual input exhibit a striking similarity-a sensitivity toward spatial frequencies that progresses in a coarse-to-fine manner. During early human development, sensitivity for higher spatial frequencies increases with age. In adulthood, when humans receive new visual input, low spatial frequencies are typically processed first before subsequent processing of higher spatial frequencies. We investigated to what extent this coarse-to-fine progression might impact visual representations in artificial vision and compared this to adult human representations. We simulated the coarse-to-fine progression of image processing in deep convolutional neural networks (CNNs) by gradually increasing spatial frequency information during training. We compared CNN performance after standard and coarse-to-fine training with a wide range of datasets from behavioral and neuroimaging experiments. In contrast to humans, CNNs that are trained using the standard protocol are very insensitive to low spatial frequency information, showing very poor performance in being able to classify such object images. By training CNNs using our coarse-to-fine method, we improved the classification accuracy of CNNs from 0% to 32% on low-pass-filtered images taken from the ImageNet dataset. The coarse-to-fine training also made the CNNs more sensitive to low spatial frequencies in hybrid images with conflicting information in different frequency bands. When comparing differently trained networks on images containing full spatial frequency information, we saw no representational differences. Overall, this integration of computational, neural, and behavioral findings shows the relevance of the exposure to and processing of inputs with variation in spatial frequency content for some aspects of high-level object representations.

The Ventral Visual Pathway Represents Animal Appearance over Animacy, Unlike Human Behavior and Deep Neural Networks.

Recent studies showed agreement between how the human brain and neural networks represent objects, suggesting that we might start to understand the underlying computations. However, we know that the human brain is prone to biases at many perceptual and cognitive levels, often shaped by learning history and evolutionary constraints. Here, we explore one such perceptual phenomenon, perceiving animacy, and use the performance of neural networks as a benchmark. We performed an fMRI study that dissociated object appearance (what an object looks like) from object category (animate or inanimate) by constructing a stimulus set that includes animate objects (e.g., a cow), typical inanimate objects (e.g., a mug), and, crucially, inanimate objects that look like the animate objects (e.g., a cow mug). Behavioral judgments and deep neural networks categorized images mainly by animacy, setting all objects (lookalike and inanimate) apart from the animate ones. In contrast, activity patterns in ventral occipitotemporal cortex (VTC) were better explained by object appearance: animals and lookalikes were similarly represented and separated from the inanimate objects. Furthermore, the appearance of an object interfered with proper object identification, such as failing to signal that a cow mug is a mug. The preference in VTC to represent a lookalike as animate was even present when participants performed a task requiring them to report the lookalikes as inanimate. In conclusion, VTC representations, in contrast to neural networks, fail to represent objects when visual appearance is dissociated from animacy, probably due to a preferred processing of visual features typical of animate objects.SIGNIFICANCE STATEMENT How does the brain represent objects that we perceive around us? Recent advances in artificial intelligence have suggested that object categorization and its neural correlates have now been approximated by neural networks. Here, we show that neural networks can predict animacy according to human behavior but do not explain visual cortex representations. In ventral occipitotemporal cortex, neural activity patterns were strongly biased toward object appearance, to the extent that objects with visual features resembling animals were represented closely to real animals and separated from other objects from the same category. This organization that privileges animals and their features over objects might be the result of learning history and evolutionary constraints.

An investigation of the effect of temporal contiguity training on size-tolerant representations in object-selective cortex.

The human visual system has a remarkable ability to reliably identify objects across variations in appearance, such as variations in viewpoint, lighting and size. Here we used fMRI in humans to test whether temporal contiguity training with natural and altered image dynamics can respectively build and break neural size tolerance for objects. Participants (N â€‹= â€‹23) were presented with sequences of images of "growing" and "shrinking" objects. In half of the trials, the object also changed identity when the size change happened. According to the temporal contiguity hypothesis, and studies with a similar paradigm in monkeys, this training process should alter size tolerance. After the training phase, BOLD responses to each of the object images were measured in the scanner. Neural patterns in LOC and V1 contained information on size, similarity and identity. In LOC, the representation of object identity was partially invariant to changes in size. However, temporal contiguity training did not affect size tolerance in LOC. Size tolerance in human object-selective cortex is more robust to variations in input statistics than expected based on prior work in monkeys supporting the temporal contiguity hypothesis.

Reduced task-dependent modulation of functional network architecture for positive versus negative affective touch processing in autism spectrum disorders.

Individuals with autism spectrum disorders (ASD) experience impairments in social communication and interaction, and often show difficulties with receiving and offering touch. Despite the high prevalence of abnormal reactions to touch in ASD, and the importance of touch communication in human relationships, the neural mechanisms underlying atypical touch processing in ASD remain largely unknown. To answer this question, we provided both pleasant and unpleasant touch stimulation to male adults with and without ASD during functional neuroimaging. By employing generalized psychophysiological interaction analysis combined with an independent component analysis approach, we characterize stimulus-dependent changes in functional connectivity patterns for processing two tactile stimuli that evoke different emotions (i.e., pleasant vs. unpleasant touch). Results reveal that neurotypical male adults showed extensive stimulus-sensitive modulations of the functional network architecture in response to the different types of touch, both at the level of brain regions and large-scale networks. Conversely, far fewer stimulus-sensitive modulations were observed in the ASD group. These aberrant functional connectivity profiles in the ASD group were marked by hypo-connectivity of the parietal operculum and major pain networks and hyper-connectivity between the semantic and limbic networks. Lastly, individuals presenting more social deficits and a more negative attitude towards social touch showed greater hyper-connectivity between the limbic and semantic networks. These findings suggest that reduced stimulus-related modulation of this functional network architecture is associated with abnormal processing of touch in ASD.

Development of visual category selectivity in ventral visual cortex does not require visual experience.

To what extent does functional brain organization rely on sensory input? Here, we show that for the penultimate visual-processing region, ventral-temporal cortex (VTC), visual experience is not the origin of its fundamental organizational property, category selectivity. In the fMRI study reported here, we presented 14 congenitally blind participants with face-, body-, scene-, and object-related natural sounds and presented 20 healthy controls with both auditory and visual stimuli from these categories. Using macroanatomical alignment, response mapping, and surface-based multivoxel pattern analysis, we demonstrated that VTC in blind individuals shows robust discriminatory responses elicited by the four categories and that these patterns of activity in blind subjects could successfully predict the visual categories in sighted controls. These findings were confirmed in a subset of blind participants born without eyes and thus deprived from all light perception since conception. The sounds also could be decoded in primary visual and primary auditory cortex, but these regions did not sustain generalization across modalities. Surprisingly, although not as strong as visual responses, selectivity for auditory stimulation in visual cortex was stronger in blind individuals than in controls. The opposite was observed in primary auditory cortex. Overall, we demonstrated a striking similarity in the cortical response layout of VTC in blind individuals and sighted controls, demonstrating that the overall category-selective map in extrastriate cortex develops independently from visual experience.

Face Repetition Probability Does Not Affect Repetition Suppression in Macaque Inferotemporal Cortex.

Repetition suppression, which refers to reduced neural activity for repeated stimuli, is typically explained by bottom-up or local adaptation mechanisms. However, recent theories have emphasized the role of top-down processes, suggesting that this response reduction reflects the fulfillment of perceptual expectations. To support this, an influential human fMRI study showed that the magnitude of suppression is modulated by the probability of a repetition. No such repetition probability effect was found in macaque inferior temporal (IT) cortex for spiking activity despite the presence of repetition suppression. Contrary to the human fMRI studies that showed an effect of repetition probability, the macaque single-unit study used a large variety of unfamiliar stimuli and the monkeys were not required to attend the stimuli. Here, as in the human fMRI studies, we used faces as stimuli and made the monkeys attend to the stimulus content. We simultaneously recorded spiking activity and local field potentials (LFPs) in the middle lateral face patch (ML) of one monkey (male) and a face-responsive region of another (female). Although we observed significant repetition suppression of spiking activity and high gamma-band LFPs in both animals, there were no effects of repetition probability even when repetitions were task relevant and repetition probability affected behavioral decisions. In conclusion, despite the use of face stimuli and a stimulus-related task, no neural signature of repetition probability was present for faces in a face responsive patch of macaque IT. This further challenges a general perceptual expectation account of repetition suppression.SIGNIFICANCE STATEMENT Repetition suppression is a reduced brain activity for repeated stimuli commonly observed across species. In the predictive coding framework, such suppression is thought to reflect fulfilled perceptual expectations. Although this hypothesis is supported by several human fMRI studies reporting an effect of repetition probability on repetition suppression, this could not be replicated in single-cell recordings in monkey inferior temporal (IT) cortex. Subsequent studies narrowed down the conditions for the effect to requiring attention and being limited to particular stimulus categories such as faces. Here, we show that, even under these conditions, repetition suppression in monkey IT neurons is still unaffected by repetition probability, even in a task with a behavioral effect, challenging the perceptual expectation account of repetition suppression.

A Varying Role for Abstraction in Models of Category Learning Constructed from Neural Representations in Early Visual Cortex.

The human capacity for visual categorization is core to how we make sense of the visible world. Although a substantive body of research in cognitive neuroscience has localized this capacity to regions of human visual cortex, relatively few studies have investigated the role of abstraction in how representations for novel object categories are constructed from the neural representation of stimulus dimensions. Using human fMRI coupled with formal modeling of observer behavior, we assess a wide range of categorization models that vary in their level of abstraction from collections of subprototypes to representations of individual exemplars. The category learning tasks range from simple linear and unidimensional category rules to complex crisscross rules that require a nonlinear combination of multiple dimensions. We show that models based on neural responses in primary visual cortex favor a variable, but often limited, extent of abstraction in the construction of representations for novel categories, which differ in degree across tasks and individuals.

The representation of symmetry in multi-voxel response patterns and functional connectivity throughout the ventral visual stream.

Several computational models explain how symmetry might be detected and represented in the human brain. However, while there is an abundance of psychophysical studies on symmetry detection and several neural studies showing where and when symmetry is detected in the brain, important questions remain about how this detection happens and how symmetric patterns are represented. We studied the representation of (vertical) symmetry in regions of the ventral visual stream, using multi-voxel pattern analyses (MVPA) and functional connectivity analyses. Our results suggest that neural representations gradually change throughout the ventral visual stream, from very similar part-based representations for symmetrical and asymmetrical stimuli in V1 and V2, over increasingly different representations for symmetrical and asymmetrical stimuli which are nevertheless still part-based in both V3 and V4, to a more holistic representation for symmetrical compared to asymmetrical stimuli in high-level LOC. This change in representations is accompanied by increased communication between left and right retinotopic areas, evidenced by higher interhemispheric functional connectivity during symmetry perception in areas V2 and V4.

Multi-method brain imaging reveals impaired representations of number as well as altered connectivity in adults with dyscalculia.

Two hypotheses have been proposed about the etiology of neurodevelopmental learning disorders, such as dyslexia and dyscalculia: representation impairments and disrupted access to representations. We implemented a multi-method brain imaging approach to directly investigate these representation and access hypotheses in dyscalculia, a highly prevalent but understudied neurodevelopmental disorder in learning to calculate. We combined several magnetic resonance imaging methods and analyses, including univariate and multivariate analyses, functional and structural connectivity. Our sample comprised 24 adults with dyscalculia and 24 carefully matched controls. Results showed a clear deficit in the non-symbolic magnitude representations in parietal, temporal and frontal regions, as well as hyper-connectivity in visual brain regions in adults with dyscalculia. Dyscalculia in adults was thereby related to both impaired number representations and altered connectivity in the brain. We conclude that dyscalculia is related to impaired number representations as well as altered access to these representations.

Domain-general and domain-specific neural changes underlying visual expertise.

Visual expertise induces changes in neural processing for many different domains of expertise. However, it is unclear how expertise effects for different domains of expertise are related. In the present fMRI study, we combine large-scale univariate and multi-voxel analyses to contrast the expertise-related neural changes associated with two different domains of expertise, bird expertise (ornithology) and mineral expertise (mineralogy). Results indicated distributed expertise-related neural changes, with effects for both domains of expertise in high-level visual cortex and effects for bird expertise even extending to low-level visual regions and the frontal lobe. Importantly, a multivariate generalization analysis showed that effects in high-level visual cortex were specific to the domain of expertise. In contrast, the neural changes in the frontal lobe relating to expertise showed significant generalization, signaling the presence of domain-independent expertise effects. In conclusion, expertise is related to a combination of domain-specific and domain-general changes in neural processing.

The multidimensional representational space of observed socio-affective touch experiences.

Observed touch interactions provide useful information on how others communicate with the external world. Previous studies revealed shared neural circuits between the direct experience and the passive observation of simple touch, such as being stroked/slapped. Here, we investigate the complexity of the neural representations underlying the understanding of others' socio-affective touch interactions. Importantly, we use a recently developed touch database that contains a larger range of more complex social and non-social touch interactions. Participants judged affective aspects of each touch event and were scanned while watching the same videos. Using correlational multivariate pattern analysis methods, we obtained neural similarity matrices in 18 regions of interest from five different networks: somatosensory, pain, the theory of mind, visual and motor regions. Among them, four networks except motor cortex represent the social nature of the touch, whereas fine-detailed affective information is reflected in more targeted areas such as social brain regions and somatosensory cortex. Lastly, individual social touch preference at the behavioral level was correlated with the involvement of somatosensory areas on representing affective information, suggesting that individuals with higher social touch preference exhibit stronger vicarious emotional responses to others' social touch experiences. Together, these results highlight the overall complexity and the individual modulation of the distributed neural representations underlying the processing of observed socio-affective touch.

Hierarchical stimulus processing in rodent primary and lateral visual cortex as assessed through neuronal selectivity and repetition suppression.

Similar to primates, visual cortex in rodents appears to be organized in two distinct hierarchical streams. However, there is still little known about how visual information is processed along those streams in rodents. In this study, we examined how repetition suppression and position and clutter tolerance of the neuronal representations evolve along the putative ventral visual stream in rats. To address this question, we recorded multiunit spiking activity in primary visual cortex (V1) and the more downstream visual laterointermediate (LI) area of head-restrained Long-Evans rats. We employed a paradigm reminiscent of the continuous carry-over design used in human neuroimaging. In both areas, stimulus repetition attenuated the early phase of the neuronal response to the repeated stimulus, with this response suppression being greater in area LI. Furthermore, stimulus preferences were more similar across positions (position tolerance) in area LI than in V1, even though the absolute responses in both areas were very sensitive to changes in position. In contrast, the neuronal representations in both areas were equally good at tolerating the presence of limited visual clutter, as modeled by the presentation of a single flank stimulus. When probing tolerance of the neuronal representations with stimulus-specific adaptation, we detected no position tolerance in either examined brain area, whereas, on the contrary, we revealed clutter tolerance in both areas. Overall, our data demonstrate similarities and discrepancies in processing of visual information along the ventral visual stream of rodents and primates. Moreover, our results stress caution in using neuronal adaptation to probe tolerance of the neuronal representations. NEW & NOTEWORTHY Rodents are emerging as a popular animal model that complement primates for studying higher level visual functions. Similar to findings in primates, we demonstrate a greater repetition suppression and position tolerance of the neuronal representations in the downstream laterointermediate area of Long-Evans rats compared with primary visual cortex. However, we report no difference in the degree of clutter tolerance between the areas. These findings provide additional evidence for hierarchical processing of visual stimuli in rodents.