Semantic Scene-Object Consistency Modulates N300/400 EEG Components, but Does Not Automatically Facilitate Object Representations.

During natural vision, objects rarely appear in isolation, but often within a semantically related scene context. Previous studies reported that semantic consistency between objects and scenes facilitates object perception and that scene-object consistency is reflected in changes in the N300 and N400 components in EEG recordings. Here, we investigate whether these N300/400 differences are indicative of changes in the cortical representation of objects. In two experiments, we recorded EEG signals, while participants viewed semantically consistent or inconsistent objects within a scene; in Experiment 1, these objects were task-irrelevant, while in Experiment 2, they were directly relevant for behavior. In both experiments, we found reliable and comparable N300/400 differences between consistent and inconsistent scene-object combinations. To probe the quality of object representations, we performed multivariate classification analyses, in which we decoded the category of the objects contained in the scene. In Experiment 1, in which the objects were not task-relevant, object category could be decoded from ~100 ms after the object presentation, but no difference in decoding performance was found between consistent and inconsistent objects. In contrast, when the objects were task-relevant in Experiment 2, we found enhanced decoding of semantically consistent, compared with semantically inconsistent, objects. These results show that differences in N300/400 components related to scene-object consistency do not index changes in cortical object representations but rather reflect a generic marker of semantic violations. Furthermore, our findings suggest that facilitatory effects between objects and scenes are task-dependent rather than automatic.

Examining the Coding Strength of Object Identity and Nonidentity Features in Human Occipito-Temporal Cortex and Convolutional Neural Networks.

A visual object is characterized by multiple visual features, including its identity, position and size. Despite the usefulness of identity and nonidentity features in vision and their joint coding throughout the primate ventral visual processing pathway, they have so far been studied relatively independently. Here in both female and male human participants, the coding of identity and nonidentity features was examined together across the human ventral visual pathway. The nonidentity features tested included two Euclidean features (position and size) and two non-Euclidean features (image statistics and spatial frequency (SF) content of an image). Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with identity outweighing the non-Euclidean but not the Euclidean features at higher levels of visual processing. In 14 convolutional neural networks (CNNs) pretrained for object categorization with varying architecture, depth, and with/without recurrent processing, nonidentity feature representation showed an initial large increase from early to mid-stage of processing, followed by a decrease at later stages of processing, different from brain responses. Additionally, from lower to higher levels of visual processing, position became more underrepresented and image statistics and SF became more overrepresented compared with identity in CNNs than in the human brain. Similar results were obtained in a CNN trained with stylized images that emphasized shape representations. Overall, by measuring the coding strength of object identity and nonidentity features together, our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.SIGNIFICANCE STATEMENT This study examined the coding strength of object identity and four types of nonidentity features along the human ventral visual processing pathway and compared brain responses with those of 14 convolutional neural networks (CNNs) pretrained to perform object categorization. Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with some notable differences among the different nonidentity features. CNNs differed from the brain in a number of aspects in their representations of identity and nonidentity features over the course of visual processing. Our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.

Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks.

Forming transformation-tolerant object representations is critical to high-level primate vision. Despite its significance, many details of tolerance in the human brain remain unknown. Likewise, despite the ability of convolutional neural networks (CNNs) to exhibit human-like object categorization performance, whether CNNs form tolerance similar to that of the human brain is unknown. Here we provide the first comprehensive documentation and comparison of three tolerance measures in the human brain and CNNs. We measured fMRI responses from human ventral visual areas to real-world objects across both Euclidean and non-Euclidean feature changes. In single fMRI voxels in higher visual areas, we observed robust object response rank-order preservation across feature changes. This is indicative of functional smoothness in tolerance at the fMRI meso-scale level that has never been reported before. At the voxel population level, we found highly consistent object representational structure across feature changes towards the end of ventral processing. Rank-order preservation, consistency, and a third tolerance measure, cross-decoding success (i.e., a linear classifier's ability to generalize performance across feature changes) showed an overall tight coupling. These tolerance measures were in general lower for Euclidean than non-Euclidean feature changes in lower visual areas, but increased over the course of ventral processing for all feature changes. These characteristics of tolerance, however, were absent in eight CNNs pretrained with ImageNet images with varying network architecture, depth, the presence/absence of recurrent processing, or whether a network was pretrained with the original or stylized ImageNet images that encouraged shape processing. CNNs do not appear to develop the same kind of tolerance as the human brain over the course of visual processing.

Visual and Semantic Representations Predict Subsequent Memory in Perceptual and Conceptual Memory Tests.

It is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used functional magnetic resonance imaging (fMRI) and representational similarity analysis to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative observed ("is round"), taxonomic ("is a fruit"), and encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later perceptual memory, conceptual memory, or both (general memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent perceptual memory in visual cortices, but also facilitated conceptual and general memory in more anterior regions. Semantic representations, in turn, predicted perceptual memory in visual cortex, conceptual memory in the perirhinal and inferior prefrontal cortex, and general memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

The contribution of object size, manipulability, and stability on neural responses to inanimate objects.

In human occipitotemporal cortex, brain responses to depicted inanimate objects have a large-scale organization by real-world object size. Critically, the size of objects in the world is systematically related to behaviorally-relevant properties: small objects are often grasped and manipulated (e.g., forks), while large objects tend to be less motor-relevant (e.g., tables), though this relationship does not always have to be true (e.g., picture frames and wheelbarrows). To determine how these two dimensions interact, we measured brain activity with functional magnetic resonance imaging while participants viewed a stimulus set of small and large objects with either low or high motor-relevance. The results revealed that the size organization was evident for objects with both low and high motor-relevance; further, a motor-relevance map was also evident across both large and small objects. Targeted contrasts revealed that typical combinations (small motor-relevant vs. large non-motor-relevant) yielded more robust topographies than the atypical covariance contrast (small non-motor-relevant vs. large motor-relevant). In subsequent exploratory analyses, a factor analysis revealed that the construct of motor-relevance was better explained by two underlying factors: one more related to manipulability, and the other to whether an object moves or is stable. The factor related to manipulability better explained responses in lateral small-object preferring regions, while the factor related to object stability (lack of movement) better explained responses in ventromedial large-object preferring regions. Taken together, these results reveal that the structure of neural responses to objects of different sizes further reflect behavior-relevant properties of manipulability and stability, and contribute to a deeper understanding of some of the factors that help the large-scale organization of object representation in high-level visual cortex.

Experience Transforms Conjunctive Object Representations: Neural Evidence for Unitization After Visual Expertise.

Certain transformations must occur within the brain to allow rapid processing of familiar experiences. Complex objects are thought to become unitized, whereby multifeature conjunctions are retrieved as rapidly as a single feature. Behavioral studies strongly support unitization theory, but a compelling neural mechanism is lacking. Here, we examined how unitization transforms conjunctive representations to become more "feature-like" by recruiting posterior regions of the ventral visual stream (VVS) whose architecture is specialized for processing single features. We used functional magnetic resonance imaging to scan humans before and after visual training with novel objects. We implemented a novel multivoxel pattern analysis to measure a conjunctive code, which represented a conjunction of object features above and beyond the sum of the parts. Importantly, a multivoxel searchlight showed that the strength of conjunctive coding in posterior VVS increased posttraining. Furthermore, multidimensional scaling revealed representational separation at the level of individual features in parallel to the changes at the level of feature conjunctions. Finally, functional connectivity between anterior and posterior VVS was higher for novel objects than for trained objects, consistent with early involvement of anterior VVS in unitizing feature conjunctions in response to novelty. These data demonstrate that the brain implements unitization as a mechanism to refine complex object representations over the course of multiple learning experiences.

Canonical Size in Haptic Drawings.

This study aimed to test differences in drawn size of familiar objects of different physical size in haptic drawings produced by blindfolded sighted participants. Using two sizes of the foil sheets on which they made convex drawings, they drew one object per foil. The results showed that the size of drawings increased linearly with the rising rank of real-world size. Although larger drawings were created on larger foils than on smaller ones, the ratio of the object drawn size within the foil sheet size did not differ across foil sizes. Hence, canonical size-a phenomenon known so far from studies on the visual domain-revealed here in a task performed in the haptic domain.

Pedestrian Detection by Novel Axis-Line Representation and Regression Pattern.

The pattern of bounding box representation and regression has long been dominant in CNN-based pedestrian detectors. Despite the method's success, it cannot accurately represent location, and introduces unnecessary background information, while pedestrian features are mainly located in axis-line areas. Other object representations, such as corner-pairs, are not easy to obtain by regression because the corners are far from the axis-line and are greatly affected by background features. In this paper, we propose a novel detection pattern, named Axis-line Representation and Regression (ALR), for pedestrian detection in road scenes. Specifically, we design a 3-d axis-line representation for pedestrians and use it as the regression target during network training. A line-box transformation method is also proposed to fit the widely used box-annotations. Meanwhile, we explore the influence of deformable convolution base-offset on detection performance and propose a base-offset initialization strategy to further promote the gain brought by ALR. Notably, the proposed ALR pattern can be introduced into both anchor-based and anchor-free frameworks. We validate the effectiveness of ALR on the Caltech-USA and CityPersons datasets. Experimental results show that our approach outperforms the baseline significantly through simple modifications and achieves competitive accuracy with other methods without bells and whistles.

Predictive coding in auditory perception: challenges and unresolved questions.

Predictive coding is arguably the currently dominant theoretical framework for the study of perception. It has been employed to explain important auditory perceptual phenomena, and it has inspired theoretical, experimental and computational modelling efforts aimed at describing how the auditory system parses the complex sound input into meaningful units (auditory scene analysis). These efforts have uncovered some vital questions, addressing which could help to further specify predictive coding and clarify some of its basic assumptions. The goal of the current review is to motivate these questions and show how unresolved issues in explaining some auditory phenomena lead to general questions of the theoretical framework. We focus on experimental and computational modelling issues related to sequential grouping in auditory scene analysis (auditory pattern detection and bistable perception), as we believe that this is the research topic where predictive coding has the highest potential for advancing our understanding. In addition to specific questions, our analysis led us to identify three more general questions that require further clarification: (1) What exactly is meant by prediction in predictive coding? (2) What governs which generative models make the predictions? and (3) What (if it exists) is the correlate of perceptual experience within the predictive coding framework?

An Information-Driven 2-Pathway Characterization of Occipitotemporal and Posterior Parietal Visual Object Representations.

Recent studies have demonstrated the existence of rich visual representations in both occipitotemporal cortex (OTC) and posterior parietal cortex (PPC). Using fMRI decoding and a bottom-up data-driven approach, we showed that although robust object category representations exist in both OTC and PPC, there is an information-driven 2-pathway separation among these regions in the representational space, with occipitotemporal regions arranging hierarchically along 1 pathway and posterior parietal regions along another pathway. We obtained 10 independent replications of this 2-pathway distinction, accounting for 58-81% of the total variance of the region-wise differences in visual representation. The separation of the PPC regions from higher occipitotemporal regions was not driven by a difference in tolerance to changes in low-level visual features, did not rely on the presence of special object categories, and was present whether or not object category was task relevant. Our information-driven 2-pathway structure differs from the well-known ventral-what and dorsal-where/how characterization of posterior brain regions. Here both pathways contain rich nonspatial visual representations. The separation we see likely reflects a difference in neural coding scheme used by PPC to represent visual information compared with that of OTC.

Conceptually Rich, Perceptually Sparse: Object Representations in 6-Month-Old Infants' Working Memory.

Six-month-old infants can store representations of multiple objects in working memory but do not always remember the objects' features (e.g., shape). Here, we asked whether infants' object representations (a) may contain conceptual content and (b) may contain this content even if perceptual features are forgotten. We hid two conceptually distinct objects (a humanlike doll and a nonhuman ball) one at a time in two separate locations and then tested infants' memory for the first-hidden object by revealing either the original hidden object or an unexpected other object. Using looking time, we found that infants remembered the categorical identity of the hidden object but failed to remember its perceptual identity. Our results suggest that young infants may encode conceptual category in a representation of an occluded object, even when perceptual features are lost.

Children's scale errors: A by-product of lexical development?

Scale errors occur when young children seriously attempt to perform an action on an object which is impossible due to its size. Children vary substantially in the incidence of scale errors with many factors potentially contributing to these differences, such as age and the type of scale errors. In particular, the evidence for an inverted U-shaped curve of scale errors involving the child's body (i.e., body scale errors), which would point to a developmental stage, is mixed. Here we re-examine how body scale errors vary with age and explore the possibility that these errors would be related to the size and properties of children's lexicon. A large sample of children aged 18-30 months (N = 125) was tested in a scale error elicitation situation. Additionally, parental questionnaires were collected to assess children's receptive and expressive lexicon. Our key findings are that scale errors linearly decrease with age in childhood, and are more likely to be found in early talkers rather than in less advanced ones. This suggests that scale errors do not correspond to a developmental stage, and that one determinant of these errors is the speed of development of the linguistic and conceptual system, as a potential explanation for the individual variability in prevalence.

Goal-Directed Visual Processing Differentially Impacts Human Ventral and Dorsal Visual Representations.

Recent studies have challenged the ventral/"what" and dorsal/"where" two-visual-processing-pathway view by showing the existence of "what" and "where" information in both pathways. Is the two-pathway distinction still valid? Here, we examined how goal-directed visual information processing may differentially impact visual representations in these two pathways. Using fMRI and multivariate pattern analysis, in three experiments on human participants (57% females), by manipulating whether color or shape was task-relevant and how they were conjoined, we examined shape-based object category decoding in occipitotemporal and parietal regions. We found that object category representations in all the regions examined were influenced by whether or not object shape was task-relevant. This task effect, however, tended to decrease as task-relevant and irrelevant features were more integrated, reflecting the well-known object-based feature encoding. Interestingly, task relevance played a relatively minor role in driving the representational structures of early visual and ventral object regions. They were driven predominantly by variations in object shapes. In contrast, the effect of task was much greater in dorsal than ventral regions, with object category and task relevance both contributing significantly to the representational structures of the dorsal regions. These results showed that, whereas visual representations in the ventral pathway are more invariant and reflect "what an object is," those in the dorsal pathway are more adaptive and reflect "what we do with it." Thus, despite the existence of "what" and "where" information in both visual processing pathways, the two pathways may still differ fundamentally in their roles in visual information representation.SIGNIFICANCE STATEMENT Visual information is thought to be processed in two distinctive pathways: the ventral pathway that processes "what" an object is and the dorsal pathway that processes "where" it is located. This view has been challenged by recent studies revealing the existence of "what" and "where" information in both pathways. Here, we found that goal-directed visual information processing differentially modulates shape-based object category representations in the two pathways. Whereas ventral representations are more invariant to the demand of the task, reflecting what an object is, dorsal representations are more adaptive, reflecting what we do with the object. Thus, despite the existence of "what" and "where" information in both pathways, visual representations may still differ fundamentally in the two pathways.

Decoding information about dynamically occluded objects in visual cortex.

During dynamic occlusion, an object passes behind an occluding surface and then later reappears. Even when completely occluded from view, such objects are experienced as continuing to exist or persist behind the occluder even though they are no longer visible. The contents and neural basis of this persistent representation remain poorly understood. Questions remain as to whether there is information maintained about the object itself (i.e. its shape or identity) or non-object-specific information such as its position or velocity as it is tracked behind an occluder, as well as which areas of visual cortex represent such information. Recent studies have found that early visual cortex is activated by "invisible" objects during visual imagery and by unstimulated regions along the path of apparent motion, suggesting that some properties of dynamically occluded objects may also be neurally represented in early visual cortex. We applied functional magnetic resonance imaging in human subjects to examine representations within visual cortex during dynamic occlusion. For gradually occluded, but not for instantly disappearing objects, there was an increase in activity in early visual cortex (V1, V2, and V3). This activity was spatially-specific, corresponding to the occluded location in the visual field. However, the activity did not encode enough information about object identity to discriminate between different kinds of occluded objects (circles vs. stars) using MVPA. In contrast, object identity could be decoded in spatially-specific subregions of higher-order, topographically organized areas such as ventral, lateral, and temporal occipital areas (VO, LO, and TO) as well as the functionally defined LOC and hMT+. These results suggest that early visual cortex may only represent the dynamically occluded object's position or motion path, while later visual areas represent object-specific information.

Trial timing and pattern-information analyses of fMRI data.

Pattern-information approaches to fMRI data analysis are becoming increasingly popular but few studies to date have investigated experimental design optimization for these analyses. Here, we tested several designs that varied in the number of trials and trial timing within fixed duration scans while participants encoded images of animals and tools. Trial timing conditions with fixed onset-to-onset timing ranged from slow 12-s trials with two repetitions of each item to quick 6-s trials with four repetitions per item. We also tested a jittered version of the quick design with 4-8s trials. We assessed the effect of trial timing on three dependent measures: category-level (animals vs. tools) decoding accuracy using a multivoxel pattern analysis, item-level (e.g., cat vs. dog vs. lion) information estimates using pattern similarity analysis, and memory effects comparing pattern similarity scores across repetitions of individual items subsequently remembered vs. forgotten. For single trial estimates, category decoding was equal across all trial timing conditions while item-level information and memory effects were better detected using slow trial timing. When modeling events on an item-by-item basis across all repetitions of a given item, a larger number of quick, regularly spaced trials provided an advantage over fewer slow trials for category decoding while item-level information was comparable across conditions. Jittered and non-jittered versions of the quick trial timing did not differ significantly in any analysis. These results will help inform experimental design choices in future studies planning to employ pattern-information analyses and demonstrate that design optimization guidelines developed for univariate analyses of a few conditions are not necessarily optimal for pattern-information analyses and condition-rich designs.

Neural Representations of Natural and Scrambled Movies Progressively Change from Rat Striate to Temporal Cortex.

In recent years, the rodent has come forward as a candidate model for investigating higher level visual abilities such as object vision. This view has been backed up substantially by evidence from behavioral studies that show rats can be trained to express visual object recognition and categorization capabilities. However, almost no studies have investigated the functional properties of rodent extrastriate visual cortex using stimuli that target object vision, leaving a gap compared with the primate literature. Therefore, we recorded single-neuron responses along a proposed ventral pathway in rat visual cortex to investigate hallmarks of primate neural object representations such as preference for intact versus scrambled stimuli and category-selectivity. We presented natural movies containing a rat or no rat as well as their phase-scrambled versions. Population analyses showed increased dissociation in representations of natural versus scrambled stimuli along the targeted stream, but without a clear preference for natural stimuli. Along the measured cortical hierarchy the neural response seemed to be driven increasingly by features that are not V1-like and destroyed by phase-scrambling. However, there was no evidence for category selectivity for the rat versus nonrat distinction. Together, these findings provide insights about differences and commonalities between rodent and primate visual cortex.

Distributed representation of visual objects by single neurons in the human brain.

It remains unclear how single neurons in the human brain represent whole-object visual stimuli. While recordings in both human and nonhuman primates have shown distributed representations of objects (many neurons encoding multiple objects), recordings of single neurons in the human medial temporal lobe, taken as subjects' discriminated objects during multiple presentations, have shown gnostic representations (single neurons encoding one object). Because some studies suggest that repeated viewing may enhance neural selectivity for objects, we had human subjects discriminate objects in a single, more naturalistic viewing session. We found that, across 432 well isolated neurons recorded in the hippocampus and amygdala, the average fraction of objects encoded was 26%. We also found that more neurons encoded several objects versus only one object in the hippocampus (28 vs 18%, p < 0.001) and in the amygdala (30 vs 19%, p < 0.001). Thus, during realistic viewing experiences, typical neurons in the human medial temporal lobe code for a considerable range of objects, across multiple semantic categories.

Population responses in V1 encode different figures by response amplitude.

The visual system simultaneously segregates between several objects presented in a visual scene. The neural code for encoding different objects or figures is not well understood. To study this question, we trained two monkeys to discriminate whether two elongated bars are either separate, thus generating two different figures, or connected, thus generating a single figure. Using voltage-sensitive dyes, we imaged at high spatial and temporal resolution V1 population responses evoked by the two bars, while keeping their local attributes similar among the two conditions. In the separate condition, unlike the connected condition, the population response to one bar is enhanced, whereas the response to the other is simultaneously suppressed. The response to the background remained unchanged between the two conditions. This divergent pattern developed ∼200 ms poststimulus onset and could discriminate well between the separate and connected single trials. The stimulus separation saliency and behavioral report were highly correlated with the differential response to the bars. In addition, the proximity and/or the specific location of the connectors seemed to have only a weak effect on this late activity pattern, further supporting the involvement of top-down influences. Additional neural codes were less informative about the separate and connected conditions, with much less consistency and discriminability compared with a response amplitude code. We suggest that V1 is involved in the encoding of each figure by different neuronal response amplitude, which can mediate their segregation and perception.

Neural signatures for sustaining object representations attributed to others in preverbal human infants.

A major feat of social beings is to encode what their conspecifics see, know or believe. While various non-human animals show precursors of these abilities, humans perform uniquely sophisticated inferences about other people's mental states. However, it is still unclear how these possibly human-specific capacities develop and whether preverbal infants, similarly to adults, form representations of other agents' mental states, specifically metarepresentations. We explored the neurocognitive bases of eight-month-olds' ability to encode the world from another person's perspective, using gamma-band electroencephalographic activity over the temporal lobes, an established neural signature for sustained object representation after occlusion. We observed such gamma-band activity when an object was occluded from the infants' perspective, as well as when it was occluded only from the other person (study 1), and also when subsequently the object disappeared, but the person falsely believed the object to be present (study 2). These findings suggest that the cognitive systems involved in representing the world from infants' own perspective are also recruited for encoding others' beliefs. Such results point to an early-developing, powerful apparatus suitable to deal with multiple concurrent representations, and suggest that infants can have a metarepresentational understanding of other minds even before the onset of language.