Pseudosparse neural coding in the visual system of primates.

When measuring sparseness in neural populations as an indicator of efficient coding, an implicit assumption is that each stimulus activates a different random set of neurons. In other words, population responses to different stimuli are, on average, uncorrelated. Here we examine neurophysiological data from four lobes of macaque monkey cortex, including V1, V2, MT, anterior inferotemporal cortex, lateral intraparietal cortex, the frontal eye fields, and perirhinal cortex, to determine how correlated population responses are. We call the mean correlation the pseudosparseness index, because high pseudosparseness can mimic statistical properties of sparseness without being authentically sparse. In every data set we find high levels of pseudosparseness ranging from 0.59-0.98, substantially greater than the value of 0.00 for authentic sparseness. This was true for synthetic and natural stimuli, as well as for single-electrode and multielectrode data. A model indicates that a key variable producing high pseudosparseness is the standard deviation of spontaneous activity across the population. Consistently high values of pseudosparseness in the data demand reconsideration of the sparse coding literature as well as consideration of the degree to which authentic sparseness provides a useful framework for understanding neural coding in the cortex.

Representation of shape, space, and attention in monkey cortex.

Attentional deficits are core to numerous developmental, neurological, and psychiatric disorders. At the single-cell level, much knowledge has been garnered from studies of shape and spatial properties, as well as from numerous demonstrations of attentional modulation of those properties. Despite this wealth of knowledge of single-cell responses across many brain regions, little is known about how these cellular characteristics relate to population level representations and how such representations relate to behavior; in particular, how these cellular responses relate to the representation of shape, space, and attention, and how these representations differ across cortical areas and streams. Here we will emphasize the role of population coding as a missing link for connecting single-cell properties with behavior. Using a data-driven intrinsic approach to population decoding, we show that both 'what' and 'where' cortical visual streams encode shape, space, and attention, yet demonstrate striking differences in these representations. We suggest that both pathways fully process shape and space, but that differences in representation may arise due to their differing functions and input and output constraints. Moreover, differences in the effects of attention on shape and spatial population representations in the two visual streams suggest two distinct strategies: in a ventral area, attention or task demands modulate the population representations themselves (perhaps to expand or enhance one part at the expense of other parts) while in a dorsal area, at a population representation level, attention effects are weak and nearly non-existent, perhaps in order to maintain veridical representations needed for visuomotor control. We show that an intrinsic approach, as opposed to theory-driven and labeled approaches, is useful for understanding how representations develop and differ across brain regions. Most importantly, these approaches help link cellular properties more tightly with behavior, a much-needed step to better understand and interpret cellular findings and key to providing insights to improve interventions in human disorders.

Face Representations via Tensorfaces of Various Complexities.

Neurons selective for faces exist in humans and monkeys. However, characteristics of face cell receptive fields are poorly understood. In this theoretical study, we explore the effects of complexity, defined as algorithmic information (Kolmogorov complexity) and logical depth, on possible ways that face cells may be organized. We use tensor decompositions to decompose faces into a set of components, called tensorfaces, and their associated weights, which can be interpreted as model face cells and their firing rates. These tensorfaces form a high-dimensional representation space in which each tensorface forms an axis of the space. A distinctive feature of the decomposition algorithm is the ability to specify tensorface complexity. We found that low-complexity tensorfaces have blob-like appearances crudely approximating faces, while high-complexity tensorfaces appear clearly face-like. Low-complexity tensorfaces require a larger population to reach a criterion face reconstruction error than medium- or high-complexity tensorfaces, and thus are inefficient by that criterion. Low-complexity tensorfaces, however, generalize better when representing statistically novel faces, which are faces falling beyond the distribution of face description parameters found in the tensorface training set. The degree to which face representations are parts based or global forms a continuum as a function of tensorface complexity, with low and medium tensorfaces being more parts based. Given the computational load imposed in creating high-complexity face cells (in the form of algorithmic information and logical depth) and in the absence of a compelling advantage to using high-complexity cells, we suggest face representations consist of a mixture of low- and medium-complexity face cells.

Extrinsic and intrinsic representations.

We extend the discussion in the target article about distinctions between extrinsic coding (external references to known things, as required by information theory) and the alternative we and the target article both favor, intrinsic coding (internal relationships within sensory and motor signals). Central to our thinking about intrinsic coding is population coding and the concept of high-dimensional neural response spaces.

Attention Effects on Neural Population Representations for Shape and Location Are Stronger in the Ventral than Dorsal Stream.

We examined how attention causes neural population representations of shape and location to change in ventral stream (AIT) and dorsal stream (LIP). Monkeys performed two identical delayed-match-to-sample (DMTS) tasks, attending either to shape or location. In AIT, shapes were more discriminable when directing attention to shape rather than location, measured by an increase in mean distance between population response vectors. In LIP, attending to location rather than shape did not increase the discriminability of different stimulus locations. Even when factoring out the change in mean vector response distance, multidimensional scaling (MDS) still showed a significant task difference in AIT, but not LIP, indicating that beyond increasing discriminability, attention also causes a nonlinear warping of representation space in AIT. Despite single-cell attentional modulations in both areas, our data show that attentional modulations of population representations are weaker in LIP, likely due to a need to maintain veridical representations for visuomotor control.

Towards building a more complex view of the lateral geniculate nucleus: Recent advances in understanding its role.

The lateral geniculate nucleus (LGN) has often been treated in the past as a linear filter that adds little to retinal processing of visual inputs. Here we review anatomical, neurophysiological, brain imaging, and modeling studies that have in recent years built up a much more complex view of LGN. These include effects related to nonlinear dendritic processing, cortical feedback, synchrony and oscillations across LGN populations, as well as involvement of LGN in higher level cognitive processing. Although recent studies have provided valuable insights into early visual processing including the role of LGN, a unified model of LGN responses to real-world objects has not yet been developed. In the light of recent data, we suggest that the role of LGN deserves more careful consideration in developing models of high-level visual processing.

Neural representation for object recognition in inferotemporal cortex.

We suggest that population representation of objects in inferotemporal cortex lie on a continuum between a purely structural, parts-based description and a purely holistic description. The intrinsic dimensionality of object representation is estimated to be around 100, perhaps with lower dimensionalities for object representations more toward the holistic end of the spectrum. Cognitive knowledge in the form of semantic information and task information feed back to inferotemporal cortex from perirhinal and prefrontal cortex respectively, providing high-level multimodal-based expectations that assist in the interpretation of object stimuli. Integration of object information across eye movements may also contribute to object recognition through a process of active vision.

Characteristics of Eye-Position Gain Field Populations Determine Geometry of Visual Space.

We have previously demonstrated differences in eye-position spatial maps for anterior inferotemporal cortex (AIT) in the ventral stream and lateral intraparietal cortex (LIP) in the dorsal stream, based on population decoding of gaze angle modulations of neural visual responses (i.e., eye-position gain fields). Here we explore the basis of such spatial encoding differences through modeling of gain field characteristics. We created a population of model neurons, each having a different eye-position gain field. This population was used to reconstruct eye-position visual space using multidimensional scaling. As gain field shapes have never been well-established experimentally, we examined different functions, including planar, sigmoidal, elliptical, hyperbolic, and mixtures of those functions. All functions successfully recovered positions, indicating weak constraints on allowable gain field shapes. We then used a genetic algorithm to modify the characteristics of model gain field populations until the recovered spatial maps closely matched those derived from monkey neurophysiological data in AIT and LIP. The primary differences found between model AIT and LIP gain fields were that AIT gain fields were more foveally dominated. That is, gain fields in AIT operated on smaller spatial scales and smaller dispersions than in LIP. Thus, we show that the geometry of eye-position visual space depends on the population characteristics of gain fields, and that differences in gain field characteristics for different cortical areas may underlie differences in the representation of space.

Dimensionality of object representations in monkey inferotemporal cortex.

We have calculated the intrinsic dimensionality of visual object representations in anterior inferotemporal (AIT) cortex, based on responses of a large sample of cells stimulated with photographs of diverse objects. Because dimensionality was dependent on data set size, we determined asymptotic dimensionality as both the number of neurons and number of stimulus image approached infinity. Our final dimensionality estimate was 93 (SD: ± 11), indicating that there is basis set of approximately 100 independent features that characterize the dimensions of neural object space. We believe this is the first estimate of the dimensionality of neural visual representations based on single-cell neurophysiological data. The dimensionality of AIT object representations was much lower than the dimensionality of the stimuli. We suggest that there may be a gradual reduction in the dimensionality of object representations in neural populations going from retina to inferotemporal cortex as receptive fields become increasingly complex.

Recovering stimulus locations using populations of eye-position modulated neurons in dorsal and ventral visual streams of non-human primates.

We recorded visual responses while monkeys fixated the same target at different gaze angles, both dorsally (lateral intraparietal cortex, LIP) and ventrally (anterior inferotemporal cortex, AIT). While eye-position modulations occurred in both areas, they were both more frequent and stronger in LIP neurons. We used an intrinsic population decoding technique, multidimensional scaling (MDS), to recover eye positions, equivalent to recovering fixated target locations. We report that eye-position based visual space in LIP was more accurate (i.e., metric). Nevertheless, the AIT spatial representation remained largely topologically correct, perhaps indicative of a categorical spatial representation (i.e., a qualitative description such as "left of" or "above" as opposed to a quantitative, metrically precise description). Additionally, we developed a simple neural model of eye position signals and illustrate that differences in single cell characteristics can influence the ability to recover target position in a population of cells. We demonstrate for the first time that the ventral stream contains sufficient information for constructing an eye-position based spatial representation. Furthermore we demonstrate, in dorsal and ventral streams as well as modeling, that target locations can be extracted directly from eye position signals in cortical visual responses without computing coordinate transforms of visual space.

Monkeys in space: primate neural data suggest volumetric representations.

The target article does not consider neural data on primate spatial representations, which we suggest provide grounds for believing that navigational space may be three-dimensional rather than quasi-two-dimensional. Furthermore, we question the authors' interpretation of rat neurophysiological data as indicating that the vertical dimension may be encoded in a neural structure separate from the two horizontal dimensions.

Population coding and the labeling problem: extrinsic versus intrinsic representations.

Current population coding methods, including weighted averaging and Bayesian estimation, are based on extrinsic representations. These require that neurons be labeled with response parameters, such as tuning curve peaks or noise distributions, which are tied to some external, world-based metric scale. Firing rates alone, without this external labeling, are insufficient to represent a variable. However, the extrinsic approach does not explain how such neural labeling is implemented. A radically different and perhaps more physiological approach is based on intrinsic representations, which have access only to firing rates. Because neurons are unlabeled, intrinsic coding represents relative, rather than absolute, values of a variable. We show that intrinsic coding has representational advantages, including invariance, categorization, and discrimination, and in certain situations it may also recover absolute stimulus values.

Unmixing binocular signals.

Incompatible images presented to the two eyes lead to perceptual oscillations in which one image at a time is visible. Early models portrayed this binocular rivalry as involving reciprocal inhibition between monocular representations of images, occurring at an early visual stage prior to binocular mixing. However, psychophysical experiments found conditions where rivalry could also occur at a higher, more abstract level of representation. In those cases, the rivalry was between image representations dissociated from eye-of-origin information, rather than between monocular representations from the two eyes. Moreover, neurophysiological recordings found the strongest rivalry correlate in inferotemporal cortex, a high-level, predominantly binocular visual area involved in object recognition, rather than early visual structures. An unresolved issue is how can the separate identities of the two images be maintained after binocular mixing in order for rivalry to be possible at higher levels? Here we demonstrate that after the two images are mixed, they can be unmixed at any subsequent stage using a physiologically plausible non-linear signal-processing algorithm, non-negative matrix factorization, previously proposed for parsing object parts during object recognition. The possibility that unmixed left and right images can be regenerated at late stages within the visual system provides a mechanism for creating various binocular representations and interactions de novo in different cortical areas for different purposes, rather than inheriting then from early areas. This is a clear example how non-linear algorithms can lead to highly non-intuitive behavior in neural information processing.

Statistics of visual responses in primate inferotemporal cortex to object stimuli.

We have characterized selectivity and sparseness in anterior inferotemporal cortex, using a large data set. Responses were collected from 674 monkey inferotemporal cells, each stimulated by 806 object photographs. This 806 × 674 matrix was examined in two ways: columnwise, looking at responses of a single neuron to all images (single-neuron selectivity), and rowwise, looking at the responses of all neurons caused by a single image (population sparseness). Selectivity and sparseness were measured as kurtosis of probability distributions. Population sparseness exceeded single-neuron selectivity, with specific values dependent on the size of the data sample. This difference was principally caused by inclusion, within the population, of neurons with a variety of dynamic ranges (standard deviations of responses over all images). Statistics of large responses were examined by quantifying how quickly the upper tail of the probability distribution decreased (tail heaviness). This analysis demonstrated that population responses had heavier tails than single-neuron responses, consistent with the difference between sparseness and selectivity measurements. Population responses with spontaneous activity subtracted had the heaviest tails, following a power law. The very light tails of single-neuron responses indicate that the critical feature for each neuron is simple enough to have a high probability of occurring within a limited stimulus set. Heavy tails of population responses indicate that there are a large number of different critical features to which different neurons are tuned. These results are inconsistent with some structural models of object recognition that posit that objects are decomposed into a small number of standard features.

Population coding of visual space: comparison of spatial representations in dorsal and ventral pathways.

Although the representation of space is as fundamental to visual processing as the representation of shape, it has received relatively little attention from neurophysiological investigations. In this study we characterize representations of space within visual cortex, and examine how they differ in a first direct comparison between dorsal and ventral subdivisions of the visual pathways. Neural activities were recorded in anterior inferotemporal cortex (AIT) and lateral intraparietal cortex (LIP) of awake behaving monkeys, structures associated with the ventral and dorsal visual pathways respectively, as a stimulus was presented at different locations within the visual field. In spatially selective cells, we find greater modulation of cell responses in LIP with changes in stimulus position. Further, using a novel population-based statistical approach (namely, multidimensional scaling), we recover the spatial map implicit within activities of neural populations, allowing us to quantitatively compare the geometry of neural space with physical space. We show that a population of spatially selective LIP neurons, despite having large receptive fields, is able to almost perfectly reconstruct stimulus locations within a low-dimensional representation. In contrast, a population of AIT neurons, despite each cell being spatially selective, provide less accurate low-dimensional reconstructions of stimulus locations. They produce instead only a topologically (categorically) correct rendition of space, which nevertheless might be critical for object and scene recognition. Furthermore, we found that the spatial representation recovered from population activity shows greater translation invariance in LIP than in AIT. We suggest that LIP spatial representations may be dimensionally isomorphic with 3D physical space, while in AIT spatial representations may reflect a more categorical representation of space (e.g., "next to" or "above").

Population coding of visual space: modeling.

We examine how the representation of space is affected by receptive field (RF) characteristics of the encoding population. Spatial responses were defined by overlapping Gaussian RFs. These responses were analyzed using multidimensional scaling to extract the representation of global space implicit in population activity. Spatial representations were based purely on firing rates, which were not labeled with RF characteristics (tuning curve peak location, for example), differentiating this approach from many other population coding models. Because responses were unlabeled, this model represents space using intrinsic coding, extracting relative positions amongst stimuli, rather than extrinsic coding where known RF characteristics provide a reference frame for extracting absolute positions. Two parameters were particularly important: RF diameter and RF dispersion, where dispersion indicates how broadly RF centers are spread out from the fovea. For large RFs, the model was able to form metrically accurate representations of physical space on low-dimensional manifolds embedded within the high-dimensional neural population response space, suggesting that in some cases the neural representation of space may be dimensionally isomorphic with 3D physical space. Smaller RF sizes degraded and distorted the spatial representation, with the smallest RF sizes (present in early visual areas) being unable to recover even a topologically consistent rendition of space on low-dimensional manifolds. Finally, although positional invariance of stimulus responses has long been associated with large RFs in object recognition models, we found RF dispersion rather than RF diameter to be the critical parameter. In fact, at a population level, the modeling suggests that higher ventral stream areas with highly restricted RF dispersion would be unable to achieve positionally-invariant representations beyond this narrow region around fixation.

Decoding Poisson spike trains by Gaussian filtering.

The temporal waveform of neural activity is commonly estimated by low-pass filtering spike train data through convolution with a gaussian kernel. However, the criteria for selecting the gaussian width sigma are not well understood. Given an ensemble of Poisson spike trains generated by an instantaneous firing rate function lambda(t), the problem was to recover an optimal estimate of lambda(t) by gaussian filtering. We provide equations describing the optimal value of sigma using an error minimization criterion and examine how the optimal sigma varies within a parameter space defining the statistics of inhomogeneous Poisson spike trains. The process was studied both analytically and through simulations. The rate functions lambda(t) were randomly generated, with the three parameters defining spike statistics being the mean of lambda(t), the variance of lambda(t), and the exponent alpha of the Fourier amplitude spectrum 1/f(alpha) of lambda(t). The value of sigma(opt) followed a power law as a function of the pooled mean interspike interval I, sigma(opt) = aI(b), where a was inversely related to the coefficient of variation C(V) of lambda(t), and b was inversely related to the Fourier spectrum exponent alpha. Besides applications for data analysis, optimal recovery of an analog signal waveform lambda(t) from spike trains may also be useful in understanding neural signal processing in vivo.

Spatial modulation of primate inferotemporal responses by eye position.

BACKGROUND: A key aspect of representations for object recognition and scene analysis in the ventral visual stream is the spatial frame of reference, be it a viewer-centered, object-centered, or scene-based coordinate system. Coordinate transforms from retinocentric space to other reference frames involve combining neural visual responses with extraretinal postural information. METHODOLOGY/PRINCIPAL FINDINGS: We examined whether such spatial information is available to anterior inferotemporal (AIT) neurons in the macaque monkey by measuring the effect of eye position on responses to a set of simple 2D shapes. We report, for the first time, a significant eye position effect in over 40% of recorded neurons with small gaze angle shifts from central fixation. Although eye position modulates responses, it does not change shape selectivity. CONCLUSIONS/SIGNIFICANCE: These data demonstrate that spatial information is available in AIT for the representation of objects and scenes within a non-retinocentric frame of reference. More generally, the availability of spatial information in AIT calls into questions the classic dichotomy in visual processing that associates object shape processing with ventral structures such as AIT but places spatial processing in a separate anatomical stream projecting to dorsal structures.

Shape selectivity in primate frontal eye field.

Previous neurophysiological studies of the frontal eye field (FEF) in monkeys have focused on its role in saccade target selection and gaze shift control. It has been argued that FEF neurons indicate the locations of behaviorally significant visual stimuli and are not inherently sensitive to specific features of the visual stimuli per se. Here, for the first time, we directly examined single cell responses to simple, two-dimensional shapes and found that shape selectivity exists in a substantial number of FEF cells during a passive fixation task or during the sample, delay (memory), and eye movement periods in a delayed match to sample (DMTS) task. Our data demonstrate that FEF neurons show sensory and mnemonic selectivity for stimulus shape features whether or not they are behaviorally significant for the task at hand. We also investigated the extent and localization of activation in the FEF using a variety of shape stimuli defined by static or dynamic cues employing functional magentic resonance imaging (fMRI) in anesthetized and paralyzed monkeys. Our fMRI results support the electrophysiological findings by showing significant FEF activation for a variety of shape stimuli and cues in the absence of attentional and motor processing. This shape selectivity in FEF is comparable to previous reports in the ventral pathway, inviting a reconsideration of the functional organization of the visual system.

Comparison of shape encoding in primate dorsal and ventral visual pathways.

Ventral and dorsal visual pathways perform fundamentally different functions. The former is involved in object recognition, whereas the latter carries out spatial localization of stimuli and visual guidance of motor actions. Despite the association of the dorsal pathway with spatial vision, recent studies have reported shape selectivity in the dorsal stream. We compared shape encoding in anterior inferotemporal cortex (AIT), a high-level ventral area, with that in lateral intraparietal cortex (LIP), a high-level dorsal area, during a fixation task. We found shape selectivities of individual neurons to be greater in anterior inferotemporal cortex than in lateral intraparietal cortex. At the neural population level, responses to different shapes were more dissimilar in AIT than LIP. Both observations suggest a greater capacity in AIT for making finer shape distinctions. Multivariate analyses of AIT data grouped together similar shapes based on neural population responses, whereas such grouping was indistinct in LIP. Thus in a first comparison of shape response properties in late stages of the two visual pathways, we report that AIT exhibits greater capability than LIP for both object discrimination and generalization. These differences in the two visual pathways provide the first neurophysiological evidence that shape encoding in the dorsal pathway is distinct from and not a mere duplication of that formed in the ventral pathway. In addition to shape selectivity, we observed stimulus-driven cognitive effects in both areas. Stimulus repetition suppression in LIP was similar to the well-known repetition suppression in AIT and may be associated with the "inhibition of return" memory effect observed during reflexive attention.