Correlations and Neuronal Population Information.

Brain function involves the activity of neuronal populations. Much recent effort has been devoted to measuring the activity of neuronal populations in different parts of the brain under various experimental conditions. Population activity patterns contain rich structure, yet many studies have focused on measuring pairwise relationships between members of a larger population-termed noise correlations. Here we review recent progress in understanding how these correlations affect population information, how information should be quantified, and what mechanisms may give rise to correlations. As population coding theory has improved, it has made clear that some forms of correlation are more important for information than others. We argue that this is a critical lesson for those interested in neuronal population responses more generally: Descriptions of population responses should be motivated by and linked to well-specified function. Within this context, we offer suggestions of where current theoretical frameworks fall short.

Decision Signals in the Local Field Potentials of Early and Mid-Level Macaque Visual Cortex.

The neural basis of perceptual decision making has typically been studied using measurements of single neuron activity, though decisions are likely based on the activity of large neuronal ensembles. Local field potentials (LFPs) may, in some cases, serve as a useful proxy for population activity and thus be useful for understanding the neural basis of perceptual decision making. However, little is known about whether LFPs in sensory areas include decision-related signals. We therefore analyzed LFPs recorded using two 48-electrode arrays implanted in primary visual cortex (V1) and area V4 of macaque monkeys trained to perform a fine orientation discrimination task. We found significant choice information in low (0-30 Hz) and higher (70-500 Hz) frequency components of the LFP, but little information in gamma frequencies (30-70 Hz). Choice information was more robust in V4 than V1 and stronger in LFPs than in simultaneously measured spiking activity. LFP-based choice information included a global component, common across electrodes within an area. Our findings reveal the presence of robust choice-related signals in the LFPs recorded in V1 and V4 and suggest that LFPs may be a useful complement to spike-based analyses of decision making.

Limited Evidence for Sensory Prediction Error Responses in Visual Cortex of Macaques and Humans.

A recent formulation of predictive coding theory proposes that a subset of neurons in each cortical area encodes sensory prediction errors, the difference between predictions relayed from higher cortex and the sensory input. Here, we test for evidence of prediction error responses in spiking responses and local field potentials (LFP) recorded in primary visual cortex and area V4 of macaque monkeys, and in complementary electroencephalographic (EEG) scalp recordings in human participants. We presented a fixed sequence of visual stimuli on most trials, and violated the expected ordering on a small subset of trials. Under predictive coding theory, pattern-violating stimuli should trigger robust prediction errors, but we found that spiking, LFP and EEG responses to expected and pattern-violating stimuli were nearly identical. Our results challenge the assertion that a fundamental computational motif in sensory cortex is to signal prediction errors, at least those based on predictions derived from temporal patterns of visual stimulation.

Predicting Perceptual Decisions Using Visual Cortical Population Responses and Choice History.

Our understanding of the neural basis of perceptual decision making has been built in part on relating co-fluctuations of single neuron responses to perceptual decisions on a trial-by-trial basis. The strength of this relationship is often compared across neurons or brain areas, recorded in different sessions, animals, or variants of a task. We sought to extend our understanding of perceptual decision making in three ways. First, we measured neuronal activity simultaneously in early [primary visual cortex (V1)] and midlevel (V4) visual cortex while macaque monkeys performed a fine orientation discrimination perceptual task. This allowed a direct comparison of choice signals in these two areas, including their dynamics. Second, we asked how our ability to predict animals' decisions would be improved by considering small simultaneously-recorded neuronal populations rather than individual units. Finally, we asked whether predictions would be improved by taking into account the animals' choice and reward histories, which can strongly influence decision making. We found that responses of individual V4 neurons were weakly predictive of decisions, but only in a brief epoch between stimulus offset and the indication of choice. In V1, few neurons showed significant decision-related activity. Analysis of neuronal population responses revealed robust choice-related information in V4 and substantially weaker signals in V1. Including choice- and reward-history information improved performance further, particularly when the recorded populations contained little decision-related information. Our work shows the power of using neuronal populations and decision history when relating neuronal responses to the perceptual decisions they are thought to underlie.SIGNIFICANCE STATEMENT Decades of research has provided a rich description of how visual information is represented in the visual cortex. Yet how cortical responses relate to visual perception remains poorly understood. Here we relate fluctuations in small neuronal population responses, recorded simultaneously in primary visual cortex (V1) and area V4 of monkeys, to perceptual reports in an orientation discrimination task. Choice-related signals were robust in V4, particularly late in the behavioral trial, but not in V1. Models that include both neuronal responses and choice-history information were able to predict a substantial portion of decisions. Our work shows the power of integrating information across neurons and including decision history in relating neuronal responses to perceptual decisions.

Temporal Contingencies Determine Whether Adaptation Strengthens or Weakens Normalization.

A fundamental and nearly ubiquitous feature of sensory encoding is that neuronal responses are strongly influenced by recent experience, or adaptation. Theoretical and computational studies have proposed that many adaptation effects may result in part from changes in the strength of normalization signals. Normalization is a "canonical" computation in which a neuron's response is modulated (normalized) by the pooled activity of other neurons. Here, we test whether adaptation can alter the strength of cross-orientation suppression, or masking, a paradigmatic form of normalization evident in primary visual cortex (V1). We made extracellular recordings of V1 neurons in anesthetized male macaques and measured responses to plaid stimuli composed of two overlapping, orthogonal gratings before and after prolonged exposure to two distinct adapters. The first adapter was a plaid consisting of orthogonal gratings and led to stronger masking. The second adapter presented the same orthogonal gratings in an interleaved manner and led to weaker masking. The strength of adaptation's effects on masking depended on the orientation of the test stimuli relative to the orientation of the adapters, but was independent of neuronal orientation preference. Changes in masking could not be explained by altered neuronal responsivity. Our results suggest that normalization signals can be strengthened or weakened by adaptation depending on the temporal contingencies of the adapting stimuli. Our findings reveal an interplay between two widespread computations in cortical circuits, adaptation and normalization, that enables flexible adjustments to the structure of the environment, including the temporal relationships among sensory stimuli.SIGNIFICANCE STATEMENT Two fundamental features of sensory responses are that they are influenced by adaptation and that they are modulated by the activity of other nearby neurons via normalization. Our findings reveal a strong interaction between these two aspects of cortical computation. Specifically, we show that cross-orientation masking, a form of normalization, can be strengthened or weakened by adaptation depending on the temporal contingencies between sensory inputs. Our findings support theoretical proposals that some adaptation effects may involve altered normalization and offer a network-based explanation for how cortex adjusts to current sensory demands.

Stimulus-Driven Population Activity Patterns in Macaque Primary Visual Cortex.

Dimensionality reduction has been applied in various brain areas to study the activity of populations of neurons. To interpret the outputs of dimensionality reduction, it is important to first understand its outputs for brain areas for which the relationship between the stimulus and neural response is well characterized. Here, we applied principal component analysis (PCA) to trial-averaged neural responses in macaque primary visual cortex (V1) to study two fundamental, population-level questions. First, we characterized how neural complexity relates to stimulus complexity, where complexity is measured using relative comparisons of dimensionality. Second, we assessed the extent to which responses to different stimuli occupy similar dimensions of the population activity space using a novel statistical method. For comparison, we performed the same dimensionality reduction analyses on the activity of a recently-proposed V1 receptive field model and a deep convolutional neural network. Our results show that the dimensionality of the population response changes systematically with alterations in the properties and complexity of the visual stimulus.

Scaling Properties of Dimensionality Reduction for Neural Populations and Network Models.

Recent studies have applied dimensionality reduction methods to understand how the multi-dimensional structure of neural population activity gives rise to brain function. It is unclear, however, how the results obtained from dimensionality reduction generalize to recordings with larger numbers of neurons and trials or how these results relate to the underlying network structure. We address these questions by applying factor analysis to recordings in the visual cortex of non-human primates and to spiking network models that self-generate irregular activity through a balance of excitation and inhibition. We compared the scaling trends of two key outputs of dimensionality reduction-shared dimensionality and percent shared variance-with neuron and trial count. We found that the scaling properties of networks with non-clustered and clustered connectivity differed, and that the in vivo recordings were more consistent with the clustered network. Furthermore, recordings from tens of neurons were sufficient to identify the dominant modes of shared variability that generalize to larger portions of the network. These findings can help guide the interpretation of dimensionality reduction outputs in regimes of limited neuron and trial sampling and help relate these outputs to the underlying network structure.

Distinguishing Neural Adaptation and Predictive Coding Hypotheses in Auditory Change Detection.

The auditory mismatch negativity (MMN) component of event-related potentials (ERPs) has served as a neural index of auditory change detection. MMN is elicited by presentation of infrequent (deviant) sounds randomly interspersed among frequent (standard) sounds. Deviants elicit a larger negative deflection in the ERP waveform compared to the standard. There is considerable debate as to whether the neural mechanism of this change detection response is due to release from neural adaptation (neural adaptation hypothesis) or from a prediction error signal (predictive coding hypothesis). Previous studies have not been able to distinguish between these explanations because paradigms typically confound the two. The current study disambiguated effects of stimulus-specific adaptation from expectation violation using a unique stimulus design that compared expectation violation responses that did and did not involve stimulus change. The expectation violation response without the stimulus change differed in timing, scalp distribution, and attentional modulation from the more typical MMN response. There is insufficient evidence from the current study to suggest that the negative deflection elicited by the expectation violation alone includes the MMN. Thus, we offer a novel hypothesis that the expectation violation response reflects a fundamentally different neural substrate than that attributed to the canonical MMN.

Mechanisms for Rapid Adaptive Control of Motion Processing in Macaque Visual Cortex.

A key feature of neural networks is their ability to rapidly adjust their function, including signal gain and temporal dynamics, in response to changes in sensory inputs. These adjustments are thought to be important for optimizing the sensitivity of the system, yet their mechanisms remain poorly understood. We studied adaptive changes in temporal integration in direction-selective cells in macaque primary visual cortex, where specific hypotheses have been proposed to account for rapid adaptation. By independently stimulating direction-specific channels, we found that the control of temporal integration of motion at one direction was independent of motion signals driven at the orthogonal direction. We also found that individual neurons can simultaneously support two different profiles of temporal integration for motion in orthogonal directions. These findings rule out a broad range of adaptive mechanisms as being key to the control of temporal integration, including untuned normalization and nonlinearities of spike generation and somatic adaptation in the recorded direction-selective cells. Such mechanisms are too broadly tuned, or occur too far downstream, to explain the channel-specific and multiplexed temporal integration that we observe in single neurons. Instead, we are compelled to conclude that parallel processing pathways are involved, and we demonstrate one such circuit using a computer model. This solution allows processing in different direction/orientation channels to be separately optimized and is sensible given that, under typical motion conditions (e.g., translation or looming), speed on the retina is a function of the orientation of image components. SIGNIFICANCE STATEMENT: Many neurons in visual cortex are understood in terms of their spatial and temporal receptive fields. It is now known that the spatiotemporal integration underlying visual responses is not fixed but depends on the visual input. For example, neurons that respond selectively to motion direction integrate signals over a shorter time window when visual motion is fast and a longer window when motion is slow. We investigated the mechanisms underlying this useful adaptation by recording from neurons as they responded to stimuli moving in two different directions at different speeds. Computer simulations of our results enabled us to rule out several candidate theories in favor of a model that integrates across multiple parallel channels that operate at different time scales.

Measuring Fisher information accurately in correlated neural populations.

Neural responses are known to be variable. In order to understand how this neural variability constrains behavioral performance, we need to be able to measure the reliability with which a sensory stimulus is encoded in a given population. However, such measures are challenging for two reasons: First, they must take into account noise correlations which can have a large influence on reliability. Second, they need to be as efficient as possible, since the number of trials available in a set of neural recording is usually limited by experimental constraints. Traditionally, cross-validated decoding has been used as a reliability measure, but it only provides a lower bound on reliability and underestimates reliability substantially in small datasets. We show that, if the number of trials per condition is larger than the number of neurons, there is an alternative, direct estimate of reliability which consistently leads to smaller errors and is much faster to compute. The superior performance of the direct estimator is evident both for simulated data and for neuronal population recordings from macaque primary visual cortex. Furthermore we propose generalizations of the direct estimator which measure changes in stimulus encoding across conditions and the impact of correlations on encoding and decoding, typically denoted by Ishuffle and Idiag respectively.

Area MT encodes three-dimensional motion.

We use visual information to determine our dynamic relationship with other objects in a three-dimensional (3D) world. Despite decades of work on visual motion processing, it remains unclear how 3D directions-trajectories that include motion toward or away from the observer-are represented and processed in visual cortex. Area MT is heavily implicated in processing visual motion and depth, yet previous work has found little evidence for 3D direction sensitivity per se. Here we use a rich ensemble of binocular motion stimuli to reveal that most neurons in area MT of the anesthetized macaque encode 3D motion information. This tuning for 3D motion arises from multiple mechanisms, including different motion preferences in the two eyes and a nonlinear interaction of these signals when both eyes are stimulated. Using a novel method for functional binocular alignment, we were able to rule out contributions of static disparity tuning to the 3D motion tuning we observed. We propose that a primary function of MT is to encode 3D motion, critical for judging the movement of objects in dynamic real-world environments.

Correlations in V1 are reduced by stimulation outside the receptive field.

The trial-to-trial response variability of nearby cortical neurons is correlated. These correlations may strongly influence population coding performance. Numerous studies have shown that correlations can be dynamically modified by attention, adaptation, learning, and potent stimulus drive. However, the mechanisms that influence correlation strength remain poorly understood. Here we test whether correlations are influenced by presenting stimuli outside the classical receptive field (RF) of visual neurons, where they recruit a normalization signal termed surround suppression. We recorded simultaneously the activity of dozens of cells using microelectrode arrays implanted in the superficial layers of V1 in anesthetized, paralyzed macaque monkeys. We presented annular stimuli that encircled--but did not impinge upon--the RFs of the recorded cells. We found that these "extra-classical" stimuli reduced correlations in the absence of stimulation of the RF, closely resembling the decorrelating effects of stimulating the RFs directly. Our results suggest that normalization signals may be an important mechanism for modulating correlations.

No consistent relationship between gamma power and peak frequency in macaque primary visual cortex.

Neural activity in the gamma frequency range ("gamma") is elevated during active cognitive states. Gamma has been proposed to play an important role in cortical function, although this is debated. Understanding what function gamma might fulfill requires a better understanding of its properties and the mechanisms that generate it. Gamma is characterized by its spectral power and peak frequency, and variations in both parameters have been associated with changes in behavioral performance. Modeling studies suggest these properties are co-modulated, but this has not been established. To test the relationship between these properties, we measured local field potentials (LFPs) and neuronal spiking responses in primary visual cortex of anesthetized monkeys, for drifting sinusoidal gratings of different sizes, contrasts, orientations and masked with different levels of noise. We find that there is no fixed relationship between LFP gamma power and peak frequency, and neither is related to the strength of spiking activity. We propose a simple model that can account for the complex stimulus dependence we observe, and suggest that separate mechanisms determine gamma power and peak frequency.

Distinct effects of brief and prolonged adaptation on orientation tuning in primary visual cortex.

Recent stimulus history-adaptation-alters neuronal responses and perception. Previous electrophysiological and perceptual studies suggest that prolonged adaptation strengthens and makes more persistent the effects seen after briefer exposures. However, no systematic comparison has been made between the effects of adaptation lasting hundreds of milliseconds, which might arise during a single fixation, and the more prolonged adaptation typically used in imaging and perceptual studies. Here we determine how 0.4, 4, and 40 s of adaptation alters orientation tuning in primary visual cortex of anesthetized macaque monkeys, and how quickly responses recover after adapter offset. We measured responses to small (1.3°) and large (7.4°) gratings because previous work has shown that adaptation effects can depend on stimulus size. Adaptation with small gratings reduced responsivity and caused tuning to shift away from the adapter. These effects strengthened with more prolonged adaptation. For responses to large gratings, brief and prolonged adaptation produced indistinguishable effects on responsivity but caused opposite shifts in tuning preference. Recovery from adaptation was notably slower after prolonged adaptation, even when this did not induce stronger effects. We show that our results can be explained by an adaptation-induced weakening of surround suppression, the dynamics of this suppression, and differential effects of brief and prolonged adaptation across response epochs. Our findings show that effects do not simply scale with adaptation duration and suggest that distinct strategies exist for adjusting to moment-to-moment fluctuations in input and to more persistent visual stimuli.

Similar adaptation effects in primary visual cortex and area MT of the macaque monkey under matched stimulus conditions.

Recent stimulus history, or adaptation, can alter neuronal response properties. Adaptation effects have been characterized in a number of visually responsive structures, from the retina to higher visual cortex. However, it remains unclear whether adaptation effects across stages of the visual system take a similar form in response to a particular sensory event. This is because studies typically probe a single structure or cortical area, using a stimulus ensemble chosen to provide potent drive to the cells of interest. Here we adopt an alternative approach and compare adaptation effects in primary visual cortex (V1) and area MT using identical stimulus ensembles. Previous work has suggested these areas adjust to recent stimulus drive in distinct ways. We show that this is not the case: adaptation effects in V1 and MT can involve weak or strong loss of responsivity and shifts in neuronal preference toward or away from the adapter, depending on stimulus size and adaptation duration. For a particular stimulus size and adaptation duration, however, effects are similar in nature and magnitude in V1 and MT. We also show that adaptation effects in MT of awake animals depend strongly on stimulus size. Our results suggest that the strategies for adjusting to recent stimulus history depend more strongly on adaptation duration and stimulus size than on the cortical area. Moreover, they indicate that different levels of the visual system adapt similarly to recent sensory experience.

Laminar dependence of neuronal correlations in visual cortex.

Neuronal responses are correlated on a range of timescales. Correlations can affect population coding and may play an important role in cortical function. Correlations are known to depend on stimulus drive, behavioral context, and experience, but the mechanisms that determine their properties are poorly understood. Here we make use of the laminar organization of cortex, with its variations in sources of input, local circuit architecture, and neuronal properties, to test whether networks engaged in similar functions but with distinct properties generate different patterns of correlation. We find that slow timescale correlations are prominent in the superficial and deep layers of primary visual cortex (V1) of macaque monkeys, but near zero in the middle layers. Brief timescale correlation (synchrony), on the other hand, was slightly stronger in the middle layers of V1, although evident at most cortical depths. Laminar variations were also apparent in the power of the local field potential, with a complementary pattern for low frequency (<10 Hz) and gamma (30-50 Hz) power. Recordings in area V2 revealed a laminar dependence similar to V1 for synchrony, but slow timescale correlations were not different between the input layers and nearby locations. Our results reveal that cortical circuits in different laminae can generate remarkably different patterns of correlations, despite being tightly interconnected.

Fast coding of orientation in primary visual cortex.

Understanding how populations of neurons encode sensory information is a major goal of systems neuroscience. Attempts to answer this question have focused on responses measured over several hundred milliseconds, a duration much longer than that frequently used by animals to make decisions about the environment. How reliably sensory information is encoded on briefer time scales, and how best to extract this information, is unknown. Although it has been proposed that neuronal response latency provides a major cue for fast decisions in the visual system, this hypothesis has not been tested systematically and in a quantitative manner. Here we use a simple 'race to threshold' readout mechanism to quantify the information content of spike time latency of primary visual (V1) cortical cells to stimulus orientation. We find that many V1 cells show pronounced tuning of their spike latency to stimulus orientation and that almost as much information can be extracted from spike latencies as from firing rates measured over much longer durations. To extract this information, stimulus onset must be estimated accurately. We show that the responses of cells with weak tuning of spike latency can provide a reliable onset detector. We find that spike latency information can be pooled from a large neuronal population, provided that the decision threshold is scaled linearly with the population size, yielding a processing time of the order of a few tens of milliseconds. Our results provide a novel mechanism for extracting information from neuronal populations over the very brief time scales in which behavioral judgments must sometimes be made.

Adaptation improves performance on a visual search task.

Temporal context, or adaptation, profoundly affects visual perception. Despite the strength and prevalence of adaptation effects, their functional role in visual processing remains unclear. The effects of spatial context and their functional role are better understood: these effects highlight features that differ from their surroundings and determine stimulus salience. Similarities in the perceptual and physiological effects of spatial and temporal context raise the possibility that they serve similar functions. We therefore tested the possibility that adaptation can enhance stimulus salience. We measured the effects of prolonged (40 s) adaptation to a counterphase grating on performance in a search task in which targets were defined by an orientation offset relative to a background of distracters. We found that, for targets with small orientation offsets, adaptation reduced reaction times and decreased the number of saccades made to find targets. Our results provide evidence that adaptation may function to highlight features that differ from the temporal context in which they are embedded.

Stimulus selectivity and spatial coherence of gamma components of the local field potential.

The gamma frequencies of the local field potential (LFP) provide a physiological correlate for numerous perceptual and cognitive phenomena and have been proposed to play a role in cortical function. Understanding the spatial extent of gamma and its relationship to spiking activity is critical for interpreting this signal and elucidating its function, but previous studies have provided widely disparate views of these properties. We addressed these issues by simultaneously recording LFPs and spiking activity using microelectrode arrays implanted in the primary visual cortex of macaque monkeys. We find that the spatial extent of gamma and its relationship to local spiking activity is stimulus dependent. Small gratings, and those masked with noise, induce a broadband increase in spectral power. This signal is tuned similarly to spiking activity and has limited spatial coherence. Large gratings, however, induce a gamma rhythm characterized by a distinctive spectral "bump," which is coherent across widely separated sites. This signal is well tuned, but its stimulus preference is similar across millimeters of cortex. The preference of this global gamma rhythm is sensitive to adaptation, in a manner consistent with its magnifying a bias in the neuronal representation of visual stimuli. Gamma thus arises from two sources that reflect different spatial scales of neural ensemble activity. Our results show that there is not a single, fixed ensemble contributing to gamma and that the selectivity of gamma cannot be used to infer its spatial extent.

The influence of surround suppression on adaptation effects in primary visual cortex.

Adaptation, the prolonged presentation of stimuli, has been used to probe mechanisms of visual processing in physiological, imaging, and perceptual studies. Previous neurophysiological studies have measured adaptation effects by using stimuli tailored to evoke robust responses in individual neurons. This approach provides an incomplete view of how an adapter alters the representation of sensory stimuli by a population of neurons with diverse functional properties. We implanted microelectrode arrays in primary visual cortex (V1) of macaque monkeys and measured orientation tuning and contrast sensitivity in populations of neurons before and after prolonged adaptation. Whereas previous studies in V1 have reported that adaptation causes stimulus-specific suppression of responsivity and repulsive shifts in tuning preference, we have found that adaptation can also lead to response facilitation and shifts in tuning toward the adapter. To explain this range of effects, we have proposed and tested a simple model that employs stimulus-specific suppression in both the receptive field and the spatial surround. The predicted effects on tuning depend on the relative drive provided by the adapter to these two receptive field components. Our data reveal that adaptation can have a much richer repertoire of effects on neuronal responsivity and tuning than previously considered and suggest an intimate mechanistic relationship between spatial and temporal contextual effects.