Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks.

Responses to natural stimuli in area V4-a mid-level area of the visual ventral stream-are well predicted by features from convolutional neural networks (CNNs) trained on image classification. This result has been taken as evidence for the functional role of V4 in object classification. However, we currently do not know if and to what extent V4 plays a role in solving other computational objectives. Here, we investigated normative accounts of V4 (and V1 for comparison) by predicting macaque single-neuron responses to natural images from the representations extracted by 23 CNNs trained on different computer vision tasks including semantic, geometric, 2D, and 3D types of tasks. We found that V4 was best predicted by semantic classification features and exhibited high task selectivity, while the choice of task was less consequential to V1 performance. Consistent with traditional characterizations of V4 function that show its high-dimensional tuning to various 2D and 3D stimulus directions, we found that diverse non-semantic tasks explained aspects of V4 function that are not captured by individual semantic tasks. Nevertheless, jointly considering the features of a pair of semantic classification tasks was sufficient to yield one of our top V4 models, solidifying V4's main functional role in semantic processing and suggesting that V4's selectivity to 2D or 3D stimulus properties found by electrophysiologists can result from semantic functional goals.

Taking subjectivity seriously: towards a unification of phenomenology, psychiatry, and neuroscience.

Nearly all psychiatric diseases involve alterations in subjective, lived experience. The scientific study of the biological basis of mental illness has generally focused on objective measures and observable behaviors, limiting the potential for our understanding of brain mechanisms of disease states and possible treatments. However, applying methods designed principally to interpret objective behavioral measures to the measurement and extrapolation of subjective states presents a number of challenges. In order to help bridge this gap, we draw on the tradition of phenomenology, a philosophical movement concerned with elucidating the structure of lived experience, which emerged in the early 20th century and influenced philosophy of mind, cognitive science, and psychiatry. A number of early phenomenologically-oriented psychiatrists made influential contributions to the field, but this approach retreated to the background as psychiatry moved towards more operationalized disease classifications. Recently, clinical-phenomenological research and viewpoints have re-emerged in the field. We argue that the potential for phenomenological research and methods to generate productive hypotheses about the neurobiological basis of psychiatric diseases has thus far been underappreciated. Using specific examples drawing on the subjective experience of mania and psychosis, we demonstrate that phenomenologically-oriented clinical studies can generate novel and fruitful propositions for neuroscientific investigation. Additionally, we outline a proposal for more rigorously integrating phenomenological investigations of subjective experience with the methods of modern neuroscience research, advocating a cross-species approach with a key role for human subjects research. Collaborative interaction between phenomenology, psychiatry, and neuroscience has the potential to move these fields towards a unified understanding of the biological basis of mental illness.

The Nested States Model: An Empirical Framework for Integrating Brain and Mind.

Philosophy of mind has made substantial progress on biologically-rooted approaches to understanding the mind and subjectivity through the enactivist perspective, but research on subjectivity within neuroscience has not kept apace. Indeed, we possess no principled means of relating experiential phenomena to neurophysiological processes. Here, we present the Nested States Model as a framework to guide empirical investigation into the relationship between subjectivity and neurobiology. Building on recent work in phenomenology and philosophy of mind, we develop an account of experiential states as layered, or nested. We argue that this nested structure is also apparent in brain activity. The recognition of this structural homology - that both experiential and brain states can be characterized as systems of nested states - brings our views of subjective mental states into broad alignment with our understanding of general principles and properties of brain activity. This alignment enables a more systematic approach to formulating specific hypotheses and predictions about how the two domains relate to one another.

Learning divisive normalization in primary visual cortex.

Divisive normalization (DN) is a prominent computational building block in the brain that has been proposed as a canonical cortical operation. Numerous experimental studies have verified its importance for capturing nonlinear neural response properties to simple, artificial stimuli, and computational studies suggest that DN is also an important component for processing natural stimuli. However, we lack quantitative models of DN that are directly informed by measurements of spiking responses in the brain and applicable to arbitrary stimuli. Here, we propose a DN model that is applicable to arbitrary input images. We test its ability to predict how neurons in macaque primary visual cortex (V1) respond to natural images, with a focus on nonlinear response properties within the classical receptive field. Our model consists of one layer of subunits followed by learned orientation-specific DN. It outperforms linear-nonlinear and wavelet-based feature representations and makes a significant step towards the performance of state-of-the-art convolutional neural network (CNN) models. Unlike deep CNNs, our compact DN model offers a direct interpretation of the nature of normalization. By inspecting the learned normalization pool of our model, we gained insights into a long-standing question about the tuning properties of DN that update the current textbook description: we found that within the receptive field oriented features were normalized preferentially by features with similar orientation rather than non-specifically as currently assumed.

Deep convolutional models improve predictions of macaque V1 responses to natural images.

Despite great efforts over several decades, our best models of primary visual cortex (V1) still predict spiking activity quite poorly when probed with natural stimuli, highlighting our limited understanding of the nonlinear computations in V1. Recently, two approaches based on deep learning have emerged for modeling these nonlinear computations: transfer learning from artificial neural networks trained on object recognition and data-driven convolutional neural network models trained end-to-end on large populations of neurons. Here, we test the ability of both approaches to predict spiking activity in response to natural images in V1 of awake monkeys. We found that the transfer learning approach performed similarly well to the data-driven approach and both outperformed classical linear-nonlinear and wavelet-based feature representations that build on existing theories of V1. Notably, transfer learning using a pre-trained feature space required substantially less experimental time to achieve the same performance. In conclusion, multi-layer convolutional neural networks (CNNs) set the new state of the art for predicting neural responses to natural images in primate V1 and deep features learned for object recognition are better explanations for V1 computation than all previous filter bank theories. This finding strengthens the necessity of V1 models that are multiple nonlinearities away from the image domain and it supports the idea of explaining early visual cortex based on high-level functional goals.

On the Structure of Neuronal Population Activity under Fluctuations in Attentional State.

Attention is commonly thought to improve behavioral performance by increasing response gain and suppressing shared variability in neuronal populations. However, both the focus and the strength of attention are likely to vary from one experimental trial to the next, thereby inducing response variability unknown to the experimenter. Here we study analytically how fluctuations in attentional state affect the structure of population responses in a simple model of spatial and feature attention. In our model, attention acts on the neural response exclusively by modulating each neuron's gain. Neurons are conditionally independent given the stimulus and the attentional gain, and correlated activity arises only from trial-to-trial fluctuations of the attentional state, which are unknown to the experimenter. We find that this simple model can readily explain many aspects of neural response modulation under attention, such as increased response gain, reduced individual and shared variability, increased correlations with firing rates, limited range correlations, and differential correlations. We therefore suggest that attention may act primarily by increasing response gain of individual neurons without affecting their correlation structure. The experimentally observed reduction in correlations may instead result from reduced variability of the attentional gain when a stimulus is attended. Moreover, we show that attentional gain fluctuations, even if unknown to a downstream readout, do not impair the readout accuracy despite inducing limited-range correlations, whereas fluctuations of the attended feature can in principle limit behavioral performance. SIGNIFICANCE STATEMENT: Covert attention is one of the most widely studied examples of top-down modulation of neural activity in the visual system. Recent studies argue that attention improves behavioral performance by shaping of the noise distribution to suppress shared variability rather than by increasing response gain. Our work shows, however, that latent, trial-to-trial fluctuations of the focus and strength of attention lead to shared variability that is highly consistent with known experimental observations. Interestingly, fluctuations in the strength of attention do not affect coding performance. As a consequence, the experimentally observed changes in response variability may not be a mechanism of attention, but rather a side effect of attentional allocation strategies in different behavioral contexts.

Monetary favors and their influence on neural responses and revealed preference.

Favors from a sender to a receiver are known to bias decisions made by the recipient, especially when the decision relates to the sender, a feature of social exchange known as reciprocity. Using an art-viewing paradigm possessing no objectively correct answer for preferring one piece of art over another, we show that sponsorship of the experiment by a company endows the logo of the company with the capacity to bias revealed preference for art displayed next to the logo. Merely offering to sponsor the experiment similarly endowed the gesturing logo of the company with the capacity to bias revealed preferences. These effects do not depend upon the size of the displayed art or the proximity of the sponsoring logo to the piece of art. We used functional magnetic resonance imaging to show that such monetary favors do not modulate a special collection of brain responses but instead modulate responses in neural networks normally activated by a wide range of preference judgments. The results raise the important possibility that monetary favors bias judgments in domains seemingly unrelated to the favor but nevertheless act in an implicit way through neural networks that underlie normal, ongoing preference judgments.

Attentional fluctuations induce shared variability in macaque primary visual cortex.

Variability in neuronal responses to identical stimuli is frequently correlated across a population. Attention is thought to reduce these correlations by suppressing noisy inputs shared by the population. However, even with precise control of the visual stimulus, the subject's attentional state varies across trials. While these state fluctuations are bound to induce some degree of correlated variability, it is currently unknown how strong their effect is, as previous studies generally do not dissociate changes in attentional strength from changes in attentional state variability. We designed a novel paradigm that does so and find both a pronounced effect of attentional fluctuations on correlated variability at long timescales and attention-dependent reductions in correlations at short timescales. These effects predominate in layers 2/3, as expected from a feedback signal such as attention. Thus, significant portions of correlated variability can be attributed to fluctuations in internally generated signals, like attention, rather than noise.

Investigating the Limits of Neurovascular Coupling.

O'Herron et al. (2016) perform two-photon imaging of vascular and neural responses in cat and rodent primary visual cortex to investigate the limits of neurovascular coupling. Their results suggest important constraints on making inferences about neuronal responses from hemodynamic activity.

Response to Comment on "Principles of connectivity among morphologically defined cell types in adult neocortex".

The critique of Barth et al centers on three points: (i) the completeness of our study is overstated; (ii) the connectivity matrix we describe is biased by technical limitations of our brain-slicing and multipatching methods; and (iii) our cell classification scheme is arbitrary and we have simply renamed previously identified interneuron types. We address these criticisms in our Response.

Spike sorting for large, dense electrode arrays.

Developments in microfabrication technology have enabled the production of neural electrode arrays with hundreds of closely spaced recording sites, and electrodes with thousands of sites are under development. These probes in principle allow the simultaneous recording of very large numbers of neurons. However, use of this technology requires the development of techniques for decoding the spike times of the recorded neurons from the raw data captured from the probes. Here we present a set of tools to solve this problem, implemented in a suite of practical, user-friendly, open-source software. We validate these methods on data from the cortex, hippocampus and thalamus of rat, mouse, macaque and marmoset, demonstrating error rates as low as 5%.

Pupil fluctuations track fast switching of cortical states during quiet wakefulness.

Neural responses are modulated by brain state, which varies with arousal, attention, and behavior. In mice, running and whisking desynchronize the cortex and enhance sensory responses, but the quiescent periods between bouts of exploratory behaviors have not been well studied. We found that these periods of "quiet wakefulness" were characterized by state fluctuations on a timescale of 1-2 s. Small fluctuations in pupil diameter tracked these state transitions in multiple cortical areas. During dilation, the intracellular membrane potential was desynchronized, sensory responses were enhanced, and population activity was less correlated. In contrast, constriction was characterized by increased low-frequency oscillations and higher ensemble correlations. Specific subtypes of cortical interneurons were differentially activated during dilation and constriction, consistent with their participation in the observed state changes. Pupillometry has been used to index attention and mental effort in humans, but the intracellular dynamics and differences in population activity underlying this phenomenon were previously unknown.

State dependence of noise correlations in macaque primary visual cortex.

Shared, trial-to-trial variability in neuronal populations has a strong impact on the accuracy of information processing in the brain. Estimates of the level of such noise correlations are diverse, ranging from 0.01 to 0.4, with little consensus on which factors account for these differences. Here we addressed one important factor that varied across studies, asking how anesthesia affects the population activity structure in macaque primary visual cortex. We found that under opioid anesthesia, activity was dominated by strong coordinated fluctuations on a timescale of 1-2 Hz, which were mostly absent in awake, fixating monkeys. Accounting for these global fluctuations markedly reduced correlations under anesthesia, matching those observed during wakefulness and reconciling earlier studies conducted under anesthesia and in awake animals. Our results show that internal signals, such as brain state transitions under anesthesia, can induce noise correlations but can also be estimated and accounted for based on neuronal population activity.