Vision-dependent specification of cell types and function in the developing cortex.

The role of postnatal experience in sculpting cortical circuitry, while long appreciated, is poorly understood at the level of cell types. We explore this in the mouse primary visual cortex (V1) using single-nucleus RNA sequencing, visual deprivation, genetics, and functional imaging. We find that vision selectively drives the specification of glutamatergic cell types in upper layers (L) (L2/3/4), while deeper-layer glutamatergic, GABAergic, and non-neuronal cell types are established prior to eye opening. L2/3 cell types form an experience-dependent spatial continuum defined by the graded expression of ∼200 genes, including regulators of cell adhesion and synapse formation. One of these genes, Igsf9b, a vision-dependent gene encoding an inhibitory synaptic cell adhesion molecule, is required for the normal development of binocular responses in L2/3. In summary, vision preferentially regulates the development of upper-layer glutamatergic cell types through the regulation of cell-type-specific gene expression programs.

Reconstruction of neocortex: Organelles, compartments, cells, circuits, and activity.

We assembled a semi-automated reconstruction of L2/3 mouse primary visual cortex from ∼250 × 140 × 90 µm3 of electron microscopic images, including pyramidal and non-pyramidal neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, nuclei, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are included. The data are publicly available, along with tools for programmatic and three-dimensional interactive access. Brief vignettes illustrate the breadth of potential applications relating structure to function in cortical circuits and neuronal cell biology. Mitochondria and synapse organization are characterized as a function of path length from the soma. Pyramidal connectivity motif frequencies are predicted accurately using a configuration model of random graphs. Pyramidal cells receiving more connections from nearby cells exhibit stronger and more reliable visual responses. Sample code shows data access and analysis.

Large-scale two-photon calcium imaging in freely moving mice.

We developed a miniaturized two-photon microscope (MINI2P) for fast, high-resolution, multiplane calcium imaging of over 1,000 neurons at a time in freely moving mice. With a microscope weight below 3 g and a highly flexible connection cable, MINI2P allowed stable imaging with no impediment of behavior in a variety of assays compared to untethered, unimplanted animals. The improved cell yield was achieved through a optical system design featuring an enlarged field of view (FOV) and a microtunable lens with increased z-scanning range and speed that allows fast and stable imaging of multiple interleaved planes, as well as 3D functional imaging. Successive imaging across multiple, adjacent FOVs enabled recordings from more than 10,000 neurons in the same animal. Large-scale proof-of-principle data were obtained from cell populations in visual cortex, medial entorhinal cortex, and hippocampus, revealing spatial tuning of cells in all areas.

High-precision coding in visual cortex.

Individual neurons in visual cortex provide the brain with unreliable estimates of visual features. It is not known whether the single-neuron variability is correlated across large neural populations, thus impairing the global encoding of stimuli. We recorded simultaneously from up to 50,000 neurons in mouse primary visual cortex (V1) and in higher order visual areas and measured stimulus discrimination thresholds of 0.35° and 0.37°, respectively, in an orientation decoding task. These neural thresholds were almost 100 times smaller than the behavioral discrimination thresholds reported in mice. This discrepancy could not be explained by stimulus properties or arousal states. Furthermore, behavioral variability during a sensory discrimination task could not be explained by neural variability in V1. Instead, behavior-related neural activity arose dynamically across a network of non-sensory brain areas. These results imply that perceptual discrimination in mice is limited by downstream decoders, not by neural noise in sensory representations.

Integrated Morphoelectric and Transcriptomic Classification of Cortical GABAergic Cells.

Neurons are frequently classified into distinct types on the basis of structural, physiological, or genetic attributes. To better constrain the definition of neuronal cell types, we characterized the transcriptomes and intrinsic physiological properties of over 4,200 mouse visual cortical GABAergic interneurons and reconstructed the local morphologies of 517 of those neurons. We find that most transcriptomic types (t-types) occupy specific laminar positions within visual cortex, and, for most types, the cells mapping to a t-type exhibit consistent electrophysiological and morphological properties. These properties display both discrete and continuous variation among t-types. Through multimodal integrated analysis, we define 28 met-types that have congruent morphological, electrophysiological, and transcriptomic properties and robust mutual predictability. We identify layer-specific axon innervation pattern as a defining feature distinguishing different met-types. These met-types represent a unified definition of cortical GABAergic interneuron types, providing a systematic framework to capture existing knowledge and bridge future analyses across different modalities.

Visuomotor Coupling Shapes the Functional Development of Mouse Visual Cortex.

The emergence of sensory-guided behavior depends on sensorimotor coupling during development. How sensorimotor experience shapes neural processing is unclear. Here, we show that the coupling between motor output and visual feedback is necessary for the functional development of visual processing in layer 2/3 (L2/3) of primary visual cortex (V1) of the mouse. Using a virtual reality system, we reared mice in conditions of normal or random visuomotor coupling. We recorded the activity of identified excitatory and inhibitory L2/3 neurons in response to transient visuomotor mismatches in both groups of mice. Mismatch responses in excitatory neurons were strongly experience dependent and driven by a transient release from inhibition mediated by somatostatin-positive interneurons. These data are consistent with a model in which L2/3 of V1 computes a difference between an inhibitory visual input and an excitatory locomotion-related input, where the balance between these two inputs is finely tuned by visuomotor experience.

Population encoding of stimulus features along the visual hierarchy.

The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.

Efficient models of cortical activity via local dynamic equilibria and coarse-grained interactions.

Biologically detailed models of brain circuitry are challenging to build and simulate due to the large number of neurons, their complex interactions, and the many unknown physiological parameters. Simplified mathematical models are more tractable, but harder to evaluate when too far removed from neuroanatomy/physiology. We propose that a multiscale model, coarse-grained (CG) while preserving local biological details, offers the best balance between biological realism and computability. This paper presents such a model. Generally, CG models focus on the interaction between groups of neurons-here termed "pixels"-rather than individual cells. In our case, dynamics are alternately updated at intra- and interpixel scales, with one informing the other, until convergence to equilibrium is achieved on both scales. An innovation is how we exploit the underlying biology: Taking advantage of the similarity in local anatomical structures across large regions of the cortex, we model intrapixel dynamics as a single dynamical system driven by "external" inputs. These inputs vary with events external to the pixel, but their ranges can be estimated a priori. Precomputing and tabulating all potential local responses speed up the updating procedure significantly compared to direct multiscale simulation. We illustrate our methodology using a model of the primate visual cortex. Except for local neuron-to-neuron variability (necessarily lost in any CG approximation) our model reproduces various features of large-scale network models at a tiny fraction of the computational cost. These include neuronal responses as a consequence of their orientation selectivity, a primary function of visual neurons.

Natural scenes reveal diverse representations of 2D and 3D body pose in the human brain.

Human pose, defined as the spatial relationships between body parts, carries instrumental information supporting the understanding of motion and action of a person. A substantial body of previous work has identified cortical areas responsive to images of bodies and different body parts. However, the neural basis underlying the visual perception of body part relationships has received less attention. To broaden our understanding of body perception, we analyzed high-resolution fMRI responses to a wide range of poses from over 4,000 complex natural scenes. Using ground-truth annotations and an application of three-dimensional (3D) pose reconstruction algorithms, we compared similarity patterns of cortical activity with similarity patterns built from human pose models with different levels of depth availability and viewpoint dependency. Targeting the challenge of explaining variance in complex natural image responses with interpretable models, we achieved statistically significant correlations between pose models and cortical activity patterns (though performance levels are substantially lower than the noise ceiling). We found that the 3D view-independent pose model, compared with two-dimensional models, better captures the activation from distinct cortical areas, including the right posterior superior temporal sulcus (pSTS). These areas, together with other pose-selective regions in the LOTC, form a broader, distributed cortical network with greater view-tolerance in more anterior patches. We interpret these findings in light of the computational complexity of natural body images, the wide range of visual tasks supported by pose structures, and possible shared principles for view-invariant processing between articulated objects and ordinary, rigid objects.

Longitudinal stability of individual brain plasticity patterns in blindness.

The primary visual cortex (V1) in blindness is engaged in a wide spectrum of tasks and sensory modalities, including audition, touch, language, and memory. This widespread involvement raises questions regarding the constancy of its role and whether it might exhibit flexibility in its function over time, connecting to diverse network functions specific to task demands. This would suggest that reorganized V1 assumes a role like multiple-demand system regions. Alternatively, varying patterns of plasticity in blind V1 may be attributed to individual factors, with different blind individuals recruiting V1 preferentially for different functions. In support of this, we recently showed that V1 functional connectivity (FC) varies greatly across blind individuals. But do these represent stable individual patterns of plasticity, or are they driven more by instantaneous changes, like a multiple-demand system now inhabiting V1? Here, we tested whether individual FC patterns from the V1 of blind individuals are stable over time. We show that over two years, FC from the V1 is unique and highly stable in a small sample of repeatedly sampled congenitally blind individuals. Further, using multivoxel pattern analysis, we demonstrate that the unique reorganization patterns of these individuals allow decoding of participant identity. Together with recent evidence for substantial individual differences in V1 connectivity, this indicates that there may be a consistent role for V1 in blindness, which may differ for each individual. Further, it suggests that the variability in visual reorganization in blindness across individuals could be used to seek stable neuromarkers for sight rehabilitation and assistive approaches.

Dynamic fading memory and expectancy effects in the monkey primary visual cortex.

In order to investigate the involvement of the primary visual cortex (V1) in working memory (WM), parallel, multisite recordings of multi-unit activity were obtained from monkey V1 while the animals performed a delayed match-to-sample (DMS) task. During the delay period, V1 population firing rate vectors maintained a lingering trace of the sample stimulus that could be reactivated by intervening impulse stimuli that enhanced neuronal firing. This fading trace of the sample did not require active engagement of the monkeys in the DMS task and likely reflects the intrinsic dynamics of recurrent cortical networks in lower visual areas. This renders an active, attention-dependent involvement of V1 in the maintenance of WM contents unlikely. By contrast, population responses to the test stimulus depended on the probabilistic contingencies between sample and test stimuli. Responses to tests that matched expectations were reduced which agrees with concepts of predictive coding.

Representations and generalization in artificial and brain neural networks.

Humans and animals excel at generalizing from limited data, a capability yet to be fully replicated in artificial intelligence. This perspective investigates generalization in biological and artificial deep neural networks (DNNs), in both in-distribution and out-of-distribution contexts. We introduce two hypotheses: First, the geometric properties of the neural manifolds associated with discrete cognitive entities, such as objects, words, and concepts, are powerful order parameters. They link the neural substrate to the generalization capabilities and provide a unified methodology bridging gaps between neuroscience, machine learning, and cognitive science. We overview recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discuss theories connecting manifold dimension and radius to generalization capacity. Second, we suggest that the theory of learning in wide DNNs, especially in the thermodynamic limit, provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. This includes the role of weight norm regularization, network architecture, and hyper-parameters. We will explore recent advances in this theory and ongoing challenges. We also discuss the dynamics of learning and its relevance to the issue of representational drift in the brain.

Dissonant music engages early visual processing.

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

Correlated variability and its attentional modulation depend on anatomical connectivity.

Visual cortical neurons show variability in their responses to repeated presentations of a stimulus and a portion of this variability is shared across neurons. Attention may enhance visual perception by reducing shared spiking variability. However, shared variability and its attentional modulation are not consistent within or across cortical areas, and depend on additional factors such as neuronal type. A critical factor that has not been tested is actual anatomical connectivity. We measured spike count correlations among pairs of simultaneously recorded neurons in the primary visual cortex (V1) for which anatomical connectivity was inferred from spiking cross-correlations. Neurons were recorded in monkeys performing a contrast-change discrimination task requiring covert shifts in visual spatial attention. Accordingly, spike count correlations were compared across trials in which attention was directed toward or away from the visual stimulus overlapping recorded neuronal receptive fields. Consistent with prior findings, attention did not significantly alter spike count correlations among random pairings of unconnected V1 neurons. However, V1 neurons connected via excitatory synapses showed a significant reduction in spike count correlations with attention. Interestingly, V1 neurons connected via inhibitory synapses demonstrated high spike count correlations overall that were not modulated by attention. Correlated variability in excitatory circuits also depended upon neuronal tuning for contrast, the task-relevant stimulus feature. These results indicate that shared variability depends on the type of connectivity in neuronal circuits. Also, attention significantly reduces shared variability in excitatory circuits, even when attention effects on randomly sampled neurons within the same area are weak.

Cognition as a Window into Neuronal Population Space.

Understanding how cognitive processes affect the responses of sensory neurons may clarify the relationship between neuronal population activity and behavior. However, tools for analyzing neuronal activity have not kept up with technological advances in recording from large neuronal populations. Here, we describe prevalent hypotheses of how cognitive processes affect sensory neurons, driven largely by a model based on the activity of single neurons or pools of neurons as the units of computation. We then use simple simulations to expand this model to a new conceptual framework that focuses on subspaces of population activity as the relevant units of computation, uses comparisons between brain areas or to behavior to guide analyses of these subspaces, and suggests that population activity is optimized to decode the large variety of stimuli and tasks that animals encounter in natural behavior. This framework provides new ways of understanding the ever-growing quantity of recorded population activity data.

Ripples in macaque V1 and V4 are modulated by top-down visual attention.

Sharp-wave ripples (SWRs) are highly synchronous neuronal activity events. They have been predominantly observed in the hippocampus during offline states such as pause in exploration, slow-wave sleep, and quiescent wakefulness. SWRs have been linked to memory consolidation, spatial navigation, and spatial decision-making. Recently, SWRs have been reported during visual search, a form of remote spatial exploration, in macaque hippocampus. However, the association between SWRs and multiple forms of awake conscious and goal-directed behavior is unknown. We report that ripple activity occurs in macaque visual areas V1 and V4 during focused spatial attention. The occurrence of ripples is modulated by stimulus characteristics, increased by attention toward the receptive field, and by the size of the attentional focus. During attention cued to the receptive field, the monkey's reaction time in detecting behaviorally relevant events was reduced by ripples. These results show that ripple activity is not limited to hippocampal activity during offline states, rather they occur in the neocortex during active attentive states and vigilance behaviors.

Different roles of response covariability and its attentional modulation in the sensory cortex and posterior parietal cortex.

The covariability of neural responses in the neuron population is highly relevant to the information encoding. Cognitive processes, such as attention, are found to modulate the covariability in the visual cortex to improve information encoding, suggesting the computational advantage of covariability modulation in the neural system. However, is the covariability modulation a general mechanism for enhanced information encoding throughout the information processing pathway, or only adopted in certain processing stages, depending on the property of neural representation? Here, with ultrahigh-field MRI, we examined the covariability, which was estimated by noise correlation, in different attention states in the early visual cortex and posterior parietal cortex (PPC) of the human brain, and its relationship to the quality of information encoding. Our results showed that while attention decreased the covariability to improve the stimulus encoding in the early visual cortex, covariability modulation was not observed in the PPC, where covariability had little impact on information encoding. Further, attention promoted the information flow between the early visual cortex and PPC, with an apparent emphasis on a flow from high- to low-dimensional representations, suggesting the existence of a reduction in the dimensionality of neural representation from the early visual cortex to PPC. Finally, the neural response patterns in the PPC could predict the amplitudes of covariability change in the early visual cortex, indicating a top-down control from the PPC to early visual cortex. Our findings reveal the specific roles of the sensory cortex and PPC during attentional modulation of covariability, determined by the complexity and fidelity of the neural representation in each cortical region.

Experience-dependent functional plasticity and visual response selectivity of surviving subplate neurons in the mouse visual cortex.

Subplate neurons are early-born cortical neurons that transiently form neural circuits during perinatal development and guide cortical maturation. Thereafter, most subplate neurons undergo cell death, while some survive and renew their target areas for synaptic connections. However, the functional properties of the surviving subplate neurons remain largely unknown. This study aimed to characterize the visual responses and experience-dependent functional plasticity of layer 6b (L6b) neurons, the remnants of subplate neurons, in the primary visual cortex (V1). Two-photon Ca2+ imaging was performed in V1 of awake juvenile mice. L6b neurons showed broader tunings for orientation, direction, and spatial frequency than did layer 2/3 (L2/3) and L6a neurons. In addition, L6b neurons showed lower matching of preferred orientation between the left and right eyes compared with other layers. Post hoc 3D immunohistochemistry confirmed that the majority of recorded L6b neurons expressed connective tissue growth factor (CTGF), a subplate neuron marker. Moreover, chronic two-photon imaging showed that L6b neurons exhibited ocular dominance (OD) plasticity by monocular deprivation during critical periods. The OD shift to the open eye depended on the response strength to the stimulation of the eye to be deprived before starting monocular deprivation. There were no significant differences in visual response selectivity prior to monocular deprivation between the OD changed and unchanged neuron groups, suggesting that OD plasticity can occur in L6b neurons showing any response features. In conclusion, our results provide strong evidence that surviving subplate neurons exhibit sensory responses and experience-dependent plasticity at a relatively late stage of cortical development.

Macaques recognize features in synthetic images derived from ventral stream neurons.

Primates can recognize features in virtually all types of images, an ability that still requires a comprehensive computational explanation. One hypothesis is that visual cortex neurons learn patterns from scenes, objects, and textures, and use these patterns to interpolate incoming visual information. We have used machine learning algorithms to instantiate visual patterns stored by neurons-we call these highly activating images prototypes. Prototypes from inferotemporal (IT) neurons often resemble parts of real-world objects, such as monkey faces and body parts, a similarity established via pretrained neural networks [C. R. Ponce et al., Cell 177, 999-1009.e10 (2019)] and naïve human participants [A. Bardon, W. Xiao, C. R. Ponce, M. S. Livingstone, G. Kreiman, Proc. Natl. Acad. Sci. U.S.A. 119, e2118705119 (2022)]. However, it is not known whether monkeys themselves perceive similarities between neuronal prototypes and real-world objects. Here, we investigated whether monkeys reported similarities between prototypes and real-world objects using a two-alternative forced choice task. We trained the animals to saccade to synthetic images of monkeys, and subsequently tested how they classified prototypes synthesized from IT and primary visual cortex (V1). We found monkeys classified IT prototypes as conspecifics more often than they did random generator images and V1 prototypes, and their choices were partially predicted by convolutional neural networks. Further, we confirmed that monkeys could abstract general shape information from images of real-world objects. Finally, we verified these results with human participants. Our results provide further evidence that prototypes from cortical neurons represent interpretable abstractions from the visual world.

Characterizing Spatiotemporal Population Receptive Fields in Human Visual Cortex with fMRI.

The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time-varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants (both females and males). We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (1) from early to later areas within a visual stream, spatial and temporal windows of pRFs progressively increase in size and show greater compressive nonlinearities, (2) later visual areas show diverging spatial and temporal windows across streams, and (3) within early visual areas (V1-V3), both spatial and temporal windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses using fMRI.