Reconstruction of neocortex: Organelles, compartments, cells, circuits, and activity.

We assembled a semi-automated reconstruction of L2/3 mouse primary visual cortex from ∼250 × 140 × 90 µm3 of electron microscopic images, including pyramidal and non-pyramidal neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, nuclei, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are included. The data are publicly available, along with tools for programmatic and three-dimensional interactive access. Brief vignettes illustrate the breadth of potential applications relating structure to function in cortical circuits and neuronal cell biology. Mitochondria and synapse organization are characterized as a function of path length from the soma. Pyramidal connectivity motif frequencies are predicted accurately using a configuration model of random graphs. Pyramidal cells receiving more connections from nearby cells exhibit stronger and more reliable visual responses. Sample code shows data access and analysis.

Vision-dependent specification of cell types and function in the developing cortex.

The role of postnatal experience in sculpting cortical circuitry, while long appreciated, is poorly understood at the level of cell types. We explore this in the mouse primary visual cortex (V1) using single-nucleus RNA sequencing, visual deprivation, genetics, and functional imaging. We find that vision selectively drives the specification of glutamatergic cell types in upper layers (L) (L2/3/4), while deeper-layer glutamatergic, GABAergic, and non-neuronal cell types are established prior to eye opening. L2/3 cell types form an experience-dependent spatial continuum defined by the graded expression of ∼200 genes, including regulators of cell adhesion and synapse formation. One of these genes, Igsf9b, a vision-dependent gene encoding an inhibitory synaptic cell adhesion molecule, is required for the normal development of binocular responses in L2/3. In summary, vision preferentially regulates the development of upper-layer glutamatergic cell types through the regulation of cell-type-specific gene expression programs.

Large-scale two-photon calcium imaging in freely moving mice.

We developed a miniaturized two-photon microscope (MINI2P) for fast, high-resolution, multiplane calcium imaging of over 1,000 neurons at a time in freely moving mice. With a microscope weight below 3 g and a highly flexible connection cable, MINI2P allowed stable imaging with no impediment of behavior in a variety of assays compared to untethered, unimplanted animals. The improved cell yield was achieved through a optical system design featuring an enlarged field of view (FOV) and a microtunable lens with increased z-scanning range and speed that allows fast and stable imaging of multiple interleaved planes, as well as 3D functional imaging. Successive imaging across multiple, adjacent FOVs enabled recordings from more than 10,000 neurons in the same animal. Large-scale proof-of-principle data were obtained from cell populations in visual cortex, medial entorhinal cortex, and hippocampus, revealing spatial tuning of cells in all areas.

High-precision coding in visual cortex.

Individual neurons in visual cortex provide the brain with unreliable estimates of visual features. It is not known whether the single-neuron variability is correlated across large neural populations, thus impairing the global encoding of stimuli. We recorded simultaneously from up to 50,000 neurons in mouse primary visual cortex (V1) and in higher order visual areas and measured stimulus discrimination thresholds of 0.35° and 0.37°, respectively, in an orientation decoding task. These neural thresholds were almost 100 times smaller than the behavioral discrimination thresholds reported in mice. This discrepancy could not be explained by stimulus properties or arousal states. Furthermore, behavioral variability during a sensory discrimination task could not be explained by neural variability in V1. Instead, behavior-related neural activity arose dynamically across a network of non-sensory brain areas. These results imply that perceptual discrimination in mice is limited by downstream decoders, not by neural noise in sensory representations.

Integrated Morphoelectric and Transcriptomic Classification of Cortical GABAergic Cells.

Neurons are frequently classified into distinct types on the basis of structural, physiological, or genetic attributes. To better constrain the definition of neuronal cell types, we characterized the transcriptomes and intrinsic physiological properties of over 4,200 mouse visual cortical GABAergic interneurons and reconstructed the local morphologies of 517 of those neurons. We find that most transcriptomic types (t-types) occupy specific laminar positions within visual cortex, and, for most types, the cells mapping to a t-type exhibit consistent electrophysiological and morphological properties. These properties display both discrete and continuous variation among t-types. Through multimodal integrated analysis, we define 28 met-types that have congruent morphological, electrophysiological, and transcriptomic properties and robust mutual predictability. We identify layer-specific axon innervation pattern as a defining feature distinguishing different met-types. These met-types represent a unified definition of cortical GABAergic interneuron types, providing a systematic framework to capture existing knowledge and bridge future analyses across different modalities.

Visuomotor Coupling Shapes the Functional Development of Mouse Visual Cortex.

The emergence of sensory-guided behavior depends on sensorimotor coupling during development. How sensorimotor experience shapes neural processing is unclear. Here, we show that the coupling between motor output and visual feedback is necessary for the functional development of visual processing in layer 2/3 (L2/3) of primary visual cortex (V1) of the mouse. Using a virtual reality system, we reared mice in conditions of normal or random visuomotor coupling. We recorded the activity of identified excitatory and inhibitory L2/3 neurons in response to transient visuomotor mismatches in both groups of mice. Mismatch responses in excitatory neurons were strongly experience dependent and driven by a transient release from inhibition mediated by somatostatin-positive interneurons. These data are consistent with a model in which L2/3 of V1 computes a difference between an inhibitory visual input and an excitatory locomotion-related input, where the balance between these two inputs is finely tuned by visuomotor experience.

Natural scenes reveal diverse representations of 2D and 3D body pose in the human brain.

Human pose, defined as the spatial relationships between body parts, carries instrumental information supporting the understanding of motion and action of a person. A substantial body of previous work has identified cortical areas responsive to images of bodies and different body parts. However, the neural basis underlying the visual perception of body part relationships has received less attention. To broaden our understanding of body perception, we analyzed high-resolution fMRI responses to a wide range of poses from over 4,000 complex natural scenes. Using ground-truth annotations and an application of three-dimensional (3D) pose reconstruction algorithms, we compared similarity patterns of cortical activity with similarity patterns built from human pose models with different levels of depth availability and viewpoint dependency. Targeting the challenge of explaining variance in complex natural image responses with interpretable models, we achieved statistically significant correlations between pose models and cortical activity patterns (though performance levels are substantially lower than the noise ceiling). We found that the 3D view-independent pose model, compared with two-dimensional models, better captures the activation from distinct cortical areas, including the right posterior superior temporal sulcus (pSTS). These areas, together with other pose-selective regions in the LOTC, form a broader, distributed cortical network with greater view-tolerance in more anterior patches. We interpret these findings in light of the computational complexity of natural body images, the wide range of visual tasks supported by pose structures, and possible shared principles for view-invariant processing between articulated objects and ordinary, rigid objects.

Efficient models of cortical activity via local dynamic equilibria and coarse-grained interactions.

Biologically detailed models of brain circuitry are challenging to build and simulate due to the large number of neurons, their complex interactions, and the many unknown physiological parameters. Simplified mathematical models are more tractable, but harder to evaluate when too far removed from neuroanatomy/physiology. We propose that a multiscale model, coarse-grained (CG) while preserving local biological details, offers the best balance between biological realism and computability. This paper presents such a model. Generally, CG models focus on the interaction between groups of neurons-here termed "pixels"-rather than individual cells. In our case, dynamics are alternately updated at intra- and interpixel scales, with one informing the other, until convergence to equilibrium is achieved on both scales. An innovation is how we exploit the underlying biology: Taking advantage of the similarity in local anatomical structures across large regions of the cortex, we model intrapixel dynamics as a single dynamical system driven by "external" inputs. These inputs vary with events external to the pixel, but their ranges can be estimated a priori. Precomputing and tabulating all potential local responses speed up the updating procedure significantly compared to direct multiscale simulation. We illustrate our methodology using a model of the primate visual cortex. Except for local neuron-to-neuron variability (necessarily lost in any CG approximation) our model reproduces various features of large-scale network models at a tiny fraction of the computational cost. These include neuronal responses as a consequence of their orientation selectivity, a primary function of visual neurons.

Population encoding of stimulus features along the visual hierarchy.

The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.

Representations and generalization in artificial and brain neural networks.

Humans and animals excel at generalizing from limited data, a capability yet to be fully replicated in artificial intelligence. This perspective investigates generalization in biological and artificial deep neural networks (DNNs), in both in-distribution and out-of-distribution contexts. We introduce two hypotheses: First, the geometric properties of the neural manifolds associated with discrete cognitive entities, such as objects, words, and concepts, are powerful order parameters. They link the neural substrate to the generalization capabilities and provide a unified methodology bridging gaps between neuroscience, machine learning, and cognitive science. We overview recent progress in studying the geometry of neural manifolds, particularly in visual object recognition, and discuss theories connecting manifold dimension and radius to generalization capacity. Second, we suggest that the theory of learning in wide DNNs, especially in the thermodynamic limit, provides mechanistic insights into the learning processes generating desired neural representational geometries and generalization. This includes the role of weight norm regularization, network architecture, and hyper-parameters. We will explore recent advances in this theory and ongoing challenges. We also discuss the dynamics of learning and its relevance to the issue of representational drift in the brain.

Dynamic fading memory and expectancy effects in the monkey primary visual cortex.

In order to investigate the involvement of the primary visual cortex (V1) in working memory (WM), parallel, multisite recordings of multi-unit activity were obtained from monkey V1 while the animals performed a delayed match-to-sample (DMS) task. During the delay period, V1 population firing rate vectors maintained a lingering trace of the sample stimulus that could be reactivated by intervening impulse stimuli that enhanced neuronal firing. This fading trace of the sample did not require active engagement of the monkeys in the DMS task and likely reflects the intrinsic dynamics of recurrent cortical networks in lower visual areas. This renders an active, attention-dependent involvement of V1 in the maintenance of WM contents unlikely. By contrast, population responses to the test stimulus depended on the probabilistic contingencies between sample and test stimuli. Responses to tests that matched expectations were reduced which agrees with concepts of predictive coding.

Cognition as a Window into Neuronal Population Space.

Understanding how cognitive processes affect the responses of sensory neurons may clarify the relationship between neuronal population activity and behavior. However, tools for analyzing neuronal activity have not kept up with technological advances in recording from large neuronal populations. Here, we describe prevalent hypotheses of how cognitive processes affect sensory neurons, driven largely by a model based on the activity of single neurons or pools of neurons as the units of computation. We then use simple simulations to expand this model to a new conceptual framework that focuses on subspaces of population activity as the relevant units of computation, uses comparisons between brain areas or to behavior to guide analyses of these subspaces, and suggests that population activity is optimized to decode the large variety of stimuli and tasks that animals encounter in natural behavior. This framework provides new ways of understanding the ever-growing quantity of recorded population activity data.

Ripples in macaque V1 and V4 are modulated by top-down visual attention.

Sharp-wave ripples (SWRs) are highly synchronous neuronal activity events. They have been predominantly observed in the hippocampus during offline states such as pause in exploration, slow-wave sleep, and quiescent wakefulness. SWRs have been linked to memory consolidation, spatial navigation, and spatial decision-making. Recently, SWRs have been reported during visual search, a form of remote spatial exploration, in macaque hippocampus. However, the association between SWRs and multiple forms of awake conscious and goal-directed behavior is unknown. We report that ripple activity occurs in macaque visual areas V1 and V4 during focused spatial attention. The occurrence of ripples is modulated by stimulus characteristics, increased by attention toward the receptive field, and by the size of the attentional focus. During attention cued to the receptive field, the monkey's reaction time in detecting behaviorally relevant events was reduced by ripples. These results show that ripple activity is not limited to hippocampal activity during offline states, rather they occur in the neocortex during active attentive states and vigilance behaviors.

Different roles of response covariability and its attentional modulation in the sensory cortex and posterior parietal cortex.

The covariability of neural responses in the neuron population is highly relevant to the information encoding. Cognitive processes, such as attention, are found to modulate the covariability in the visual cortex to improve information encoding, suggesting the computational advantage of covariability modulation in the neural system. However, is the covariability modulation a general mechanism for enhanced information encoding throughout the information processing pathway, or only adopted in certain processing stages, depending on the property of neural representation? Here, with ultrahigh-field MRI, we examined the covariability, which was estimated by noise correlation, in different attention states in the early visual cortex and posterior parietal cortex (PPC) of the human brain, and its relationship to the quality of information encoding. Our results showed that while attention decreased the covariability to improve the stimulus encoding in the early visual cortex, covariability modulation was not observed in the PPC, where covariability had little impact on information encoding. Further, attention promoted the information flow between the early visual cortex and PPC, with an apparent emphasis on a flow from high- to low-dimensional representations, suggesting the existence of a reduction in the dimensionality of neural representation from the early visual cortex to PPC. Finally, the neural response patterns in the PPC could predict the amplitudes of covariability change in the early visual cortex, indicating a top-down control from the PPC to early visual cortex. Our findings reveal the specific roles of the sensory cortex and PPC during attentional modulation of covariability, determined by the complexity and fidelity of the neural representation in each cortical region.

Macaques recognize features in synthetic images derived from ventral stream neurons.

Primates can recognize features in virtually all types of images, an ability that still requires a comprehensive computational explanation. One hypothesis is that visual cortex neurons learn patterns from scenes, objects, and textures, and use these patterns to interpolate incoming visual information. We have used machine learning algorithms to instantiate visual patterns stored by neurons-we call these highly activating images prototypes. Prototypes from inferotemporal (IT) neurons often resemble parts of real-world objects, such as monkey faces and body parts, a similarity established via pretrained neural networks [C. R. Ponce et al., Cell 177, 999-1009.e10 (2019)] and naïve human participants [A. Bardon, W. Xiao, C. R. Ponce, M. S. Livingstone, G. Kreiman, Proc. Natl. Acad. Sci. U.S.A. 119, e2118705119 (2022)]. However, it is not known whether monkeys themselves perceive similarities between neuronal prototypes and real-world objects. Here, we investigated whether monkeys reported similarities between prototypes and real-world objects using a two-alternative forced choice task. We trained the animals to saccade to synthetic images of monkeys, and subsequently tested how they classified prototypes synthesized from IT and primary visual cortex (V1). We found monkeys classified IT prototypes as conspecifics more often than they did random generator images and V1 prototypes, and their choices were partially predicted by convolutional neural networks. Further, we confirmed that monkeys could abstract general shape information from images of real-world objects. Finally, we verified these results with human participants. Our results provide further evidence that prototypes from cortical neurons represent interpretable abstractions from the visual world.

Experience-dependent functional plasticity and visual response selectivity of surviving subplate neurons in the mouse visual cortex.

Subplate neurons are early-born cortical neurons that transiently form neural circuits during perinatal development and guide cortical maturation. Thereafter, most subplate neurons undergo cell death, while some survive and renew their target areas for synaptic connections. However, the functional properties of the surviving subplate neurons remain largely unknown. This study aimed to characterize the visual responses and experience-dependent functional plasticity of layer 6b (L6b) neurons, the remnants of subplate neurons, in the primary visual cortex (V1). Two-photon Ca2+ imaging was performed in V1 of awake juvenile mice. L6b neurons showed broader tunings for orientation, direction, and spatial frequency than did layer 2/3 (L2/3) and L6a neurons. In addition, L6b neurons showed lower matching of preferred orientation between the left and right eyes compared with other layers. Post hoc 3D immunohistochemistry confirmed that the majority of recorded L6b neurons expressed connective tissue growth factor (CTGF), a subplate neuron marker. Moreover, chronic two-photon imaging showed that L6b neurons exhibited ocular dominance (OD) plasticity by monocular deprivation during critical periods. The OD shift to the open eye depended on the response strength to the stimulation of the eye to be deprived before starting monocular deprivation. There were no significant differences in visual response selectivity prior to monocular deprivation between the OD changed and unchanged neuron groups, suggesting that OD plasticity can occur in L6b neurons showing any response features. In conclusion, our results provide strong evidence that surviving subplate neurons exhibit sensory responses and experience-dependent plasticity at a relatively late stage of cortical development.

Characterizing Spatiotemporal Population Receptive Fields in Human Visual Cortex with fMRI.

The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time-varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants (both females and males). We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (1) from early to later areas within a visual stream, spatial and temporal windows of pRFs progressively increase in size and show greater compressive nonlinearities, (2) later visual areas show diverging spatial and temporal windows across streams, and (3) within early visual areas (V1-V3), both spatial and temporal windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses using fMRI.

Contrast Sensitivity of ON and OFF Human Retinal Pathways in Myopia.

The human visual cortex processes light and dark stimuli with ON and OFF pathways that are differently modulated by luminance contrast. We have previously demonstrated that ON cortical pathways have higher contrast sensitivity than OFF cortical pathways and the difference increases with luminance range (defined as the maximum minus minimum luminance in the scene). Here, we demonstrate that these ON-OFF cortical differences are already present in the human retina and that retinal responses measured with electroretinography are more affected by reductions in luminance range than cortical responses measured with electroencephalography. Moreover, we show that ON-OFF pathway differences measured with electroretinography become more pronounced in myopia, a visual disorder that elongates the eye and blurs vision at far distance. We find that, as the eye axial length increases across subjects, ON retinal pathways become less responsive, slower in response latency, less sensitive, and less effective and slower at driving pupil constriction. Based on these results, we conclude that myopia is associated with a deficit in ON pathway function that decreases the ability of the retina to process low contrast and regulate retinal illuminance in bright environments.

Category Learning Selectively Enhances Representations of Boundary-Adjacent Exemplars in Early Visual Cortex.

Category learning and visual perception are fundamentally interactive processes, such that successful categorization often depends on the ability to make fine visual discriminations between stimuli that vary on continuously valued dimensions. Research suggests that category learning can improve perceptual discrimination along the stimulus dimensions that predict category membership and that these perceptual enhancements are a byproduct of functional plasticity in the visual system. However, the precise mechanisms underlying learning-dependent sensory modulation in categorization are not well understood. We hypothesized that category learning leads to a representational sharpening of underlying sensory populations tuned to values at or near the category boundary. Furthermore, such sharpening should occur largely during active learning of new categories. These hypotheses were tested using fMRI and a theoretically constrained model of vision to quantify changes in the shape of orientation representations while human adult subjects learned to categorize physically identical stimuli based on either an orientation rule (N = 12) or an orthogonal spatial frequency rule (N = 13). Consistent with our predictions, modeling results revealed relatively enhanced reconstructed representations of stimulus orientation in visual cortex (V1-V3) only for orientation rule learners. Moreover, these reconstructed representations varied as a function of distance from the category boundary, such that representations for challenging stimuli near the boundary were significantly sharper than those for stimuli at the category centers. These results support an efficient model of plasticity wherein only the sensory populations tuned to the most behaviorally relevant regions of feature space are enhanced during category learning.

Mnemonic But Not Contextual Feedback Signals Defy Dedifferentiation in the Aging Early Visual Cortex.

Perception is an intricate interplay between feedforward visual input and internally generated feedback signals that comprise concurrent contextual and time-distant mnemonic (episodic and semantic) information. Yet, an unresolved question is how the composition of feedback signals changes across the lifespan and to what extent feedback signals undergo age-related dedifferentiation, that is, a decline in neural specificity. Previous research on this topic has focused on feedforward perceptual representation and episodic memory reinstatement, suggesting reduced fidelity of neural representations at the item and category levels. In this fMRI study, we combined an occlusion paradigm that filters feedforward input to the visual cortex and multivariate analysis techniques to investigate the information content in cortical feedback, focusing on age-related differences in its composition. We further asked to what extent differentiation in feedback signals (in the occluded region) is correlated to differentiation in feedforward signals. Comparing younger (18-30 years) and older female and male adults (65-75 years), we found that contextual but not mnemonic feedback was prone to age-related dedifferentiation. Semantic feedback signals were even better differentiated in older adults, highlighting the growing importance of generalized knowledge across ages. We also found that differentiation in feedforward signals was correlated with differentiation in episodic but not semantic feedback signals. Our results provide evidence for age-related adjustments in the composition of feedback signals and underscore the importance of examining dedifferentiation in aging for both feedforward and feedback processing.