Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior.

Neural dynamics can reflect intrinsic dynamics or dynamic inputs, such as sensory inputs or inputs from other brain regions. To avoid misinterpreting temporally structured inputs as intrinsic dynamics, dynamical models of neural activity should account for measured inputs. However, incorporating measured inputs remains elusive in joint dynamical modeling of neural-behavioral data, which is important for studying neural computations of behavior. We first show how training dynamical models of neural activity while considering behavior but not input or input but not behavior may lead to misinterpretations. We then develop an analytical learning method for linear dynamical models that simultaneously accounts for neural activity, behavior, and measured inputs. The method provides the capability to prioritize the learning of intrinsic behaviorally relevant neural dynamics and dissociate them from both other intrinsic dynamics and measured input dynamics. In data from a simulated brain with fixed intrinsic dynamics that performs different tasks, the method correctly finds the same intrinsic dynamics regardless of the task while other methods can be influenced by the task. In neural datasets from three subjects performing two different motor tasks with task instruction sensory inputs, the method reveals low-dimensional intrinsic neural dynamics that are missed by other methods and are more predictive of behavior and/or neural activity. The method also uniquely finds that the intrinsic behaviorally relevant neural dynamics are largely similar across the different subjects and tasks, whereas the overall neural dynamics are not. These input-driven dynamical models of neural-behavioral data can uncover intrinsic dynamics that may otherwise be missed.

Scalable Gaussian process inference of neural responses to natural images.

Predicting the responses of sensory neurons is a long-standing neuroscience goal. However, while there has been much progress in modeling neural responses to simple and/or artificial stimuli, predicting responses to natural stimuli remains an ongoing challenge. On the one hand, deep neural networks perform very well on certain datasets but can fail when data are limited. On the other hand, Gaussian processes (GPs) perform well on limited data but are poor at predicting responses to high-dimensional stimuli, such as natural images. Here, we show how structured priors, e.g., for local and smooth receptive fields, can be used to scale up GPs to model neural responses to high-dimensional stimuli. With this addition, GPs largely outperform a deep neural network trained to predict retinal responses to natural images, with the largest differences observed when both models are trained on a small dataset. Further, since they allow us to quantify the uncertainty in their predictions, GPs are well suited to closed-loop experiments, where stimuli are chosen actively so as to collect "informative" neural data. We show how GPs can be used to actively select which stimuli to present, so as to i) efficiently learn a model of retinal responses to natural images, using few data, and ii) rapidly distinguish between competing models (e.g., a linear vs. a nonlinear model). In the future, our approach could be applied to other sensory areas, beyond the retina.

Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space.

Word embedding representations have been shown to be effective in predicting human neural responses to lingual stimuli. While these representations are sensitive to the textual context, they lack the extratextual sources of context such as prior knowledge, thoughts, and beliefs, all of which constitute the listener's perspective. In this study, we propose conceptualizing the listeners' perspective as a source that induces changes in the embedding space. We relied on functional magnetic resonance imaging data collected by Yeshurun Y, Swanson S, Simony E, Chen J, Lazaridi C, Honey CJ, Hasson U. Same story, different story: the neural representation of interpretive frameworks. Psychol Sci. 2017:28(3):307-319, in which two groups of human listeners (n = 40) were listening to the same story but with different perspectives. Using a dedicated fine-tuning process, we created two modified versions of a word embedding space, corresponding to the two groups of listeners. We found that each transformed space was better fitted with neural responses of the corresponding group, and that the spatial distances between these spaces reflect both interpretational differences between the perspectives and the group-level neural differences. Together, our results demonstrate how aligning a continuous embedding space to a specific context can provide a novel way of modeling listeners' intrinsic perspectives.

Idiosyncratic neural coding and neuromodulation of olfactory individuality in Drosophila.

Innate behavioral biases and preferences can vary significantly among individuals of the same genotype. Though individuality is a fundamental property of behavior, it is not currently understood how individual differences in brain structure and physiology produce idiosyncratic behaviors. Here we present evidence for idiosyncrasy in olfactory behavior and neural responses in Drosophila We show that individual female Drosophila from a highly inbred laboratory strain exhibit idiosyncratic odor preferences that persist for days. We used in vivo calcium imaging of neural responses to compare projection neuron (second-order neurons that convey odor information from the sensory periphery to the central brain) responses to the same odors across animals. We found that, while odor responses appear grossly stereotyped, upon closer inspection, many individual differences are apparent across antennal lobe (AL) glomeruli (compact microcircuits corresponding to different odor channels). Moreover, we show that neuromodulation, environmental stress in the form of altered nutrition, and activity of certain AL local interneurons affect the magnitude of interfly behavioral variability. Taken together, this work demonstrates that individual Drosophila exhibit idiosyncratic olfactory preferences and idiosyncratic neural responses to odors, and that behavioral idiosyncrasies are subject to neuromodulation and regulation by neurons in the AL.

NeuroGen: Activation optimized image synthesis for discovery neuroscience.

Functional MRI (fMRI) is a powerful technique that has allowed us to characterize visual cortex responses to stimuli, yet such experiments are by nature constructed based on a priori hypotheses, limited to the set of images presented to the individual while they are in the scanner, are subject to noise in the observed brain responses, and may vary widely across individuals. In this work, we propose a novel computational strategy, which we call NeuroGen, to overcome these limitations and develop a powerful tool for human vision neuroscience discovery. NeuroGen combines an fMRI-trained neural encoding model of human vision with a deep generative network to synthesize images predicted to achieve a target pattern of macro-scale brain activation. We demonstrate that the reduction of noise that the encoding model provides, coupled with the generative network's ability to produce images of high fidelity, results in a robust discovery architecture for visual neuroscience. By using only a small number of synthetic images created by NeuroGen, we demonstrate that we can detect and amplify differences in regional and individual human brain response patterns to visual stimuli. We then verify that these discoveries are reflected in the several thousand observed image responses measured with fMRI. We further demonstrate that NeuroGen can create synthetic images predicted to achieve regional response patterns not achievable by the best-matching natural images. The NeuroGen framework extends the utility of brain encoding models and opens up a new avenue for exploring, and possibly precisely controlling, the human visual system.

A large and rich EEG dataset for modeling human visual object recognition.

The human brain achieves visual object recognition through multiple stages of linear and nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models' prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.

Risk-taking bias in human decision-making is encoded via a right-left brain push-pull system.

A person's decisions vary even when options stay the same, like when a gambler changes bets despite constant odds of winning. Internal bias (e.g., emotion) contributes to this variability and is shaped by past outcomes, yet its neurobiology during decision-making is not well understood. To map neural circuits encoding bias, we administered a gambling task to 10 participants implanted with intracerebral depth electrodes in cortical and subcortical structures. We predicted the variability in betting behavior within and across patients by individual bias, which is estimated through a dynamical model of choice. Our analysis further revealed that high-frequency activity increased in the right hemisphere when participants were biased toward risky bets, while it increased in the left hemisphere when participants were biased away from risky bets. Our findings provide electrophysiological evidence that risk-taking bias is a lateralized push-pull neural system governing counterintuitive and highly variable decision-making in humans.

Spatial Arrangement Drastically Changes the Neural Representation of Multiple Visual Stimuli That Compete in More Than One Feature Domain.

Natural scenes often contain multiple objects and surfaces. However, how neurons in the visual cortex represent multiple visual stimuli is not well understood. Previous studies have shown that, when multiple stimuli compete in one feature domain, the evoked neuronal response is biased toward the stimulus that has a stronger signal strength. We recorded from two male macaques to investigate how neurons in the middle temporal cortex (MT) represent multiple stimuli that compete in more than one feature domain. Visual stimuli were two random-dot patches moving in different directions. One stimulus had low luminance contrast and moved with high coherence, whereas the other had high contrast and moved with low coherence. We found that how MT neurons represent multiple stimuli depended on the spatial arrangement. When two stimuli were overlapping, MT responses were dominated by the stimulus component that had high contrast. When two stimuli were spatially separated within the receptive fields, the contrast dominance was abolished. We found the same results when using contrast to compete with motion speed. Our neural data and computer simulations using a V1-MT model suggest that the contrast dominance found with overlapping stimuli is due to normalization occurring at an input stage fed to MT, and MT neurons cannot overturn this bias based on their own feature selectivity. The interaction between spatially separated stimuli can largely be explained by normalization within MT. Our results revealed new rules on stimulus competition and highlighted the impact of hierarchical processing on representing multiple stimuli in the visual cortex.SIGNIFICANCE STATEMENT Previous studies have shown that the neural representation of multiple visual stimuli can be accounted for by a divisive normalization model. By using multiple stimuli that compete in more than one feature domain, we found that luminance contrast has a dominant effect in determining competition between multiple stimuli when they are overlapping but not spatially separated. Our results revealed that neuronal responses to multiple stimuli in a given cortical area cannot be simply predicted by the population neural responses elicited in that area by the individual stimulus components. To understand the neural representation of multiple stimuli, rather than considering response normalization only within the area of interest, one must consider the computations including normalization occurring along the hierarchical visual pathway.

Effect of spinal cord injury on neural encoding of spontaneous postural perturbations in the hindlimb sensorimotor cortex.

Supraspinal signals play a significant role in compensatory responses to postural perturbations. Although the cortex is not necessary for basic postural tasks in intact animals, its role in responding to unexpected postural perturbations after spinal cord injury (SCI) has not been studied. To better understand how SCI impacts cortical encoding of postural perturbations, the activity of single neurons in the hindlimb sensorimotor cortex (HLSMC) was recorded in the rat during unexpected tilts before and after a complete midthoracic spinal transection. In a subset of animals, limb ground reaction forces were also collected. HLSMC activity was strongly modulated in response to different tilt profiles. As the velocity of the tilt increased, more information was conveyed by the HLSMC neurons about the perturbation due to increases in both the number of recruited neurons and the magnitude of their responses. SCI led to attenuated and delayed hindlimb ground reaction forces. However, HLSMC neurons remained responsive to tilts after injury but with increased latencies and decreased tuning to slower tilts. Information conveyed by cortical neurons about the tilts was therefore reduced after SCI, requiring more cells to convey the same amount of information as before the transection. Given that reorganization of the hindlimb sensorimotor cortex in response to therapy after complete midthoracic SCI is necessary for behavioral recovery, this sustained encoding of information after SCI could be a substrate for the reorganization that uses sensory information from above the lesion to control trunk muscles that permit weight-supported stepping and postural control.NEW & NOTEWORTHY The role of cortical circuits in the encoding of posture and balance is of interest for developing therapies for spinal cord injury. This work demonstrated that unexpected postural perturbations are encoded in the hindlimb sensorimotor cortex even in the absence of hindlimb sensory feedback. In fact, the hindlimb sensorimotor cortex continues to encode for postural perturbations after complete spinal transection.

Neural-inspired sensors enable sparse, efficient classification of spatiotemporal data.

Sparse sensor placement is a central challenge in the efficient characterization of complex systems when the cost of acquiring and processing data is high. Leading sparse sensing methods typically exploit either spatial or temporal correlations, but rarely both. This work introduces a sparse sensor optimization that is designed to leverage the rich spatiotemporal coherence exhibited by many systems. Our approach is inspired by the remarkable performance of flying insects, which use a few embedded strain-sensitive neurons to achieve rapid and robust flight control despite large gust disturbances. Specifically, we identify neural-inspired sensors at a few key locations on a flapping wing that are able to detect body rotation. This task is particularly challenging as the rotational twisting mode is three orders of magnitude smaller than the flapping modes. We show that nonlinear filtering in time, built to mimic strain-sensitive neurons, is essential to detect rotation, whereas instantaneous measurements fail. Optimized sparse sensor placement results in efficient classification with approximately 10 sensors, achieving the same accuracy and noise robustness as full measurements consisting of hundreds of sensors. Sparse sensing with neural-inspired encoding establishes an alternative paradigm in hyperefficient, embodied sensing of spatiotemporal data and sheds light on principles of biological sensing for agile flight control.

How environmental movement constraints shape the neural code for space.

Study of the neural code for space in rodents has many insights to offer for how mammals, including humans, construct a mental representation of space. This code is centered on the hippocampal place cells, which are active in particular places in the environment. Place cells are informed by numerous other spatial cell types including grid cells, which provide a signal for distance and direction and are thought to help anchor the place cell signal. These neurons combine self-motion and environmental information to create and update their map-like representation. Study of their activity patterns in complex environments of varying structure has revealed that this "cognitive map" of space is not a fixed and rigid entity that permeates space, but rather is variably affected by the movement constraints of the environment. These findings are pointing toward a more flexible spatial code in which the map is adapted to the movement possibilities of the space. An as-yet-unanswered question is whether these different forms of representation have functional consequences, as suggested by an enactivist view of spatial cognition.

Diverse coactive neurons encode stimulus-driven and stimulus-independent variables.

Both experimenter-controlled stimuli and stimulus-independent variables impact cortical neural activity. A major hurdle to understanding neural representation is distinguishing between qualitatively different causes of the fluctuating population activity. We applied an unsupervised low-rank tensor decomposition analysis to the recorded population activity in the visual cortex of awake mice in response to repeated presentations of naturalistic visual stimuli. We found that neurons covaried largely independently of individual neuron stimulus response reliability and thus encoded both stimulus-driven and stimulus-independent variables. Importantly, a neuron's response reliability and the neuronal coactivation patterns substantially reorganized for different external visual inputs. Analysis of recurrent balanced neural network models revealed that both the stimulus specificity and the mixed encoding of qualitatively different variables can arise from clustered external inputs. These results establish that coactive neurons with diverse response reliability mediate a mixed representation of stimulus-driven and stimulus-independent variables in the visual cortex.NEW & NOTEWORTHY V1 neurons covary largely independently of individual neuron's response reliability. A single neuron's response reliability imposes only a weak constraint on its encoding capabilities. Visual stimulus instructs a neuron's reliability and coactivation pattern. Network models revealed using clustered external inputs.

Population coding in the cerebellum: a machine learning perspective.

The cere resembles a feedforward, three-layer network of neurons in which the "hidden layer" consists of Purkinje cells (P-cells) and the output layer consists of deep cerebellar nucleus (DCN) neurons. In this analogy, the output of each DCN neuron is a prediction that is compared with the actual observation, resulting in an error signal that originates in the inferior olive. Efficient learning requires that the error signal reach the DCN neurons, as well as the P-cells that project onto them. However, this basic rule of learning is violated in the cerebellum: the olivary projections to the DCN are weak, particularly in adulthood. Instead, an extraordinarily strong signal is sent from the olive to the P-cells, producing complex spikes. Curiously, P-cells are grouped into small populations that converge onto single DCN neurons. Why are the P-cells organized in this way, and what is the membership criterion of each population? Here, I apply elementary mathematics from machine learning and consider the fact that P-cells that form a population exhibit a special property: they can synchronize their complex spikes, which in turn suppress activity of DCN neuron they project to. Thus complex spikes cannot only act as a teaching signal for a P-cell, but through complex spike synchrony, a P-cell population may act as a surrogate teacher for the DCN neuron that produced the erroneous output. It appears that grouping of P-cells into small populations that share a preference for error satisfies a critical requirement of efficient learning: providing error information to the output layer neuron (DCN) that was responsible for the error, as well as the hidden layer neurons (P-cells) that contributed to it. This population coding may account for several remarkable features of behavior during learning, including multiple timescales, protection from erasure, and spontaneous recovery of memory.

Semicircular canal biomechanics in health and disease.

The semicircular canals are responsible for sensing angular head motion in three-dimensional space and for providing neural inputs to the central nervous system (CNS) essential for agile mobility, stable vision, and autonomic control of the cardiovascular and other gravity-sensitive systems. Sensation relies on fluid mechanics within the labyrinth to selectively convert angular head acceleration into sensory hair bundle displacements in each of three inner ear sensory organs. Canal afferent neurons encode the direction and time course of head movements over a broad range of movement frequencies and amplitudes. Disorders altering canal mechanics result in pathological inputs to the CNS, often leading to debilitating symptoms. Vestibular disorders and conditions with mechanical substrates include benign paroxysmal positional nystagmus, direction-changing positional nystagmus, alcohol positional nystagmus, caloric nystagmus, Tullio phenomena, and others. Here, the mechanics of angular motion transduction and how it contributes to neural encoding by the semicircular canals is reviewed in both health and disease.

Neural Tuning to Low-Level Features of Speech throughout the Perisylvian Cortex.

Despite a large body of research, we continue to lack a detailed account of how auditory processing of continuous speech unfolds in the human brain. Previous research showed the propagation of low-level acoustic features of speech from posterior superior temporal gyrus toward anterior superior temporal gyrus in the human brain (Hullett et al., 2016). In this study, we investigate what happens to these neural representations past the superior temporal gyrus and how they engage higher-level language processing areas such as inferior frontal gyrus. We used low-level sound features to model neural responses to speech outside of the primary auditory cortex. Two complementary imaging techniques were used with human participants (both males and females): electrocorticography (ECoG) and fMRI. Both imaging techniques showed tuning of the perisylvian cortex to low-level speech features. With ECoG, we found evidence of propagation of the temporal features of speech sounds along the ventral pathway of language processing in the brain toward inferior frontal gyrus. Increasingly coarse temporal features of speech spreading from posterior superior temporal cortex toward inferior frontal gyrus were associated with linguistic features such as voice onset time, duration of the formant transitions, and phoneme, syllable, and word boundaries. The present findings provide the groundwork for a comprehensive bottom-up account of speech comprehension in the human brain.SIGNIFICANCE STATEMENT We know that, during natural speech comprehension, a broad network of perisylvian cortical regions is involved in sound and language processing. Here, we investigated the tuning to low-level sound features within these regions using neural responses to a short feature film. We also looked at whether the tuning organization along these brain regions showed any parallel to the hierarchy of language structures in continuous speech. Our results show that low-level speech features propagate throughout the perisylvian cortex and potentially contribute to the emergence of "coarse" speech representations in inferior frontal gyrus typically associated with high-level language processing. These findings add to the previous work on auditory processing and underline a distinctive role of inferior frontal gyrus in natural speech comprehension.

Transferring and generalizing deep-learning-based neural encoding models across subjects.

Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision.

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.

Primate beta oscillations and rhythmic behaviors.

The study of non-human primates in complex behaviors such as rhythm perception and entrainment is critical to understand the neurophysiological basis of human cognition. Next to reviewing the role of beta oscillations in human beat perception, here we discuss the role of primate putaminal oscillatory activity in the control of rhythmic movements that are guided by a sensory metronome or internally gated. The analysis of the local field potentials of the behaving macaques showed that gamma-oscillations reflect local computations associated with stimulus processing of the metronome, whereas beta-activity involves the entrainment of large putaminal circuits, probably in conjunction with other elements of cortico-basal ganglia-thalamo-cortical circuit, during internally driven rhythmic tapping. Thus, this review emphasizes the need of parametric neurophysiological observations in non-human primates that display a well-controlled behavior during high-level cognitive processes.

Neural population coding of multiple stimuli.

In natural scenes, objects generally appear together with other objects. Yet, theoretical studies of neural population coding typically focus on the encoding of single objects in isolation. Experimental studies suggest that neural responses to multiple objects are well described by linear or nonlinear combinations of the responses to constituent objects, a phenomenon we call stimulus mixing. Here, we present a theoretical analysis of the consequences of common forms of stimulus mixing observed in cortical responses. We show that some of these mixing rules can severely compromise the brain's ability to decode the individual objects. This cost is usually greater than the cost incurred by even large reductions in the gain or large increases in neural variability, explaining why the benefits of attention can be understood primarily in terms of a stimulus selection, or demixing, mechanism rather than purely as a gain increase or noise reduction mechanism. The cost of stimulus mixing becomes even higher when the number of encoded objects increases, suggesting a novel mechanism that might contribute to set size effects observed in myriad psychophysical tasks. We further show that a specific form of neural correlation and heterogeneity in stimulus mixing among the neurons can partially alleviate the harmful effects of stimulus mixing. Finally, we derive simple conditions that must be satisfied for unharmful mixing of stimuli.

Effect of background noise on neuronal coding of interaural level difference cues in rat inferior colliculus.

Humans can accurately localize sounds even in unfavourable signal-to-noise conditions. To investigate the neural mechanisms underlying this, we studied the effect of background wide-band noise on neural sensitivity to variations in interaural level difference (ILD), the predominant cue for sound localization in azimuth for high-frequency sounds, at the characteristic frequency of cells in rat inferior colliculus (IC). Binaural noise at high levels generally resulted in suppression of responses (55.8%), but at lower levels resulted in enhancement (34.8%) as well as suppression (30.3%). When recording conditions permitted, we then examined if any binaural noise effects were related to selective noise effects at each of the two ears, which we interpreted in light of well-known differences in input type (excitation and inhibition) from each ear shaping particular forms of ILD sensitivity in the IC. At high signal-to-noise ratios (SNR), in most ILD functions (41%), the effect of background noise appeared to be due to effects on inputs from both ears, while for a large percentage (35.8%) appeared to be accounted for by effects on excitatory input. However, as SNR decreased, change in excitation became the dominant contributor to the change due to binaural background noise (63.6%). These novel findings shed light on the IC neural mechanisms for sound localization in the presence of continuous background noise. They also suggest that some effects of background noise on encoding of sound location reported to be emergent in upstream auditory areas can also be observed at the level of the midbrain.