Effective gene expression prediction from sequence by integrating long-range interactions.

How noncoding DNA determines gene expression in different cell types is a major unsolved problem, and critical downstream applications in human genetics depend on improved solutions. Here, we report substantially improved gene expression prediction accuracy from DNA sequences through the use of a deep learning architecture, called Enformer, that is able to integrate information from long-range interactions (up to 100 kb away) in the genome. This improvement yielded more accurate variant effect predictions on gene expression for both natural genetic variants and saturation mutagenesis measured by massively parallel reporter assays. Furthermore, Enformer learned to predict enhancer-promoter interactions directly from the DNA sequence competitively with methods that take direct experimental data as input. We expect that these advances will enable more effective fine-mapping of human disease associations and provide a framework to interpret cis-regulatory evolution.

Overcoming catastrophic forgetting in neural networks.

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

A probabilistic approach to demixing odors.

The olfactory system faces a hard problem: on the basis of noisy information from olfactory receptor neurons (the neurons that transduce chemicals to neural activity), it must figure out which odors are present in the world. Odors almost never occur in isolation, and different odors excite overlapping populations of olfactory receptor neurons, so the central challenge of the olfactory system is to demix its input. Because of noise and the large number of possible odors, demixing is fundamentally a probabilistic inference task. We propose that the early olfactory system uses approximate Bayesian inference to solve it. The computations involve a dynamical loop between the olfactory bulb and the piriform cortex, with cortex explaining incoming activity from the olfactory receptor neurons in terms of a mixture of odors. The model is compatible with known anatomy and physiology, including pattern decorrelation, and it performs better than other models at demixing odors.

Hybrid computing using a neural network with dynamic external memory.

Artificial neural networks are remarkably adept at sensory processing, sequence learning and reinforcement learning, but are limited in their ability to represent variables and data structures and to store data over long timescales, owing to the lack of an external memory. Here we introduce a machine learning model called a differentiable neural computer (DNC), which consists of a neural network that can read from and write to an external memory matrix, analogous to the random-access memory in a conventional computer. Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data. When trained with supervised learning, we demonstrate that a DNC can successfully answer synthetic questions designed to emulate reasoning and inference problems in natural language. We show that it can learn tasks such as finding the shortest path between specified points and inferring the missing links in randomly generated graphs, and then generalize these tasks to specific graphs such as transport networks and family trees. When trained with reinforcement learning, a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols. Taken together, our results demonstrate that DNCs have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read-write memory.

How well do mean field theories of spiking quadratic-integrate-and-fire networks work in realistic parameter regimes?

We use mean field techniques to compute the distribution of excitatory and inhibitory firing rates in large networks of randomly connected spiking quadratic integrate and fire neurons. These techniques are based on the assumption that activity is asynchronous and Poisson. For most parameter settings these assumptions are strongly violated; nevertheless, so long as the networks are not too synchronous, we find good agreement between mean field prediction and network simulations. Thus, much of the intuition developed for randomly connected networks in the asynchronous regime applies to mildly synchronous networks.

The 9th annual computational and systems neuroscience (cosyne) meeting.

The 9th annual Computational and Systems Neuroscience meeting (Cosyne) was held 23-26 February in Salt Lake City, Utah. Cosyne meeting is the forum for exchange of experimental and theoretical/computational approaches to studying systems neuroscience.

Orientation selective or not? - Measuring significance of tuning to a circular parameter.

Orientation and direction tuning are among the most studied features of the visual system and are routinely measured during experiments to estimate the quality of neuronal responses. However, standard approaches to report orientation selectivity are only narrowly quantitative and strongly depend on the signal quality, while the more sophisticated ones are computationally exhaustive, making them difficult to use during ongoing experiments. We propose a fast and efficient method for reporting the reliability of coding applicable to any circular parameter. Similar to standard deviation in the linear statistics, reproducibility measures trial-to-trial variability of a circular response parameter. Reproducibility is a normalized measure easily transformed to p-values, which provide explicit information about significance of the estimated orientation preference. The proposed approach is applicable to a wide range of signal types. Here, we discuss examples from optical imaging and electrophysiological recordings, and provide a more thorough examination based on tuning curves modeled in silico.

Nasal chemosensory-stimulation evoked activity patterns in the rat trigeminal ganglion visualized by in vivo voltage-sensitive dye imaging.

Mammalian nasal chemosensation is predominantly mediated by two independent neuronal pathways, the olfactory and the trigeminal system. Within the early olfactory system, spatiotemporal responses of the olfactory bulb to various odorants have been mapped in great detail. In contrast, far less is known about the representation of volatile chemical stimuli at an early stage in the trigeminal system, the trigeminal ganglion (TG), which contains neurons directly projecting to the nasal cavity. We have established an in vivo preparation that allows high-resolution imaging of neuronal population activity from a large region of the rat TG using voltage-sensitive dyes (VSDs). Application of different chemical stimuli to the nasal cavity elicited distinct, stimulus-category specific, spatiotemporal activation patterns that comprised activated as well as suppressed areas. Thus, our results provide the first direct insights into the spatial representation of nasal chemosensory information within the trigeminal ganglion imaged at high temporal resolution.

Dominant vertical orientation processing without clustered maps: early visual brain dynamics imaged with voltage-sensitive dye in the pigeon visual Wulst.

The pigeon is a widely established behavioral model of visual cognition, but the processes along its most basic visual pathways remain mostly unexplored. Here, we report the neuronal population dynamics of the visual Wulst, an assumed homolog of the mammalian striate cortex, captured for the first time with voltage-sensitive dye imaging. Responses to drifting gratings were characterized by focal emergence of activity that spread extensively across the entire Wulst, followed by rapid adaptation that was most effective in the surround. Using additional electrophysiological recordings, we found cells that prefer a variety of orientations. However, analysis of the imaged spatiotemporal activation patterns revealed no clustered orientation map-like arrangements as typically found in the primary visual cortices of many mammalian species. Instead, the vertical orientation was overrepresented, both in terms of the imaged population signal, as well as the number of neurons preferring the vertical orientation. Such enhanced selectivity for the vertical orientation may result from horizontal motion vectors that trigger adaptation to the extensive flow field input during natural behavior. Our findings suggest that, although the avian visual Wulst is homologous to the primary visual cortex in terms of its gross anatomical connectivity and topology, its detailed operation and internal organization is still shaped according to specific input characteristics.

Contrast independence of cardinal preference: stable oblique effect in orientation maps of ferret visual cortex.

The oblique effect was first described as enhanced detection and discrimination of cardinal orientations compared with oblique orientations. Such biases in visual processing are believed to originate from a functional adaptation to environmental statistics dominated by cardinal contours. At the neuronal level, the oblique orientation effect corresponds to the numerical overrepresentation and narrower tuning bandwidths of cortical neurons representing the cardinal axes. The anisotropic distribution of orientation preferences over large cortical regions was revealed with optical imaging, providing further evidence for the cortical oblique effect in several mammalian species. Our present study explores whether the dominant representation of cardinal contours persists at different stimulus contrasts. Performing intrinsic optical imaging in the ferret visual cortex and presenting drifting gratings at various orientations and contrasts (100%, 30% and 10%), we found that the overrepresentation of vertical and horizontal contours was invariant across stimulus contrasts. In addition, the responses to cardinal orientations were also more robust and evoked larger modulation depths than responses to oblique orientations. We conclude that orientation maps remain constant across the full range of contrast levels down to detection thresholds. Thus, a stable layout of the functional architecture dedicated to processing oriented edges seems to reflect a fundamental coding strategy of the early visual cortex.

Establishment of a scaffold for orientation maps in primary visual cortex of higher mammals.

In higher mammals, environmentally driven patterns of neural activity do not play a role in the establishment of orientation specificity and maps. It has been proposed that specific long-range interactions provide the scaffold for developing orientation maps. Our model aims at explaining how such a scaffold could develop in the first place. Broad spontaneous activity waves and locally evoked spatially periodic response pattern are used. The model is discussed in relation to biological evidence, and experiments to test the model are proposed. We show that reliable orientation specificity cannot be a result of haphazard cortical wiring, as has been proposed.

A model of event-related EEG synchronization changes in beta and gamma frequency bands.

During preparation, execution and recovery from simple movements, the EEG power spectrum undergoes a sequence of changes. The power in the beta band (13-25 Hz) decreases during preparation and execution of movement, but during recovery it reaches a level higher than that in the reference period (not affected by the event). These effects are known as event-related beta desynchronization and beta rebound. The power in the gamma band (>30 Hz) increases significantly just before the onset of the movement. This effect is known as event-related gamma synchronization. There are numerous observations concerning these effects but the underlying physiological mechanisms and functional role are not clear. We propose a lumped computational model of a cortical circuit. The model consists only of a pyramidal and an interneuronal population. Each population represents averaged properties of constituting neurons. The output of the model represents a local field potential, with a power spectrum peak either in the beta or in the gamma band. The model elucidates the mechanisms of transition between slower and faster rhythms, gamma synchronization and beta desynchronization and rebound effects. The sufficient conditions to observe the effects in the model are changes of the external excitation level and of the connection strength between excitatory and inhibitory populations attributed to short-time plasticity. The present model presents the role of the pyramidal neurons to interneuron connection in the oscillatory behavior of the two populations. We conclude that the pronounced facilitation of the pyramidal to fast spiking interneuron connections, initiated by robust excitation of the motor cortex neurons, may be essential for the effect of beta rebound. Further experiments concerning short-time plasticity during behavioral tasks would be of great value in studies of functional local cortical circuits.