Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm.

A critical bottleneck for the training of large neural networks (NNs) is communication with off-chip memory. A promising mitigation effort consists of integrating crossbar arrays of analogue memories in the Back-End-Of-Line, to store the NN parameters and efficiently perform the required synaptic operations. The "Tiki-Taka" algorithm was developed to facilitate NN training in the presence of device nonidealities. However, so far, a resistive switching device exhibiting all the fundamental Tiki-Taka requirements, which are many programmable states, a centered symmetry point, and low programming noise, was not yet demonstrated. Here, a complementary metal-oxide semiconductor (CMOS)-compatible resistive random access memory (RRAM), showing more than 30 programmable states with low noise and a symmetry point with only 5% skew from the center, is presented for the first time. These results enable generalization of Tiki-Taka training from small fully connected networks to larger long-/short-term-memory types of NN.

Selective Modulation of Axonal Sodium Channel Subtypes by 5-HT1A Receptor in Cortical Pyramidal Neuron.

Serotonergic innervation of the prefrontal cortex (PFC) modulates neuronal activity and PFC functions. However, the cellular mechanism for serotonergic modulation of neuronal excitability remains unclear. We performed patch-clamp recording at the axon of layer-5 pyramidal neurons in rodent PFC slices. We found surprisingly that the activation of 5-HT1A receptors selectively inhibits Na+ currents obtained at the axon initial segment (AIS) but not those at the axon trunk. In addition, Na+ channel subtype NaV1.2 but not NaV1.6 at the AIS is selectively modulated by 5-HT1A receptors. Further experiments revealed that the inhibitory effect is attributable to a depolarizing shift of the activation curve and a facilitation of slow inactivation of AIS Na+ currents. Consistently, dual somatic and axonal recording and simulation results demonstrate that the activation of 5-HT1A receptors could decrease the success rate of action potential (AP) backpropagation toward the somatodendritic compartments, enhancing the segregation of axonal and dendritic activities. Together, our results reveal a selective modulation of NaV1.2 distributed at the proximal AIS region and AP backpropagation by 5-HT1A receptors, suggesting a potential mechanism for serotonergic regulation of functional polarization in the dendro-axonal axis, synaptic plasticity and PFC functions.

Decentralized Multisensory Information Integration in Neural Systems.

How multiple sensory cues are integrated in neural circuitry remains a challenge. The common hypothesis is that information integration might be accomplished in a dedicated multisensory integration area receiving feedforward inputs from the modalities. However, recent experimental evidence suggests that it is not a single multisensory brain area, but rather many multisensory brain areas that are simultaneously involved in the integration of information. Why many mutually connected areas should be needed for information integration is puzzling. Here, we investigated theoretically how information integration could be achieved in a distributed fashion within a network of interconnected multisensory areas. Using biologically realistic neural network models, we developed a decentralized information integration system that comprises multiple interconnected integration areas. Studying an example of combining visual and vestibular cues to infer heading direction, we show that such a decentralized system is in good agreement with anatomical evidence and experimental observations. In particular, we show that this decentralized system can integrate information optimally. The decentralized system predicts that optimally integrated information should emerge locally from the dynamics of the communication between brain areas and sheds new light on the interpretation of the connectivity between multisensory brain areas. SIGNIFICANCE STATEMENT: To extract information reliably from ambiguous environments, the brain integrates multiple sensory cues, which provide different aspects of information about the same entity of interest. Here, we propose a decentralized architecture for multisensory integration. In such a system, no processor is in the center of the network topology and information integration is achieved in a distributed manner through reciprocally connected local processors. Through studying the inference of heading direction with visual and vestibular cues, we show that the decentralized system can integrate information optimally, with the reciprocal connections between processers determining the extent of cue integration. Our model reproduces known multisensory integration behaviors observed in experiments and sheds new light on our understanding of how information is integrated in the brain.

Binocular input coincidence mediates critical period plasticity in the mouse primary visual cortex.

Classical studies on the development of ocular dominance (OD) organization in primary visual cortex (V1) have revealed a postnatal critical period (CP), during which visual inputs between the two eyes are most effective in shaping cortical circuits through synaptic competition. A brief closure of one eye during CP caused a pronounced shift of response preference of V1 neurons toward the open eye, a form of CP plasticity in the developing V1. However, it remains unclear what particular property of binocular inputs during CP is responsible for mediating this experience-dependent OD plasticity. Using whole-cell recording in mouse V1, we found that visually driven synaptic inputs from the two eyes to binocular cells in layers 2/3 and 4 became highly coincident during CP. Enhancing cortical GABAergic transmission activity by brain infusion with diazepam not only caused a precocious onset of the high coincidence of binocular inputs and OD plasticity in pre-CP mice, but rescued both of them in dark-reared mice, suggesting a tight link between coincident binocular inputs and CP plasticity. In Thy1-ChR2 mice, chronic disruption of this binocular input coincidence during CP by asynchronous optogenetic activation of retinal ganglion cells abolished the OD plasticity. Computational simulation using a feed-forward network model further suggests that the coincident inputs could mediate this CP plasticity through a homeostatic synaptic learning mechanism with synaptic competition. These results suggest that the high-level correlation of binocular inputs is a hallmark of the CP of developing V1 and serves as neural substrate for the induction of OD plasticity.

Laterality effect for faces in chimpanzees (Pan troglodytes).

Face perception in humans is governed more by right-hemispheric than left-hemispheric neural correlate. Some but not all neurophysiological studies depict a right-side dominance for face responsive neurons in the brains of macaques. Hence, it is an open question whether and to what extent a right-hemisphere preference of processing faces exists across primate brains. We investigated chimpanzees discriminating chimeric faces of chimpanzees and humans, i.e., the combination of either left or right sides of a face vertically flipped and merged into a whole face. We found an effect of choosing the left-chimeric face more often than the right-chimeric face as being the one of the two that is closer to the original face, reflecting an advantage for the right side of the brain to process faces, as reported in humans. Moreover, we found a modulation by age of the participants, suggesting that the exposure history with a particular category shapes the right-hemispheric neural correlate to a configural/holistic processing strategy. In other words, the findings in chimpanzee participants parallel those in human participants and are suggestive for similar neural machineries in the occipital-temporal cortices in both species.

Fast and robust analog in-memory deep neural network training.

Analog in-memory computing is a promising future technology for efficiently accelerating deep learning networks. While using in-memory computing to accelerate the inference phase has been studied extensively, accelerating the training phase has received less attention, despite its arguably much larger compute demand to accelerate. While some analog in-memory training algorithms have been suggested, they either invoke significant amount of auxiliary digital compute-accumulating the gradient in digital floating point precision, limiting the potential speed-up-or suffer from the need for near perfectly programming reference conductance values to establish an algorithmic zero point. Here, we propose two improved algorithms for in-memory training, that retain the same fast runtime complexity while resolving the requirement of a precise zero point. We further investigate the limits of the algorithms in terms of conductance noise, symmetry, retention, and endurance which narrow down possible device material choices adequate for fast and robust in-memory deep neural network training.

Quantitative inference of population response properties across eccentricity from motion-induced maps in macaque V1.

Interpreting population responses in the primary visual cortex (V1) remains a challenge especially with the advent of techniques measuring activations of large cortical areas simultaneously with high precision. For successful interpretation, a quantitatively precise model prediction is of great importance. In this study, we investigate how accurate a spatiotemporal filter (STF) model predicts average response profiles to coherently drifting random dot motion obtained by optical imaging of intrinsic signals in V1 of anesthetized macaques. We establish that orientation difference maps, obtained by subtracting orthogonal axis-of-motion, invert with increasing drift speeds, consistent with the motion streak effect. Consistent with perception, the speed at which the map inverts (the critical speed) depends on cortical eccentricity and systematically increases from foveal to parafoveal. We report that critical speeds and response maps to drifting motion are excellently reproduced by the STF model. Our study thus suggests that the STF model is quantitatively accurate enough to be used as a first model of choice for interpreting responses obtained with intrinsic imaging methods in V1. We show further that this good quantitative correspondence opens the possibility to infer otherwise not easily accessible population receptive field properties from responses to complex stimuli, such as drifting random dot motions.

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators.

Analog in-memory computing-a promising approach for energy-efficient acceleration of deep learning workloads-computes matrix-vector multiplications but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable inference accuracy. Here, we develop an hardware-aware retraining approach to systematically examine the accuracy of analog in-memory computing across multiple network topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a realistic crossbar model, we improve significantly on earlier retraining approaches. We show that many larger-scale deep neural networks-including convnets, recurrent networks, and transformers-can in fact be successfully retrained to show iso-accuracy with the floating point implementation. Our results further suggest that nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on accuracy, and that recurrent networks are particularly robust to all nonidealities.

Optimised weight programming for analogue memory-based deep neural networks.

Analogue memory-based deep neural networks provide energy-efficiency and per-area throughput gains relative to state-of-the-art digital counterparts such as graphics processing units. Recent advances focus largely on hardware-aware algorithmic training and improvements to circuits, architectures, and memory devices. Optimal translation of software-trained weights into analogue hardware weights-given the plethora of complex memory non-idealities-represents an equally important task. We report a generalised computational framework that automates the crafting of complex weight programming strategies to minimise accuracy degradations during inference, particularly over time. The framework is agnostic to network structure and generalises well across recurrent, convolutional, and transformer neural networks. As a highly flexible numerical heuristic, the approach accommodates arbitrary device-level complexity, making it potentially relevant for a variety of analogue memories. By quantifying the limit of achievable inference accuracy, it also enables analogue memory-based deep neural network accelerators to reach their full inference potential.

Statistical comparison of spike responses to natural stimuli in monkey area V1 with simulated responses of a detailed laminar network model for a patch of V1.

A major goal of computational neuroscience is the creation of computer models for cortical areas whose response to sensory stimuli resembles that of cortical areas in vivo in important aspects. It is seldom considered whether the simulated spiking activity is realistic (in a statistical sense) in response to natural stimuli. Because certain statistical properties of spike responses were suggested to facilitate computations in the cortex, acquiring a realistic firing regimen in cortical network models might be a prerequisite for analyzing their computational functions. We present a characterization and comparison of the statistical response properties of the primary visual cortex (V1) in vivo and in silico in response to natural stimuli. We recorded from multiple electrodes in area V1 of 4 macaque monkeys and developed a large state-of-the-art network model for a 5 × 5-mm patch of V1 composed of 35,000 neurons and 3.9 million synapses that integrates previously published anatomical and physiological details. By quantitative comparison of the model response to the "statistical fingerprint" of responses in vivo, we find that our model for a patch of V1 responds to the same movie in a way which matches the statistical structure of the recorded data surprisingly well. The deviation between the firing regimen of the model and the in vivo data are on the same level as deviations among monkeys and sessions. This suggests that, despite strong simplifications and abstractions of cortical network models, they are nevertheless capable of generating realistic spiking activity. To reach a realistic firing state, it was not only necessary to include both N-methyl-d-aspartate and GABA(B) synaptic conductances in our model, but also to markedly increase the strength of excitatory synapses onto inhibitory neurons (>2-fold) in comparison to literature values, hinting at the importance to carefully adjust the effect of inhibition for achieving realistic dynamics in current network models.

Phase-of-firing coding of natural visual stimuli in primary visual cortex.

We investigated the hypothesis that neurons encode rich naturalistic stimuli in terms of their spike times relative to the phase of ongoing network fluctuations rather than only in terms of their spike count. We recorded local field potentials (LFPs) and multiunit spikes from the primary visual cortex of anaesthetized macaques while binocularly presenting a color movie. We found that both the spike counts and the low-frequency LFP phase were reliably modulated by the movie and thus conveyed information about it. Moreover, movie periods eliciting higher firing rates also elicited a higher reliability of LFP phase across trials. To establish whether the LFP phase at which spikes were emitted conveyed visual information that could not be extracted by spike rates alone, we compared the Shannon information about the movie carried by spike counts to that carried by the phase of firing. We found that at low LFP frequencies, the phase of firing conveyed 54% additional information beyond that conveyed by spike counts. The extra information available in the phase of firing was crucial for the disambiguation between stimuli eliciting high spike rates of similar magnitude. Thus, phase coding may allow primary cortical neurons to represent several effective stimuli in an easily decodable format.

Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices.

Recent advances in deep learning have been driven by ever-increasing model sizes, with networks growing to millions or even billions of parameters. Such enormous models call for fast and energy-efficient hardware accelerators. We study the potential of Analog AI accelerators based on Non-Volatile Memory, in particular Phase Change Memory (PCM), for software-equivalent accurate inference of natural language processing applications. We demonstrate a path to software-equivalent accuracy for the GLUE benchmark on BERT (Bidirectional Encoder Representations from Transformers), by combining noise-aware training to combat inherent PCM drift and noise sources, together with reduced-precision digital attention-block computation down to INT6.

RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays.

Analog arrays are a promising emerging hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they employ analog circuitry to compute matrix-vector products in constant time, irrespective of the size of the matrix. However, ConvNets map very unfavorably onto analog arrays when done in a straight-forward manner, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times. Here, we propose to parallelize the training by replicating the kernel matrix of a convolution layer on distinct analog arrays, and randomly divide parts of the compute among them. With this modification, analog arrays execute ConvNets with a large acceleration factor that is proportional to the number of kernel matrices used per layer (here tested 16-1024). Despite having more free parameters, we show analytically and in numerical experiments that this new convolution architecture is self-regularizing and implicitly learns similar filters across arrays. We also report superior performance on a number of datasets and increased robustness to adversarial attacks. Our investigation suggests to revise the notion that emerging hardware architectures that feature analog arrays for fast matrix-vector multiplication are not suitable for ConvNets.

Training LSTM Networks With Resistive Cross-Point Devices.

In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to the device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. Here we attempt to study the validity of the RPU approach by simulating large scale networks. For instance, the models studied here are roughly 1500 times larger than the more often studied multilayer perceptron models trained on the MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch.

Autapses enhance bursting and coincidence detection in neocortical pyramidal cells.

Autapses are synaptic contacts of a neuron's axon onto its own dendrite and soma. In the neocortex, self-inhibiting autapses in GABAergic interneurons are abundant in number and play critical roles in regulating spike precision and network activity. Here we examine whether the principal glutamatergic pyramidal cells (PCs) also form functional autapses. In patch-clamp recording from both rodent and human PCs, we isolated autaptic responses and found that these occur predominantly in layer-5 PCs projecting to subcortical regions, with very few in those projecting to contralateral prefrontal cortex and layer 2/3 PCs. Moreover, PC autapses persist during development into adulthood. Surprisingly, they produce giant postsynaptic responses (∼5 fold greater than recurrent PC-PC synapses) that are exclusively mediated by AMPA receptors. Upon activation, autapses enhance burst firing, neuronal responsiveness and coincidence detection of synaptic inputs. These findings indicate that PC autapses are functional and represent an important circuit element in the neocortex.

Integrating structured biological data by Kernel Maximum Mean Discrepancy.

MOTIVATION: Many problems in data integration in bioinformatics can be posed as one common question: Are two sets of observations generated by the same distribution? We propose a kernel-based statistical test for this problem, based on the fact that two distributions are different if and only if there exists at least one function having different expectation on the two distributions. Consequently we use the maximum discrepancy between function means as the basis of a test statistic. The Maximum Mean Discrepancy (MMD) can take advantage of the kernel trick, which allows us to apply it not only to vectors, but strings, sequences, graphs, and other common structured data types arising in molecular biology. RESULTS: We study the practical feasibility of an MMD-based test on three central data integration tasks: Testing cross-platform comparability of microarray data, cancer diagnosis, and data-content based schema matching for two different protein function classification schemas. In all of these experiments, including high-dimensional ones, MMD is very accurate in finding samples that were generated from the same distribution, and outperforms its best competitors. CONCLUSIONS: We have defined a novel statistical test of whether two samples are from the same distribution, compatible with both multivariate and structured data, that is fast, easy to implement, and works well, as confirmed by our experiments. AVAILABILITY: http://www.dbs.ifi.lmu.de/~borgward/MMD.

Distinct Inhibitory Circuits Orchestrate Cortical beta and gamma Band Oscillations.

Distinct subtypes of inhibitory interneuron are known to shape diverse rhythmic activities in the cortex, but how they interact to orchestrate specific band activity remains largely unknown. By recording optogenetically tagged interneurons of specific subtypes in the primary visual cortex of behaving mice, we show that spiking of somatostatin (SOM)- and parvalbumin (PV)-expressing interneurons preferentially correlates with cortical beta and gamma band oscillations, respectively. Suppression of SOM cell spiking reduces the spontaneous low-frequency band (<30-Hz) oscillations and selectively reduces visually induced enhancement of beta oscillation. In comparison, suppressing PV cell activity elevates the synchronization of spontaneous activity across a broad frequency range and further precludes visually induced changes in beta and gamma oscillations. Rhythmic activation of SOM and PV cells in the local circuit entrains resonant activity in the narrow 5- to 30-Hz band and the wide 20- to 80-Hz band, respectively. Together, these findings reveal differential and cooperative roles of SOM and PV inhibitory neurons in orchestrating specific cortical oscillations.

A distinct entorhinal cortex to hippocampal CA1 direct circuit for olfactory associative learning.

Lateral and medial parts of entorhinal cortex (EC) convey nonspatial 'what' and spatial 'where' information, respectively, into hippocampal CA1, via both the indirect EC layer 2→ hippocampal dentate gyrus→CA3→CA1 and the direct EC layer 3→CA1 paths. However, it remains elusive how the direct path transfers distinct information and contributes to hippocampal learning functions. Here we report that lateral EC projection neurons selectively form direct excitatory synapses onto a subpopulation of morphologically complex, calbindin-expressing pyramidal cells (PCs) in the dorsal CA1 (dCA1), while medial EC neurons uniformly innervate all dCA1 PCs. Optogenetically inactivating the distinct lateral EC-dCA1 connections or the postsynaptic dCA1 calbindin-expressing PC activity slows olfactory associative learning. Moreover, optetrode recordings reveal that dCA1 calbindin-expressing PCs develop more selective spiking responses to odor cues during learning. Thus, our results identify a direct lateral EC→dCA1 circuit that is required for olfactory associative learning.

Integration or separation in the processing of facial properties--a computational view.

A face recognition system ought to read out information about the identity, facial expression and invariant properties of faces, such as sex and race. A current debate is whether separate neural units in the brain deal with these face properties individually or whether a single neural unit processes in parallel all aspects of faces. While the focus of studies has been directed toward the processing of identity and facial expression, little research exists on the processing of invariant aspects of faces. In a theoretical framework we tested whether a system can deal with identity in combination with sex, race or facial expression using the same underlying mechanism. We used dimension reduction to describe how the representational face space organizes face properties when trained on different aspects of faces. When trained to learn identities, the system not only successfully recognized identities, but also was immediately able to classify sex and race, suggesting that no additional system for the processing of invariant properties is needed. However, training on identity was insufficient for the recognition of facial expressions and vice versa. We provide a theoretical approach on the interconnection of invariant facial properties and the separation of variant and invariant facial properties.

Selective Maturation of Temporal Dynamics of Intracortical Excitatory Transmission at the Critical Period Onset.

Although the developmental maturation of cortical inhibitory synapses is known to be a critical factor in gating the onset of critical period (CP) for experience-dependent cortical plasticity, how synaptic transmission dynamics of other cortical synapses are regulated during the transition to CP remains unknown. Here, by systematically examining various intracortical synapses within layer 4 of the mouse visual cortex, we demonstrate that synaptic temporal dynamics of intracortical excitatory synapses on principal cells (PCs) and inhibitory parvalbumin- or somatostatin-expressing cells are selectively regulated before the CP onset, whereas those of intracortical inhibitory synapses and long-range thalamocortical excitatory synapses remain unchanged. This selective maturation of synaptic dynamics results from a ubiquitous reduction of presynaptic release and is dependent on visual experience. These findings provide an additional essential circuit mechanism for regulating CP timing in the developing visual cortex.