Analog Resistive Switching Devices for Training Deep Neural Networks with the Novel Tiki-Taka Algorithm.

A critical bottleneck for the training of large neural networks (NNs) is communication with off-chip memory. A promising mitigation effort consists of integrating crossbar arrays of analogue memories in the Back-End-Of-Line, to store the NN parameters and efficiently perform the required synaptic operations. The "Tiki-Taka" algorithm was developed to facilitate NN training in the presence of device nonidealities. However, so far, a resistive switching device exhibiting all the fundamental Tiki-Taka requirements, which are many programmable states, a centered symmetry point, and low programming noise, was not yet demonstrated. Here, a complementary metal-oxide semiconductor (CMOS)-compatible resistive random access memory (RRAM), showing more than 30 programmable states with low noise and a symmetry point with only 5% skew from the center, is presented for the first time. These results enable generalization of Tiki-Taka training from small fully connected networks to larger long-/short-term-memory types of NN.

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators.

Analog in-memory computing-a promising approach for energy-efficient acceleration of deep learning workloads-computes matrix-vector multiplications but only approximately, due to nonidealities that often are non-deterministic or nonlinear. This can adversely impact the achievable inference accuracy. Here, we develop an hardware-aware retraining approach to systematically examine the accuracy of analog in-memory computing across multiple network topologies, and investigate sensitivity and robustness to a broad set of nonidealities. By introducing a realistic crossbar model, we improve significantly on earlier retraining approaches. We show that many larger-scale deep neural networks-including convnets, recurrent networks, and transformers-can in fact be successfully retrained to show iso-accuracy with the floating point implementation. Our results further suggest that nonidealities that add noise to the inputs or outputs, not the weights, have the largest impact on accuracy, and that recurrent networks are particularly robust to all nonidealities.

Optimised weight programming for analogue memory-based deep neural networks.

Analogue memory-based deep neural networks provide energy-efficiency and per-area throughput gains relative to state-of-the-art digital counterparts such as graphics processing units. Recent advances focus largely on hardware-aware algorithmic training and improvements to circuits, architectures, and memory devices. Optimal translation of software-trained weights into analogue hardware weights-given the plethora of complex memory non-idealities-represents an equally important task. We report a generalised computational framework that automates the crafting of complex weight programming strategies to minimise accuracy degradations during inference, particularly over time. The framework is agnostic to network structure and generalises well across recurrent, convolutional, and transformer neural networks. As a highly flexible numerical heuristic, the approach accommodates arbitrary device-level complexity, making it potentially relevant for a variety of analogue memories. By quantifying the limit of achievable inference accuracy, it also enables analogue memory-based deep neural network accelerators to reach their full inference potential.

Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices.

Recent advances in deep learning have been driven by ever-increasing model sizes, with networks growing to millions or even billions of parameters. Such enormous models call for fast and energy-efficient hardware accelerators. We study the potential of Analog AI accelerators based on Non-Volatile Memory, in particular Phase Change Memory (PCM), for software-equivalent accurate inference of natural language processing applications. We demonstrate a path to software-equivalent accuracy for the GLUE benchmark on BERT (Bidirectional Encoder Representations from Transformers), by combining noise-aware training to combat inherent PCM drift and noise sources, together with reduced-precision digital attention-block computation down to INT6.

RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays.

Analog arrays are a promising emerging hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they employ analog circuitry to compute matrix-vector products in constant time, irrespective of the size of the matrix. However, ConvNets map very unfavorably onto analog arrays when done in a straight-forward manner, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times. Here, we propose to parallelize the training by replicating the kernel matrix of a convolution layer on distinct analog arrays, and randomly divide parts of the compute among them. With this modification, analog arrays execute ConvNets with a large acceleration factor that is proportional to the number of kernel matrices used per layer (here tested 16-1024). Despite having more free parameters, we show analytically and in numerical experiments that this new convolution architecture is self-regularizing and implicitly learns similar filters across arrays. We also report superior performance on a number of datasets and increased robustness to adversarial attacks. Our investigation suggests to revise the notion that emerging hardware architectures that feature analog arrays for fast matrix-vector multiplication are not suitable for ConvNets.

Training LSTM Networks With Resistive Cross-Point Devices.

In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to the device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. Here we attempt to study the validity of the RPU approach by simulating large scale networks. For instance, the models studied here are roughly 1500 times larger than the more often studied multilayer perceptron models trained on the MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch.

Autapses enhance bursting and coincidence detection in neocortical pyramidal cells.

Autapses are synaptic contacts of a neuron's axon onto its own dendrite and soma. In the neocortex, self-inhibiting autapses in GABAergic interneurons are abundant in number and play critical roles in regulating spike precision and network activity. Here we examine whether the principal glutamatergic pyramidal cells (PCs) also form functional autapses. In patch-clamp recording from both rodent and human PCs, we isolated autaptic responses and found that these occur predominantly in layer-5 PCs projecting to subcortical regions, with very few in those projecting to contralateral prefrontal cortex and layer 2/3 PCs. Moreover, PC autapses persist during development into adulthood. Surprisingly, they produce giant postsynaptic responses (∼5 fold greater than recurrent PC-PC synapses) that are exclusively mediated by AMPA receptors. Upon activation, autapses enhance burst firing, neuronal responsiveness and coincidence detection of synaptic inputs. These findings indicate that PC autapses are functional and represent an important circuit element in the neocortex.

Distinct Inhibitory Circuits Orchestrate Cortical beta and gamma Band Oscillations.

Distinct subtypes of inhibitory interneuron are known to shape diverse rhythmic activities in the cortex, but how they interact to orchestrate specific band activity remains largely unknown. By recording optogenetically tagged interneurons of specific subtypes in the primary visual cortex of behaving mice, we show that spiking of somatostatin (SOM)- and parvalbumin (PV)-expressing interneurons preferentially correlates with cortical beta and gamma band oscillations, respectively. Suppression of SOM cell spiking reduces the spontaneous low-frequency band (<30-Hz) oscillations and selectively reduces visually induced enhancement of beta oscillation. In comparison, suppressing PV cell activity elevates the synchronization of spontaneous activity across a broad frequency range and further precludes visually induced changes in beta and gamma oscillations. Rhythmic activation of SOM and PV cells in the local circuit entrains resonant activity in the narrow 5- to 30-Hz band and the wide 20- to 80-Hz band, respectively. Together, these findings reveal differential and cooperative roles of SOM and PV inhibitory neurons in orchestrating specific cortical oscillations.

A distinct entorhinal cortex to hippocampal CA1 direct circuit for olfactory associative learning.

Lateral and medial parts of entorhinal cortex (EC) convey nonspatial 'what' and spatial 'where' information, respectively, into hippocampal CA1, via both the indirect EC layer 2→ hippocampal dentate gyrus→CA3→CA1 and the direct EC layer 3→CA1 paths. However, it remains elusive how the direct path transfers distinct information and contributes to hippocampal learning functions. Here we report that lateral EC projection neurons selectively form direct excitatory synapses onto a subpopulation of morphologically complex, calbindin-expressing pyramidal cells (PCs) in the dorsal CA1 (dCA1), while medial EC neurons uniformly innervate all dCA1 PCs. Optogenetically inactivating the distinct lateral EC-dCA1 connections or the postsynaptic dCA1 calbindin-expressing PC activity slows olfactory associative learning. Moreover, optetrode recordings reveal that dCA1 calbindin-expressing PCs develop more selective spiking responses to odor cues during learning. Thus, our results identify a direct lateral EC→dCA1 circuit that is required for olfactory associative learning.

Selective Modulation of Axonal Sodium Channel Subtypes by 5-HT1A Receptor in Cortical Pyramidal Neuron.

Serotonergic innervation of the prefrontal cortex (PFC) modulates neuronal activity and PFC functions. However, the cellular mechanism for serotonergic modulation of neuronal excitability remains unclear. We performed patch-clamp recording at the axon of layer-5 pyramidal neurons in rodent PFC slices. We found surprisingly that the activation of 5-HT1A receptors selectively inhibits Na+ currents obtained at the axon initial segment (AIS) but not those at the axon trunk. In addition, Na+ channel subtype NaV1.2 but not NaV1.6 at the AIS is selectively modulated by 5-HT1A receptors. Further experiments revealed that the inhibitory effect is attributable to a depolarizing shift of the activation curve and a facilitation of slow inactivation of AIS Na+ currents. Consistently, dual somatic and axonal recording and simulation results demonstrate that the activation of 5-HT1A receptors could decrease the success rate of action potential (AP) backpropagation toward the somatodendritic compartments, enhancing the segregation of axonal and dendritic activities. Together, our results reveal a selective modulation of NaV1.2 distributed at the proximal AIS region and AP backpropagation by 5-HT1A receptors, suggesting a potential mechanism for serotonergic regulation of functional polarization in the dendro-axonal axis, synaptic plasticity and PFC functions.

Selective Maturation of Temporal Dynamics of Intracortical Excitatory Transmission at the Critical Period Onset.

Although the developmental maturation of cortical inhibitory synapses is known to be a critical factor in gating the onset of critical period (CP) for experience-dependent cortical plasticity, how synaptic transmission dynamics of other cortical synapses are regulated during the transition to CP remains unknown. Here, by systematically examining various intracortical synapses within layer 4 of the mouse visual cortex, we demonstrate that synaptic temporal dynamics of intracortical excitatory synapses on principal cells (PCs) and inhibitory parvalbumin- or somatostatin-expressing cells are selectively regulated before the CP onset, whereas those of intracortical inhibitory synapses and long-range thalamocortical excitatory synapses remain unchanged. This selective maturation of synaptic dynamics results from a ubiquitous reduction of presynaptic release and is dependent on visual experience. These findings provide an additional essential circuit mechanism for regulating CP timing in the developing visual cortex.

Integration or separation in the processing of facial properties--a computational view.

A face recognition system ought to read out information about the identity, facial expression and invariant properties of faces, such as sex and race. A current debate is whether separate neural units in the brain deal with these face properties individually or whether a single neural unit processes in parallel all aspects of faces. While the focus of studies has been directed toward the processing of identity and facial expression, little research exists on the processing of invariant aspects of faces. In a theoretical framework we tested whether a system can deal with identity in combination with sex, race or facial expression using the same underlying mechanism. We used dimension reduction to describe how the representational face space organizes face properties when trained on different aspects of faces. When trained to learn identities, the system not only successfully recognized identities, but also was immediately able to classify sex and race, suggesting that no additional system for the processing of invariant properties is needed. However, training on identity was insufficient for the recognition of facial expressions and vice versa. We provide a theoretical approach on the interconnection of invariant facial properties and the separation of variant and invariant facial properties.

Decentralized Multisensory Information Integration in Neural Systems.

How multiple sensory cues are integrated in neural circuitry remains a challenge. The common hypothesis is that information integration might be accomplished in a dedicated multisensory integration area receiving feedforward inputs from the modalities. However, recent experimental evidence suggests that it is not a single multisensory brain area, but rather many multisensory brain areas that are simultaneously involved in the integration of information. Why many mutually connected areas should be needed for information integration is puzzling. Here, we investigated theoretically how information integration could be achieved in a distributed fashion within a network of interconnected multisensory areas. Using biologically realistic neural network models, we developed a decentralized information integration system that comprises multiple interconnected integration areas. Studying an example of combining visual and vestibular cues to infer heading direction, we show that such a decentralized system is in good agreement with anatomical evidence and experimental observations. In particular, we show that this decentralized system can integrate information optimally. The decentralized system predicts that optimally integrated information should emerge locally from the dynamics of the communication between brain areas and sheds new light on the interpretation of the connectivity between multisensory brain areas. SIGNIFICANCE STATEMENT: To extract information reliably from ambiguous environments, the brain integrates multiple sensory cues, which provide different aspects of information about the same entity of interest. Here, we propose a decentralized architecture for multisensory integration. In such a system, no processor is in the center of the network topology and information integration is achieved in a distributed manner through reciprocally connected local processors. Through studying the inference of heading direction with visual and vestibular cues, we show that the decentralized system can integrate information optimally, with the reciprocal connections between processers determining the extent of cue integration. Our model reproduces known multisensory integration behaviors observed in experiments and sheds new light on our understanding of how information is integrated in the brain.

Experience-dependent emergence of beta and gamma band oscillations in the primary visual cortex during the critical period.

Neural oscillatory activities have been shown to play important roles in neural information processing and the shaping of circuit connections during development. However, it remains unknown whether and how specific neural oscillations emerge during a postnatal critical period (CP), in which neuronal connections are most substantially modified by neural activity and experience. By recording local field potentials (LFPs) and single unit activity in developing primary visual cortex (V1) of head-fixed awake mice, we here demonstrate an emergence of characteristic oscillatory activities during the CP. From the pre-CP to CP, the peak frequency of spontaneous fast oscillatory activities shifts from the beta band (15-35 Hz) to the gamma band (40-70 Hz), accompanied by a decrease of cross-frequency coupling (CFC) and broadband spike-field coherence (SFC). Moreover, visual stimulation induced a large increase of beta-band activity but a reduction of gamma-band activity specifically from the CP onwards. Dark rearing of animals from the birth delayed this emergence of oscillatory activities during the CP, suggesting its dependence on early visual experience. These findings suggest that the characteristic neuronal oscillatory activities emerged specifically during the CP may represent as neural activity trait markers for the experience-dependent maturation of developing visual cortical circuits.

Circuit motifs for contrast-adaptive differentiation in early sensory systems: the role of presynaptic inhibition and short-term plasticity.

In natural signals, such as the luminance value across of a visual scene, abrupt changes in intensity value are often more relevant to an organism than intensity values at other positions and times. Thus to reduce redundancy, sensory systems are specialized to detect the times and amplitudes of informative abrupt changes in the input stream rather than coding the intensity values at all times. In theory, a system that responds transiently to fast changes is called a differentiator. In principle, several different neural circuit mechanisms exist that are capable of responding transiently to abrupt input changes. However, it is unclear which circuit would be best suited for early sensory systems, where the dynamic range of the natural input signals can be very wide. We here compare the properties of different simple neural circuit motifs for implementing signal differentiation. We found that a circuit motif based on presynaptic inhibition (PI) is unique in a sense that the vesicle resources in the presynaptic site can be stably maintained over a wide range of stimulus intensities, making PI a biophysically plausible mechanism to implement a differentiator with a very wide dynamical range. Moreover, by additionally considering short-term plasticity (STP), differentiation becomes contrast adaptive in the PI-circuit but not in other potential neural circuit motifs. Numerical simulations show that the behavior of the adaptive PI-circuit is consistent with experimental observations suggesting that adaptive presynaptic inhibition might be a good candidate neural mechanism to achieve differentiation in early sensory systems.

A Phenomenological Synapse Model for Asynchronous Neurotransmitter Release.

Neurons communicate with each other via synapses. Action potentials cause release of neurotransmitters at the axon terminal. Typically, this neurotransmitter release is tightly time-locked to the arrival of an action potential and is thus called synchronous release. However, neurotransmitter release is stochastic and the rate of release of small quanta of neurotransmitters can be considerably elevated even long after the ceasing of spiking activity, leading to asynchronous release of neurotransmitters. Such asynchronous release varies for tissue and neuron types and has been shown recently to be pronounced in fast-spiking neurons. Notably, it was found that asynchronous release is enhanced in human epileptic tissue implicating a possibly important role in generating abnormal neural activity. Current neural network models for simulating and studying neural activity virtually only consider synchronous release and ignore asynchronous transmitter release. Here, we develop a phenomenological model for asynchronous neurotransmitter release, which, on one hand, captures the fundamental features of the asynchronous release process, and, on the other hand, is simple enough to be incorporated in large-size network simulations. Our proposed model is based on the well-known equations for short-term dynamical synaptic interactions and includes an additional stochastic term for modeling asynchronous release. We use experimental data obtained from inhibitory fast-spiking synapses of human epileptic tissue to fit the model parameters, and demonstrate that our model reproduces the characteristics of realistic asynchronous transmitter release.

Own-race and own-species advantages in face perception: a computational view.

The frequency to which an organism is exposed to a particular type of face influences recognition performance. For example, Asians are better in individuating Asian than Caucasian faces, known as the own-race advantage. Similarly, humans in general are better in individuating human than monkey faces, known as the own-species advantage. It is an open question whether the underlying mechanisms causing these effects are similar. We hypothesize that these processes are governed by neural plasticity of the face discrimination system to retain optimal discrimination performance in its environment. Using common face features derived from a set of images from various face classes, we show that maximizing the feature variance between different individuals while ensuring minimal variance within individuals achieved good discrimination performances on own-class faces when selecting a subset of feature dimensions. Further, the selected subset of features does not necessarily lead to an optimal performance on the other class of faces. Thus, the face discrimination system continuously re-optimizes its space constraint face representation to optimize recognition performance on the current distribution of faces in its environment. This model can account for both, the own-race and own-species advantages. We name this approach Space Constraint Optimized Representational Embedding (SCORE).

The other-race and other-species effects in face perception - a subordinate-level analysis.

The ability of face discrimination is modulated by the frequency of exposure to a category of faces. In other words, lower discrimination performance was measured for infrequently encountered faces as opposed to frequently encountered ones. This phenomenon has been described in the literature: the own-race advantage, a benefit in processing own-race as opposed to the other-race faces, and the own-species advantage, a benefit in processing the conspecific type of faces as opposed to the heterospecific type. So far, the exact parameters that drive either of these two effects are not fully understood. In the following we present a full assessment of data in human participants describing the discrimination performances across two races (Asian and Caucasian) as well as a range of non-human primate faces (chimpanzee, Rhesus macaque and marmoset). We measured reaction times of Asian participants performing a delayed matching-to-sample task, and correlated the results with similarity estimates of facial configuration and face parts. We found faster discrimination of own-race above other-race/species faces. Further, we found a strong reliance on configural information in upright own-species/-race faces and on individual face parts in all inverted face classes, supporting the assumption of specialized processing for the face class of most frequent exposure.

Perceptual training continuously refines neuronal population codes in primary visual cortex.

Perceptual learning substantially improves visual discrimination and detection ability, which has been associated with visual cortical plasticity. However, little is known about the dynamic changes in neuronal response properties over the course of training. Using chronically implanted multielectrode arrays, we were able to capture day-by-day spatiotemporal dynamics of neurons in the primary visual cortex (V1) of monkeys trained to detect camouflaged visual contours. We found progressive strengthening and accelerating in both facilitation of neurons encoding the contour elements and suppression of neurons responding to the background components. The enhancement of this figure-ground contrast in V1 was closely correlated with improved behavioral performance on a daily basis. Decoding accuracy of a simple linear classifier based on V1 population responses also paralleled the animal's behavioral changes. Our results indicate that perceptual learning shapes the V1 population code to allow a more efficient readout of task-relevant information.

Binocular input coincidence mediates critical period plasticity in the mouse primary visual cortex.

Classical studies on the development of ocular dominance (OD) organization in primary visual cortex (V1) have revealed a postnatal critical period (CP), during which visual inputs between the two eyes are most effective in shaping cortical circuits through synaptic competition. A brief closure of one eye during CP caused a pronounced shift of response preference of V1 neurons toward the open eye, a form of CP plasticity in the developing V1. However, it remains unclear what particular property of binocular inputs during CP is responsible for mediating this experience-dependent OD plasticity. Using whole-cell recording in mouse V1, we found that visually driven synaptic inputs from the two eyes to binocular cells in layers 2/3 and 4 became highly coincident during CP. Enhancing cortical GABAergic transmission activity by brain infusion with diazepam not only caused a precocious onset of the high coincidence of binocular inputs and OD plasticity in pre-CP mice, but rescued both of them in dark-reared mice, suggesting a tight link between coincident binocular inputs and CP plasticity. In Thy1-ChR2 mice, chronic disruption of this binocular input coincidence during CP by asynchronous optogenetic activation of retinal ganglion cells abolished the OD plasticity. Computational simulation using a feed-forward network model further suggests that the coincident inputs could mediate this CP plasticity through a homeostatic synaptic learning mechanism with synaptic competition. These results suggest that the high-level correlation of binocular inputs is a hallmark of the CP of developing V1 and serves as neural substrate for the induction of OD plasticity.