On the visual analytic intelligence of neural networks.

Visual oddity task was conceived to study universal ethnic-independent analytic intelligence of humans from a perspective of comprehension of spatial concepts. Advancements in artificial intelligence led to important breakthroughs, yet excelling at such abstract tasks remains challenging. Current approaches typically resort to non-biologically-plausible architectures with ever-growing models consuming substantially more energy than the brain. Motivated by the brain's efficiency and reasoning capabilities, we present a biologically inspired system that receives inputs from synthetic eye movements - reminiscent of saccades, and processes them with neuronal units incorporating dynamics of neocortical neurons. We introduce a procedurally generated visual oddity dataset to train an architecture extending conventional relational networks and our proposed system. We demonstrate that both approaches are capable of abstract problem-solving at high accuracy, and we uncover that both share the same essential underlying mechanism of reasoning in seemingly unrelated aspects of their architectures. Finally, we show that the biologically inspired network achieves superior accuracy, learns faster and requires fewer parameters than the conventional network.

Online Spatio-Temporal Learning in Deep Neural Networks.

Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm ic framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm ic framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.

Introducing principles of synaptic integration in the optimization of deep neural networks.

Plasticity circuits in the brain are known to be influenced by the distribution of the synaptic weights through the mechanisms of synaptic integration and local regulation of synaptic strength. However, the complex interplay of stimulation-dependent plasticity with local learning signals is disregarded by most of the artificial neural network training algorithms devised so far. Here, we propose a novel biologically inspired optimizer for artificial and spiking neural networks that incorporates key principles of synaptic plasticity observed in cortical dendrites: GRAPES (Group Responsibility for Adjusting the Propagation of Error Signals). GRAPES implements a weight-distribution-dependent modulation of the error signal at each node of the network. We show that this biologically inspired mechanism leads to a substantial improvement of the performance of artificial and spiking networks with feedforward, convolutional, and recurrent architectures, it mitigates catastrophic forgetting, and it is optimally suited for dedicated hardware implementations. Overall, our work indicates that reconciling neurophysiology insights with machine intelligence is key to boosting the performance of neural networks.

Accelerating Inference of Convolutional Neural Networks Using In-memory Computing.

In-memory computing (IMC) is a non-von Neumann paradigm that has recently established itself as a promising approach for energy-efficient, high throughput hardware for deep learning applications. One prominent application of IMC is that of performing matrix-vector multiplication in O ( 1 ) time complexity by mapping the synaptic weights of a neural-network layer to the devices of an IMC core. However, because of the significantly different pattern of execution compared to previous computational paradigms, IMC requires a rethinking of the architectural design choices made when designing deep-learning hardware. In this work, we focus on application-specific, IMC hardware for inference of Convolution Neural Networks (CNNs), and provide methodologies for implementing the various architectural components of the IMC core. Specifically, we present methods for mapping synaptic weights and activations on the memory structures and give evidence of the various trade-offs therein, such as the one between on-chip memory requirements and execution latency. Lastly, we show how to employ these methods to implement a pipelined dataflow that offers throughput and latency beyond state-of-the-art for image classification tasks.

Convergence Behavior of DNNs with Mutual-Information-Based Regularization.

Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.

Publisher Correction: Memory devices and applications for in-memory computing.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Mixed-Precision Deep Learning Based on Computational Memory.

Deep neural networks (DNNs) have revolutionized the field of artificial intelligence and have achieved unprecedented success in cognitive tasks such as image and speech recognition. Training of large DNNs, however, is computationally intensive and this has motivated the search for novel computing architectures targeting this application. A computational memory unit with nanoscale resistive memory devices organized in crossbar arrays could store the synaptic weights in their conductance states and perform the expensive weighted summations in place in a non-von Neumann manner. However, updating the conductance states in a reliable manner during the weight update process is a fundamental challenge that limits the training accuracy of such an implementation. Here, we propose a mixed-precision architecture that combines a computational memory unit performing the weighted summations and imprecise conductance updates with a digital processing unit that accumulates the weight updates in high precision. A combined hardware/software training experiment of a multilayer perceptron based on the proposed architecture using a phase-change memory (PCM) array achieves 97.73% test accuracy on the task of classifying handwritten digits (based on the MNIST dataset), within 0.6% of the software baseline. The architecture is further evaluated using accurate behavioral models of PCM on a wide class of networks, namely convolutional neural networks, long-short-term-memory networks, and generative-adversarial networks. Accuracies comparable to those of floating-point implementations are achieved without being constrained by the non-idealities associated with the PCM devices. A system-level study demonstrates 172 × improvement in energy efficiency of the architecture when used for training a multilayer perceptron compared with a dedicated fully digital 32-bit implementation.

Accurate deep neural network inference using computational phase-change memory.

In-memory computing using resistive memory devices is a promising non-von Neumann approach for making energy-efficient deep learning inference hardware. However, due to device variability and noise, the network needs to be trained in a specific way so that transferring the digitally trained weights to the analog resistive memory devices will not result in significant loss of accuracy. Here, we introduce a methodology to train ResNet-type convolutional neural networks that results in no appreciable accuracy loss when transferring weights to phase-change memory (PCM) devices. We also propose a compensation technique that exploits the batch normalization parameters to improve the accuracy retention over time. We achieve a classification accuracy of 93.7% on CIFAR-10 and a top-1 accuracy of 71.6% on ImageNet benchmarks after mapping the trained weights to PCM. Our hardware results on CIFAR-10 with ResNet-32 demonstrate an accuracy above 93.5% retained over a one-day period, where each of the 361,722 synaptic weights is programmed on just two PCM devices organized in a differential configuration.

Experimental Demonstration of Supervised Learning in Spiking Neural Networks with Phase-Change Memory Synapses.

Spiking neural networks (SNN) are computational models inspired by the brain's ability to naturally encode and process information in the time domain. The added temporal dimension is believed to render them more computationally efficient than the conventional artificial neural networks, though their full computational capabilities are yet to be explored. Recently, in-memory computing architectures based on non-volatile memory crossbar arrays have shown great promise to implement parallel computations in artificial and spiking neural networks. In this work, we evaluate the feasibility to realize high-performance event-driven in-situ supervised learning systems using nanoscale and stochastic analog memory synapses. For the first time, the potential of analog memory synapses to generate precisely timed spikes in SNNs is experimentally demonstrated. The experiment targets applications which directly integrates spike encoded signals generated from bio-mimetic sensors with in-memory computing based learning systems to generate precisely timed control signal spikes for neuromorphic actuators. More than 170,000 phase-change memory (PCM) based synapses from our prototype chip were trained based on an event-driven learning rule, to generate spike patterns with more than 85% of the spikes within a 25 ms tolerance interval in a 1250 ms long spike pattern. We observe that the accuracy is mainly limited by the imprecision related to device programming and temporal drift of conductance values. We show that an array level scaling scheme can significantly improve the retention of the trained SNN states in the presence of conductance drift in the PCM. Combining the computational potential of supervised SNNs with the parallel compute power of in-memory computing, this work paves the way for next-generation of efficient brain-inspired systems.

Memory devices and applications for in-memory computing.

Traditional von Neumann computing systems involve separate processing and memory units. However, data movement is costly in terms of time and energy and this problem is aggravated by the recent explosive growth in highly data-centric applications related to artificial intelligence. This calls for a radical departure from the traditional systems and one such non-von Neumann computational approach is in-memory computing. Hereby certain computational tasks are performed in place in the memory itself by exploiting the physical attributes of the memory devices. Both charge-based and resistance-based memory devices are being explored for in-memory computing. In this Review, we provide a broad overview of the key computational primitives enabled by these memory devices as well as their applications spanning scientific computing, signal processing, optimization, machine learning, deep learning and stochastic computing.

Neuromorphic computing with multi-memristive synapses.

Neuromorphic computing has emerged as a promising avenue towards building the next generation of intelligent computing systems. It has been proposed that memristive devices, which exhibit history-dependent conductivity modulation, could efficiently represent the synaptic weights in artificial neural networks. However, precise modulation of the device conductance over a wide dynamic range, necessary to maintain high network accuracy, is proving to be challenging. To address this, we present a multi-memristive synaptic architecture with an efficient global counter-based arbitration scheme. We focus on phase change memory devices, develop a comprehensive model and demonstrate via simulations the effectiveness of the concept for both spiking and non-spiking neural networks. Moreover, we present experimental results involving over a million phase change memory devices for unsupervised learning of temporal correlations using a spiking neural network. The work presents a significant step towards the realization of large-scale and energy-efficient neuromorphic computing systems.

Temporal correlation detection using computational phase-change memory.

Conventional computers based on the von Neumann architecture perform computation by repeatedly transferring data between their physically separated processing and memory units. As computation becomes increasingly data centric and the scalability limits in terms of performance and power are being reached, alternative computing paradigms with collocated computation and storage are actively being sought. A fascinating such approach is that of computational memory where the physics of nanoscale memory devices are used to perform certain computational tasks within the memory unit in a non-von Neumann manner. We present an experimental demonstration using one million phase change memory devices organized to perform a high-level computational primitive by exploiting the crystallization dynamics. Its result is imprinted in the conductance states of the memory devices. The results of using such a computational memory for processing real-world data sets show that this co-existence of computation and storage at the nanometer scale could enable ultra-dense, low-power, and massively-parallel computing systems.

All-memristive neuromorphic computing with level-tuned neurons.

In the new era of cognitive computing, systems will be able to learn and interact with the environment in ways that will drastically enhance the capabilities of current processors, especially in extracting knowledge from vast amount of data obtained from many sources. Brain-inspired neuromorphic computing systems increasingly attract research interest as an alternative to the classical von Neumann processor architecture, mainly because of the coexistence of memory and processing units. In these systems, the basic components are neurons interconnected by synapses. The neurons, based on their nonlinear dynamics, generate spikes that provide the main communication mechanism. The computational tasks are distributed across the neural network, where synapses implement both the memory and the computational units, by means of learning mechanisms such as spike-timing-dependent plasticity. In this work, we present an all-memristive neuromorphic architecture comprising neurons and synapses realized by using the physical properties and state dynamics of phase-change memristors. The architecture employs a novel concept of interconnecting the neurons in the same layer, resulting in level-tuned neuronal characteristics that preferentially process input information. We demonstrate the proposed architecture in the tasks of unsupervised learning and detection of multiple temporal correlations in parallel input streams. The efficiency of the neuromorphic architecture along with the homogenous neuro-synaptic dynamics implemented with nanoscale phase-change memristors represent a significant step towards the development of ultrahigh-density neuromorphic co-processors.

Stochastic phase-change neurons.

Artificial neuromorphic systems based on populations of spiking neurons are an indispensable tool in understanding the human brain and in constructing neuromimetic computational systems. To reach areal and power efficiencies comparable to those seen in biological systems, electroionics-based and phase-change-based memristive devices have been explored as nanoscale counterparts of synapses. However, progress on scalable realizations of neurons has so far been limited. Here, we show that chalcogenide-based phase-change materials can be used to create an artificial neuron in which the membrane potential is represented by the phase configuration of the nanoscale phase-change device. By exploiting the physics of reversible amorphous-to-crystal phase transitions, we show that the temporal integration of postsynaptic potentials can be achieved on a nanosecond timescale. Moreover, we show that this is inherently stochastic because of the melt-quench-induced reconfiguration of the atomic structure occurring when the neuron is reset. We demonstrate the use of these phase-change neurons, and their populations, in the detection of temporal correlations in parallel data streams and in sub-Nyquist representation of high-bandwidth signals.

Oxygenated amorphous carbon for resistive memory applications.

Carbon-based electronics is a promising alternative to traditional silicon-based electronics as it could enable faster, smaller and cheaper transistors, interconnects and memory devices. However, the development of carbon-based memory devices has been hampered either by the complex fabrication methods of crystalline carbon allotropes or by poor performance. Here we present an oxygenated amorphous carbon (a-COx) produced by physical vapour deposition that has several properties in common with graphite oxide. Moreover, its simple fabrication method ensures excellent reproducibility and tuning of its properties. Memory devices based on a-COx exhibit outstanding non-volatile resistive memory performance, such as switching times on the order of 10 ns and cycling endurance in excess of 10(4) times. A detailed investigation of the pristine, SET and RESET states indicates a switching mechanism based on the electrochemical redox reaction of carbon. These results suggest that a-COx could play a key role in non-volatile memory technology and carbon-based electronics.

Projected phase-change memory devices.

Nanoscale memory devices, whose resistance depends on the history of the electric signals applied, could become critical building blocks in new computing paradigms, such as brain-inspired computing and memcomputing. However, there are key challenges to overcome, such as the high programming power required, noise and resistance drift. Here, to address these, we present the concept of a projected memory device, whose distinguishing feature is that the physical mechanism of resistance storage is decoupled from the information-retrieval process. We designed and fabricated projected memory devices based on the phase-change storage mechanism and convincingly demonstrate the concept through detailed experimentation, supported by extensive modelling and finite-element simulations. The projected memory devices exhibit remarkably low drift and excellent noise performance. We also demonstrate active control and customization of the programming characteristics of the device that reliably realize a multitude of resistance states.

Scanning probe microscopy based on magnetoresistive sensing.

Integrated sensors are essential for scanning probe microscopy (SPM) based systems that employ arrays of microcantilevers for high throughput. Common integrated sensors, such as piezoresistive, piezoelectric, capacitive and thermoelectric sensors, suffer from low bandwidth and/or low resolution. In this paper, a novel magnetoresistive-sensor-based scanning probe microscopy (MR-SPM) technique is presented. The principle of MR-SPM is first demonstrated using experiments with magnetic cantilevers and commercial MR sensors. A new cantilever design tailored to MR-SPM is then presented and micromagnetic simulations are employed to evaluate the achievable resolution. A remarkable resolution of 0.84 Å over a bandwidth of 1 MHz is estimated, which would significantly outperform state-of-the-art optical deflection sensors. Due to its combination of high resolution at high bandwidth, and its amenability to integration in probe arrays, MR-SPM holds great promise for low-cost, high-throughput SPM.

High-throughput intermittent-contact scanning probe microscopy.

Large arrays of micro-cantilevers operating in parallel are essential for achieving high throughput in such applications as life sciences, nanofabrication and semiconductor metrology. A novel intermittent-contact mode operation is presented that is suitable for such applications. The cantilevers are electrostatically actuated. The oscillation amplitude is kept small to enable high-frequency operation and to reduce the tip-sample interaction force, and thus the tip and sample wear. Input shaping of the actuation signal is employed for high-speed reliable operation in the presence of the tip-sample adhesion forces. The deflection signal is sampled once per oscillation cycle to enable high-speed imaging. Experimental results are shown which demonstrate the efficacy of the proposed scheme. In particular, during continuous high-speed imaging, the tip diameter is maintained over a remarkable 140 m of tip travel.

A flow visualization study of an anatomic coronary artery anastomosis model with an implant.

Flow Streamlining Devices is a new tool in Coronary Artery Bypass Grafting (CABG). They aim in: a) Performing a sutureless anastomosis to reduce thrombosis at the veno-arterial junction, and b) Providing a hemodynamically efficient scaffolding to reduce secondary flow disturbances. Thrombosis and flow disturbances are factors that have been reported as contributing factors to the development of intimal hyperplasia (IH) and failure of the graft. By reducing thrombosis and flow disturbances, it is expected that IH will be inhibited and the lifetime of the graft extended. To evaluate the hemodynamic benefits of such an implant, two models were designed and fabricated. One simulated the geometry of the conventional anastomosis without an implant, and the other simulated an anastomosis with a flow streamlining implant. Identical flow conditions relevant to a coronary anastomosis were imposed on both models and flow visualization was performed with dye injection and a digital camera. Results showed reduction of disturbances in the presence of the implant. This reduction seems to be favorable to hemodynamic streamlining which may create conditions that may inhibit the initialization of IH. However, the compliance and geometric mismatch between the anastomosis and the implant created a disturbance at the rigid compliant wall interface, which should be eliminated prior to clinical applications.

Effect of a flow-streamlining implant at the distal anastomosis of a coronary artery bypass graft.

Intimal thickening in the coronary artery bypass graft (CABG) distal anastomosis has been implicated as the major cause of restenosis and long-term graft failure. Several studies point to the interplay between nonuniform hemodynamics including disturbed flows and recirculation zones, wall shear stress, and long particle residence time as possible etiologies. The hemodynamic features of two anatomic models of saphenous-vein CABGs were studied and compared. One simulated an anastomosis with both diameter and compliance mismatch and a curvature at the connection, analogous to the geometry observed in a conventional cardiothoracic procedure. The other, simulated an anastomosis with a flow stabilizing anastomotic implant connector which improves current cardiothoracic procedures by eliminating the distal vein bulging and curvature. Physiologic flow conditions were imposed on both models and qualitative analysis of the flow was performed with dye injection and a digital camera. Quantitative analysis was performed with laser Doppler velocimetry. Results showed that the presence of the bulge at the veno-arterial junction, contributed to the formation of accentuated secondary structures (helices), which progress into the flow divider and significantly affect radial velocity components at the host vessel up to four diameters downstream of the junction. The model with the implant, achieved more hemodynamically efficient conditions on the host vessel with higher mean and maximum axial velocities and lower radial velocities than the conventional model. The presence of the sinus may also affect the magnitude and shape of the shear stress at locations where intimal thickening occurs. Thus, the presence of the implant creates a more streamlined environment with more primary and less secondary flow components which may then inhibit the development of intimal thickening, restenosis, and ultimate failure of the saphenous vein graft.