Compute in-Memory with Non-Volatile Elements for Neural Networks: A Review from a Co-Design Perspective.

Deep learning has become ubiquitous, touching daily lives across the globe. Today, traditional computer architectures are stressed to their limits in efficiently executing the growing complexity of data and models. Compute-in-memory (CIM) can potentially play an important role in developing efficient hardware solutions that reduce data movement from compute-unit to memory, known as the von Neumann bottleneck. At its heart is a cross-bar architecture with nodal non-volatile-memory elements that performs an analog multiply-and-accumulate operation, enabling the matrix-vector-multiplications repeatedly used in all neural network workloads. The memory materials can significantly influence final system-level characteristics and chip performance, including speed, power, and classification accuracy. With an over-arching co-design viewpoint, this review assesses the use of cross-bar based CIM for neural networks, connecting the material properties and the associated design constraints and demands to application, architecture, and performance. Both digital and analog memory are considered, assessing the status for training and inference, and providing metrics for the collective set of properties non-volatile memory materials will need to demonstrate for a successful CIM technology.

Neural Network Training With Asymmetric Crosspoint Elements.

Analog crossbar arrays comprising programmable non-volatile resistors are under intense investigation for acceleration of deep neural network training. However, the ubiquitous asymmetric conductance modulation of practical resistive devices critically degrades the classification performance of networks trained with conventional algorithms. Here we first describe the fundamental reasons behind this incompatibility. Then, we explain the theoretical underpinnings of a novel fully-parallel training algorithm that is compatible with asymmetric crosspoint elements. By establishing a powerful analogy with classical mechanics, we explain how device asymmetry can be exploited as a useful feature for analog deep learning processors. Instead of conventionally tuning weights in the direction of the error function gradient, network parameters can be programmed to successfully minimize the total energy (Hamiltonian) of the system that incorporates the effects of device asymmetry. Our technique enables immediate realization of analog deep learning accelerators based on readily available device technologies.

Algorithm for Training Neural Networks on Resistive Device Arrays.

Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware, however, strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the "Tiki-Taka" algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the "Tiki-Taka" algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that the accuracy achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics is matched in accuracy achieved using the "Tiki-Taka" algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.

RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays.

Analog arrays are a promising emerging hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they employ analog circuitry to compute matrix-vector products in constant time, irrespective of the size of the matrix. However, ConvNets map very unfavorably onto analog arrays when done in a straight-forward manner, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times. Here, we propose to parallelize the training by replicating the kernel matrix of a convolution layer on distinct analog arrays, and randomly divide parts of the compute among them. With this modification, analog arrays execute ConvNets with a large acceleration factor that is proportional to the number of kernel matrices used per layer (here tested 16-1024). Despite having more free parameters, we show analytically and in numerical experiments that this new convolution architecture is self-regularizing and implicitly learns similar filters across arrays. We also report superior performance on a number of datasets and increased robustness to adversarial attacks. Our investigation suggests to revise the notion that emerging hardware architectures that feature analog arrays for fast matrix-vector multiplication are not suitable for ConvNets.

Training LSTM Networks With Resistive Cross-Point Devices.

In our previous work we have shown that resistive cross point devices, so called resistive processing unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to the device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. Here we attempt to study the validity of the RPU approach by simulating large scale networks. For instance, the models studied here are roughly 1500 times larger than the more often studied multilayer perceptron models trained on the MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch.

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices.

In a previous work we have detailed the requirements for obtaining maximal deep learning performance benefit by implementing fully connected deep neural networks (DNN) in the form of arrays of resistive devices. Here we extend the concept of Resistive Processing Unit (RPU) devices to convolutional neural networks (CNNs). We show how to map the convolutional layers to fully connected RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed by the analog nature of the computations performed on the arrays significantly affect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and that can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that a combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of the RPU approach to a large class of neural network architectures.

High-speed logic integrated circuits with solution-processed self-assembled carbon nanotubes.

As conventional monolithic silicon technology struggles to meet the requirements for the 7-nm technology node, there has been tremendous progress in demonstrating the scalability of carbon nanotube field-effect transistors down to the size that satisfies the 3-nm node and beyond. However, to date, circuits built with carbon nanotubes have overlooked key aspects of a practical logic technology and have stalled at simple functionality demonstrations. Here, we report high-performance complementary carbon nanotube ring oscillators using fully manufacturable processes, with a stage switching frequency of 2.82 GHz. The circuit was built on solution-processed, self-assembled carbon nanotube arrays with over 99.9% semiconducting purity, and the complementary feature was achieved by employing two different work function electrodes.

End-bonded contacts for carbon nanotube transistors with low, size-independent resistance.

Moving beyond the limits of silicon transistors requires both a high-performance channel and high-quality electrical contacts. Carbon nanotubes provide high-performance channels below 10 nanometers, but as with silicon, the increase in contact resistance with decreasing size becomes a major performance roadblock. We report a single-walled carbon nanotube (SWNT) transistor technology with an end-bonded contact scheme that leads to size-independent contact resistance to overcome the scaling limits of conventional side-bonded or planar contact schemes. A high-performance SWNT transistor was fabricated with a sub-10-nanometer contact length, showing a device resistance below 36 kilohms and on-current above 15 microampere per tube. The p-type end-bonded contact, formed through the reaction of molybdenum with the SWNT to form carbide, also exhibited no Schottky barrier. This strategy promises high-performance SWNT transistors, enabling future ultimately scaled device technologies.

Origins and characteristics of the threshold voltage variability of quasiballistic single-walled carbon nanotube field-effect transistors.

Ultrascaled transistors based on single-walled carbon nanotubes are identified as one of the top candidates for future microprocessor chips as they provide significantly better device performance and scaling properties than conventional silicon technologies. From the perspective of the chip performance, the device variability is as important as the device performance for practical applications. This paper presents a systematic investigation on the origins and characteristics of the threshold voltage (VT) variability of scaled quasiballistic nanotube transistors. Analysis of experimental results from variable-temperature measurement as well as gate oxide thickness scaling studies shows that the random variation from fixed charges present on the oxide surface close to nanotubes dominates the VT variability of nanotube transistors. The VT variability of single-tube transistors has a figure of merit that is quantitatively comparable with that of current silicon devices; and it could be reduced with the adoption of improved device passivation schemes, which might be necessary for practical devices incorporating multiple nanotubes, whose area normalized VT variability becomes worse due to the synergic effects from the limited surface coverage of nanotubes and the nonlinearity of the device off-state leakage current, as predicted by the Monte Carlo simulation.

Toward high-performance digital logic technology with carbon nanotubes.

The slow-down in traditional silicon complementary metal-oxide-semiconductor (CMOS) scaling (Moore's law) has created an opportunity for a disruptive innovation to bring the semiconductor industry into a postsilicon era. Due to their ultrathin body and ballistic transport, carbon nanotubes (CNTs) have the intrinsic transport and scaling properties to usher in this new era. The remaining challenges are largely materials-related and include obtaining purity levels suitable for logic technology, placement of CNTs at very tight (∼5 nm) pitch to allow for density scaling and source/drain contact scaling. This review examines the potential performance advantages of a CNT-based computing technology, outlines the remaining challenges, and describes the recent progress on these fronts. Although overcoming these issues will be challenging and will require a large, sustained effort from both industry and academia, the recent progress in the field is a cause for optimism that these materials can have an impact on future technologies.

Defining and overcoming the contact resistance challenge in scaled carbon nanotube transistors.

Carbon nanotubes (CNTs) continue to show strong promise as the channel material for an aggressively scaled, high-performance transistor technology. However, there has been concern regarding the contact resistance (Rc) in CNT field-effect transistors (CNTFETs) limiting the ultimate performance, especially at scaled contact lengths. In this work, the contact resistance in CNTFETs is defined in the context of a high-performance scaled transistor, including how the demonstrated Rc relates to technology targets. The impact of different source/drain contact metals (Pd, Pt, Au, Rh, Ni, and Ti) on the scaling of Rc versus contact length is presented. It is discovered that the most optimal contact metal at long contact lengths (Pd) is not necessarily the best for scaled devices, where a newly explored scaled metal contact, Rh, yields the best scaling trend. When extrapolated for a sub-10 nm transistor technology, these results show that the Rc in scaled CNTFETs is within a factor of 2 of the technology target with much potential for improvement through enhanced understanding and engineering of transport at the metal-CNT interface.

Graphene radio frequency receiver integrated circuit.

Graphene has attracted much interest as a future channel material in radio frequency electronics because of its superior electrical properties. Fabrication of a graphene integrated circuit without significantly degrading transistor performance has proven to be challenging, posing one of the major bottlenecks to compete with existing technologies. Here we present a fabrication method fully preserving graphene transistor quality, demonstrated with the implementation of a high-performance three-stage graphene integrated circuit. The circuit operates as a radio frequency receiver performing signal amplification, filtering and downconversion mixing. All circuit components are integrated into 0.6 mm(2) area and fabricated on 200 mm silicon wafers, showing the unprecedented graphene circuit complexity and silicon complementary metal-oxide-semiconductor process compatibility. The demonstrated circuit performance allow us to use graphene integrated circuit to perform practical wireless communication functions, receiving and restoring digital text transmitted on a 4.3-GHz carrier signal.

High-performance air-stable n-type carbon nanotube transistors with erbium contacts.

So far, realization of reproducible n-type carbon nanotube (CNT) transistors suitable for integrated digital applications has been a difficult task. In this work, hundreds of n-type CNT transistors from three different low work function metals-erbium, lanthanum, and yttrium-are studied and benchmarked against p-type devices with palladium contacts. The crucial role of metal type and deposition conditions is elucidated with respect to overall yield and performance of the n-type devices. It is found that high oxidation rates and sensitivity to deposition conditions are the major causes for the lower yield and large variation in performance of n-type CNT devices with low work function metal contacts. Considerable improvement in device yield is attained using erbium contacts evaporated at high deposition rates. Furthermore, the air-stability of our n-type transistors is studied in light of the extreme sensitivity of these metals to oxidation.

Carbon nanotube complementary wrap-gate transistors.

Among the challenges hindering the integration of carbon nanotube (CNT) transistors in digital technology are the lack of a scalable self-aligned gate and complementary n- and p-type devices. We report CNT transistors with self-aligned gates scaled down to 20 nm in the ideal gate-all-around geometry. Uniformity of the gate wrapping the nanotube channels is confirmed, and the process is shown not to damage the CNTs. Further, both n- and p-type transistors were realized by using the appropriate gate dielectric-HfO2 yielded n-type and Al2O3 yielded p-type-with quantum simulations used to explore the impact of important device parameters on performance. These discoveries not only provide a promising platform for further research into gate-all-around CNT devices but also demonstrate that scalable digital switches with realistic technological potential can be achieved with carbon nanotubes.

Arrays of single-walled carbon nanotubes with full surface coverage for high-performance electronics.

Single-walled carbon nanotubes have exceptional electronic properties and have been proposed as a replacement for silicon in applications such as low-cost thin-film transistors and high-performance logic devices. However, practical devices will require dense, aligned arrays of electronically pure nanotubes to optimize performance, maximize device packing density and provide sufficient drive current (or power output) for each transistor. Here, we show that aligned arrays of semiconducting carbon nanotubes can be assembled using the Langmuir-Schaefer method. The arrays have a semiconducting nanotube purity of 99% and can fully cover a surface with a nanotube density of more than 500 tubes/µm. The nanotube pitch is self-limited by the diameter of the nanotube plus the van der Waals separation, and the intrinsic mobility of the nanotubes is preserved after array assembly. Transistors fabricated using this approach exhibit significant device performance characteristics with a drive current density of more than 120 µA µm(-1), transconductance greater than 40 µS µm(-1) and on/off ratios of ∼1 × 10(3).

High-density integration of carbon nanotubes via chemical self-assembly.

Carbon nanotubes have potential in the development of high-speed and power-efficient logic applications. However, for such technologies to be viable, a high density of semiconducting nanotubes must be placed at precise locations on a substrate. Here, we show that ion-exchange chemistry can be used to fabricate arrays of individually positioned carbon nanotubes with a density as high as 1 × 10(9) cm(-2)-two orders of magnitude higher than previous reports. With this approach, we assembled a high density of carbon-nanotube transistors in a conventional semiconductor fabrication line and then electrically tested more than 10,000 devices in a single chip. The ability to characterize such large distributions of nanotube devices is crucial for analysing transistor performance, yield and semiconducting nanotube purity.

Evaluation of field-effect mobility and contact resistance of transistors that use solution-processed single-walled carbon nanotubes.

Solution-processed single-walled carbon nanotubes (SWNTs) offer many unique processing advantages over nanotubes grown by the chemical vapor deposition (CVD) method, including capabilities of separating the nanotubes by electronic type and depositing them onto various substrates in the form of ultradensely aligned arrays at low temperature. However, long-channel transistors that use solution-processed SWNTs generally demonstrate inferior device performance, which poses concerns over the feasibility of using these nanotubes in high-performance logic applications. This paper presents the first systematic study of contact resistance, intrinsic field-effect mobility (µ(FE)), and conductivity (σ(m)) of solution-processed SWNTs based on both the transmission line method and the Y function method. The results indicate that, compared to CVD nanotubes, although solution-processed SWNTs have much lower µ(FE) for semiconducting nanotubes and lower σ(m) for metallic nanotubes due to the presence of a higher level of structural defects, such defects do not affect the quality of electric contacts between the nanotube and metal source/drain electrodes. Therefore, solution-processed SWNTs are expected to offer performance comparable to that of CVD nanotubes in ultimately scaled field-effect transistors, where contacts will dominate electron transport instead of electron scattering in the channel region. These results show promise for using solution-processed SWNTs for high-performance nanoelectronic devices.

Variability in carbon nanotube transistors: improving device-to-device consistency.

The large amount of hysteresis and threshold voltage variation in carbon nanotube transistors impedes their use in highly integrated digital applications. The origin of this variability is elucidated by employing a top-coated, hydrophobic monolayer to passivate bottom-gated devices. Compared to passivating only the supporting substrate, it is found that covering the nanotube channel proves highly effective and robust at improving device-to-device consistency-hysteresis and threshold voltage variation are reduced by an average of 84 and 53%, respectively. The effect of gate and drain-source bias on hysteresis is considered, showing strong dependence that must be accounted for when analyzing the effectiveness of a passivation layer. These results provide both key insight into the origin of variability in carbon nanotube transistors and a promising path for resolving this significant obstacle.