A vision chip with complementary pathways for open-world sensing.

Image sensors face substantial challenges when dealing with dynamic, diverse and unpredictable scenes in open-world applications. However, the development of image sensors towards high speed, high resolution, large dynamic range and high precision is limited by power and bandwidth. Here we present a complementary sensing paradigm inspired by the human visual system that involves parsing visual information into primitive-based representations and assembling these primitives to form two complementary vision pathways: a cognition-oriented pathway for accurate cognition and an action-oriented pathway for rapid response. To realize this paradigm, a vision chip called Tianmouc is developed, incorporating a hybrid pixel array and a parallel-and-heterogeneous readout architecture. Leveraging the characteristics of the complementary vision pathway, Tianmouc achieves high-speed sensing of up to 10,000 fps, a dynamic range of 130 dB and an advanced figure of merit in terms of spatial resolution, speed and dynamic range. Furthermore, it adaptively reduces bandwidth by 90%. We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications.

Advancing brain-inspired computing with hybrid neural networks.

Brain-inspired computing, drawing inspiration from the fundamental structure and information-processing mechanisms of the human brain, has gained significant momentum in recent years. It has emerged as a research paradigm centered on brain-computer dual-driven and multi-network integration. One noteworthy instance of this paradigm is the hybrid neural network (HNN), which integrates computer-science-oriented artificial neural networks (ANNs) with neuroscience-oriented spiking neural networks (SNNs). HNNs exhibit distinct advantages in various intelligent tasks, including perception, cognition and learning. This paper presents a comprehensive review of HNNs with an emphasis on their origin, concepts, biological perspective, construction framework and supporting systems. Furthermore, insights and suggestions for potential research directions are provided aiming to propel the advancement of the HNN paradigm.

[Research progress in pharmacological effects and chemical components of processed Angelicae Sinensis Radix products].

Angelicae Sinensis Radix is one of the main Chinese medicinal materials with both medicinal and edible values. It has the functions of tonifying and activating blood, regulating menstruation and relieving pain, and moistening intestines to relieve constipation. It is mainly produced in the southeastern Gansu province, and that produced in Minxian, Gansu is praised for the best quality. The chemical components of Angelicae Sinensis Radix mainly include volatile oils, organic acids, and polysaccharides, which have anti-inflammatory, pain-relieving, anti-tumor, anti-oxidation, immunomodulatory and other pharmacological effects. Therefore, this medicinal material is widely used in clinical practice. By reviewing the relevant literature, this study systematically introduced the research status about the chemical constituents and pharmacological effects of processed Angelicae Sinensis Radix products, aiming to provide a theoretical reference and support for the future research, development, and clinical application of related drugs.

Brain-inspired multimodal hybrid neural network for robot place recognition.

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.

A framework for the general design and computation of hybrid neural networks.

There is a growing trend to design hybrid neural networks (HNNs) by combining spiking neural networks and artificial neural networks to leverage the strengths of both. Here, we propose a framework for general design and computation of HNNs by introducing hybrid units (HUs) as a linkage interface. The framework not only integrates key features of these computing paradigms but also decouples them to improve flexibility and efficiency. HUs are designable and learnable to promote transmission and modulation of hybrid information flows in HNNs. Through three cases, we demonstrate that the framework can facilitate hybrid model design. The hybrid sensing network implements multi-pathway sensing, achieving high tracking accuracy and energy efficiency. The hybrid modulation network implements hierarchical information abstraction, enabling meta-continual learning of multiple tasks. The hybrid reasoning network performs multimodal reasoning in an interpretable, robust and parallel manner. This study advances cross-paradigm modeling for a broad range of intelligent tasks.

Neuromorphic computing chip with spatiotemporal elasticity for multi-intelligent-tasking robots.

Recent advances in artificial intelligence have enhanced the abilities of mobile robots in dealing with complex and dynamic scenarios. However, to enable computationally intensive algorithms to be executed locally in multitask robots with low latency and high efficiency, innovations in computing hardware are required. Here, we report TianjicX, a neuromorphic computing hardware that can support true concurrent execution of multiple cross-computing-paradigm neural network (NN) models with various coordination manners for robotics. With spatiotemporal elasticity, TianjicX can support adaptive allocation of computing resources and scheduling of execution time for each task. Key to this approach is a high-level model, "Rivulet," which bridges the gap between robotic-level requirements and hardware implementations. It abstracts the execution of NN tasks through distribution of static data and streaming of dynamic data to form the basic activity context, adopts time and space slices to achieve elastic resource allocation for each activity, and performs configurable hybrid synchronous-asynchronous grouping. Thereby, Rivulet is capable of supporting independent and interactive execution. Building on Rivulet with hardware design for realizing spatiotemporal elasticity, a 28-nanometer TianjicX neuromorphic chip with event-driven, high parallelism, low latency, and low power was developed. Using a single TianjicX chip and a specially developed compiler stack, we built a multi-intelligent-tasking mobile robot, Tianjicat, to perform a cat-and-mouse game. Multiple tasks, including sound recognition and tracking, object recognition, obstacle avoidance, and decision-making, can be concurrently executed. Compared with NVIDIA Jetson TX2, latency is substantially reduced by 79.09 times, and dynamic power is reduced by 50.66%.

Brain-inspired global-local learning incorporated with neuromorphic computing.

There are two principle approaches for learning in artificial intelligence: error-driven global learning and neuroscience-oriented local learning. Integrating them into one network may provide complementary learning capabilities for versatile learning scenarios. At the same time, neuromorphic computing holds great promise, but still needs plenty of useful algorithms and algorithm-hardware co-designs to fully exploit its advantages. Here, we present a neuromorphic global-local synergic learning model by introducing a brain-inspired meta-learning paradigm and a differentiable spiking model incorporating neuronal dynamics and synaptic plasticity. It can meta-learn local plasticity and receive top-down supervision information for multiscale learning. We demonstrate the advantages of this model in multiple different tasks, including few-shot learning, continual learning, and fault-tolerance learning in neuromorphic vision sensors. It achieves significantly higher performance than single-learning methods. We further implement the model in the Tianjic neuromorphic platform by exploiting algorithm-hardware co-designs and prove that the model can fully utilize neuromorphic many-core architecture to develop hybrid computation paradigm.

End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip.

Integration of computer-science oriented artificial neural networks (ANNs) and neuroscience oriented spiking neural networks (SNNs) has emerged as a highly promising direction to achieve further breakthroughs in artificial intelligence through complementary advantages. This integration needs to support individual modeling of ANNs and SNNs as well as their hybrid modeling, which not only simultaneously calculates single-paradigm networks but also converts their different information representations. It remains challenging to realize effective calculation and signal conversion on the existing dedicated hardware platforms. To solve this problem, we propose an end-to-end mapping framework for implementing various hybrid neural networks on many-core neuromorphic architectures based on the cross-paradigm Tianjic chip. We construct hardware configuration schemes for four typical signal conversions and establish a global timing adjustment mechanism among different heterogeneous modules. Experimental results show that our framework can implement these hybrid models with low execution latency and low power consumption with nearly no accuracy degradation. This work provides a new approach of developing hybrid neural network models for brain-inspired computing chips and further tapping the potential of these models.

Spontaneous sparse learning for PCM-based memristor neural networks.

Neural networks trained by backpropagation have achieved tremendous successes on numerous intelligent tasks. However, naïve gradient-based training and updating methods on memristors impede applications due to intrinsic material properties. Here, we built a 39 nm 1 Gb phase change memory (PCM) memristor array and quantified the unique resistance drift effect. On this basis, spontaneous sparse learning (SSL) scheme that leverages the resistance drift to improve PCM-based memristor network training is developed. During training, SSL regards the drift effect as spontaneous consistency-based distillation process that reinforces the array weights at the high-resistance state continuously unless the gradient-based method switches them to low resistance. Experiments show that the SSL not only helps the convergence of network with better performance and sparsity controllability without additional computation in handwritten digit classification. This work promotes the learning algorithms with the intrinsic properties of memristor devices, opening a new direction for development of neuromorphic computing chips.

Adversarial symmetric GANs: Bridging adversarial samples and adversarial networks.

Generative adversarial networks have achieved remarkable performance on various tasks but suffer from training instability. Despite many training strategies proposed to improve training stability, this issue remains as a challenge. In this paper, we investigate the training instability from the perspective of adversarial samples and reveal that adversarial training on fake samples is implemented in vanilla GANs, but adversarial training on real samples has long been overlooked. Consequently, the discriminator is extremely vulnerable to adversarial perturbation and the gradient given by the discriminator contains non-informative adversarial noises, which hinders the generator from catching the pattern of real samples. Here, we develop adversarial symmetric GANs (AS-GANs) that incorporate adversarial training of the discriminator on real samples into vanilla GANs, making adversarial training symmetrical. The discriminator is therefore more robust and provides more informative gradient with less adversarial noise, thereby stabilizing training and accelerating convergence. The effectiveness of the AS-GANs is verified on image generation on CIFAR-10, CIFAR-100, CelebA, and LSUN with varied network architectures. Not only the training is more stabilized, but the FID scores of generated samples are consistently improved by a large margin compared to the baseline. Theoretical analysis is also conducted to explain why AS-GAN can improve training. The bridging of adversarial samples and adversarial networks provides a new approach to further develop adversarial networks.

A hybrid and scalable brain-inspired robotic platform.

Recent years have witnessed tremendous progress of intelligent robots brought about by mimicking human intelligence. However, current robots are still far from being able to handle multiple tasks in a dynamic environment as efficiently as humans. To cope with complexity and variability, further progress toward scalability and adaptability are essential for intelligent robots. Here, we report a brain-inspired robotic platform implemented by an unmanned bicycle that exhibits scalability of network scale, quantity and diversity to handle the changing needs of different scenarios. The platform adopts rich coding schemes and a trainable and scalable neural state machine, enabling flexible cooperation of hybrid networks. In addition, an embedded system is developed using a cross-paradigm neuromorphic chip to facilitate the implementation of diverse neural networks in spike or non-spike form. The platform achieved various real-time tasks concurrently in different real-world scenarios, providing a new pathway to enhance robots' intelligence.

A system hierarchy for brain-inspired computing.

Neuromorphic computing draws inspiration from the brain to provide computing technology and architecture with the potential to drive the next wave of computer engineering1-13. Such brain-inspired computing also provides a promising platform for the development of artificial general intelligence14,15. However, unlike conventional computing systems, which have a well established computer hierarchy built around the concept of Turing completeness and the von Neumann architecture16-18, there is currently no generalized system hierarchy or understanding of completeness for brain-inspired computing. This affects the compatibility between software and hardware, impairing the programming flexibility and development productivity of brain-inspired computing. Here we propose 'neuromorphic completeness', which relaxes the requirement for hardware completeness, and a corresponding system hierarchy, which consists of a Turing-complete software-abstraction model and a versatile abstract neuromorphic architecture. Using this hierarchy, various programs can be described as uniform representations and transformed into the equivalent executable on any neuromorphic complete hardware-that is, it ensures programming-language portability, hardware completeness and compilation feasibility. We implement toolchain software to support the execution of different types of program on various typical hardware platforms, demonstrating the advantage of our system hierarchy, including a new system-design dimension introduced by the neuromorphic completeness. We expect that our study will enable efficient and compatible progress in all aspects of brain-inspired computing systems, facilitating the development of various applications, including artificial general intelligence.

Robust Event-Based Object Tracking Combining Correlation Filter and CNN Representation.

Object tracking based on the event-based camera or dynamic vision sensor (DVS) remains a challenging task due to the noise events, rapid change of event-stream shape, chaos of complex background textures, and occlusion. To address the challenges, this paper presents a robust event-stream object tracking method based on correlation filter mechanism and convolutional neural network (CNN) representation. In the proposed method, rate coding is used to encode the event-stream object. Feature representations from hierarchical convolutional layers of a pre-trained CNN are used to represent the appearance of the rate encoded event-stream object. Results prove that the proposed method not only achieves good tracking performance in many complicated scenes with noise events, complex background textures, occlusion, and intersected trajectories, but also is robust to variable scale, variable pose, and non-rigid deformations. In addition, the correlation filter-based method has the advantage of high speed. The proposed approach will promote the potential applications of these event-based vision sensors in autonomous driving, robots and many other high-speed scenes.

Towards artificial general intelligence with hybrid Tianjic chip architecture.

There are two general approaches to developing artificial general intelligence (AGI)1: computer-science-oriented and neuroscience-oriented. Because of the fundamental differences in their formulations and coding schemes, these two approaches rely on distinct and incompatible platforms2-8, retarding the development of AGI. A general platform that could support the prevailing computer-science-based artificial neural networks as well as neuroscience-inspired models and algorithms is highly desirable. Here we present the Tianjic chip, which integrates the two approaches to provide a hybrid, synergistic platform. The Tianjic chip adopts a many-core architecture, reconfigurable building blocks and a streamlined dataflow with hybrid coding schemes, and can not only accommodate computer-science-based machine-learning algorithms, but also easily implement brain-inspired circuits and several coding schemes. Using just one chip, we demonstrate the simultaneous processing of versatile algorithms and models in an unmanned bicycle system, realizing real-time object detection, tracking, voice control, obstacle avoidance and balance control. Our study is expected to stimulate AGI development by paving the way to more generalized hardware platforms.

Self-Doping Memristors with Equivalently Synaptic Ion Dynamics for Neuromorphic Computing.

The accumulation and extrusion of Ca2+ ions in the pre- and post-synaptic terminals play crucial roles in initiating short- and long-term plasticity (STP and LTP) in biological synapses, respectively. Mimicking these synaptic behaviors by electronic devices represents a vital step toward realization of neuromorphic computing. However, the majority of reported synaptic devices usually focus on the emulation of qualitatively synaptic behaviors; devices that can truly emulate the physical behavior of the synaptic Ca2+ ion dynamics in STP and LTP are rarely reported. In this work, Ag/Ag:Ta2O5/Pt self-doping memristors were developed to equivalently emulate the Ca2+ ion dynamics of biological synapses. With conductive filaments from double sources, these memristors produced unique double-switching behavior under voltage sweeps and demonstrated several essential synaptic behaviors under pulse stimuli, including STP, LTP, STP to LTP transition, and spike-rate-dependent plasticity. Experimental results and nanoparticle dynamic simulations both showed that Ag atoms from double sources could mimic Ca2+ dynamics in the pre- and post-synaptic terminals under stimuli. A perceptron network with an STP to LTP transition layer based on the self-doping memristors was also introduced and evaluated; simulations showed that this network could solve noisy figure recognition tasks efficiently. All of these results indicate that the self-doping memristors are promising components for future hardware creation of neuromorphic systems and emulate the characteristics of the brain.

L1 -Norm Batch Normalization for Efficient Training of Deep Neural Networks.

Batch normalization (BN) has recently become a standard component for accelerating and improving the training of deep neural networks (DNNs). However, BN brings in additional calculations, consumes more memory, and significantly slows down the training iteration. Furthermore, the nonlinear square and sqrt operations in the normalization process impede low bit-width quantization techniques, which draw much attention to the deep learning hardware community. In this paper, we propose an L1 -norm BN (L1BN) with only linear operations in both forward and backward propagations during training. L1BN is approximately equivalent to the conventional L2 -norm BN (L2BN) by multiplying a scaling factor that equals (π/2)1/2 . Experiments on various convolutional neural networks and generative adversarial networks reveal that L1BN can maintain the same performance and convergence rate as L2BN but with higher computational efficiency. In real application-specified integrated circuit synthesis with reduced resources, L1BN achieves 25% speedup and 37% energy saving compared to the original L2BN. Our hardware-friendly normalization method not only surpasses L2BN in speed but also simplifies the design of deep learning accelerators. Last but not least, L1BN promises a fully quantized training of DNNs, which empowers future artificial intelligence applications on mobile devices with transfer and continual learning capability.

Truly Concomitant and Independently Expressed Short- and Long-Term Plasticity in a Bi2 O2 Se-Based Three-Terminal Memristor.

Concomitance of diverse synaptic plasticity across different timescales produces complex cognitive processes. To achieve comparable cognitive complexity in memristive neuromorphic systems, devices that are capable of emulating short-term (STP) and long-term plasticity (LTP) concomitantly are essential. In existing memristors, however, STP and LTP can only be induced selectively because of the inability to be decoupled using different loci and mechanisms. In this work, the first demonstration of truly concomitant STP and LTP is reported in a three-terminal memristor that uses independent physical phenomena to represent each form of plasticity. The emerging layered material Bi2 O2 Se is used for memristors for the first time, opening up the prospects for ultrathin, high-speed, and low-power neuromorphic devices. The concerted action of STP and LTP allows full-range modulation of the transient synaptic efficacy, from depression to facilitation, by stimulus frequency or intensity, providing a versatile device platform for neuromorphic function implementation. A heuristic recurrent neural circuitry model is developed to simulate the intricate "sleep-wake cycle autoregulation" process, in which the concomitance of STP and LTP is posited as a key factor in enabling this neural homeostasis. This work sheds new light on the development of generic memristor platforms for highly dynamic neuromorphic computing.

Highly Compact Artificial Memristive Neuron with Low Energy Consumption.

Neuromorphic systems aim to implement large-scale artificial neural network on hardware to ultimately realize human-level intelligence. The recent development of nonsilicon nanodevices has opened the huge potential of full memristive neural networks (FMNN), consisting of memristive neurons and synapses, for neuromorphic applications. Unlike the widely reported memristive synapses, the development of artificial neurons on memristive devices has less progress. Sophisticated neural dynamics is the major obstacle behind the lagging. Here a rich dynamics-driven artificial neuron is demonstrated, which successfully emulates partial essential neural features of neural processing, including leaky integration, automatic threshold-driven fire, and self-recovery, in a unified manner. The realization of bioplausible artificial neurons on a single device with ultralow power consumption paves the way for constructing energy-efficient large-scale FMNN and may boost the development of neuromorphic systems with high density, low power, and fast speed.

Native point defects of semiconducting layered Bi2O2Se.

Bi2O2Se is an emerging semiconducting, air-stable layered material (Nat. Nanotechnol. 2017, 12, 530; Nano Lett. 2017, 17, 3021), potentially exceeding MoS2 and phosphorene in electron mobility and rivalling typical Van der Waals stacked layered materials in the next-generation high-speed and low-power electronics. Holding the promise of functional versatility, it is arousing rapidly growing interest from various disciplines, including optoelectronics, thermoelectronics and piezoelectronics. In this work, we comprehensively study the electrical properties of the native point defects in Bi2O2Se, as an essential step toward understanding the fundamentals of this material. The defect landscapes dependent on both Fermi energy and the chemical potentials of atomic constituents are investigated. Along with the bulk defect analysis, a complementary inspection of the surface properties, within the simple context of charge neutrality level model, elucidates the observed n-type characteristics of Bi2O2Se based FETs. This work provides important guide to engineer the defects of Bi2O2Se for desired properties, which is key to the successful application of this emerging layered material27.

Matrix function optimization under weighted boundary constraints and its applications in network control.

The matrix function optimization under weighted boundary constraints on the matrix variables is investigated in this work. An "index-notation-arrangement based chain rule" (I-Chain rule) is introduced to obtain the gradient of a matrix function. By doing this, we propose the weighted trace-constraint-based projected gradient method (WTPGM) and weighted orthornormal-constraint-based projected gradient method (WOPGM) to locate a point of minimum of an objective/cost function of matrix variables iteratively subject to weighted trace constraint and weighted orthonormal constraint, respectively. New techniques are implemented to establish the convergence property of both algorithms. In addition, compared with the existing scheme termed "orthornormal-constraint-based projected gradient method" (OPGM) that requires the gradient has to be represented by the multiplication of a symmetrical matrix and the matrix variable itself, such a condition has been relaxed in WOPGM. Simulation results show the effectiveness of our methods not only in network control but also in other learning problems. We believe that the results reveal interesting physical insights in the field of network control and allow extensive applications of matrix function optimization problems in science and engineering.