Spiking neural networks for nonlinear regression.

Spiking neural networks (SNN), also often referred to as the third generation of neural networks, carry the potential for a massive reduction in memory and energy consumption over traditional, second-generation neural networks. Inspired by the undisputed efficiency of the human brain, they introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. Energy efficiency plays a crucial role in many engineering applications, for instance, in structural health monitoring. Machine learning in engineering contexts, especially in data-driven mechanics, focuses on regression. While regression with SNN has already been discussed in a variety of publications, in this contribution, we provide a novel formulation for its accuracy and energy efficiency. In particular, a network topology for decoding binary spike trains to real numbers is introduced, using the membrane potential of spiking neurons. Several different spiking neural architectures, ranging from simple spiking feed-forward to complex spiking long short-term memory neural networks, are derived. Since the proposed architectures do not contain any dense layers, they exploit the full potential of SNN in terms of energy efficiency. At the same time, the accuracy of the proposed SNN architectures is demonstrated by numerical examples, namely different material models. Linear and nonlinear, as well as history-dependent material models, are examined. While this contribution focuses on mechanical examples, the interested reader may regress any custom function by adapting the published source code.

Modeling and hardware implementation of universal interface-based floating fractional-order mem-elements.

Fractional-order systems generalize classical differential systems and have empirically shown to achieve fine-grain modeling of the temporal dynamics and frequency responses of certain real-world phenomena. Although the study of integer-order memory element (mem-element) emulators has persisted for several years, the study of fractional-order mem-elements has received little attention. To promote the study of the characteristics and applications of mem-element systems in fractional calculus and memory systems, a novel universal fractional-order mem-elements interface for constructing three types of fractional-order mem-element emulators is proposed in this paper. With the same circuit topology, the floating fractional-order memristor, the fractional-order memcapacitor, and fractional-order meminductor emulators can be implemented by simply combining the impedances of different passive elements. PSPICE circuit simulation and printed circuit board hardware experiments validate the dynamical behaviors and effectiveness of our proposed emulators. In addition, the dynamic relationship between fractional-order parameters and values of fractional-order impedance is explored in MATLAB simulation. The proposed fractional-order mem-element emulators built based on the universal interface are constructed with a small number of active and passive elements, which not only reduces the cost but also promotes the development of fractional-order mem-element emulators and application research for the future.

Weak self-supervised learning for seizure forecasting: a feasibility study.

This paper proposes an artificial intelligence system that continuously improves over time at event prediction using initially unlabelled data by using self-supervised learning. Time-series data are inherently autocorrelated. By using a detection model to generate weak labels on the fly, which are concurrently used as targets to train a prediction model on a time-shifted input data stream, this autocorrelation can effectively be harnessed to reduce the burden of manual labelling. This is critical in medical patient monitoring, as it enables the development of personalized forecasting models without demanding the annotation of long sequences of physiological signal recordings. We perform a feasibility study on seizure prediction, which is identified as an ideal test case, as pre-ictal brainwaves are patient-specific, and tailoring models to individual patients is known to improve forecasting performance significantly. Our self-supervised approach is used to train individualized forecasting models for 10 patients, showing an average relative improvement in sensitivity by 14.30% and a reduction in false alarms by 19.61% in early seizure forecasting. This proof-of-concept on the feasibility of using a continuous stream of time-series neurophysiological data paves the way towards a low-power neuromorphic neuromodulation system.

A Multimodal AI System for Out-of-Distribution Generalization of Seizure Identification.

Artificial intelligence (AI) and health sensory data-fusion hold the potential to automate many laborious and time-consuming processes in hospitals or ambulatory settings, e.g. home monitoring and telehealth. One such unmet challenge is rapid and accurate epileptic seizure annotation. An accurate and automatic approach can provide an alternative way to label seizures in epilepsy or deliver a substitute for inaccurate patient self-reports. Multimodal sensory fusion is believed to provide an avenue to improve the performance of AI systems in seizure identification. We propose a state-of-the-art performing AI system that combines electroencephalogram (EEG) and electrocardiogram (ECG) for seizure identification, tested on clinical data with early evidence demonstrating generalization across hospitals. The model was trained and validated on the publicly available Temple University Hospital (TUH) dataset. To evaluate performance in a clinical setting, we conducted non-patient-specific pseudo-prospective inference tests on three out-of-distribution datasets, including EPILEPSIAE (30 patients) and the Royal Prince Alfred Hospital (RPAH) in Sydney, Australia (31 neurologists-shortlisted patients and 30 randomly selected). Our multimodal approach improves the area under the receiver operating characteristic curve (AUC-ROC) by an average margin of 6.71% and 14.42% for deep learning techniques using EEG-only and ECG-only, respectively. Our model's state-of-the-art performance and robustness to out-of-distribution datasets show the accuracy and efficiency necessary to improve epilepsy diagnoses. To the best of our knowledge, this is the first pseudo-prospective study of an AI system combining EEG and ECG modalities for automatic seizure annotation achieved with fusion of two deep learning networks.

Investigating Selectorless Property within Niobium Devices for Storage Applications.

Resistive random-access memory (RRAM) crossbar arrays have shown significant promise as drivers of neuromorphic computing, in-memory computing, and high-density storage-class memory applications. However, leakage current through parasitic sneak paths is one of the dominant obstacles for large-scale commercial deployment of RRAM arrays. To overcome this issue without compromising on the structural simplicity, the use of inherent selectors native to switching is one of the most promising ways to reduce sneak path currents without sacrificing density associated with the simple two-electrode structure. In this study, niobium oxide (NbOx) was chosen as the resistive switching layer since it co-exhibits non-volatile memory and metal-insulator-transition selector behavior. Experimental results demonstrate abnormal phenomena in the reset process: a rapid decrease in current, followed by an increase when reset from the on state. The current conduction mechanism was examined through statistical analysis, and a conduction filament physical model was developed to explain the abnormal phenomenon. Under optimized operation conditions, non-linearity of ∼500 and fast switching speeds of 30 ns set and 50 ns reset were obtained. The switching behaviors with the intrinsic selector property make the NbOx device an attractive candidate for future memory and in-memory computing applications.

Low-Power Wireless Sensor Network Using Fine-Grain Control of Sensor Module Power Mode.

Wireless sensor nodes are heavily resource-constrained due to their edge form factor, which has motivated increasing battery life through low-power techniques. This paper proposes a power management method that leads to less energy consumption in an idle state than conventional power management systems used in wireless sensor nodes. We analyze and benchmark the power consumption between Sleep, Idle, and Run modes. To reduce sensor node power consumption, we develop fine-grained power modes (FGPM) with five states which modulate energy consumption according to the sensor node's communication status. We evaluate the proposed method on a test bench Mica2. As a result, the power consumed is 74.2% lower than that of conventional approaches. The proposed method targets the reduction of power consumption in IoT sensor modules with long sleep mode or short packet data in which most networks operate.

Quantized Convolutional Neural Network Implementation on a Parallel-Connected Memristor Crossbar Array for Edge AI Platforms.

There are many challenges in the hardware implementation of a neural network using nanoscale memristor crossbar arrays where the use of analog cells is concerned. Multi-state or analog cells introduce more stringent noise margins, which are difficult to adhere to in light of variability. We propose a potential solution using a 1-bit memristor that stores binary values "0" or "1" with their memristive states, denoted as a high-resistance state (HRS) and a low-resistance state (LRS). In addition, we propose a new architecture consisting of 4-parallel 1-bit memristors at each crosspoint on the array. The four 1-bit memristors connected in parallel represent 5 decimal values according to the number of activated memristors. This is then mapped to a synaptic weight, which corresponds to the state of an artificial neuron in a neural network. We implement a convolutional neural network (CNN) model on a framework (tensorflow) using an equivalent quantized weight mapping model that demonstrates learning results almost identical to a high-precision CNN model. This radix-5 CNN is mapped to hardware on the proposed parallel-connected memristor crossbar array. Also, we propose a method for negative weight representation on a memristor crossbar array. Then, we verify the CNN hardware on an edge-AI (e-AI) platform, developed on a field-programmable gate array (FPGA). In this e-AI platform, we represent five weights per crosspoint using CLB logics. We test the learning results of the CNN hardware using an e-AI platform with a dataset consisting of 4×4 images in three classes. We verify the functionality of our radix-5 CNN implementation showing comparable classification accuracy to high-precision use cases, with reduction of the area of the memristor crossbar array by half, all verified on a FPGA. Implementing the CNN model on the FPGA board can contribute to the practical use of edge-AI.

A Leaky-Integrate-and-Fire Neuron Model of Spontaneous Reset of Thin-Film Metal-Oxide Resistive Switches.

Resistive switches in crossbar arrays introduce one potential option to push past the limits of CMOS process scaling, with advantages including low switching thresholds (<3 V), high integrability with CMOS, and fast switching speeds (<10 ns). These typically employ a 1T1R scheme for each cell, where the transistor is deployed for selection and sneak path mitigation. However, when conductive filaments are formed in metal-oxide resistive switches, it is often the case that analog states are not thermodynamically favorable, and will spontaneously set or reset to a more stable state. This causes stochastic switching, variability, and non-reproducibility, in a manner which cannot be harnessed in stochastic gradient descent. Equally important is the memory leakage problem that is introduced. In this work, we present a generalized neuron model of resistive switching in the development of a phase plane characterization, and verify its operation by comparing it to our own in-house fabricated thin-film titanium-oxide memristor array. We show an alternative design methodology that draws inspiration from the leaky-integrate-and-fire neuron model. The advantages exhibited by such a methodology are to provide more biologically accurate neuronal model and to enable large scale simulations, demonstrated by the 30% improvement in speed over similar device models.

Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications.

The advent of dedicated Deep Learning (DL) accelerators and neuromorphic processors has brought on new opportunities for applying both Deep and Spiking Neural Network (SNN) algorithms to healthcare and biomedical applications at the edge. This can facilitate the advancement of medical Internet of Things (IoT) systems and Point of Care (PoC) devices. In this paper, we provide a tutorial describing how various technologies including emerging memristive devices, Field Programmable Gate Arrays (FPGAs), and Complementary Metal Oxide Semiconductor (CMOS) can be used to develop efficient DL accelerators to solve a wide variety of diagnostic, pattern recognition, and signal processing problems in healthcare. Furthermore, we explore how spiking neuromorphic processors can complement their DL counterparts for processing biomedical signals. The tutorial is augmented with case studies of the vast literature on neural network and neuromorphic hardware as applied to the healthcare domain. We benchmark various hardware platforms by performing a sensor fusion signal processing task combining electromyography (EMG) signals with computer vision. Comparisons are made between dedicated neuromorphic processors and embedded AI accelerators in terms of inference latency and energy. Finally, we provide our analysis of the field and share a perspective on the advantages, disadvantages, challenges, and opportunities that various accelerators and neuromorphic processors introduce to healthcare and biomedical domains.

Adaptive Synaptic Memory via Lithium Ion Modulation in RRAM Devices.

Biologically plausible computing systems require fine-grain tuning of analog synaptic characteristics. In this study, lithium-doped silicate resistive random access memory with a titanium nitride (TiN) electrode mimicking biological synapses is demonstrated. Biological plausibility of this RRAM device is thought to occur due to the low ionization energy of lithium ions, which enables controllable forming and filamentary retraction spontaneously or under an applied voltage. The TiN electrode can effectively store lithium ions, a principle widely adopted from battery construction, and allows state-dependent decay to be reliably achieved. As a result, this device offers multi-bit functionality and synaptic plasticity for simulating various strengths in neuronal connections. Both short-term memory and long-term memory are emulated across dynamical timescales. Spike-timing-dependent plasticity and paired-pulse facilitation are also demonstrated. These mechanisms are capable of self-pruning to generate efficient neural networks. Time-dependent resistance decay is observed for different conductance values, which mimics both biological and artificial memory pruning and conforms to the trend of the biological brain that prunes weak synaptic connections. By faithfully emulating learning rules that exist in human's higher cortical areas from STDP to synaptic pruning, the device has the capacity to drive forward the development of highly efficient neuromorphic computing systems.

Nano-Programmable Logics Based on Double-Layer Anti-Facing Memristors.

The memristor, as theorized by Chua in 1971 (L. Chua, IEEE Trans. Circuit Theory 18, 507 (1971)), is a two-terminal device whose resistance state is based on the history of charge flow brought about as a result of the voltage applied across its terminals. High-density regular fabrics for nanoscale memristors, such as crossbar arrays, are emerging architectures for system-on-chip (SoC) implementation, which provide both simplified structure and improved performance (W. H. Yu, et al., IEEE Trans. VLSI 20, 1012 (2012)). The advantage of using memristors as the switching devices within crossbar arrays is their nanoscale switching capability, which specifically changes their resistance state between high and low. In this paper, we propose a new nano-programmable logic array (PLA) device in the form of an on anti-facing double-layer memristor array. The PLA is composed of an AND plane and an OR plane merged onto the same layer. The AND and OR planes are stacked vertically such that each layer forms a crossbar architecture; thus, a cross section reveals two anti-facing memristors with 5 layers: the bottom metal layer, a memristive layer, the intermediate metal layer, an anti-facing memristive layer, and the top metal layer. The intermediate metal layer provides its output at the AND plane which is the input of the OR plane, and as such, the input and output nodes of the two logic functions are shared. Thus, the proposed architecture reduces the propagation delay of the AND plane by 70% by sharing the OR plane input wires. Additionally, the anti-facing architecture makes it easy to determine appropriate values for the pull-up and pull-down registers of the PLA.

Analysis and generation of chaos using compositely connected coupled memristors.

In large-scale high-density integrated circuits, memristors in close proximity to one another both influence, and are influenced by, the behavior of nearby memristors. However, the previous analyses of memristors-based circuit applications have seldom considered the possibility of coupling effects between memristors which invariably influences the response of all memristors, thus rendering much previous research as incomplete. In this paper, the circuit dynamics of memristive Chua's circuits are systematically analyzed based on a pair of compositely connected flux-controlled memristors characterized by cubic nonlinearity as a typical example. A theoretical analysis is undertaken and verified via MATLAB. While tuning the coupling strength, variations in circuit dynamics are characterized by phase portraits, bifurcation diagrams, and Lyapunov exponents. A new floating memristor emulator with coupling ports, described by cubic nonlinearity, is designed using off-the-shelf circuit devices and is shown to be successfully used in building chaotic circuits in hardware experiments, verifying theoretical results in simulations. This paper provides a new way through which memristors-based circuit dynamics can be influenced by tuning the coupling strength between memristors without changing other circuit parameters. It is further highlighted that when designing future memristors-based circuits, the coupling action between memristors should be considered if necessary and compensated when causing undesired circuit responses.

Formulation and Implementation of Nonlinear Integral Equations to Model Neural Dynamics Within the Vertebrate Retina.

Existing computational models of the retina often compromise between the biophysical accuracy and a hardware-adaptable methodology of implementation. When compared to the current modes of vision restoration, algorithmic models often contain a greater correlation between stimuli and the affected neural network, but lack physical hardware practicality. Thus, if the present processing methods are adapted to complement very-large-scale circuit design techniques, it is anticipated that it will engender a more feasible approach to the physical construction of the artificial retina. The computational model presented in this research serves to provide a fast and accurate predictive model of the retina, a deeper understanding of neural responses to visual stimulation, and an architecture that can realistically be transformed into a hardware device. Traditionally, implicit (or semi-implicit) ordinary differential equations (OES) have been used for optimal speed and accuracy. We present a novel approach that requires the effective integration of different dynamical time scales within a unified framework of neural responses, where the rod, cone, amacrine, bipolar, and ganglion cells correspond to the implemented pathways. Furthermore, we show that adopting numerical integration can both accelerate retinal pathway simulations by more than 50% when compared with traditional ODE solvers in some cases, and prove to be a more realizable solution for the hardware implementation of predictive retinal models.