Neuronal diversity can improve machine learning for physics and beyond.

Diversity conveys advantages in nature, yet homogeneous neurons typically comprise the layers of artificial neural networks. Here we construct neural networks from neurons that learn their own activation functions, quickly diversify, and subsequently outperform their homogeneous counterparts on image classification and nonlinear regression tasks. Sub-networks instantiate the neurons, which meta-learn especially efficient sets of nonlinear responses. Examples include conventional neural networks classifying digits and forecasting a van der Pol oscillator and physics-informed Hamiltonian neural networks learning Hénon-Heiles stellar orbits and the swing of a video recorded pendulum clock. Such learned diversity provides examples of dynamical systems selecting diversity over uniformity and elucidates the role of diversity in natural and artificial systems.

Physics-enhanced neural networks learn order and chaos.

Artificial neural networks are universal function approximators. They can forecast dynamics, but they may need impractically many neurons to do so, especially if the dynamics is chaotic. We use neural networks that incorporate Hamiltonian dynamics to efficiently learn phase space orbits even as nonlinear systems transition from order to chaos. We demonstrate Hamiltonian neural networks on a widely used dynamics benchmark, the Hénon-Heiles potential, and on nonperturbative dynamical billiards. We introspect to elucidate the Hamiltonian neural network forecasting.

Nonlinear dynamics based machine learning: Utilizing dynamics-based flexibility of nonlinear circuits to implement different functions.

The core element of machine learning is a flexible, universal function approximator that can be trained and fit into the data. One of the main challenges in modern machine learning is to understand the role of nonlinearity and complexity in these universal function approximators. In this research, we focus on nonlinear complex systems, and show their capability in representation and learning of different functions. Complex nonlinear dynamics and chaos naturally yield an almost infinite diversity of dynamical behaviors and functions. Physical, biological and engineered systems can utilize this diversity to implement adaptive, robust behaviors and operations. A nonlinear dynamical system can be considered as an embodiment of a collection of different possible behaviors or functions, from which different behaviors or functions can be chosen as a response to different conditions or problems. This process of selection can be manual in the sense that one can manually pick and choose the right function through directly setting parameters. Alternatively, we can automate the process and allow the system itself learn how to do it. This creates an approach to machine learning, wherein the nonlinear dynamics represents and embodies different possible functions, and it learns through training how to pick the right function from this function space. We report on how we utilized nonlinear dynamics and chaos to design and fabricate nonlinear dynamics based, morphable hardware in silicon as a physical embodiment for different possible functions. We demonstrate how this flexible, morphable hardware learns through learning and searching algorithms such as genetic algorithm to implement different desired functions. In this approach, we combine two powerful natural and biological phenomenon, Darwinian evolution and nonlinear dynamics and chaos, as a dynamics-oriented approach to designing intelligent, adaptive systems with applications. Nonlinear dynamics embodies different functions at the hardware level, while an evolutionary method is utilized in order to find the parameters to implement the right function.

Chaotic attractor hopping yields logic operations.

Certain nonlinear systems can switch between dynamical attractors occupying different regions of phase space, under variation of parameters or initial states. In this work we exploit this feature to obtain reliable logic operations. With logic output 0/1 mapped to dynamical attractors bounded in distinct regions of phase space, and logic inputs encoded by a very small bias parameter, we explicitly demonstrate that the system hops consistently in response to an external input stream, operating effectively as a reliable logic gate. This system offers the advantage that very low-amplitude inputs yield highly amplified outputs. Additionally, different dynamical variables in the system yield complementary logic operations in parallel. Further, we show that in certain parameter regions noise aids the reliability of logic operations, and is actually necessary for obtaining consistent outputs. This leads us to a generalization of the concept of Logical Stochastic Resonance to attractors more complex than fixed point states, such as periodic or chaotic attractors. Lastly, the results are verified in electronic circuit experiments, demonstrating the robustness of the phenomena. So we have combined the research directions of Chaos Computing and Logical Stochastic Resonance here, and this approach has potential to be realized in wide-ranging systems.

Detecting and characterizing high-frequency oscillations in epilepsy: a case study of big data analysis.

We develop a framework to uncover and analyse dynamical anomalies from massive, nonlinear and non-stationary time series data. The framework consists of three steps: preprocessing of massive datasets to eliminate erroneous data segments, application of the empirical mode decomposition and Hilbert transform paradigm to obtain the fundamental components embedded in the time series at distinct time scales, and statistical/scaling analysis of the components. As a case study, we apply our framework to detecting and characterizing high-frequency oscillations (HFOs) from a big database of rat electroencephalogram recordings. We find a striking phenomenon: HFOs exhibit on-off intermittency that can be quantified by algebraic scaling laws. Our framework can be generalized to big data-related problems in other fields such as large-scale sensor data and seismic data analysis.

Nonlinear dynamics as an engine of computation.

Control of chaos teaches that control theory can tame the complex, random-like behaviour of chaotic systems. This alliance between control methods and physics-cybernetical physics-opens the door to many applications, including dynamics-based computing. In this article, we introduce nonlinear dynamics and its rich, sometimes chaotic behaviour as an engine of computation. We review our work that has demonstrated how to compute using nonlinear dynamics. Furthermore, we investigate the interrelationship between invariant measures of a dynamical system and its computing power to strengthen the bridge between physics and computation.This article is part of the themed issue 'Horizons of cybernetical physics'.

Superlinearly scalable noise robustness of redundant coupled dynamical systems.

We illustrate through theory and numerical simulations that redundant coupled dynamical systems can be extremely robust against local noise in comparison to uncoupled dynamical systems evolving in the same noisy environment. Previous studies have shown that the noise robustness of redundant coupled dynamical systems is linearly scalable and deviations due to noise can be minimized by increasing the number of coupled units. Here, we demonstrate that the noise robustness can actually be scaled superlinearly if some conditions are met and very high noise robustness can be realized with very few coupled units. We discuss these conditions and show that this superlinear scalability depends on the nonlinearity of the individual dynamical units. The phenomenon is demonstrated in discrete as well as continuous dynamical systems. This superlinear scalability not only provides us an opportunity to exploit the nonlinearity of physical systems without being bogged down by noise but may also help us in understanding the functional role of coupled redundancy found in many biological systems. Moreover, engineers can exploit superlinear noise suppression by starting a coupled system near (not necessarily at) the appropriate initial condition.

Exploiting chaos for applications.

We discuss how understanding the nature of chaotic dynamics allows us to control these systems. A controlled chaotic system can then serve as a versatile pattern generator that can be used for a range of application. Specifically, we will discuss the application of controlled chaos to the design of novel computational paradigms. Thus, we present an illustrative research arc, starting with ideas of control, based on the general understanding of chaos, moving over to applications that influence the course of building better devices.

Nonlinear dynamics based digital logic and circuits.

We discuss the role and importance of dynamics in the brain and biological neural networks and argue that dynamics is one of the main missing elements in conventional Boolean logic and circuits. We summarize a simple dynamics based computing method, and categorize different techniques that we have introduced to realize logic, functionality, and programmability. We discuss the role and importance of coupled dynamics in networks of biological excitable cells, and then review our simple coupled dynamics based method for computing. In this paper, for the first time, we show how dynamics can be used and programmed to implement computation in any given base, including but not limited to base two.

Strange nonchaotic stars.

The unprecedented light curves of the Kepler space telescope document how the brightness of some stars pulsates at primary and secondary frequencies whose ratios are near the golden mean, the most irrational number. A nonlinear dynamical system driven by an irrational ratio of frequencies generically exhibits a strange but nonchaotic attractor. For Kepler's "golden" stars, we present evidence of the first observation of strange nonchaotic dynamics in nature outside the laboratory. This discovery could aid the classification and detailed modeling of variable stars.

Noise tolerant spatiotemporal chaos computing.

We introduce and design a noise tolerant chaos computing system based on a coupled map lattice (CML) and the noise reduction capabilities inherent in coupled dynamical systems. The resulting spatiotemporal chaos computing system is more robust to noise than a single map chaos computing system. In this CML based approach to computing, under the coupled dynamics, the local noise from different nodes of the lattice diffuses across the lattice, and it attenuates each other's effects, resulting in a system with less noise content and a more robust chaos computing architecture.

Phase shift in the 24-hour rhythm of hippocampal EEG spiking activity in a rat model of temporal lobe epilepsy.

For over a century epileptic seizures have been known to cluster at specific times of the day. Recent studies have suggested that the circadian regulatory system may become permanently altered in epilepsy, but little is known about how this affects neural activity and the daily pattern of seizures. To investigate, we tracked long-term changes in the rate of spontaneous hippocampal EEG spikes (SPKs) in a rat model of temporal lobe epilepsy. In healthy animals, SPKs oscillated with near 24-h period; however, after injury by status epilepticus, a persistent phase shift of ∼12 h emerged in animals that later went on to develop chronic spontaneous seizures. Additional measurements showed that global 24-h rhythms, including core body temperature and theta state transitions, did not phase shift. Instead, we hypothesized that locally impaired circadian input to the hippocampus might be responsible for the SPK phase shift. This was investigated with a biophysical computer model in which we showed that subtle changes in the relative strengths of circadian input could produce a phase shift in hippocampal neural activity. MRI provided evidence that the medial septum, a putative circadian relay center for the hippocampus, exhibits signs of damage and therefore could contribute to local circadian impairment. Our results suggest that balanced circadian input is critical to maintaining natural circadian phase in the hippocampus and that damage to circadian relay centers, such as the medial septum, may disrupt this balance. We conclude by discussing how abnormal circadian regulation may contribute to the daily rhythms of epileptic seizures and related cognitive dysfunction.

Chaos computing in terms of periodic orbits.

The complex dynamics of chaotic systems can perform computations. The parameters and/or the initial conditions of a dynamical system are the data inputs and the resulting system state is the output of the computation. By controlling how inputs are mapped to outputs, a specific function can be performed. Previously no clear connection has been drawn between the structure of the dynamics and the computation. In this paper we demonstrate how chaos computation can be explained, modeled, and even predicted in terms of the dynamics of the underlying chaotic system, specifically the periodic orbit structure of the system. Knowing the dynamical equations of the system, we compute the system's periodic orbits as well as its stability in terms of its eigenvalues, thereby demonstrating how, how well, and what the chaotic system can compute.

Stochastic amplification of calcium-activated potassium currents in Ca2+ microdomains.

Small conductance (SK) calcium-activated potassium channels are found in many tissues throughout the body and open in response to elevations in intracellular calcium. In hippocampal neurons, SK channels are spatially co-localized with L-Type calcium channels. Due to the restriction of calcium transients into microdomains, only a limited number of L-Type Ca(2+) channels can activate SK and, thus, stochastic gating becomes relevant. Using a stochastic model with calcium microdomains, we predict that intracellular Ca(2+) fluctuations resulting from Ca(2+) channel gating can increase SK2 subthreshold activity by 1-2 orders of magnitude. This effectively reduces the value of the Hill coefficient. To explain the underlying mechanism, we show how short, high-amplitude calcium pulses associated with stochastic gating of calcium channels are much more effective at activating SK2 channels than the steady calcium signal produced by a deterministic simulation. This stochastic amplification results from two factors: first, a supralinear rise in the SK2 channel's steady-state activation curve at low calcium levels and, second, a momentary reduction in the channel's time constant during the calcium pulse, causing the channel to approach its steady-state activation value much faster than it decays. Stochastic amplification can potentially explain subthreshold SK2 activation in unified models of both sub- and suprathreshold regimes. Furthermore, we expect it to be a general phenomenon relevant to many proteins that are activated nonlinearly by stochastic ligand release.

Unstable periodic orbits and noise in chaos computing.

Different methods to utilize the rich library of patterns and behaviors of a chaotic system have been proposed for doing computation or communication. Since a chaotic system is intrinsically unstable and its nearby orbits diverge exponentially from each other, special attention needs to be paid to the robustness against noise of chaos-based approaches to computation. In this paper unstable periodic orbits, which form the skeleton of any chaotic system, are employed to build a model for the chaotic system to measure the sensitivity of each orbit to noise, and to select the orbits whose symbolic representations are relatively robust against the existence of noise. Furthermore, since unstable periodic orbits are extractable from time series, periodic orbit-based models can be extracted from time series too. Chaos computing can be and has been implemented on different platforms, including biological systems. In biology noise is always present; as a result having a clear model for the effects of noise on any given biological implementation has profound importance. Also, since in biology it is hard to obtain exact dynamical equations of the system under study, the time series techniques we introduce here are of critical importance.

Logical stochastic resonance with correlated internal and external noises in a synthetic biological logic block.

Following the advent of synthetic biology, several gene networks have been engineered to emulate digital devices, with the ability to program cells for different applications. In this work, we adapt the concept of logical stochastic resonance to a synthetic gene network derived from a bacteriophage λ. The intriguing results of this study show that it is possible to build a biological logic block that can emulate or switch from the AND to the OR gate functionalities through externally tuning the system parameters. Moreover, this behavior and the robustness of the logic gate are underpinned by the presence of an optimal amount of random fluctuations. We extend our earlier work in this field, by taking into account the effects of correlated external (additive) and internal (multiplicative or state-dependent) noise. Results obtained through analytical calculations as well as numerical simulations are presented.

Introduction to focus issue: intrinsic and designed computation: information processing in dynamical systems--beyond the digital hegemony.

How dynamical systems store and process information is a fundamental question that touches a remarkably wide set of contemporary issues: from the breakdown of Moore's scaling laws--that predicted the inexorable improvement in digital circuitry--to basic philosophical problems of pattern in the natural world. It is a question that also returns one to the earliest days of the foundations of dynamical systems theory, probability theory, mathematical logic, communication theory, and theoretical computer science. We introduce the broad and rather eclectic set of articles in this Focus Issue that highlights a range of current challenges in computing and dynamical systems.

Chaogates: morphing logic gates that exploit dynamical patterns.

Chaotic systems can yield a wide variety of patterns. Here we use this feature to generate all possible fundamental logic gate functions. This forms the basis of the design of a dynamical computing device, a chaogate, that can be rapidly morphed to become any desired logic gate. Here we review the basic concepts underlying this and present an extension of the formalism to include asymmetric logic functions.

Support vector machines for seizure detection in an animal model of chronic epilepsy.

We compare the performance of three support vector machine (SVM) types: weighted SVM, one-class SVM and support vector data description (SVDD) for the application of seizure detection in an animal model of chronic epilepsy. Large EEG datasets (273 h and 91 h respectively, with a sampling rate of 1 kHz) from two groups of rats with chronic epilepsy were used in this study. For each of these EEG datasets, we extracted three energy-based seizure detection features: mean energy, mean curve length and wavelet energy. Using these features we performed twofold cross-validation to obtain the performance statistics: sensitivity (S), specificity (K) and detection latency (tau) as a function of control parameters for the given SVM. Optimal control parameters for each SVM type that produced the best seizure detection statistics were then identified using two independent strategies. Performance of each SVM type is ranked based on the overall seizure detection performance through an optimality index metric (O). We found that SVDD not only performed better than the other SVM types in terms of highest value of the mean optimality index metric (O⁻) but also gave a more reliable performance across the two EEG datasets.

Effects of phase on homeostatic spike rates.

Recent experimental results by Talathi et al. (Neurosci Lett 455:145-149, 2009) showed a divergence in the spike rates of two types of population spike events, representing the putative activity of the excitatory and inhibitory neurons in the CA1 area of an animal model for temporal lobe epilepsy. The divergence in the spike rate was accompanied by a shift in the phase of oscillations between these spike rates leading to a spontaneous epileptic seizure. In this study, we propose a model of homeostatic synaptic plasticity which assumes that the target spike rate of populations of excitatory and inhibitory neurons in the brain is a function of the phase difference between the excitatory and inhibitory spike rates. With this model of homeostatic synaptic plasticity, we are able to simulate the spike rate dynamics seen experimentally by Talathi et al. in a large network of interacting excitatory and inhibitory neurons using two different spiking neuron models. A drift analysis of the spike rates resulting from the homeostatic synaptic plasticity update rule allowed us to determine the type of synapse that may be primarily involved in the spike rate imbalance in the experimental observation by Talathi et al. We find excitatory neurons, particularly those in which the excitatory neuron is presynaptic, have the most influence in producing the diverging spike rates and causing the spike rates to be anti-phase. Our analysis suggests that the excitatory neuronal population, more specifically the excitatory to excitatory synaptic connections, could be implicated in a methodology designed to control epileptic seizures.