Network inference from short, noisy, low time-resolution, partial measurements: Application to C. elegans neuronal calcium dynamics.

Network link inference from measured time series data of the behavior of dynamically interacting network nodes is an important problem with wide-ranging applications, e.g., estimating synaptic connectivity among neurons from measurements of their calcium fluorescence. Network inference methods typically begin by using the measured time series to assign to any given ordered pair of nodes a numerical score reflecting the likelihood of a directed link between those two nodes. In typical cases, the measured time series data may be subject to limitations, including limited duration, low sampling rate, observational noise, and partial nodal state measurement. However, it is unknown how the performance of link inference techniques on such datasets depends on these experimental limitations of data acquisition. Here, we utilize both synthetic data generated from coupled chaotic systems as well as experimental data obtained from Caenorhabditis elegans neural activity to systematically assess the influence of data limitations on the character of scores reflecting the likelihood of a directed link between a given node pair. We do this for three network inference techniques: Granger causality, transfer entropy, and, a machine learning-based method. Furthermore, we assess the ability of appropriate surrogate data to determine statistical confidence levels associated with the results of link-inference techniques.

Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems.

The ability of machine learning (ML) models to "extrapolate" to situations outside of the range spanned by their training data is crucial for predicting the long-term behavior of non-stationary dynamical systems (e.g., prediction of terrestrial climate change), since the future trajectories of such systems may (perhaps after crossing a tipping point) explore regions of state space which were not explored in past time-series measurements used as training data. We investigate the extent to which ML methods can yield useful results by extrapolation of such training data in the task of forecasting non-stationary dynamics, as well as conditions under which such methods fail. In general, we find that ML can be surprisingly effective even in situations that might appear to be extremely challenging, but do (as one would expect) fail when "too much" extrapolation is required. For the latter case, we show that good results can potentially be obtained by combining the ML approach with an available inaccurate conventional model based on scientific knowledge.

Parallel Machine Learning for Forecasting the Dynamics of Complex Networks.

Forecasting the dynamics of large, complex, sparse networks from previous time series data is important in a wide range of contexts. Here we present a machine learning scheme for this task using a parallel architecture that mimics the topology of the network of interest. We demonstrate the utility and scalability of our method implemented using reservoir computing on a chaotic network of oscillators. Two levels of prior knowledge are considered: (i) the network links are known, and (ii) the network links are unknown and inferred via a data-driven approach to approximately optimize prediction.

Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate,regime transitions, and the effect of stochasticity.

We develop and test machine learning techniques for successfully using past state time series data and knowledge of a time-dependent system parameter to predict the evolution of the "climate" associated with the long-term behavior of a non-stationary dynamical system, where the non-stationary dynamical system is itself unknown. By the term climate, we mean the statistical properties of orbits rather than their precise trajectories in time. By the term non-stationary, we refer to systems that are, themselves, varying with time. We show that our methods perform well on test systems predicting both continuous gradual climate evolution as well as relatively sudden climate changes (which we refer to as "regime transitions"). We consider not only noiseless (i.e., deterministic) non-stationary dynamical systems, but also climate prediction for non-stationary dynamical systems subject to stochastic forcing (i.e., dynamical noise), and we develop a method for handling this latter case. The main conclusion of this paper is that machine learning has great promise as a new and highly effective approach to accomplishing data driven prediction of non-stationary systems.

Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components.

We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data are in the form of noisy partial measurements of the past and present state of the dynamical system. Recently, there have been several promising data-driven approaches to forecasting of chaotic dynamical systems using machine learning. Particularly promising among these are hybrid approaches that combine machine learning with a knowledge-based model, where a machine-learning technique is used to correct the imperfections in the knowledge-based model. Such imperfections may be due to incomplete understanding and/or limited resolution of the physical processes in the underlying dynamical system, e.g., the atmosphere or the ocean. Previously proposed data-driven forecasting approaches tend to require, for training, measurements of all the variables that are intended to be forecast. We describe a way to relax this assumption by combining data assimilation with machine learning. We demonstrate this technique using the Ensemble Transform Kalman Filter to assimilate synthetic data for the three-variable Lorenz 1963 system and for the Kuramoto-Sivashinsky system, simulating a model error in each case by a misspecified parameter value. We show that by using partial measurements of the state of the dynamical system, we can train a machine-learning model to improve predictions made by an imperfect knowledge-based model.

Separation of chaotic signals by reservoir computing.

We demonstrate the utility of machine learning in the separation of superimposed chaotic signals using a technique called reservoir computing. We assume no knowledge of the dynamical equations that produce the signals and require only training data consisting of finite-time samples of the component signals. We test our method on signals that are formed as linear combinations of signals from two Lorenz systems with different parameters. Comparing our nonlinear method with the optimal linear solution to the separation problem, the Wiener filter, we find that our method significantly outperforms the Wiener filter in all the scenarios we study. Furthermore, this difference is particularly striking when the component signals have similar frequency spectra. Indeed, our method works well when the component frequency spectra are indistinguishable-a case where a Wiener filter performs essentially no separation.

Combining machine learning with knowledge-based modeling for scalable forecasting and subgrid-scale closure of large, complex, spatiotemporal systems.

We consider the commonly encountered situation (e.g., in weather forecast) where the goal is to predict the time evolution of a large, spatiotemporally chaotic dynamical system when we have access to both time series data of previous system states and an imperfect model of the full system dynamics. Specifically, we attempt to utilize machine learning as the essential tool for integrating the use of past data into predictions. In order to facilitate scalability to the common scenario of interest where the spatiotemporally chaotic system is very large and complex, we propose combining two approaches: (i) a parallel machine learning prediction scheme and (ii) a hybrid technique for a composite prediction system composed of a knowledge-based component and a machine learning-based component. We demonstrate that not only can this method combining (i) and (ii) be scaled to give excellent performance for very large systems but also that the length of time series data needed to train our multiple, parallel machine learning components is dramatically less than that necessary without parallelization. Furthermore, considering cases where computational realization of the knowledge-based component does not resolve subgrid-scale processes, our scheme is able to use training data to incorporate the effect of the unresolved short-scale dynamics upon the resolved longer-scale dynamics (subgrid-scale closure).

Observing microscopic transitions from macroscopic bursts: Instability-mediated resetting in the incoherent regime of the D-dimensional generalized Kuramoto model.

This paper considers a recently introduced D-dimensional generalized Kuramoto model for many (N≫1) interacting agents, in which the agent states are D-dimensional unit vectors. It was previously shown that, for even (but not odd) D, similar to the original Kuramoto model (D=2), there exists a continuous dynamical phase transition from incoherence to coherence of the time asymptotic attracting state (time t→∞) as the coupling parameter K increases through a critical value which we denote Kc (+)>0. We consider this transition from the point of view of the stability of an incoherent state, where an incoherent state is defined as one for which the N→∞ distribution function is time-independent and the macroscopic order parameter is zero. In contrast with D=2, for even D>2, there is an infinity of possible incoherent equilibria, each of which becomes unstable with increasing K at a different point K=Kc. Although there are incoherent equilibria for which Kc=Kc (+), there are also incoherent equilibria with a range of possible Kc values below Kc (+), (Kc (+)/2)≤KcKc (+)? We find, for a given incoherent equilibrium, that, if K is rapidly increased from K

Complexity reduction ansatz for systems of interacting orientable agents: Beyond the Kuramoto model.

Previous results have shown that a large class of complex systems consisting of many interacting heterogeneous phase oscillators exhibit an attracting invariant manifold. This result has enabled reduced analytic system descriptions from which all the long term dynamics of these systems can be calculated. Although very useful, these previous results are limited by the restriction that the individual interacting system components have one-dimensional dynamics, with states described by a single, scalar, angle-like variable (e.g., the Kuramoto model). In this paper, we consider a generalization to an appropriate class of coupled agents with higher-dimensional dynamics. For this generalized class of model systems, we demonstrate that the dynamics again contain an invariant manifold, hence enabling previously inaccessible analysis and improved numerical study, allowing a similar simplified description of these systems. We also discuss examples illustrating the potential utility of our results for a wide range of interesting situations.

Scattering statistics in nonlinear wave chaotic systems.

The Random Coupling Model (RCM) is a statistical approach for studying the scattering properties of linear wave chaotic systems in the semi-classical regime. Its success has been experimentally verified in various over-moded wave settings, including both microwave and acoustic systems. It is of great interest to extend its use in nonlinear systems. This paper studies the impact of a nonlinear port on the measured statistical electromagnetic properties of a ray-chaotic complex enclosure in the short wavelength limit. A Vector Network Analyzer is upgraded with a high power option, which enables calibrated scattering (S) parameter measurements up to +43dBm. By attaching a diode to the excitation antenna, amplitude-dependent S-parameters and Wigner reaction matrix (impedance) statistics are observed. We have systematically studied how the key components in the RCM are affected by this nonlinear port, including the radiation impedance, short ray orbit corrections, and statistical properties. By applying the newly developed radiation efficiency extension to the RCM, we find that the diode admittance increases with the excitation amplitude. This reduces the amount of power entering the cavity through the port so that the diode effectively acts as a protection element. As a result, we have developed a quantitative understanding of the statistical scattering properties of a semi-classical wave chaotic system with a nonlinear coupling channel.

Using machine learning to assess short term causal dependence and infer network links.

We introduce and test a general machine-learning-based technique for the inference of short term causal dependence between state variables of an unknown dynamical system from time-series measurements of its state variables. Our technique leverages the results of a machine learning process for short time prediction to achieve our goal. The basic idea is to use the machine learning to estimate the elements of the Jacobian matrix of the dynamical flow along an orbit. The type of machine learning that we employ is reservoir computing. We present numerical tests on link inference of a network of interacting dynamical nodes. It is seen that dynamical noise can greatly enhance the effectiveness of our technique, while observational noise degrades the effectiveness. We believe that the competition between these two opposing types of noise will be the key factor determining the success of causal inference in many of the most important application situations.

Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach.

We demonstrate the effectiveness of using machine learning for model-free prediction of spatiotemporally chaotic systems of arbitrarily large spatial extent and attractor dimension purely from observations of the system's past evolution. We present a parallel scheme with an example implementation based on the reservoir computing paradigm and demonstrate the scalability of our scheme using the Kuramoto-Sivashinsky equation as an example of a spatiotemporally chaotic system.

Attractor reconstruction by machine learning.

A machine-learning approach called "reservoir computing" has been used successfully for short-term prediction and attractor reconstruction of chaotic dynamical systems from time series data. We present a theoretical framework that describes conditions under which reservoir computing can create an empirical model capable of skillful short-term forecasts and accurate long-term ergodic behavior. We illustrate this theory through numerical experiments. We also argue that the theory applies to certain other machine learning methods for time series prediction.

Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model.

A model-based approach to forecasting chaotic dynamical systems utilizes knowledge of the mechanistic processes governing the dynamics to build an approximate mathematical model of the system. In contrast, machine learning techniques have demonstrated promising results for forecasting chaotic systems purely from past time series measurements of system state variables (training data), without prior knowledge of the system dynamics. The motivation for this paper is the potential of machine learning for filling in the gaps in our underlying mechanistic knowledge that cause widely-used knowledge-based models to be inaccurate. Thus, we here propose a general method that leverages the advantages of these two approaches by combining a knowledge-based model and a machine learning technique to build a hybrid forecasting scheme. Potential applications for such an approach are numerous (e.g., improving weather forecasting). We demonstrate and test the utility of this approach using a particular illustrative version of a machine learning known as reservoir computing, and we apply the resulting hybrid forecaster to a low-dimensional chaotic system, as well as to a high-dimensional spatiotemporal chaotic system. These tests yield extremely promising results in that our hybrid technique is able to accurately predict for a much longer period of time than either its machine-learning component or its model-based component alone.

Frequency and phase synchronization in large groups: Low dimensional description of synchronized clapping, firefly flashing, and cricket chirping.

A common observation is that large groups of oscillatory biological units often have the ability to synchronize. A paradigmatic model of such behavior is provided by the Kuramoto model, which achieves synchronization through coupling of the phase dynamics of individual oscillators, while each oscillator maintains a different constant inherent natural frequency. Here we consider the biologically likely possibility that the oscillatory units may be capable of enhancing their synchronization ability by adaptive frequency dynamics. We propose a simple augmentation of the Kuramoto model which does this. We also show that, by the use of a previously developed technique [Ott and Antonsen, Chaos 18, 037113 (2008)], it is possible to reduce the resulting dynamics to a lower dimensional system for the macroscopic evolution of the oscillator ensemble. By employing this reduction, we investigate the dynamics of our system, finding a characteristic hysteretic behavior and enhancement of the quality of the achieved synchronization.

Uncovering low dimensional macroscopic chaotic dynamics of large finite size complex systems.

In the last decade, it has been shown that a large class of phase oscillator models admit low dimensional descriptions for the macroscopic system dynamics in the limit of an infinite number N of oscillators. The question of whether the macroscopic dynamics of other similar systems also have a low dimensional description in the infinite N limit has, however, remained elusive. In this paper, we show how techniques originally designed to analyze noisy experimental chaotic time series can be used to identify effective low dimensional macroscopic descriptions from simulations with a finite number of elements. We illustrate and verify the effectiveness of our approach by applying it to the dynamics of an ensemble of globally coupled Landau-Stuart oscillators for which we demonstrate low dimensional macroscopic chaotic behavior with an effective 4-dimensional description. By using this description, we show that one can calculate dynamical invariants such as Lyapunov exponents and attractor dimensions. One could also use the reconstruction to generate short-term predictions of the macroscopic dynamics.

Nonlinear wave chaos: statistics of second harmonic fields.

Concepts from the field of wave chaos have been shown to successfully predict the statistical properties of linear electromagnetic fields in electrically large enclosures. The Random Coupling Model (RCM) describes these properties by incorporating both universal features described by Random Matrix Theory and the system-specific features of particular system realizations. In an effort to extend this approach to the nonlinear domain, we add an active nonlinear frequency-doubling circuit to an otherwise linear wave chaotic system, and we measure the statistical properties of the resulting second harmonic fields. We develop an RCM-based model of this system as two linear chaotic cavities coupled by means of a nonlinear transfer function. The harmonic field strengths are predicted to be the product of two statistical quantities and the nonlinearity characteristics. Statistical results from measurement-based calculation, RCM-based simulation, and direct experimental measurements are compared and show good agreement over many decades of power.

Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.

We use recent advances in the machine learning area known as "reservoir computing" to formulate a method for model-free estimation from data of the Lyapunov exponents of a chaotic process. The technique uses a limited time series of measurements as input to a high-dimensional dynamical system called a "reservoir." After the reservoir's response to the data is recorded, linear regression is used to learn a large set of parameters, called the "output weights." The learned output weights are then used to form a modified autonomous reservoir designed to be capable of producing an arbitrarily long time series whose ergodic properties approximate those of the input signal. When successful, we say that the autonomous reservoir reproduces the attractor's "climate." Since the reservoir equations and output weights are known, we can compute the derivatives needed to determine the Lyapunov exponents of the autonomous reservoir, which we then use as estimates of the Lyapunov exponents for the original input generating system. We illustrate the effectiveness of our technique with two examples, the Lorenz system and the Kuramoto-Sivashinsky (KS) equation. In the case of the KS equation, we note that the high dimensional nature of the system and the large number of Lyapunov exponents yield a challenging test of our method, which we find the method successfully passes.

Reservoir observers: Model-free inference of unmeasured variables in chaotic systems.

Deducing the state of a dynamical system as a function of time from a limited number of concurrent system state measurements is an important problem of great practical utility. A scheme that accomplishes this is called an "observer." We consider the case in which a model of the system is unavailable or insufficiently accurate, but "training" time series data of the desired state variables are available for a short period of time, and a limited number of other system variables are continually measured. We propose a solution to this problem using networks of neuron-like units known as "reservoir computers." The measurements that are continually available are input to the network, which is trained with the limited-time data to output estimates of the desired state variables. We demonstrate our method, which we call a "reservoir observer," using the Rössler system, the Lorenz system, and the spatiotemporally chaotic Kuramoto-Sivashinsky equation. Subject to the condition of observability (i.e., whether it is in principle possible, by any means, to infer the desired unmeasured variables from the measured variables), we show that the reservoir observer can be a very effective and versatile tool for robustly reconstructing unmeasured dynamical system variables.

Modeling the network dynamics of pulse-coupled neurons.

We derive a mean-field approximation for the macroscopic dynamics of large networks of pulse-coupled theta neurons in order to study the effects of different network degree distributions and degree correlations (assortativity). Using the ansatz of Ott and Antonsen [Chaos 18, 037113 (2008)], we obtain a reduced system of ordinary differential equations describing the mean-field dynamics, with significantly lower dimensionality compared with the complete set of dynamical equations for the system. We find that, for sufficiently large networks and degrees, the dynamical behavior of the reduced system agrees well with that of the full network. This dimensional reduction allows for an efficient characterization of system phase transitions and attractors. For networks with tightly peaked degree distributions, the macroscopic behavior closely resembles that of fully connected networks previously studied by others. In contrast, networks with highly skewed degree distributions exhibit different macroscopic dynamics due to the emergence of degree dependent behavior of different oscillators. For nonassortative networks (i.e., networks without degree correlations), we observe the presence of a synchronously firing phase that can be suppressed by the presence of either assortativity or disassortativity in the network. We show that the results derived here can be used to analyze the effects of network topology on macroscopic behavior in neuronal networks in a computationally efficient fashion.