Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components.

We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data are in the form of noisy partial measurements of the past and present state of the dynamical system. Recently, there have been several promising data-driven approaches to forecasting of chaotic dynamical systems using machine learning. Particularly promising among these are hybrid approaches that combine machine learning with a knowledge-based model, where a machine-learning technique is used to correct the imperfections in the knowledge-based model. Such imperfections may be due to incomplete understanding and/or limited resolution of the physical processes in the underlying dynamical system, e.g., the atmosphere or the ocean. Previously proposed data-driven forecasting approaches tend to require, for training, measurements of all the variables that are intended to be forecast. We describe a way to relax this assumption by combining data assimilation with machine learning. We demonstrate this technique using the Ensemble Transform Kalman Filter to assimilate synthetic data for the three-variable Lorenz 1963 system and for the Kuramoto-Sivashinsky system, simulating a model error in each case by a misspecified parameter value. We show that by using partial measurements of the state of the dynamical system, we can train a machine-learning model to improve predictions made by an imperfect knowledge-based model.

Separation of chaotic signals by reservoir computing.

We demonstrate the utility of machine learning in the separation of superimposed chaotic signals using a technique called reservoir computing. We assume no knowledge of the dynamical equations that produce the signals and require only training data consisting of finite-time samples of the component signals. We test our method on signals that are formed as linear combinations of signals from two Lorenz systems with different parameters. Comparing our nonlinear method with the optimal linear solution to the separation problem, the Wiener filter, we find that our method significantly outperforms the Wiener filter in all the scenarios we study. Furthermore, this difference is particularly striking when the component signals have similar frequency spectra. Indeed, our method works well when the component frequency spectra are indistinguishable-a case where a Wiener filter performs essentially no separation.

Dynamics of analog logic-gate networks for machine learning.

We describe the continuous-time dynamics of networks implemented on Field Programable Gate Arrays (FPGAs). The networks can perform Boolean operations when the FPGA is in the clocked (digital) mode; however, we run the programed FPGA in the unclocked (analog) mode. Our motivation is to use these FPGA networks as ultrafast machine-learning processors, using the technique of reservoir computing. We study both the undriven dynamics and the input response of these networks as we vary network design parameters, and we relate the dynamics to accuracy on two machine-learning tasks.

Attractor reconstruction by machine learning.

A machine-learning approach called "reservoir computing" has been used successfully for short-term prediction and attractor reconstruction of chaotic dynamical systems from time series data. We present a theoretical framework that describes conditions under which reservoir computing can create an empirical model capable of skillful short-term forecasts and accurate long-term ergodic behavior. We illustrate this theory through numerical experiments. We also argue that the theory applies to certain other machine learning methods for time series prediction.

Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model.

A model-based approach to forecasting chaotic dynamical systems utilizes knowledge of the mechanistic processes governing the dynamics to build an approximate mathematical model of the system. In contrast, machine learning techniques have demonstrated promising results for forecasting chaotic systems purely from past time series measurements of system state variables (training data), without prior knowledge of the system dynamics. The motivation for this paper is the potential of machine learning for filling in the gaps in our underlying mechanistic knowledge that cause widely-used knowledge-based models to be inaccurate. Thus, we here propose a general method that leverages the advantages of these two approaches by combining a knowledge-based model and a machine learning technique to build a hybrid forecasting scheme. Potential applications for such an approach are numerous (e.g., improving weather forecasting). We demonstrate and test the utility of this approach using a particular illustrative version of a machine learning known as reservoir computing, and we apply the resulting hybrid forecaster to a low-dimensional chaotic system, as well as to a high-dimensional spatiotemporal chaotic system. These tests yield extremely promising results in that our hybrid technique is able to accurately predict for a much longer period of time than either its machine-learning component or its model-based component alone.

Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.

We use recent advances in the machine learning area known as "reservoir computing" to formulate a method for model-free estimation from data of the Lyapunov exponents of a chaotic process. The technique uses a limited time series of measurements as input to a high-dimensional dynamical system called a "reservoir." After the reservoir's response to the data is recorded, linear regression is used to learn a large set of parameters, called the "output weights." The learned output weights are then used to form a modified autonomous reservoir designed to be capable of producing an arbitrarily long time series whose ergodic properties approximate those of the input signal. When successful, we say that the autonomous reservoir reproduces the attractor's "climate." Since the reservoir equations and output weights are known, we can compute the derivatives needed to determine the Lyapunov exponents of the autonomous reservoir, which we then use as estimates of the Lyapunov exponents for the original input generating system. We illustrate the effectiveness of our technique with two examples, the Lorenz system and the Kuramoto-Sivashinsky (KS) equation. In the case of the KS equation, we note that the high dimensional nature of the system and the large number of Lyapunov exponents yield a challenging test of our method, which we find the method successfully passes.

Defining chaos.

In this paper, we propose, discuss, and illustrate a computationally feasible definition of chaos which can be applied very generally to situations that are commonly encountered, including attractors, repellers, and non-periodically forced systems. This definition is based on an entropy-like quantity, which we call "expansion entropy," and we define chaos as occurring when this quantity is positive. We relate and compare expansion entropy to the well-known concept of topological entropy to which it is equivalent under appropriate conditions. We also present example illustrations, discuss computational implementations, and point out issues arising from attempts at giving definitions of chaos that are not entropy-based.

Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing.

Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics ("climate") can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth ("instability"). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.

Comment on "Long time evolution of phase oscillator systems" [Chaos 19, 023117 (2009)].

In a recent paper by Ott and Antonsen [Chaos 19, 023117 (2009)], it was shown for the case of Lorentzian distributions of oscillator frequencies that the dynamics of a very general class of large systems of coupled phase oscillators time-asymptotes to a particular simplified form given by Ott and Antonsen [Chaos 18, 037113 (2008)]. This comment extends this previous result to a broad class of oscillator distribution functions.

Duplication count distributions in DNA sequences.

We study quantitative features of complex repetitive DNA in several genomes by studying sequences that are sufficiently long that they are unlikely to have repeated by chance. For each genome we study, we determine the number of identical copies, the "duplication count," of each sequence of length 40, that is of each "40-mer." We say a 40-mer is "repeated" if its duplication count is at least 2. We focus mainly on "complex" 40-mers, those without short internal repetitions. We find that we can classify most of the complex repeated 40-mers into two categories: one category has its copies clustered closely together on one chromosome, the other has its copies distributed widely across multiple chromosomes. For each genome and each of the categories above, we compute N(c), the number of 40-mers that have duplication count c, for each integer c. In each case, we observe a power-law-like decay in N(c) as c increases from 3 to 50 or higher. In particular, we find that N(c) decays much more slowly than would be predicted by evolutionary models where each 40-mer is equally likely to be duplicated. We also analyze an evolutionary model that does reflect the slow decay of N(c).

Approximating the largest eigenvalue of network adjacency matrices.

The largest eigenvalue of the adjacency matrix of a network plays an important role in several network processes (e.g., synchronization of oscillators, percolation on directed networks, and linear stability of equilibria of network coupled systems). In this paper we develop approximations to the largest eigenvalue of adjacency matrices and discuss the relationships between these approximations. Numerical experiments on simulated networks are used to test our results.

Onset of synchronization in large networks of coupled oscillators.

We study the transition from incoherence to coherence in large networks of coupled phase oscillators. We present various approximations that describe the behavior of an appropriately defined order parameter past the transition and generalize recent results for the critical coupling strength. We find that, under appropriate conditions, the coupling strength at which the transition occurs is determined by the largest eigenvalue of the adjacency matrix. We show how, with an additional assumption, a mean-field approximation recently proposed is recovered from our results. We test our theory with numerical simulations and find that it describes the transition when our assumptions are satisfied. We find that our theory describes the transition well in situations in which the mean-field approximation fails. We study the finite-size effects caused by nodes with small degree and find that they cause the critical coupling strength to increase.

Formation of multifractal population patterns from reproductive growth and local resettlement.

We consider the general character of the spatial distribution of a population that grows through reproduction and subsequent local resettlement of new population members. We present several simple one- and two-dimensional point placement models to illustrate possible generic behavior of these distributions. We show, numerically and analytically, that these models all lead to multifractal spatial distributions of population. Additionally, we make qualitative links between our models and the example of the Earth at Night image, showing the Earth's nighttime man-made light as seen from space. The Earth at Night data suffer from saturation of the sensing photodetectors at high brightness ("clipping"), and we account for how this influences the determined dimension spectrum of the light intensity distribution.

A preprocessor for shotgun assembly of large genomes.

The whole-genome shotgun (WGS) assembly technique has been remarkably successful in efforts to determine the sequence of bases that make up a genome. WGS assembly begins with a large collection of short fragments that have been selected at random from a genome. The sequence of bases at each end of the fragment is determined, albeit imprecisely, resulting in a sequence of letters called a "read." Each letter in a read is assigned a quality value, which estimates the probability that a sequencing error occurred in determining that letter. Reads are typically cut off after about 500 letters, where sequencing errors become endemic. We report on a set of procedures that (1) corrects most of the sequencing errors, (2) changes quality values accordingly, and (3) produces a list of "overlaps," i.e., pairs of reads that plausibly come from overlapping parts of the genome. Our procedures, which we call collectively the "UMD Overlapper," can be run iteratively and as a preprocessor for other assemblers. We tested the UMD Overlapper on Celera's Drosophila reads. When we replaced Celera's overlap procedures in the front end of their assembler, it was able to produce a significantly improved genome.

Bifurcation scenarios for bubbling transition.

Dynamical systems with chaos on an invariant submanifold can exhibit a type of behavior called bubbling, whereby a small random or fixed perturbation to the system induces intermittent bursting. The bifurcation to bubbling occurs when a periodic orbit embedded in the chaotic attractor in the invariant manifold becomes unstable to perturbations transverse to the invariant manifold. Generically the periodic orbit can become transversely unstable through a pitchfork, transcritical, period-doubling, or Hopf bifurcation. In this paper a unified treatment of the four types of bubbling bifurcation is presented. Conditions are obtained determining whether the transition to bubbling is soft or hard; that is, whether the maximum burst amplitude varies continuously or discontinuously with variation of the parameter through its critical value. For soft bubbling transitions, the scaling of the maximum burst amplitude with the parameter is derived. For both hard and soft transitions the scaling of the average interburst time with the bifurcation parameter is deduced. Both random (noise) and fixed (mismatch) perturbations are considered. Results of numerical experiments testing our theoretical predictions are presented.

Anomalous diffusion in infinite horizon billiards.

We consider the long time dependence for the moments of displacement <|r|(q)> of infinite horizon billiards, given a bounded initial distribution of particles. For a variety of billiard models we find <|r|(q)> approximately t(gamma(q)) (up to factors of ln t). The time exponent, gamma(q), is piecewise linear and equal to q/2 for q<2 and q-1 for q>2. We discuss the lack of dependence of this result on the initial distribution of particles and resolve apparent discrepancies between this time dependence and a prior result. The lack of dependence on initial distribution follows from a remarkable scaling result that we obtain for the time evolution of the distribution function of the angle of a particle's velocity vector.

Growing networks with geographical attachment preference: emergence of small worlds.

We introduce a simple mechanism for the evolution of small world networks. Our model is a growing network in which all connections are made locally to geographically nearby sites. Although connections are made purely locally, network growth leads to stretching of old connections and to high clustering. Our results suggest that the abundance of small world networks in geographically constrained systems is a natural consequence of system growth and local interactions.

Spatial patterns of desynchronization bursts in networks.

We adapt a previous model and analysis method (the master stability function), extensively used for studying the stability of the synchronous state of networks of identical chaotic oscillators, to the case of oscillators that are similar but not exactly identical. We find that bubbling induced desynchronization bursts occur for some parameter values. These bursts have spatial patterns, which can be predicted from the network connectivity matrix and the unstable periodic orbits embedded in the attractor. We test the analysis of bursts by comparison with numerical experiments. In the case that no bursting occurs, we discuss the deviations from the exactly synchronous state caused by the mismatch between oscillators.

Weighted percolation on directed networks.

We present and numerically test an analysis of the percolation transition for general node removal strategies valid for locally treelike directed networks. On the basis of heuristic arguments we predict that, if the probability of removing node i is p(i), the network disintegrates if p(i) is such that the largest eigenvalue of the matrix with entries A(ij)(1-p(i)) is less than 1, where A is the adjacency matrix of the network. The knowledge or applicability of a Markov network model is not required by our theory, thus making it applicable to situations not covered by previous works.

Improving Phrap-based assembly of the rat using "reliable" overlaps.

The assembly methods used for whole-genome shotgun (WGS) data have a major impact on the quality of resulting draft genomes. We present a novel algorithm to generate a set of "reliable" overlaps based on identifying repeat k-mers. To demonstrate the benefits of using reliable overlaps, we have created a version of the Phrap assembly program that uses only overlaps from a specific list. We call this version PhrapUMD. Integrating PhrapUMD and our "reliable-overlap" algorithm with the Baylor College of Medicine assembler, Atlas, we assemble the BACs from the Rattus norvegicus genome project. Starting with the same data as the Nov. 2002 Atlas assembly, we compare our results and the Atlas assembly to the 4.3 Mb of rat sequence in the 21 BACs that have been finished. Our version of the draft assembly of the 21 BACs increases the coverage of finished sequence from 93.4% to 96.3%, while simultaneously reducing the base error rate from 4.5 to 1.1 errors per 10,000 bases. There are a number of ways of assessing the relative merits of assemblies when the finished sequence is available. If one views the overall quality of an assembly as proportional to the inverse of the product of the error rate and sequence missed, then the assembly presented here is seven times better. The UMD Overlapper with options for reliable overlaps is available from the authors at http://www.genome.umd.edu. We also provide the changes to the Phrap source code enabling it to use only the reliable overlaps.