Learning transfer operators by kernel density estimation.

Inference of transfer operators from data is often formulated as a classical problem that hinges on the Ulam method. The conventional description, known as the Ulam-Galerkin method, involves projecting onto basis functions represented as characteristic functions supported over a fine grid of rectangles. From this perspective, the Ulam-Galerkin approach can be interpreted as density estimation using the histogram method. In this study, we recast the problem within the framework of statistical density estimation. This alternative perspective allows for an explicit and rigorous analysis of bias and variance, thereby facilitating a discussion on the mean square error. Through comprehensive examples utilizing the logistic map and a Markov map, we demonstrate the validity and effectiveness of this approach in estimating the eigenvectors of the Frobenius-Perron operator. We compare the performance of histogram density estimation (HDE) and kernel density estimation (KDE) methods and find that KDE generally outperforms HDE in terms of accuracy. However, it is important to note that KDE exhibits limitations around boundary points and jumps. Based on our research findings, we suggest the possibility of incorporating other density estimation methods into this field and propose future investigations into the application of KDE-based estimation for high-dimensional maps. These findings provide valuable insights for researchers and practitioners working on estimating the Frobenius-Perron operator and highlight the potential of density estimation techniques in this area of study.

Stability analysis of intralayer synchronization in time-varying multilayer networks with generic coupling functions.

The stability analysis of synchronization patterns on generalized network structures is of immense importance nowadays. In this article, we scrutinize the stability of intralayer synchronous state in temporal multilayer hypernetworks, where each dynamic units in a layer communicate with others through various independent time-varying connection mechanisms. Here, dynamical units within and between layers may be interconnected through arbitrary generic coupling functions. We show that intralayer synchronous state exists as an invariant solution. Using fast-switching stability criteria, we derive the condition for stable coherent state in terms of associated time-averaged network structure, and in some instances we are able to separate the transverse subspace optimally. Using simultaneous block diagonalization of coupling matrices, we derive the synchronization stability condition without considering time-averaged network structure. Finally, we verify our analytically derived results through a series of numerical simulations on synthetic and real-world neuronal networked systems.

Spanning trees of recursive scale-free graphs.

We present a link-by-link rule-based method for constructing all members of the ensemble of spanning trees for any recursively generated, finitely articulated graph, such as the Dorogovtsev-Goltsev-Mendes (DGM) net. The recursions allow for many large-scale properties of the ensemble of spanning trees to be analytically solved exactly. We show how a judicious application of the prescribed growth rules selects for certain subsets of the spanning trees with particular desired properties (small world, extended diameter, degree distribution, etc.), and thus approximates and/or provides solutions to several optimization problems on undirected and unweighted networks. The analysis of spanning trees enhances the usefulness of recursive graphs as sophisticated models for everyday life complex networks.

Beach-level 24-hour forecasts of Florida red tide-induced respiratory irritation.

An accurate forecast of the red tide respiratory irritation level would improve the lives of many people living in areas affected by algal blooms. Using a decades-long database of daily beach conditions, two conceptually different models to forecast the respiratory irritation risk level one day ahead of time are trained. One model is wind-based, using the current days' respiratory level and the predicted wind direction of the following day. The other model is a probabilistic self-exciting Hawkes process model. Both models are trained on beaches in Florida during 2011--2017 and applied to the red tide bloom during 2018-2019. For beaches where there is enough historical data to develop a model, the model which performs best depends on the beach. The wind-based model is the most accurate at half the beaches, correctly predicting the respiratory risk level on average about 84% of the time. The Hawkes model is the most accurate (81% accuracy) at nearly all of the remaining beaches.

The essential synchronization backbone problem.

Network optimization strategies for the process of synchronization have generally focused on the re-wiring or re-weighting of links in order to (1) expand the range of coupling strengths that achieve synchronization, (2) expand the basin of attraction for the synchronization manifold, or (3) lower the average time to synchronization. A new optimization goal is proposed in seeking the minimum subset of the edge set of the original network that enables the same essential ability to synchronize in that the synchronization manifolds have conjugate stability. We call this type of minimal spanning subgraph an essential synchronization backbone of the original system, and we present two algorithms: one is a strategy for an exhaustive search for a true solution, while the other is a method of approximation for this combinatorial problem. The solution spaces that result from different choices of dynamical systems and coupling schemes vary with the level of a hierarchical structure present and also the number of interwoven central cycles. Applications can include the important problem in civil engineering of power grid hardening, where new link creation may be costly, and the defense of certain key links to the functional process may be prioritized.

On Geometry of Information Flow for Causal Inference.

Causal inference is perhaps one of the most fundamental concepts in science, beginning originally from the works of some of the ancient philosophers, through today, but also weaved strongly in current work from statisticians, machine learning experts, and scientists from many other fields. This paper takes the perspective of information flow, which includes the Nobel prize winning work on Granger-causality, and the recently highly popular transfer entropy, these being probabilistic in nature. Our main contribution will be to develop analysis tools that will allow a geometric interpretation of information flow as a causal inference indicated by positive transfer entropy. We will describe the effective dimensionality of an underlying manifold as projected into the outcome space that summarizes information flow. Therefore, contrasting the probabilistic and geometric perspectives, we will introduce a new measure of causal inference based on the fractal correlation dimension conditionally applied to competing explanations of future forecasts, which we will write G e o C y → x . This avoids some of the boundedness issues that we show exist for the transfer entropy, T y → x . We will highlight our discussions with data developed from synthetic models of successively more complex nature: these include the Hénon map example, and finally a real physiological example relating breathing and heart rate function.

Stochastic and mixed flower graphs.

Stochasticity is introduced to a well studied class of recursively grown graphs: (u,v)-flower nets, which have power-law degree distributions as well as small-world properties (when u=1). The stochastic variant interpolates between different (deterministic) flower graphs thus adding flexibility to the model. The random multiplicative growth process involved, however, leads to a spread ensemble of networks with finite variance for the number of links, nodes, and loops. Nevertheless, the degree exponent and loopiness exponent attain unique values in the thermodynamic limit of infinitely large graphs. We also study a class of mixed flower networks, closely related to the stochastic flowers, but which are grown recursively in a deterministic way. The deterministic growth of mixed flower-nets eliminates ensemble spreads, and their recursive growth allows for exact analysis of their (uniquely defined) mixed properties.

Manifold learning for organizing unstructured sets of process observations.

Data mining is routinely used to organize ensembles of short temporal observations so as to reconstruct useful, low-dimensional realizations of an underlying dynamical system. In this paper, we use manifold learning to organize unstructured ensembles of observations ("trials") of a system's response surface. We have no control over where every trial starts, and during each trial, operating conditions are varied by turning "agnostic" knobs, which change system parameters in a systematic, but unknown way. As one (or more) knobs "turn," we record (possibly partial) observations of the system response. We demonstrate how such partial and disorganized observation ensembles can be integrated into coherent response surfaces whose dimension and parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through different types of observations. We demonstrate our approach by organizing unstructured observations of response surfaces, including the reconstruction of a cusp bifurcation surface for hydrogen combustion in a continuous stirred tank reactor. Finally, we demonstrate how this observation-based reconstruction naturally leads to informative transport maps between the input parameter space and output/state variable spaces.

Open or closed? Information flow decided by transfer operators and forecastability quality metric.

A basic systems question concerns the concept of closure, meaning autonomy (closed) in the sense of describing the (sub)system as fully consistent within itself. Alternatively, the system may be nonautonomous (open), meaning it receives influence from an outside subsystem. We assert here that the concept of information flow and the related concept of causation inference are summarized by this simple question of closure as we define herein. We take the forecasting perspective of Weiner-Granger causality that describes a causal relationship exists if a subsystem's forecast quality depends on considering states of another subsystem. Here, we develop a new direct analytic discussion, rather than a data oriented approach. That is, we refer to the underlying Frobenius-Perron (FP) transfer operator that moderates evolution of densities of ensembles of orbits, and two alternative forms of the restricted Frobenius-Perron operator, interpreted as if either closed (deterministic FP) or not closed (the unaccounted outside influence seems stochastic and we show correspondingly requires the stochastic FP operator). Thus follows contrasting the kernels of the variants of the operators, as if densities in their own rights. However, the corresponding differential entropy comparison by Kullback-Leibler divergence, as one would typically use when developing transfer entropy, becomes ill-defined. Instead, we build our Forecastability Quality Metric (FQM) upon the "symmetrized" variant known as Jensen-Shannon divergence, and we are also able to point out several useful resulting properties. We illustrate the FQM by a simple coupled chaotic system. Our analysis represents a new theoretical direction, but we do describe data oriented directions for the future.

Anatomy of leadership in collective behaviour.

Understanding the mechanics behind the coordinated movement of mobile animal groups (collective motion) provides key insights into their biology and ecology, while also yielding algorithms for bio-inspired technologies and autonomous systems. It is becoming increasingly clear that many mobile animal groups are composed of heterogeneous individuals with differential levels and types of influence over group behaviors. The ability to infer this differential influence, or leadership, is critical to understanding group functioning in these collective animal systems. Due to the broad interpretation of leadership, many different measures and mathematical tools are used to describe and infer "leadership," e.g., position, causality, influence, and information flow. But a key question remains: which, if any, of these concepts actually describes leadership? We argue that instead of asserting a single definition or notion of leadership, the complex interaction rules and dynamics typical of a group imply that leadership itself is not merely a binary classification (leader or follower), but rather, a complex combination of many different components. In this paper, we develop an anatomy of leadership, identify several principal components, and provide a general mathematical framework for discussing leadership. With the intricacies of this taxonomy in mind, we present a set of leadership-oriented toy models that should be used as a proving ground for leadership inference methods going forward. We believe this multifaceted approach to leadership will enable a broader understanding of leadership and its inference from data in mobile animal groups and beyond.

Introduction to Focus Issue: Causation inference and information flow in dynamical systems: Theory and applications.

Questions of causation are foundational across science and often relate further to problems of control, policy decisions, and forecasts. In nonlinear dynamics and complex systems science, causation inference and information flow are closely related concepts, whereby "information" or knowledge of certain states can be thought of as coupling influence onto the future states of other processes in a complex system. While causation inference and information flow are by now classical topics, incorporating methods from statistics and time series analysis, information theory, dynamical systems, and statistical mechanics, to name a few, there remain important advancements in continuing to strengthen the theory, and pushing the context of applications, especially with the ever-increasing abundance of data collected across many fields and systems. This Focus Issue considers different aspects of these questions, both in terms of founding theory and several topical applications.

Geometric k-nearest neighbor estimation of entropy and mutual information.

Nonparametric estimation of mutual information is used in a wide range of scientific problems to quantify dependence between variables. The k-nearest neighbor (knn) methods are consistent, and therefore expected to work well for a large sample size. These methods use geometrically regular local volume elements. This practice allows maximum localization of the volume elements, but can also induce a bias due to a poor description of the local geometry of the underlying probability measure. We introduce a new class of knn estimators that we call geometric knn estimators (g-knn), which use more complex local volume elements to better model the local geometry of the probability measures. As an example of this class of estimators, we develop a g-knn estimator of entropy and mutual information based on elliptical volume elements, capturing the local stretching and compression common to a wide range of dynamical system attractors. A series of numerical examples in which the thickness of the underlying distribution and the sample sizes are varied suggest that local geometry is a source of problems for knn methods such as the Kraskov-Stögbauer-Grassberger estimator when local geometric effects cannot be removed by global preprocessing of the data. The g-knn method performs well despite the manipulation of the local geometry. In addition, the examples suggest that the g-knn estimators can be of particular relevance to applications in which the system is large, but the data size is limited.

An Emergent Space for Distributed Data with Hidden Internal Order through Manifold Learning.

Manifold-learning techniques are routinely used in mining complex spatiotemporal data to extract useful, parsimonious data representations/parametrizations; these are, in turn, useful in nonlinear model identification tasks. We focus here on the case of time series data that can ultimately be modelled as a spatially distributed system (e.g. a partial differential equation, PDE), but where we do not know the space in which this PDE should be formulated. Hence, even the spatial coordinates for the distributed system themselves need to be identified - to "emerge from"-the data mining process. We will first validate this "emergent space" reconstruction for time series sampled without space labels in known PDEs; this brings up the issue of observability of physical space from temporal observation data, and the transition from spatially resolved to lumped (order-parameter-based) representations by tuning the scale of the data mining kernels. We will then present actual emergent space "discovery" illustrations. Our illustrative examples include chimera states (states of coexisting coherent and incoherent dynamics), and chaotic as well as quasiperiodic spatiotemporal dynamics, arising in partial differential equations and/or in heterogeneous networks. We also discuss how data-driven "spatial" coordinates can be extracted in ways invariant to the nature of the measuring instrument. Such gauge-invariant data mining can go beyond the fusion of heterogeneous observations of the same system, to the possible matching of apparently different systems. For an older version of this article, including other examples, see https://arxiv.org/abs/1708.05406.

Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator.

Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD)51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems.

Modeling the lowest-cost splitting of a herd of cows by optimizing a cost function.

Animals live in groups to defend against predation and to obtain food. However, for some animals-especially ones that spend long periods of time feeding-there are costs if a group chooses to move on before their nutritional needs are satisfied. If the conflict between feeding and keeping up with a group becomes too large, it may be advantageous for some groups of animals to split into subgroups with similar nutritional needs. We model the costs and benefits of splitting in a herd of cows using a cost function that quantifies individual variation in hunger, desire to lie down, and predation risk. We model the costs associated with hunger and lying desire as the standard deviations of individuals within a group, and we model predation risk as an inverse exponential function of the group size. We minimize the cost function over all plausible groups that can arise from a given herd and study the dynamics of group splitting. We examine how the cow dynamics and cost function depend on the parameters in the model and consider two biologically-motivated examples: (1) group switching and group fission in a herd of relatively homogeneous cows, and (2) a herd with an equal number of adult males (larger animals) and adult females (smaller animals).

An observer for an occluded reaction-diffusion system with spatially varying parameters.

Spatially dependent parameters of a two-component chaotic reaction-diffusion partial differential equation (PDE) model describing ocean ecology are observed by sampling a single species. We estimate the model parameters and the other species in the system by autosynchronization, where quantities of interest are evolved according to misfit between model and observations, to only partially observed data. Our motivating example comes from oceanic ecology as viewed by remote sensing data, but where noisy occluded data are realized in the form of cloud cover. We demonstrate a method to learn a large-scale coupled synchronizing system that represents the spatio-temporal dynamics and apply a network approach to analyze manifold stability.

Detecting phase transitions in collective behavior using manifold's curvature.

If a given behavior of a multi-agent system restricts the phase variable to an invariant manifold, then we define a phase transition as a change of physical characteristics such as speed, coordination, and structure. We define such a phase transition as splitting an underlying manifold into two sub-manifolds with distinct dimensionalities around the singularity where the phase transition physically exists. Here, we propose a method of detecting phase transitions and splitting the manifold into phase transitions free sub-manifolds. Therein, we firstly utilize a relationship between curvature and singular value ratio of points sampled in a curve, and then extend the assertion into higher-dimensions using the shape operator. Secondly, we attest that the same phase transition can also be approximated by singular value ratios computed locally over the data in a neighborhood on the manifold. We validate the Phase Transition Detection (PTD) method using one particle simulation and three real world examples.

Stretching and folding in finite time.

Complex flows mix efficiently, and this process can be understood by considering the stretching and folding of material volumes. Although many metrics have been devised to characterize stretching, fewer are able to capture folding in a quantitative way in spatiotemporally variable flows. Here, we extend our previous methods based on the finite-time curving of fluid-element trajectories to nonzero scales and show that this finite-scale finite-time curvature contains information about both stretching and folding. We compare this metric to the more commonly used finite-time Lyapunov exponent and illustrate our methods using experimental flow-field data from a quasi-two-dimensional laboratory flow. Our new analysis tools add to the growing set of Lagrangian methods for characterizing mixing in complex, aperiodic fluid flows.

Causation entropy from symbolic representations of dynamical systems.

Identification of causal structures and quantification of direct information flows in complex systems is a challenging yet important task, with practical applications in many fields. Data generated by dynamical processes or large-scale systems are often symbolized, either because of the finite resolution of the measurement apparatus, or because of the need of statistical estimation. By algorithmic application of causation entropy, we investigated the effects of symbolization on important concepts such as Markov order and causal structure of the tent map. We uncovered that these quantities depend nonmonotonically and, most of all, sensitively on the choice of symbolization. Indeed, we show that Markov order and causal structure do not necessarily converge to their original analog counterparts as the resolution of the partitioning becomes finer.

Classification of collective behavior: a comparison of tracking and machine learning methods to study the effect of ambient light on fish shoaling.

Traditional approaches for the analysis of collective behavior entail digitizing the position of each individual, followed by evaluation of pertinent group observables, such as cohesion and polarization. Machine learning may enable considerable advancements in this area by affording the classification of these observables directly from images. While such methods have been successfully implemented in the classification of individual behavior, their potential in the study collective behavior is largely untested. In this paper, we compare three methods for the analysis of collective behavior: simple tracking (ST) without resolving occlusions, machine learning with real data (MLR), and machine learning with synthetic data (MLS). These methods are evaluated on videos recorded from an experiment studying the effect of ambient light on the shoaling tendency of Giant danios. In particular, we compute average nearest-neighbor distance (ANND) and polarization using the three methods and compare the values with manually-verified ground-truth data. To further assess possible dependence on sampling rate for computing ANND, the comparison is also performed at a low frame rate. Results show that while ST is the most accurate at higher frame rate for both ANND and polarization, at low frame rate for ANND there is no significant difference in accuracy between the three methods. In terms of computational speed, MLR and MLS take significantly less time to process an image, with MLS better addressing constraints related to generation of training data. Finally, all methods are able to successfully detect a significant difference in ANND as the ambient light intensity is varied irrespective of the direction of intensity change.