Data-driven modelling of brain activity using neural networks, diffusion maps, and the Koopman operator.

We propose a machine-learning approach to construct reduced-order models (ROMs) to predict the long-term out-of-sample dynamics of brain activity (and in general, high-dimensional time series), focusing mainly on task-dependent high-dimensional fMRI time series. Our approach is a three stage one. First, we exploit manifold learning and, in particular, diffusion maps (DMs) to discover a set of variables that parametrize the latent space on which the emergent high-dimensional fMRI time series evolve. Then, we construct ROMs on the embedded manifold via two techniques: Feedforward Neural Networks (FNNs) and the Koopman operator. Finally, for predicting the out-of-sample long-term dynamics of brain activity in the ambient fMRI space, we solve the pre-image problem, i.e., the construction of a map from the low-dimensional manifold to the original high-dimensional (ambient) space by coupling DMs with Geometric Harmonics (GH) when using FNNs and the Koopman modes per se. For our illustrations, we have assessed the performance of the two proposed schemes using two benchmark fMRI time series: (i) a simplistic five-dimensional model of stochastic discrete-time equations used just for a "transparent" illustration of the approach, thus knowing a priori what one expects to get, and (ii) a real fMRI dataset with recordings during a visuomotor task. We show that the proposed Koopman operator approach provides, for any practical purposes, equivalent results to the FNN-GH approach, thus bypassing the need to train a non-linear map and to use GH to extrapolate predictions in the ambient space; one can use instead the low-frequency truncation of the DMs function space of L2-integrable functions to predict the entire list of coordinate functions in the ambient space and to solve the pre-image problem.

Intrinsic-Strain Engineering by Dislocation Imprint in Bulk Ferroelectrics.

We report an intrinsic strain engineering, akin to thin filmlike approaches, via irreversible high-temperature plastic deformation of a tetragonal ferroelectric single-crystal BaTiO_{3}. Dislocations well-aligned along the [001] axis and associated strain fields in plane defined by the [110]/[1[over ¯]10] plane are introduced into the volume, thus nucleating only in-plane domain variants. By combining direct experimental observations and theoretical analyses, we reveal that domain instability and extrinsic degradation processes can both be mitigated during the aging and fatigue processes, and demonstrate that this requires careful strain tuning of the ratio of in-plane and out-of-plane domain variants. Our findings advance the understanding of structural defects that drive domain nucleation and instabilities in ferroic materials and are essential for mitigating device degradation.

Learning effective stochastic differential equations from microscopic simulations: Linking stochastic numerics to deep learning.

We identify effective stochastic differential equations (SDEs) for coarse observables of fine-grained particle- or agent-based simulations; these SDEs then provide useful coarse surrogate models of the fine scale dynamics. We approximate the drift and diffusivity functions in these effective SDEs through neural networks, which can be thought of as effective stochastic ResNets. The loss function is inspired by, and embodies, the structure of established stochastic numerical integrators (here, Euler-Maruyama and Milstein); our approximations can thus benefit from backward error analysis of these underlying numerical schemes. They also lend themselves naturally to "physics-informed" gray-box identification when approximate coarse models, such as mean field equations, are available. Existing numerical integration schemes for Langevin-type equations and for stochastic partial differential equations can also be used for training; we demonstrate this on a stochastically forced oscillator and the stochastic wave equation. Our approach does not require long trajectories, works on scattered snapshot data, and is designed to naturally handle different time steps per snapshot. We consider both the case where the coarse collective observables are known in advance, as well as the case where they must be found in a data-driven manner.

Anisotropic dislocation-domain wall interactions in ferroelectrics.

Dislocations are usually expected to degrade electrical, thermal and optical functionality and to tune mechanical properties of materials. Here, we demonstrate a general framework for the control of dislocation-domain wall interactions in ferroics, employing an imprinted dislocation network. Anisotropic dielectric and electromechanical properties are engineered in barium titanate crystals via well-controlled line-plane relationships, culminating in extraordinary and stable large-signal dielectric permittivity (≈23100) and piezoelectric coefficient (≈2470 pm V-1). In contrast, a related increase in properties utilizing point-plane relation prompts a dramatic cyclic degradation. Observed dielectric and piezoelectric properties are rationalized using transmission electron microscopy and time- and cycle-dependent nuclear magnetic resonance paired with X-ray diffraction. Succinct mechanistic understanding is provided by phase-field simulations and driving force calculations of the described dislocation-domain wall interactions. Our 1D-2D defect approach offers a fertile ground for tailoring functionality in a wide range of functional material systems.

Learning emergent partial differential equations in a learned emergent space.

We propose an approach to learn effective evolution equations for large systems of interacting agents. This is demonstrated on two examples, a well-studied system of coupled normal form oscillators and a biologically motivated example of coupled Hodgkin-Huxley-like neurons. For such types of systems there is no obvious space coordinate in which to learn effective evolution laws in the form of partial differential equations. In our approach, we accomplish this by learning embedding coordinates from the time series data of the system using manifold learning as a first step. In these emergent coordinates, we then show how one can learn effective partial differential equations, using neural networks, that do not only reproduce the dynamics of the oscillator ensemble, but also capture the collective bifurcations when system parameters vary. The proposed approach thus integrates the automatic, data-driven extraction of emergent space coordinates parametrizing the agent dynamics, with machine-learning assisted identification of an emergent PDE description of the dynamics in this parametrization.

Learning the temporal evolution of multivariate densities via normalizing flows.

In this work, we propose a method to learn multivariate probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker-Planck equations). We analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker-Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a multivariate normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can approximate probability density function evolutions in time from observed sampled data for systems driven by both Brownian and Lévy noise. We present examples with two- and three-dimensional, uni- and multimodal distributions to validate the method.

On the parameter combinations that matter and on those that do not: data-driven studies of parameter (non)identifiability.

We present a data-driven approach to characterizing nonidentifiability of a model's parameters and illustrate it through dynamic as well as steady kinetic models. By employing Diffusion Maps and their extensions, we discover the minimal combinations of parameters required to characterize the output behavior of a chemical system: a set of effective parameters for the model. Furthermore, we introduce and use a Conformal Autoencoder Neural Network technique, as well as a kernel-based Jointly Smooth Function technique, to disentangle the redundant parameter combinations that do not affect the output behavior from the ones that do. We discuss the interpretability of our data-driven effective parameters, and demonstrate the utility of the approach both for behavior prediction and parameter estimation. In the latter task, it becomes important to describe level sets in parameter space that are consistent with a particular output behavior. We validate our approach on a model of multisite phosphorylation, where a reduced set of effective parameters (nonlinear combinations of the physical ones) has previously been established analytically.

Local conformal autoencoder for standardized data coordinates.

We propose a local conformal autoencoder (LOCA) for standardized data coordinates. LOCA is a deep learning-based method for obtaining standardized data coordinates from scientific measurements. Data observations are modeled as samples from an unknown, nonlinear deformation of an underlying Riemannian manifold, which is parametrized by a few normalized, latent variables. We assume a repeated measurement sampling strategy, common in scientific measurements, and present a method for learning an embedding in [Formula: see text] that is isometric to the latent variables of the manifold. The coordinates recovered by our method are invariant to diffeomorphisms of the manifold, making it possible to match between different instrumental observations of the same phenomenon. Our embedding is obtained using LOCA, which is an algorithm that learns to rectify deformations by using a local z-scoring procedure, while preserving relevant geometric information. We demonstrate the isometric embedding properties of LOCA in various model settings and observe that it exhibits promising interpolation and extrapolation capabilities, superior to the current state of the art. Finally, we demonstrate LOCA's efficacy in single-site Wi-Fi localization data and for the reconstruction of three-dimensional curved surfaces from two-dimensional projections.

Manifold learning for organizing unstructured sets of process observations.

Data mining is routinely used to organize ensembles of short temporal observations so as to reconstruct useful, low-dimensional realizations of an underlying dynamical system. In this paper, we use manifold learning to organize unstructured ensembles of observations ("trials") of a system's response surface. We have no control over where every trial starts, and during each trial, operating conditions are varied by turning "agnostic" knobs, which change system parameters in a systematic, but unknown way. As one (or more) knobs "turn," we record (possibly partial) observations of the system response. We demonstrate how such partial and disorganized observation ensembles can be integrated into coherent response surfaces whose dimension and parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through different types of observations. We demonstrate our approach by organizing unstructured observations of response surfaces, including the reconstruction of a cusp bifurcation surface for hydrogen combustion in a continuous stirred tank reactor. Finally, we demonstrate how this observation-based reconstruction naturally leads to informative transport maps between the input parameter space and output/state variable spaces.

Linking Gaussian process regression with data-driven manifold embeddings for nonlinear data fusion.

In statistical modelling with Gaussian process regression, it has been shown that combining (few) high-fidelity data with (many) low-fidelity data can enhance prediction accuracy, compared to prediction based on the few high-fidelity data only. Such information fusion techniques for multi-fidelity data commonly approach the high-fidelity model f h(t) as a function of two variables (t, s), and then use f l(t) as the s data. More generally, the high-fidelity model can be written as a function of several variables (t, s 1, s 2….); the low-fidelity model f l and, say, some of its derivatives can then be substituted for these variables. In this paper, we will explore mathematical algorithms for multi-fidelity information fusion that use such an approach towards improving the representation of the high-fidelity function with only a few training data points. Given that f h may not be a simple function-and sometimes not even a function-of f l, we demonstrate that using additional functions of t, such as derivatives or shifts of f l, can drastically improve the approximation of f h through Gaussian processes. We also point out a connection with 'embedology' techniques from topology and dynamical systems. Our illustrative examples range from instructive caricatures to computational biology models, such as Hodgkin-Huxley neural oscillations.

On learning Hamiltonian systems from data.

Concise, accurate descriptions of physical systems through their conserved quantities abound in the natural sciences. In data science, however, current research often focuses on regression problems, without routinely incorporating additional assumptions about the system that generated the data. Here, we propose to explore a particular type of underlying structure in the data: Hamiltonian systems, where an "energy" is conserved. Given a collection of observations of such a Hamiltonian system over time, we extract phase space coordinates and a Hamiltonian function of them that acts as the generator of the system dynamics. The approach employs an autoencoder neural network component to estimate the transformation from observations to the phase space of a Hamiltonian system. An additional neural network component is used to approximate the Hamiltonian function on this constructed space, and the two components are trained jointly. As an alternative approach, we also demonstrate the use of Gaussian processes for the estimation of such a Hamiltonian. After two illustrative examples, we extract an underlying phase space as well as the generating Hamiltonian from a collection of movies of a pendulum. The approach is fully data-driven and does not assume a particular form of the Hamiltonian function.

An Emergent Space for Distributed Data with Hidden Internal Order through Manifold Learning.

Manifold-learning techniques are routinely used in mining complex spatiotemporal data to extract useful, parsimonious data representations/parametrizations; these are, in turn, useful in nonlinear model identification tasks. We focus here on the case of time series data that can ultimately be modelled as a spatially distributed system (e.g. a partial differential equation, PDE), but where we do not know the space in which this PDE should be formulated. Hence, even the spatial coordinates for the distributed system themselves need to be identified - to "emerge from"-the data mining process. We will first validate this "emergent space" reconstruction for time series sampled without space labels in known PDEs; this brings up the issue of observability of physical space from temporal observation data, and the transition from spatially resolved to lumped (order-parameter-based) representations by tuning the scale of the data mining kernels. We will then present actual emergent space "discovery" illustrations. Our illustrative examples include chimera states (states of coexisting coherent and incoherent dynamics), and chaotic as well as quasiperiodic spatiotemporal dynamics, arising in partial differential equations and/or in heterogeneous networks. We also discuss how data-driven "spatial" coordinates can be extracted in ways invariant to the nature of the measuring instrument. Such gauge-invariant data mining can go beyond the fusion of heterogeneous observations of the same system, to the possible matching of apparently different systems. For an older version of this article, including other examples, see https://arxiv.org/abs/1708.05406.

Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator.

Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD)51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems.

Gradient navigation model for pedestrian dynamics.

We present a microscopic ordinary differential equation (ODE)-based model for pedestrian dynamics: the gradient navigation model. The model uses a superposition of gradients of distance functions to directly change the direction of the velocity vector. The velocity is then integrated to obtain the location. The approach differs fundamentally from force-based models needing only three equations to derive the ODE system, as opposed to four in, e.g., the social force model. Also, as a result, pedestrians are no longer subject to inertia. Several other advantages ensue: Model-induced oscillations are avoided completely since no actual forces are present. The derivatives in the equations of motion are smooth and therefore allow the use of fast and accurate high-order numerical integrators. At the same time, the existence and uniqueness of the solution to the ODE system follow almost directly from the smoothness properties. In addition, we introduce a method to calibrate parameters by theoretical arguments based on empirically validated assumptions rather than by numerical tests. These parameters, combined with the accurate integration, yield simulation results with no collisions of pedestrians. Several empirically observed system phenomena emerge without the need to recalibrate the parameter set for each scenario: obstacle avoidance, lane formation, stop-and-go waves, and congestion at bottlenecks. The density evolution in the latter is shown to be quantitatively close to controlled experiments. Likewise, we observe a dependence of the crowd velocity on the local density that compares well with benchmark fundamental diagrams.

Inter-Golgi transport mediated by COPI-containing vesicles carrying small cargoes.

A core prediction of the vesicular transport model is that COPI vesicles are responsible for trafficking anterograde cargoes forward. In this study, we test this prediction by examining the properties and requirements of inter-Golgi transport within fused cells, which requires mobile carriers in order for exchange of constituents to occur. We report that both small soluble and membrane-bound secretory cargo and exogenous Golgi resident glycosyl-transferases are exchanged between separated Golgi. Large soluble aggregates, which traverse individual stacks, do not transfer between Golgi, implying that small cargoes (which can fit in a typical transport vesicle) are transported by a different mechanism. Super-resolution microscopy reveals that the carriers of both anterograde and retrograde cargoes are the size of COPI vesicles, contain coatomer, and functionally require ARF1 and coatomer for transport. The data suggest that COPI vesicles traffic both small secretory cargo and steady-state Golgi resident enzymes among stacked cisternae that are stationary. DOI:http://dx.doi.org/10.7554/eLife.01296.001.