Transformations establishing equivalence across neural networks: When have two networks learned the same task?

Transformations are a key tool in the qualitative study of dynamical systems: transformations to a normal form, for example, underpin the study of instabilities and bifurcations. In this work, we test, and when possible establish, an equivalence between two different artificial neural networks by attempting to construct a data-driven transformation between them, using diffusion maps with a Mahalanobis-like metric. If the construction succeeds, the two networks can be thought of as belonging to the same equivalence class. We first discuss transformation functions between only the outputs of the two networks; we then also consider transformations that take into account outputs (activations) of a number of internal neurons from each network. Whitney's theorem dictates the number of (generic) measurements from one of the networks required to reconstruct each and every feature of the second network. The construction of the transformation function relies on a consistent, intrinsic representation of the network input space. We illustrate our algorithm by matching neural network pairs trained to learn (a) observations of scalar functions, (b) observations of two-dimensional vector fields, and (c) representations of images of a moving three-dimensional object (a rotating horse). We also demonstrate reconstruction of a network's input (and output) from minimal partial observations of intermediate neuron activations. The construction of equivalences across different network instantiations clearly relates to transfer learning and will also be valuable in establishing equivalence between different machine learning-based tools.

Tipping points of evolving epidemiological networks: Machine learning-assisted, data-driven effective modeling.

We study the tipping point collective dynamics of an adaptive susceptible-infected-susceptible (SIS) epidemiological network in a data-driven, machine learning-assisted manner. We identify a parameter-dependent effective stochastic differential equation (eSDE) in terms of physically meaningful coarse mean-field variables through a deep-learning ResNet architecture inspired by numerical stochastic integrators. We construct an approximate effective bifurcation diagram based on the identified drift term of the eSDE and contrast it with the mean-field SIS model bifurcation diagram. We observe a subcritical Hopf bifurcation in the evolving network's effective SIS dynamics that causes the tipping point behavior; this takes the form of large amplitude collective oscillations that spontaneously-yet rarely-arise from the neighborhood of a (noisy) stationary state. We study the statistics of these rare events both through repeated brute force simulations and by using established mathematical/computational tools exploiting the right-hand side of the identified SDE. We demonstrate that such a collective SDE can also be identified (and the rare event computations also performed) in terms of data-driven coarse observables, obtained here via manifold learning techniques, in particular, Diffusion Maps. The workflow of our study is straightforwardly applicable to other complex dynamic problems exhibiting tipping point dynamics.

Task-oriented machine learning surrogates for tipping points of agent-based models.

We present a machine learning framework bridging manifold learning, neural networks, Gaussian processes, and Equation-Free multiscale approach, for the construction of different types of effective reduced order models from detailed agent-based simulators and the systematic multiscale numerical analysis of their emergent dynamics. The specific tasks of interest here include the detection of tipping points, and the uncertainty quantification of rare events near them. Our illustrative examples are an event-driven, stochastic financial market model describing the mimetic behavior of traders, and a compartmental stochastic epidemic model on an Erdös-Rényi network. We contrast the pros and cons of the different types of surrogate models and the effort involved in learning them. Importantly, the proposed framework reveals that, around the tipping points, the emergent dynamics of both benchmark examples can be effectively described by a one-dimensional stochastic differential equation, thus revealing the intrinsic dimensionality of the normal form of the specific type of the tipping point. This allows a significant reduction in the computational cost of the tasks of interest.

Learning black- and gray-box chemotactic PDEs/closures from agent based Monte Carlo simulation data.

We propose a machine learning framework for the data-driven discovery of macroscopic chemotactic Partial Differential Equations (PDEs)-and the closures that lead to them- from high-fidelity, individual-based stochastic simulations of Escherichia coli bacterial motility. The fine scale, chemomechanical, hybrid (continuum-Monte Carlo) simulation model embodies the underlying biophysics, and its parameters are informed from experimental observations of individual cells. Using a parsimonious set of collective observables, we learn effective, coarse-grained "Keller-Segel class" chemotactic PDEs using machine learning regressors: (a) (shallow) feedforward neural networks and (b) Gaussian Processes. The learned laws can be black-box (when no prior knowledge about the PDE law structure is assumed) or gray-box when parts of the equation (e.g. the pure diffusion part) is known and "hardwired" in the regression process. More importantly, we discuss data-driven corrections (both additive and functional), to analytically known, approximate closures.

Gentlest Ascent Dynamics on Manifolds Defined by Adaptively Sampled Point-Clouds.

Finding saddle points of dynamical systems is an important problem in practical applications, such as the study of rare events of molecular systems. Gentlest ascent dynamics (GAD) (10.1088/0951-7715/24/6/008) is one of a number of algorithms in existence that attempt to find saddle points. It works by deriving a new dynamical system in which saddle points of the original system become stable equilibria. GAD has been recently generalized to the study of dynamical systems on manifolds (differential algebraic equations) described by equality constraints (10.1007/s10915-022-01838-3) and given in an extrinsic formulation. In this paper, we present an extension of GAD to manifolds defined by point-clouds, formulated by using an intrinsic viewpoint. These point-clouds are adaptively sampled during an iterative process that drives the system from the initial conformation (typically in the neighborhood of a stable equilibrium) to a saddle point. Our method requires the reactant (initial conformation), does not require the explicit constraint equations to be specified, and is purely data-driven.

Machine learning-assisted crystal engineering of a zeolite.

It is shown that Machine Learning (ML) algorithms can usefully capture the effect of crystallization composition and conditions (inputs) on key microstructural characteristics (outputs) of faujasite type zeolites (structure types FAU, EMT, and their intergrowths), which are widely used zeolite catalysts and adsorbents. The utility of ML (in particular, Geometric Harmonics) toward learning input-output relationships of interest is demonstrated, and a comparison with Neural Networks and Gaussian Process Regression, as alternative approaches, is provided. Through ML, synthesis conditions were identified to enhance the Si/Al ratio of high purity FAU zeolite to the hitherto highest level (i.e., Si/Al = 3.5) achieved via direct (not seeded), and organic structure-directing-agent-free synthesis from sodium aluminosilicate sols. The analysis of the ML algorithms' results offers the insight that reduced Na2O content is key to formulating FAU materials with high Si/Al ratio. An acid catalyst prepared by partial ion exchange of the high-Si/Al-ratio FAU (Si/Al = 3.5) exhibits improved proton reactivity (as well as specific activity, per unit mass of catalyst) in propane cracking and dehydrogenation compared to the catalyst prepared from the previously reported highest Si/Al ratio (Si/Al = 2.8).

Depolymerization of plastics by means of electrified spatiotemporal heating.

Depolymerization is a promising strategy for recycling waste plastic into constituent monomers for subsequent repolymerization1. However, many commodity plastics cannot be selectively depolymerized using conventional thermochemical approaches, as it is difficult to control the reaction progress and pathway. Although catalysts can improve the selectivity, they are susceptible to performance degradation2. Here we present a catalyst-free, far-from-equilibrium thermochemical depolymerization method that can generate monomers from commodity plastics (polypropylene (PP) and poly(ethylene terephthalate) (PET)) by means of pyrolysis. This selective depolymerization process is realized by two features: (1) a spatial temperature gradient and (2) a temporal heating profile. The spatial temperature gradient is achieved using a bilayer structure of porous carbon felt, in which the top electrically heated layer generates and conducts heat down to the underlying reactor layer and plastic. The resulting temperature gradient promotes continuous melting, wicking, vaporization and reaction of the plastic as it encounters the increasing temperature traversing the bilayer, enabling a high degree of depolymerization. Meanwhile, pulsing the electrical current through the top heater layer generates a temporal heating profile that features periodic high peak temperatures (for example, about 600 °C) to enable depolymerization, yet the transient heating duration (for example, 0.11 s) can suppress unwanted side reactions. Using this approach, we depolymerized PP and PET to their monomers with yields of about 36% and about 43%, respectively. Overall, this electrified spatiotemporal heating (STH) approach potentially offers a solution to the global plastic waste problem.

Black and gray box learning of amplitude equations: Application to phase field systems.

We present a data-driven approach to learning surrogate models for amplitude equations and illustrate its application to interfacial dynamics of phase field systems. In particular, we demonstrate learning effective partial differential equations describing the evolution of phase field interfaces from full phase field data. We illustrate this on a model phase field system, where analytical approximate equations for the dynamics of the phase field interface (a higher-order eikonal equation and its approximation, the Kardar-Parisi-Zhang equation) are known. For this system, we discuss data-driven approaches for the identification of equations that accurately describe the front interface dynamics. When the analytical approximate models mentioned above become inaccurate, as we move beyond the region of validity of the underlying assumptions, the data-driven equations outperform them. In these regimes, going beyond black box identification, we explore different approaches to learning data-driven corrections to the analytically approximate models, leading to effective gray box partial differential equations.

Learning effective stochastic differential equations from microscopic simulations: Linking stochastic numerics to deep learning.

We identify effective stochastic differential equations (SDEs) for coarse observables of fine-grained particle- or agent-based simulations; these SDEs then provide useful coarse surrogate models of the fine scale dynamics. We approximate the drift and diffusivity functions in these effective SDEs through neural networks, which can be thought of as effective stochastic ResNets. The loss function is inspired by, and embodies, the structure of established stochastic numerical integrators (here, Euler-Maruyama and Milstein); our approximations can thus benefit from backward error analysis of these underlying numerical schemes. They also lend themselves naturally to "physics-informed" gray-box identification when approximate coarse models, such as mean field equations, are available. Existing numerical integration schemes for Langevin-type equations and for stochastic partial differential equations can also be used for training; we demonstrate this on a stochastically forced oscillator and the stochastic wave equation. Our approach does not require long trajectories, works on scattered snapshot data, and is designed to naturally handle different time steps per snapshot. We consider both the case where the coarse collective observables are known in advance, as well as the case where they must be found in a data-driven manner.

Limits of entrainment of circadian neuronal networks.

Circadian rhythmicity lies at the center of various important physiological and behavioral processes in mammals, such as sleep, metabolism, homeostasis, mood changes, and more. Misalignment of intrinsic neuronal oscillations with the external day-night cycle can disrupt such processes and lead to numerous disorders. In this work, we computationally determine the limits of circadian synchronization to external light signals of different frequency, duty cycle, and simulated amplitude. Instead of modeling circadian dynamics with generic oscillator models (e.g., Kuramoto-type), we use a detailed computational neuroscience model, which integrates biomolecular dynamics, neuronal electrophysiology, and network effects. This allows us to investigate the effect of small drug molecules, such as Longdaysin, and connect our results with experimental findings. To combat the high dimensionality of such a detailed model, we employ a matrix-free approach, while our entire algorithmic pipeline enables numerical continuation and construction of bifurcation diagrams using only direct simulation. We, thus, computationally explore the effect of heterogeneity in the circadian neuronal network, as well as the effect of the corrective therapeutic intervention of Longdaysin. Last, we employ unsupervised learning to construct a data-driven embedding space for representing neuronal heterogeneity.

Learning Poisson Systems and Trajectories of Autonomous Systems via Poisson Neural Networks.

We propose the Poisson neural networks (PNNs) to learn Poisson systems and trajectories of autonomous systems from data. Based on the Darboux-Lie theorem, the phase flow of a Poisson system can be written as the composition of: 1) a coordinate transformation; 2) an extended symplectic map; and 3) the inverse of the transformation. In this work, we extend this result to the unknotted trajectories of autonomous systems. We employ structured neural networks with physical priors to approximate the three aforementioned maps. We demonstrate through several simulations that PNNs are capable of handling very accurately several challenging tasks, including the motion of a particle in the electromagnetic potential, the nonlinear Schrödinger equation, and pixel observations of the two-body problem.

Long-term p21 and p53 dynamics regulate the frequency of mitosis events and cell cycle arrest following radiation damage.

Radiation exposure of healthy cells can halt cell cycle temporarily or permanently. In this work, we analyze the time evolution of p21 and p53 from two single cell datasets of retinal pigment epithelial cells exposed to several levels of radiation, and in particular, the effect of radiation on cell cycle arrest. Employing various quantification methods from signal processing, we show how p21 levels, and to a lesser extent p53 levels, dictate whether the cells are arrested in their cell cycle and how frequently these mitosis events are likely to occur. We observed that single cells exposed to the same dose of DNA damage exhibit heterogeneity in cellular outcomes and that the frequency of cell division is a more accurate monitor of cell damage rather than just radiation level. Finally, we show how heterogeneity in DNA damage signaling is manifested early in the response to radiation exposure level and has potential to predict long-term fate.

Machine Learning Memory Kernels as Closure for Non-Markovian Stochastic Processes.

Finding the dynamical law of observable quantities lies at the core of physics. Within the particular field of statistical mechanics, the generalized Langevin equation (GLE) comprises a general model for the evolution of observables covering a great deal of physical systems with many degrees of freedom and an inherently stochastic nature. Although formally exact, GLE brings its own great challenges. It depends on the complete history of the observables under scrutiny, as well as the microscopic degrees of freedom, all of which are often inaccessible. We show that these drawbacks can be overcome by adopting elements of machine learning from empirical data, in particular coupling a multilayer perceptron (MLP) with the formal structure of GLE and calibrating the MLP with the data. This yields a powerful computational tool capable of describing noisy complex systems beyond the realms of statistical mechanics. It is exemplified with a number of representative examples from different fields: from a single colloidal particle and particle chains in a thermal bath to climatology and finance, showing in all cases excellent agreement with the actual observable dynamics. The new framework offers an alternative perspective for the study of nonequilibrium processes opening also a new route for stochastic modeling.

Time-series forecasting using manifold learning, radial basis function interpolation, and geometric harmonics.

We address a three-tier numerical framework based on nonlinear manifold learning for the forecasting of high-dimensional time series, relaxing the "curse of dimensionality" related to the training phase of surrogate/machine learning models. At the first step, we embed the high-dimensional time series into a reduced low-dimensional space using nonlinear manifold learning (local linear embedding and parsimonious diffusion maps). Then, we construct reduced-order surrogate models on the manifold (here, for our illustrations, we used multivariate autoregressive and Gaussian process regression models) to forecast the embedded dynamics. Finally, we solve the pre-image problem, thus lifting the embedded time series back to the original high-dimensional space using radial basis function interpolation and geometric harmonics. The proposed numerical data-driven scheme can also be applied as a reduced-order model procedure for the numerical solution/propagation of the (transient) dynamics of partial differential equations (PDEs). We assess the performance of the proposed scheme via three different families of problems: (a) the forecasting of synthetic time series generated by three simplistic linear and weakly nonlinear stochastic models resembling electroencephalography signals, (b) the prediction/propagation of the solution profiles of a linear parabolic PDE and the Brusselator model (a set of two nonlinear parabolic PDEs), and (c) the forecasting of a real-world data set containing daily time series of ten key foreign exchange rates spanning the time period 3 September 2001-29 October 2020.

Learning emergent partial differential equations in a learned emergent space.

We propose an approach to learn effective evolution equations for large systems of interacting agents. This is demonstrated on two examples, a well-studied system of coupled normal form oscillators and a biologically motivated example of coupled Hodgkin-Huxley-like neurons. For such types of systems there is no obvious space coordinate in which to learn effective evolution laws in the form of partial differential equations. In our approach, we accomplish this by learning embedding coordinates from the time series data of the system using manifold learning as a first step. In these emergent coordinates, we then show how one can learn effective partial differential equations, using neural networks, that do not only reproduce the dynamics of the oscillator ensemble, but also capture the collective bifurcations when system parameters vary. The proposed approach thus integrates the automatic, data-driven extraction of emergent space coordinates parametrizing the agent dynamics, with machine-learning assisted identification of an emergent PDE description of the dynamics in this parametrization.

Programmable heating and quenching for efficient thermochemical synthesis.

Conventional thermochemical syntheses by continuous heating under near-equilibrium conditions face critical challenges in improving the synthesis rate, selectivity, catalyst stability and energy efficiency, owing to the lack of temporal control over the reaction temperature and time, and thus the reaction pathways1-3. As an alternative, we present a non-equilibrium, continuous synthesis technique that uses pulsed heating and quenching (for example, 0.02 s on, 1.08 s off) using a programmable electric current to rapidly switch the reaction between high (for example, up to 2,400 K) and low temperatures. The rapid quenching ensures high selectivity and good catalyst stability, as well as lowers the average temperature to reduce the energy cost. Using CH4 pyrolysis as a model reaction, our programmable heating and quenching technique leads to high selectivity to value-added C2 products (>75% versus <35% by the conventional non-catalytic method and versus <60% by most conventional methods using optimized catalysts). Our technique can be extended to a range of thermochemical reactions, such as NH3 synthesis, for which we achieve a stable and high synthesis rate of about 6,000 µmol gFe-1 h-1 at ambient pressure for >100 h using a non-optimized catalyst. This study establishes a new model towards highly efficient non-equilibrium thermochemical synthesis.

Learning the temporal evolution of multivariate densities via normalizing flows.

In this work, we propose a method to learn multivariate probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker-Planck equations). We analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker-Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a multivariate normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can approximate probability density function evolutions in time from observed sampled data for systems driven by both Brownian and Lévy noise. We present examples with two- and three-dimensional, uni- and multimodal distributions to validate the method.

High-entropy nanoparticles: Synthesis-structure-property relationships and data-driven discovery.

High-entropy nanoparticles have become a rapidly growing area of research in recent years. Because of their multielemental compositions and unique high-entropy mixing states (i.e., solid-solution) that can lead to tunable activity and enhanced stability, these nanoparticles have received notable attention for catalyst design and exploration. However, this strong potential is also accompanied by grand challenges originating from their vast compositional space and complex atomic structure, which hinder comprehensive exploration and fundamental understanding. Through a multidisciplinary view of synthesis, characterization, catalytic applications, high-throughput screening, and data-driven materials discovery, this review is dedicated to discussing the important progress of high-entropy nanoparticles and unveiling the critical needs for their future development for catalysis, energy, and sustainability applications.

On the parameter combinations that matter and on those that do not: data-driven studies of parameter (non)identifiability.

We present a data-driven approach to characterizing nonidentifiability of a model's parameters and illustrate it through dynamic as well as steady kinetic models. By employing Diffusion Maps and their extensions, we discover the minimal combinations of parameters required to characterize the output behavior of a chemical system: a set of effective parameters for the model. Furthermore, we introduce and use a Conformal Autoencoder Neural Network technique, as well as a kernel-based Jointly Smooth Function technique, to disentangle the redundant parameter combinations that do not affect the output behavior from the ones that do. We discuss the interpretability of our data-driven effective parameters, and demonstrate the utility of the approach both for behavior prediction and parameter estimation. In the latter task, it becomes important to describe level sets in parameter space that are consistent with a particular output behavior. We validate our approach on a model of multisite phosphorylation, where a reduced set of effective parameters (nonlinear combinations of the physical ones) has previously been established analytically.

Initializing LSTM internal states via manifold learning.

We present an approach, based on learning an intrinsic data manifold, for the initialization of the internal state values of long short-term memory (LSTM) recurrent neural networks, ensuring consistency with the initial observed input data. Exploiting the generalized synchronization concept, we argue that the converged, "mature" internal states constitute a function on this learned manifold. The dimension of this manifold then dictates the length of observed input time series data required for consistent initialization. We illustrate our approach through a partially observed chemical model system, where initializing the internal LSTM states in this fashion yields visibly improved performance. Finally, we show that learning this data manifold enables the transformation of partially observed dynamics into fully observed ones, facilitating alternative identification paths for nonlinear dynamical systems.