Extrapolating tipping points and simulating non-stationary dynamics of complex systems using efficient machine learning.

Model-free and data-driven prediction of tipping point transitions in nonlinear dynamical systems is a challenging and outstanding task in complex systems science. We propose a novel, fully data-driven machine learning algorithm based on next-generation reservoir computing to extrapolate the bifurcation behavior of nonlinear dynamical systems using stationary training data samples. We show that this method can extrapolate tipping point transitions. Furthermore, it is demonstrated that the trained next-generation reservoir computing architecture can be used to predict non-stationary dynamics with time-varying bifurcation parameters. In doing so, post-tipping point dynamics of unseen parameter regions can be simulated.

Optimizing the combination of data-driven and model-based elements in hybrid reservoir computing.

Hybrid reservoir computing combines purely data-driven machine learning predictions with a physical model to improve the forecasting of complex systems. In this study, we investigate in detail the predictive capabilities of three different architectures for hybrid reservoir computing: the input hybrid (IH), output hybrid (OH), and full hybrid (FH), which combines IH and OH. By using nine different three-dimensional chaotic model systems and the high-dimensional spatiotemporal chaotic Kuramoto-Sivashinsky system, we demonstrate that all hybrid reservoir computing approaches significantly improve the prediction results, provided that the model is sufficiently accurate. For accurate models, we find that the OH and FH results are equivalent and significantly outperform the IH results, especially for smaller reservoir sizes. For totally inaccurate models, the predictive capabilities of IH and FH may decrease drastically, while the OH architecture remains as accurate as the purely data-driven results. Furthermore, OH allows for the separation of the reservoir and the model contributions to the output predictions. This enables an interpretation of the roles played by the data-driven and model-based elements in output hybrid reservoir computing, resulting in higher explainability of the prediction results. Overall, our findings suggest that the OH approach is the most favorable architecture for hybrid reservoir computing, when taking accuracy, interpretability, robustness to model error, and simplicity into account.

A novel approach to minimal reservoir computing.

Reservoir computers are powerful machine learning algorithms for predicting nonlinear systems. Unlike traditional feedforward neural networks, they work on small training data sets, operate with linear optimization, and therefore require minimal computational resources. However, the traditional reservoir computer uses random matrices to define the underlying recurrent neural network and has a large number of hyperparameters that need to be optimized. Recent approaches show that randomness can be taken out by running regressions on a large library of linear and nonlinear combinations constructed from the input data and their time lags and polynomials thereof. However, for high-dimensional and nonlinear data, the number of these combinations explodes. Here, we show that a few simple changes to the traditional reservoir computer architecture further minimizing computational resources lead to significant and robust improvements in short- and long-term predictive performances compared to similar models while requiring minimal sizes of training data sets.

Efficient forecasting of chaotic systems with block-diagonal and binary reservoir computing.

The prediction of complex nonlinear dynamical systems with the help of machine learning has become increasingly popular in different areas of science. In particular, reservoir computers, also known as echo-state networks, turned out to be a very powerful approach, especially for the reproduction of nonlinear systems. The reservoir, the key component of this method, is usually constructed as a sparse, random network that serves as a memory for the system. In this work, we introduce block-diagonal reservoirs, which implies that a reservoir can be composed of multiple smaller reservoirs, each with its own dynamics. Furthermore, we take out the randomness of the reservoir by using matrices of ones for the individual blocks. This breaks with the widespread interpretation of the reservoir as a single network. In the example of the Lorenz and Halvorsen systems, we analyze the performance of block-diagonal reservoirs and their sensitivity to hyperparameters. We find that the performance is comparable to sparse random networks and discuss the implications with regard to scalability, explainability, and hardware realizations of reservoir computers.

Identifying causality drivers and deriving governing equations of nonlinear complex systems.

Identifying and describing the dynamics of complex systems is a central challenge in various areas of science, such as physics, finance, or climatology. While machine learning algorithms are increasingly overtaking traditional approaches, their inner workings and, thus, the drivers of causality remain elusive. In this paper, we analyze the causal structure of chaotic systems using Fourier transform surrogates and three different inference techniques: While we confirm that Granger causality is exclusively able to detect linear causality, transfer entropy and convergent cross-mapping indicate that causality is determined to a significant extent by nonlinear properties. For the Lorenz and Halvorsen systems, we find that their contribution is independent of the strength of the nonlinear coupling. Furthermore, we show that a simple rationale and calibration algorithm are sufficient to extract the governing equations directly from the causal structure of the data. Finally, we illustrate the applicability of the framework to real-world dynamical systems using financial data before and after the COVID-19 outbreak. It turns out that the pandemic triggered a fundamental rupture in the world economy, which is reflected in the causal structure and the resulting equations.

Symmetry kills the square in a multifunctional reservoir computer.

The learning capabilities of a reservoir computer (RC) can be stifled due to symmetry in its design. Including quadratic terms in the training of a RC produces a "square readout matrix" that breaks the symmetry to quell the influence of "mirror-attractors," which are inverted copies of the RC's solutions in state space. In this paper, we prove analytically that certain symmetries in the training data forbid the square readout matrix to exist. These analytical results are explored numerically from the perspective of "multifunctionality," by training the RC to specifically reconstruct a coexistence of the Lorenz attractor and its mirror-attractor. We demonstrate that the square readout matrix emerges when the position of one attractor is slightly altered, even if there are overlapping regions between the attractors or if there is a second pair of attractors. We also find that at large spectral radius values of the RC's internal connections, the square readout matrix reappears prior to the RC crossing the edge of chaos.

Controlling nonlinear dynamical systems into arbitrary states using machine learning.

Controlling nonlinear dynamical systems is a central task in many different areas of science and engineering. Chaotic systems can be stabilized (or chaotified) with small perturbations, yet existing approaches either require knowledge about the underlying system equations or large data sets as they rely on phase space methods. In this work we propose a novel and fully data driven scheme relying on machine learning (ML), which generalizes control techniques of chaotic systems without requiring a mathematical model for its dynamics. Exploiting recently developed ML-based prediction capabilities, we demonstrate that nonlinear systems can be forced to stay in arbitrary dynamical target states coming from any initial state. We outline and validate our approach using the examples of the Lorenz and the Rössler system and show how these systems can very accurately be brought not only to periodic, but even to intermittent and different chaotic behavior. Having this highly flexible control scheme with little demands on the amount of required data on hand, we briefly discuss possible applications ranging from engineering to medicine.

Breaking symmetries of the reservoir equations in echo state networks.

Reservoir computing has repeatedly been shown to be extremely successful in the prediction of nonlinear time-series. However, there is no complete understanding of the proper design of a reservoir yet. We find that the simplest popular setup has a harmful symmetry, which leads to the prediction of what we call mirror-attractor. We prove this analytically. Similar problems can arise in a general context, and we use them to explain the success or failure of some designs. The symmetry is a direct consequence of the hyperbolic tangent activation function. Furthermore, four ways to break the symmetry are compared numerically: A bias in the output, a shift in the input, a quadratic term in the readout, and a mixture of even and odd activation functions. First, we test their susceptibility to the mirror-attractor. Second, we evaluate their performance on the task of predicting Lorenz data with the mean shifted to zero. The short-time prediction is measured with the forecast horizon while the largest Lyapunov exponent and the correlation dimension are used to represent the climate. Finally, the same analysis is repeated on a combined dataset of the Lorenz attractor and the Halvorsen attractor, which we designed to reveal potential problems with symmetry. We find that all methods except the output bias are able to fully break the symmetry with input shift and quadratic readout performing the best overall.

Reducing network size and improving prediction stability of reservoir computing.

Reservoir computing is a very promising approach for the prediction of complex nonlinear dynamical systems. Besides capturing the exact short-term trajectories of nonlinear systems, it has also proved to reproduce its characteristic long-term properties very accurately. However, predictions do not always work equivalently well. It has been shown that both short- and long-term predictions vary significantly among different random realizations of the reservoir. In order to gain an understanding on when reservoir computing works best, we investigate some differential properties of the respective realization of the reservoir in a systematic way. We find that removing nodes that correspond to the largest weights in the output regression matrix reduces outliers and improves overall prediction quality. Moreover, this allows to effectively reduce the network size and, therefore, increase computational efficiency. In addition, we use a nonlinear scaling factor in the hyperbolic tangent of the activation function. This adjusts the response of the activation function to the range of values of the input variables of the nodes. As a consequence, this reduces the number of outliers significantly and increases both the short- and long-term prediction quality for the nonlinear systems investigated in this study. Our results demonstrate that a large optimization potential lies in the systematical refinement of the differential reservoir properties for a given dataset.

Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing.

The prediction of complex nonlinear dynamical systems with the help of machine learning techniques has become increasingly popular. In particular, reservoir computing turned out to be a very promising approach especially for the reproduction of the long-term properties of a nonlinear system. Yet, a thorough statistical analysis of the forecast results is missing. Using the Lorenz and Rössler system, we statistically analyze the quality of prediction for different parametrizations-both the exact short-term prediction as well as the reproduction of the long-term properties (the "climate") of the system as estimated by the correlation dimension and largest Lyapunov exponent. We find that both short- and long-term predictions vary significantly among the realizations. Thus, special care must be taken in selecting the good predictions as realizations, which deliver better short-term prediction also tend to better resemble the long-term climate of the system. Instead of only using purely random Erdös-Renyi networks, we also investigate the benefit of alternative network topologies such as small world or scale-free networks and show which effect they have on the prediction quality. Our results suggest that the overall performance with respect to the reproduction of the climate of both the Lorenz and Rössler system is worst for scale-free networks. For the Lorenz system, there seems to be a slight benefit of using small world networks, while for the Rössler system, small world and Erdös-Renyi networks performed equivalently well. In general, the observation is that reservoir computing works for all network topologies investigated here.

Image Registration with Particles, Examplified with the Complex Plasma Laboratory PK-4 on Board the International Space Station.

Often, in complex plasmas and beyond, images of particles are recorded with a side-by-side camera setup. These images ideally need to be joined to create a large combined image. This is, for instance, the case in the PK-4 Laboratory on board the International Space Station (the next generation of complex plasma laboratories in space). It enables observations of microparticles embedded in an elongated low temperature DC plasma tube. The microparticles acquire charges from the surrounding plasma and interact strongly with each other. A sheet of laser light illuminates the microparticles, and two cameras record the motion of the microparticles inside this laser sheet. The fields of view of these cameras slightly overlap. In this article, we present two methods to combine the associated image pairs into one image, namely the SimpleElastix toolkit based on comparing the mutual information and a method based on detecting the particle positions. We found that the method based on particle positions performs slightly better than that based on the mutual information, and conclude with recommendations for other researchers wanting to solve a related problem.

Phase walk analysis of leptokurtic time series.

The Fourier phase information play a key role for the quantified description of nonlinear data. We present a novel tool for time series analysis that identifies nonlinearities by sensitively detecting correlations among the Fourier phases. The method, being called phase walk analysis, is based on well established measures from random walk analysis, which are now applied to the unwrapped Fourier phases of time series. We provide an analytical description of its functionality and demonstrate its capabilities on systematically controlled leptokurtic noise. Hereby, we investigate the properties of leptokurtic time series and their influence on the Fourier phases of time series. The phase walk analysis is applied to measured and simulated intermittent time series, whose probability density distribution is approximated by power laws. We use the day-to-day returns of the Dow-Jones industrial average, a synthetic time series with tailored nonlinearities mimicing the power law behavior of the Dow-Jones and the acceleration of the wind at an Atlantic offshore site. Testing for nonlinearities by means of surrogates shows that the new method yields strong significances for nonlinear behavior. Due to the drastically decreased computing time as compared to embedding space methods, the number of surrogate realizations can be increased by orders of magnitude. Thereby, the probability distribution of the test statistics can very accurately be derived and parameterized, which allows for much more precise tests on nonlinearities.

Linear and nonlinear market correlations: Characterizing financial crises and portfolio optimization.

Pearson correlation and mutual information-based complex networks of the day-to-day returns of U.S. S&P500 stocks between 1985 and 2015 have been constructed to investigate the mutual dependencies of the stocks and their nature. We show that both networks detect qualitative differences especially during (recent) turbulent market periods, thus indicating strongly fluctuating interconnections between the stocks of different companies in changing economic environments. A measure for the strength of nonlinear dependencies is derived using surrogate data and leads to interesting observations during periods of financial market crises. In contrast to the expectation that dependencies reduce mainly to linear correlations during crises, we show that (at least in the 2008 crisis) nonlinear effects are significantly increasing. It turns out that the concept of centrality within a network could potentially be used as some kind of an early warning indicator for abnormal market behavior as we demonstrate with the example of the 2008 subprime mortgage crisis. Finally, we apply a Markowitz mean variance portfolio optimization and integrate the measure of nonlinear dependencies to scale the investment exposure. This leads to significant outperformance as compared to a fully invested portfolio.

Surrogate-assisted network analysis of nonlinear time series.

The performance of recurrence networks and symbolic networks to detect weak nonlinearities in time series is compared to the nonlinear prediction error. For the synthetic data of the Lorenz system, the network measures show a comparable performance. In the case of relatively short and noisy real-world data from active galactic nuclei, the nonlinear prediction error yields more robust results than the network measures. The tests are based on surrogate data sets. The correlations in the Fourier phases of data sets from some surrogate generating algorithms are also examined. The phase correlations are shown to have an impact on the performance of the tests for nonlinearity.

Collective effects in vortex movements in complex plasmas.

We study the onset and characteristics of vortices in complex (dusty) plasmas using two-dimensional simulations in a setup modeled after the PK-3 Plus laboratory. A small number of microparticles initially self-arranges in a monolayer around the void. As additional particles are introduced, an extended system of vortices develops due to a nonzero curl of the plasma forces. We demonstrate a shear-thinning effect in the vortices. Velocity structure functions and the energy and enstrophy spectra show that vortex flow turbulence is present that is in essence of the "classical" Kolmogorov type.

Trabecular bone structure analysis of the spine using clinical MDCT: can it predict vertebral bone strength?

Recent technical improvements have made it possible to determine trabecular bone structure parameters of the spine using clinical multi-detector computed tomography (MDCT). Therefore, the purpose of this study was to analyze trabecular bone structure parameters obtained from clinical MDCT in relation to high resolution peripheral quantitative computed tomography (HR-pQCT) as a standard of reference and to investigate whether clinical MDCT can predict vertebral bone strength. Fourteen functional spinal segment units between T7 and L3 were harvested from 14 formalin-fixed human cadavers (11 women and 3 men; age 84 ± 10 years). All functional spinal segment units were examined using HR-pQCT (isotropic voxel size of 41 µm(3)) and a clinical whole-body MDCT (interpolated voxel size of 146 × 146 × 300 µm(3)). Trabecular bone structure analyses (histomorphometric and texture measures) were performed in the HR-pQCT as well as MDCT images. Vertebral failure load (FL) of the functional spinal segment units was determined in an uniaxial biomechanical test. The HR-pQCT and MDCT derived trabecular bone structure parameters showed correlations ranging from r = 0.60 to r = 0.90 (p < 0.05). Correlations between trabecular bone structure parameters and FL amounted up to r = 0.86 (p < 0.05) using the HR-pQCT images, and up to r = 0.79 (p < 0.05) using the MDCT images. Correlation coefficients of FL versus trabecular bone structure parameters obtained with HR-pQCT and MDCT were not significantly different (p > 0.05). In this cadaver model, the spatial resolution of clinically available whole-body MDCT scanners was suitable for trabecular bone structure analysis of the spine and to predict vertebral bone strength.

Scaling relations between trabecular bone volume fraction and microstructure at different skeletal sites.

In this study, we investigated the scaling relations between trabecular bone volume fraction (BV/TV) and parameters of the trabecular microstructure at different skeletal sites. Cylindrical bone samples with a diameter of 8mm were harvested from different skeletal sites of 154 human donors in vitro: 87 from the distal radius, 59/69 from the thoracic/lumbar spine, 51 from the femoral neck, and 83 from the greater trochanter. µCT images were obtained with an isotropic spatial resolution of 26µm. BV/TV and trabecular microstructure parameters (TbN, TbTh, TbSp, scaling indices (< > and σ of α and αz), and Minkowski Functionals (Surface, Curvature, Euler)) were computed for each sample. The regression coefficient ß was determined for each skeletal site as the slope of a linear fit in the double-logarithmic representations of the correlations of BV/TV versus the respective microstructure parameter. Statistically significant correlation coefficients ranging from r=0.36 to r=0.97 were observed for BV/TV versus microstructure parameters, except for Curvature and Euler. The regression coefficients ß were 0.19 to 0.23 (TbN), 0.21 to 0.30 (TbTh), -0.28 to -0.24 (TbSp), 0.58 to 0.71 (Surface) and 0.12 to 0.16 (<α>), 0.07 to 0.11 (<αz>), -0.44 to -0.30 (σ(α)), and -0.39 to -0.14 (σ(αz)) at the different skeletal sites. The 95% confidence intervals of ß overlapped for almost all microstructure parameters at the different skeletal sites. The scaling relations were independent of vertebral fracture status and similar for subjects aged 60-69, 70-79, and >79years. In conclusion, the bone volume fraction-microstructure scaling relations showed a rather universal character.

Cortical and trabecular bone structure analysis at the distal radius-prediction of biomechanical strength by DXA and MRI.

The purpose of this study was to investigate whether the combination of dual-energy X-ray absorptiometry (DXA)-based bone mass and magnetic resonance imaging (MRI)-based cortical and trabecular structural measures improves the prediction of radial bone strength. Thirty-eight left forearms were harvested from formalin-fixed human cadavers. Bone mineral content (BMC) and bone mineral density (BMD) of the distal radius were measured using DXA. Cortical and trabecular structural measures of the distal radius were computed in high-resolution 1.5T MR images. Cortical measures included average cortical thickness and cross-sectional area. Trabecular measures included morphometric and texture parameters. The forearms were biomechanically tested in a fall simulation to measure absolute radial bone strength (failure load). Relative radial bone strength was determined by dividing radial failure loads by age, body mass index, radius length, and average radius cross-sectional area, respectively. DXA derived BMC and BMD showed statistically significant (p < 0.05) correlations with absolute and relative radial bone strength (r ≤ 0.78). Correlation coefficients for cortical and trabecular structural measures with absolute and relative radial bone strength amounted up to r = 0.59 and r = 0.74, respectively, (p < 0.05). In combination with DXA-based bone mass, trabecular but not, cortical structural measures, added in multiple regression models significant (p < 0.05) information in predicting absolute and relative radial bone strength (up to R adj = 0.88). Thus, a combination of DXA-based bone mass and MRI-based trabecular structural measures most accurately predicted absolute and relative radial bone strength, whereas structural measures of the cortex did not provide significant additional information in combination with DXA.

Reproducibility of trabecular bone structure measurements of the distal radius at 1.5 and 3.0 T magnetic resonance imaging.

The purpose of this study was to assess and compare the reproducibility of trabecular bone structure measurements of the distal radius at 1.5 and 3.0 T magnetic resonance imaging (MRI). Root mean square reproducibility errors ranged from 0.69% to 4.94% at 1.5 T MRI and from 0.38% to 5.80% at 3.0 T MRI. Thus, reproducibility errors of trabecular bone structure measurements are overall in an acceptable range and similar at 1.5 and 3.0 T MRI.

Detection and quantification of breast tumor necrosis with MR imaging: value of the necrosis-avid contrast agent Gadophrin-3.

RATIONALE AND OBJECTIVES: The authors evaluated the use of T1-weighted magnetic resonance (MR) imaging with Gadophrin-3 enhancement and of plain T2-weighted MR imaging to detect and quantify breast tumor necrosis. MATERIALS AND METHODS: Twenty EMT-6 tumors (mouse mammary sarcoma), implanted into the mammary fat pad of BALB/c-AnNCrl mice, underwent MR imaging with plain T2-weighted and T1-weighted fast field echo sequences before and 24 hours after injection of Gadophrin-3, a new necrosis-avid contrast agent. Tumor necrosis on MR images was quantified by means of a dedicated segmentation program and was correlated with histologic findings. RESULTS: In all tumors a central necrosis was revealed by histopathologic analysis, and central enhancement was seen with Gadophrin-3 on T1-weighted images. Small tumors (diameter, < 1 cm) showed an inhomogeneous central enhancement, whereas larger tumors (diameter, > 1 cm) enhanced mainly in the periphery of necrotic tissue. Plain T2-weighted images showed a hyperintense central area in only three of 20 cases with a large central necrosis. CONCLUSION: Gadophrin-3-enhanced T1-weighted images are superior to plain T2-weighted images for the detection of necrosis in a murine tumor xenograft model.