Sampling with flows, diffusion, and autoregressive neural networks from a spin-glass perspective.

Recent years witnessed the development of powerful generative models based on flows, diffusion, or autoregressive neural networks, achieving remarkable success in generating data from examples with applications in a broad range of areas. A theoretical analysis of the performance and understanding of the limitations of these methods remain, however, challenging. In this paper, we undertake a step in this direction by analyzing the efficiency of sampling by these methods on a class of problems with a known probability distribution and comparing it with the sampling performance of more traditional methods such as the Monte Carlo Markov chain and Langevin dynamics. We focus on a class of probability distribution widely studied in the statistical physics of disordered systems that relate to spin glasses, statistical inference, and constraint satisfaction problems. We leverage the fact that sampling via flow-based, diffusion-based, or autoregressive networks methods can be equivalently mapped to the analysis of a Bayes optimal denoising of a modified probability measure. Our findings demonstrate that these methods encounter difficulties in sampling stemming from the presence of a first-order phase transition along the algorithm's denoising path. Our conclusions go both ways: We identify regions of parameters where these methods are unable to sample efficiently, while that is possible using standard Monte Carlo or Langevin approaches. We also identify regions where the opposite happens: standard approaches are inefficient while the discussed generative methods work well.

The overlap gap property: A topological barrier to optimizing over random structures.

The problem of optimizing over random structures emerges in many areas of science and engineering, ranging from statistical physics to machine learning and artificial intelligence. For many such structures, finding optimal solutions by means of fast algorithms is not known and often is believed not to be possible. At the same time, the formal hardness of these problems in the form of the complexity-theoretic NP-hardness is lacking. A new approach for algorithmic intractability in random structures is described in this article, which is based on the topological disconnectivity property of the set of pairwise distances of near-optimal solutions, called the Overlap Gap Property. The article demonstrates how this property 1) emerges in most models known to exhibit an apparent algorithmic hardness; 2) is consistent with the hardness/tractability phase transition for many models analyzed to the day; and, importantly, 3) allows to mathematically rigorously rule out a large class of algorithms as potential contenders, specifically the algorithms that exhibit the input stability (insensitivity).

Strong ergodicity breaking in aging of mean-field spin glasses.

Out-of-equilibrium relaxation processes show aging if they become slower as time passes. Aging processes are ubiquitous and play a fundamental role in the physics of glasses and spin glasses and in other applications (e.g., in algorithms minimizing complex cost/loss functions). The theory of aging in the out-of-equilibrium dynamics of mean-field spin glass models has achieved a fundamental role, thanks to the asymptotic analytic solution found by Cugliandolo and Kurchan. However, this solution is based on assumptions (e.g., the weak ergodicity breaking hypothesis) which have never been put under a strong test until now. In the present work, we present the results of an extraordinary large set of numerical simulations of the prototypical mean-field spin glass models, namely the Sherrington-Kirkpatrick and the Viana-Bray models. Thanks to a very intensive use of graphics processing units (GPUs), we have been able to run the latter model for more than [Formula: see text] spin updates and thus safely extrapolate the numerical data both in the thermodynamical limit and in the large times limit. The measurements of the two-times correlation functions in isothermal aging after a quench from a random initial configuration to a temperature [Formula: see text] provides clear evidence that, at large times, such correlations do not decay to zero as expected by assuming weak ergodicity breaking. We conclude that strong ergodicity breaking takes place in mean-field spin glasses aging dynamics which, asymptotically, takes place in a confined configurational space. Theoretical models for the aging dynamics need to be revised accordingly.

The Onset of Parisi's Complexity in a Mismatched Inference Problem.

We show that a statistical mechanics model where both the Sherringhton-Kirkpatrick and Hopfield Hamiltonians appear, which is equivalent to a high-dimensional mismatched inference problem, is described by a replica symmetry-breaking Parisi solution.

The Mpemba effect in spin glasses is a persistent memory effect.

The Mpemba effect occurs when a hot system cools faster than an initially colder one, when both are refrigerated in the same thermal reservoir. Using the custom-built supercomputer Janus II, we study the Mpemba effect in spin glasses and show that it is a nonequilibrium process, governed by the coherence length ξ of the system. The effect occurs when the bath temperature lies in the glassy phase, but it is not necessary for the thermal protocol to cross the critical temperature. In fact, the Mpemba effect follows from a strong relationship between the internal energy and ξ that turns out to be a sure-tell sign of being in the glassy phase. Thus, the Mpemba effect presents itself as an intriguing avenue for the experimental study of the coherence length in supercooled liquids and other glass formers.

Support for the value 5/2 for the spin glass lower critical dimension at zero magnetic field.

We study numerically various properties of the free energy barriers in the Edwards-Anderson model of spin glasses in the low-temperature region in both three and four spatial dimensions. In particular, we investigated the dependence of height of free energy barriers on system size and on the distance between the initial and final states (i.e., the overlap distance). A related quantity is the distribution of large local fluctuations of the overlap in large 3D samples at equilibrium. Our results for both quantities (barriers and large deviations) are in agreement with the prediction obtained in the framework of mean-field theory. In addition, our result supports [Formula: see text] as the lower critical dimension of the model.

Physical foundations of biological complexity.

Biological systems reach hierarchical complexity that has no counterpart outside the realm of biology. Undoubtedly, biological entities obey the fundamental physical laws. Can today's physics provide an explanatory framework for understanding the evolution of biological complexity? We argue that the physical foundation for understanding the origin and evolution of complexity can be gleaned at the interface between the theory of frustrated states resulting in pattern formation in glass-like media and the theory of self-organized criticality (SOC). On the one hand, SOC has been shown to emerge in spin-glass systems of high dimensionality. On the other hand, SOC is often viewed as the most appropriate physical description of evolutionary transitions in biology. We unify these two faces of SOC by showing that emergence of complex features in biological evolution typically, if not always, is triggered by frustration that is caused by competing interactions at different organizational levels. Such competing interactions lead to SOC, which represents the optimal conditions for the emergence of complexity. Competing interactions and frustrated states permeate biology at all organizational levels and are tightly linked to the ubiquitous competition for limiting resources. This perspective extends from the comparatively simple phenomena occurring in glasses to large-scale events of biological evolution, such as major evolutionary transitions. Frustration caused by competing interactions in multidimensional systems could be the general driving force behind the emergence of complexity, within and beyond the domain of biology.

A statics-dynamics equivalence through the fluctuation-dissipation ratio provides a window into the spin-glass phase from nonequilibrium measurements.

We have performed a very accurate computation of the nonequilibrium fluctuation-dissipation ratio for the 3D Edwards-Anderson Ising spin glass, by means of large-scale simulations on the special-purpose computers Janus and Janus II. This ratio (computed for finite times on very large, effectively infinite, systems) is compared with the equilibrium probability distribution of the spin overlap for finite sizes. Our main result is a quantitative statics-dynamics dictionary, which could allow the experimental exploration of important features of the spin-glass phase without requiring uncontrollable extrapolations to infinite times or system sizes.

Spin Glasses in a Field Show a Phase Transition Varying the Distance among Real Replicas (And How to Exploit It to Find the Critical Line in a Field).

We discuss a phase transition in spin glass models that have been rarely considered in the past, namely, the phase transition that may take place when two real replicas are forced to be at a larger distance (i.e., at a smaller overlap) than the typical one. In the first part of the work, by solving analytically the Sherrington-Kirkpatrick model in a field close to its critical point, we show that, even in a paramagnetic phase, the forcing of two real replicas to an overlap small enough leads the model to a phase transition where the symmetry between replicas is spontaneously broken. More importantly, this phase transition is related to the de Almeida-Thouless (dAT) critical line. In the second part of the work, we exploit the phase transition in the overlap between two real replicas to identify the critical line in a field in finite dimensional spin glasses. This is a notoriously difficult computational problem, because of considerable finite size corrections. We introduce a new method of analysis of Monte Carlo data for disordered systems, where the overlap between two real replicas is used as a conditioning variate. We apply this analysis to equilibrium measurements collected in the paramagnetic phase in a field, h > 0 and T c ( h ) < T < T c ( h = 0 ) , of the d = 1 spin glass model with long range interactions decaying fast enough to be outside the regime of validity of the mean field theory. We thus provide very reliable estimates for the thermodynamic critical temperature in a field.

Heterometallic Na6Co3 Phenylsilsesquioxane Exhibiting Slow Dynamic Behavior in its Magnetization.

A heterometallic phenylsilsesquioxane [(PhSiO1,5)22(CoO)3(NaO0.5)6]⋅(EtOH)6⋅(H2O) 1 cage architecture of Co(II) ions in a triangular topology exhibits a slow dynamic behavior in its magnetization, induced by the freezing of the spins of individual molecules.

Regularization, early-stopping and dreaming: A Hopfield-like setup to address generalization and overfitting.

In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.

Optimization Algorithms for Multi-species Spherical Spin Glasses.

This paper develops approximate message passing algorithms to optimize multi-species spherical spin glasses. We first show how to efficiently achieve the algorithmic threshold energy identified in our companion work (Huang and Sellke in arXiv preprint, 2023. arXiv:2303.12172), thus confirming that the Lipschitz hardness result proved therein is tight. Next we give two generalized algorithms which produce multiple outputs and show all of them are approximate critical points. Namely, in an r-species model we construct 2r approximate critical points when the external field is stronger than a "topological trivialization" phase boundary, and exponentially many such points in the complementary regime. We also compute the local behavior of the Hamiltonian around each. These extensions are relevant for another companion work (Huang and Sellke in arXiv preprint, 2023. arXiv:2308.09677) on topological trivialization of the landscape.

Evolution of magnetic surfboards and spin glass behavior in (Fe1-pGap)2TiO5.

The unusual anisotropy of the spin glass (SG) transition in the pseudobrookite system Fe2TiO5has been interpreted as arising from an induced, van der Waals-like, interaction among magnetic clusters. Here we present susceptibility (χ) and specific heat data (C) for Fe2TiO5diluted with non-magnetic Ga, (Fe1-pGap)2TiO5, for disorder parameterp= 0, 0.11, and 0.42, and elastic neutron scattering data forp= 0.20. A uniform suppression ofTgis observed upon increasingp, along with a value ofχTgthat increases asTgdecreases, i.e.dχ(Tg)/dTg<0We also observeCT∝T2in the low temperature limit. The observed behavior places (Fe1-pGap)2TiO5in the category of a strongly geometrically frustrated SG.

A Spin Glass Model for the Loss Surfaces of Generative Adversarial Networks.

We present a novel mathematical model that seeks to capture the key design feature of generative adversarial networks (GANs). Our model consists of two interacting spin glasses, and we conduct an extensive theoretical analysis of the complexity of the model's critical points using techniques from Random Matrix Theory. The result is insights into the loss surfaces of large GANs that build upon prior insights for simpler networks, but also reveal new structure unique to this setting which explains the greater difficulty of training GANs.