Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations.

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

Exact Quench Dynamics of the Floquet Quantum East Model at the Deterministic Point.

We study the nonequilibrium dynamics of the Floquet quantum East model (a Trotterized version of the kinetically constrained quantum East spin chain) at its "deterministic point," where evolution is defined in terms of CNOT permutation gates. We solve exactly the thermalization dynamics for a broad class of initial product states by means of "space evolution." We prove: (i) the entanglement of a block of spins grows at most at one-half the maximal speed allowed by locality (i.e., half the speed of dual-unitary circuits); (ii) if the block of spins is initially prepared in a classical configuration, speed of entanglement is a quarter of the maximum; (iii) thermalization to the infinite temperature state is reached exactly in a time that scales with the size of the block.

Optimal Sampling of Dynamical Large Deviations in Two Dimensions via Tensor Networks.

We use projected entangled-pair states (PEPS) to calculate the large deviation statistics of the dynamical activity of the two-dimensional East model, and the two-dimensional symmetric simple exclusion process (SSEP) with open boundaries, in lattices of up to 40×40 sites. We show that at long times both models have phase transitions between active and inactive dynamical phases. For the 2D East model we find that this trajectory transition is of the first order, while for the SSEP we find indications of a second order transition. We then show how the PEPS can be used to implement a trajectory sampling scheme capable of directly accessing rare trajectories. We also discuss how the methods described here can be extended to study rare events at finite times.

Reaction-Limited Quantum Reaction-Diffusion Dynamics.

We consider the quantum nonequilibrium dynamics of systems where fermionic particles coherently hop on a one-dimensional lattice and are subject to dissipative processes analogous to those of classical reaction-diffusion models. Particles can either annihilate in pairs, A+A→0, or coagulate upon contact, A+A→A, and possibly also branch, A→A+A. In classical settings, the interplay between these processes and particle diffusion leads to critical dynamics as well as to absorbing-state phase transitions. Here, we analyze the impact of coherent hopping and of quantum superposition, focusing on the so-called reaction-limited regime. Here, spatial density fluctuations are quickly smoothed out due to fast hopping, which for classical systems is described by a mean-field approach. By exploiting the time-dependent generalized Gibbs ensemble method, we demonstrate that quantum coherence and destructive interference play a crucial role in these systems and are responsible for the emergence of locally protected dark states and collective behavior beyond mean field. This can manifest both at stationarity and during the relaxation dynamics. Our analytical results highlight fundamental differences between classical nonequilibrium dynamics and their quantum counterpart and show that quantum effects indeed change collective universal behavior.

General Upper Bounds on Fluctuations of Trajectory Observables.

Thermodynamic uncertainty relations (TURs) are general lower bounds on the size of fluctuations of dynamical observables. They have important consequences, one being that the precision of estimation of a current is limited by the amount of entropy production. Here, we prove the existence of general upper bounds on the size of fluctuations of any linear combination of fluxes (including all time-integrated currents or dynamical activities) for continuous-time Markov chains. We obtain these general relations by means of concentration bound techniques. These "inverse TURs" are valid for all times and not only in the long time limit. We illustrate our analytical results with a simple model, and discuss wider implications of these new relations.

Trajectory phase transitions in non-interacting systems: all-to-all dynamics and the random energy model.

We study the fluctuations of time-additive random observables in the stochastic dynamics of a system of [Formula: see text] non-interacting Ising spins. We mainly consider the case of all-to-all dynamics where transitions are possible between any two spin configurations with uniform rates. We show that the cumulant generating function of the time-integral of a normally distributed quenched random function of configurations, i.e. the energy function of the random energy model (REM), has a phase transition in the large [Formula: see text] limit for trajectories of any time extent. We prove this by determining the exact limit of the scaled cumulant generating function. This is accomplished by connecting the dynamical problem to a spectral analysis of the all-to-all quantum REM. We also discuss finite [Formula: see text] corrections as observed in numerical simulations. This article is part of the theme issue 'Quantum annealing and computation: challenges and perspectives'.

Finite Time Large Deviations via Matrix Product States.

Recent work has shown the effectiveness of tensor network methods for computing large deviation functions in constrained stochastic models in the infinite time limit. Here we show that these methods can also be used to study the statistics of dynamical observables at arbitrary finite time. This is a harder problem because, in contrast to the infinite time case, where only the extremal eigenstate of a tilted Markov generator is relevant, for finite time the whole spectrum plays a role. We show that finite time dynamical partition sums can be computed efficiently and accurately in one dimension using matrix product states and describe how to use such results to generate rare event trajectories on demand. We apply our methods to the Fredrickson-Andersen and East kinetically constrained models and to the symmetric simple exclusion process, unveiling dynamical phase diagrams in terms of counting field and trajectory time. We also discuss extensions of this method to higher dimensions.

Reinforcement learning of rare diffusive dynamics.

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, such as those relevant in the study of reactive events, and trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, such as those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

Nonequilibrium Quantum Many-Body Rydberg Atom Engine.

The standard approach to quantum engines is based on equilibrium systems and on thermodynamic transformations between Gibbs states. However, nonequilibrium quantum systems offer enhanced experimental flexibility in the control of their parameters and, if used as engines, a more direct interpretation of the type of work they deliver. Here we introduce an out-of-equilibrium quantum engine inspired by recent experiments with cold atoms. Our system is connected to a single environment and produces mechanical work from many-body interparticle interactions arising between atoms in highly excited Rydberg states. As such, it is not a heat engine but an isothermal one. We perform many-body simulations to show that this system can produce work. The setup we introduce and investigate represents a promising platform for devising new types of microscopic machines and for exploring quantum effects in thermodynamic processes.

Using Matrix Product States to Study the Dynamical Large Deviations of Kinetically Constrained Models.

Here we demonstrate that tensor network techniques-originally devised for the analysis of quantum many-body problems-are well suited for the detailed study of rare event statistics in kinetically constrained models (KCMs). As concrete examples, we consider the Fredrickson-Andersen and East models, two paradigmatic KCMs relevant to the modeling of glasses. We show how variational matrix product states allow us to numerically approximate-systematically and with high accuracy-the leading eigenstates of the tilted dynamical generators, which encode the large deviation statistics of the dynamics. Via this approach, we can study system sizes beyond what is possible with other methods, allowing us to characterize in detail the finite size scaling of the trajectory-space phase transition of these models, the behavior of spectral gaps, and the spatial structure and "entanglement" properties of dynamical phases. We discuss the broader implications of our results.

Theory for Glassy Behavior of Supercooled Liquid Mixtures.

We present a model for glassy dynamics in supercooled liquid mixtures. Given the relaxation behavior of individual supercooled liquids, the model predicts the relaxation times of their mixtures as temperature is decreased. The model is based on dynamical facilitation theory for glassy dynamics, which provides a physical basis for relaxation and vitrification of a supercooled liquid. This is in contrast to empirical linear interpolations such as the Gordon-Taylor equation typically used to predict glass transition temperatures of liquid mixtures. To understand the behavior of supercooled liquid mixtures we consider a multicomponent variant of the kinetically constrained East model in which components have a different energy scale and can also diffuse when locally mobile regions, i.e., excitations, are present. Using a variational approach we determine an effective single component model with a single effective energy scale that best approximates a mixture. When scaled by this single effective energy, we show that experimental relaxation times of many liquid mixtures all collapse onto the "parabolic law" predicted by dynamical facilitation theory. The model can be used to predict transport properties and glass transition temperatures of mixtures of glassy materials, with implications in atmospheric chemistry, biology, and pharmaceuticals.

Unraveling the Large Deviation Statistics of Markovian Open Quantum Systems.

We analyze dynamical large deviations of quantum trajectories in Markovian open quantum systems in their full generality. We derive a quantum level-2.5 large deviation principle for these systems, which describes the joint fluctuations of time-averaged quantum jump rates and of the time-averaged quantum state for long times. Like its level-2.5 counterpart for classical continuous-time Markov chains (which it contains as a special case), this description is both explicit and complete, as the statistics of arbitrary time-extensive dynamical observables can be obtained by contraction from the explicit level-2.5 rate functional we derive. Our approach uses an unraveled representation of the quantum dynamics which allows these statistics to be obtained by analyzing a classical stochastic process in the space of pure states. For quantum reset processes we show that the unraveled dynamics is semi-Markovian and derive bounds on the asymptotic variance of the number of quantum jumps which generalize classical thermodynamic uncertainty relations. We finish by discussing how our level-2.5 approach can be used to study large deviations of nonlinear functions of the state, such as measures of entanglement.

Unified Thermodynamic Uncertainty Relations in Linear Response.

Thermodynamic uncertainty relations (TURs) are recently established relations between the relative uncertainty of time-integrated currents and entropy production in nonequilibrium systems. For small perturbations away from equilibrium, linear response (LR) theory provides the natural framework to study generic nonequilibrium processes. Here, we use LR to derive TURs in a straightforward and unified way. Our approach allows us to generalize TURs to systems without local time-reversal symmetry, including, e.g., ballistic transport and periodically driven classical and quantum systems. We find that, for broken time reversal, the bounds on the relative uncertainty are controlled both by dissipation and by a parameter encoding the asymmetry of the Onsager matrix. We illustrate our results with an example from mesoscopic physics. We also extend our approach beyond linear response: for Markovian dynamics, it reveals a connection between the TUR and current fluctuation theorems.

Solvation in Space-time: Pretransition Effects in Trajectory Space.

We demonstrate pretransition effects in space-time in trajectories of systems in which the dynamics displays a first-order phase transition between distinct dynamical phases. These effects are analogous to those observed for thermodynamic first-order phase transitions, most notably the hydrophobic effect in water. Considering the (infinite temperature) East model as an elementary example, we study the properties of "space-time solvation" by examining trajectories where finite space-time regions are conditioned to be inactive in an otherwise active phase. We find that solvating an inactive region of space-time within an active trajectory shows two regimes in the dynamical equivalent of solvation free energy: an "entropic" small solute regime in which uncorrelated fluctuations are sufficient to evacuate activity from the solute, and an "energetic" large solute regime which involves the formation of a solute-induced inactive domain with an associated active-inactive interface bearing a dynamical interfacial tension. We also show that as a result of this dynamical interfacial tension there is a dynamical analog of the hydrophobic collapse that drives the assembly of large hydrophobes in water. We discuss the general relevance of these results to the properties of dynamical fluctuations in systems with slow collective relaxation such as glass formers.

Quantum Slow Relaxation and Metastability due to Dynamical Constraints.

One of the general mechanisms that give rise to the slow cooperative relaxation characteristic of classical glasses is the presence of kinetic constraints in the dynamics. Here we show that dynamical constraints can similarly lead to slow thermalization and metastability in translationally invariant quantum many-body systems. We illustrate this general idea by considering two simple models: (i) a one-dimensional quantum analogue to classical constrained lattice gases where excitation hopping is constrained by the state of neighboring sites, mimicking excluded-volume interactions of dense fluids; and (ii) fully packed quantum dimers on the square lattice. Both models have a Rokhsar-Kivelson (RK) point at which kinetic and potential energy constants are equal. To one side of the RK point, where kinetic energy dominates, thermalization is fast. To the other, where potential energy dominates, thermalization is slow, memory of initial conditions persists for long times, and separation of timescales leads to pronounced metastability before eventual thermalization. Furthermore, in analogy with what occurs in the relaxation of classical glasses, the slow-thermalization regime displays dynamical heterogeneity as manifested by spatially segregated growth of entanglement.

Experimental Determination of Dynamical Lee-Yang Zeros.

Statistical physics provides the concepts and methods to explain the phase behavior of interacting many-body systems. Investigations of Lee-Yang zeros-complex singularities of the free energy in systems of finite size-have led to a unified understanding of equilibrium phase transitions. The ideas of Lee and Yang, however, are not restricted to equilibrium phenomena. Recently, Lee-Yang zeros have been used to characterize nonequilibrium processes such as dynamical phase transitions in quantum systems after a quench or dynamic order-disorder transitions in glasses. Here, we experimentally realize a scheme for determining Lee-Yang zeros in such nonequilibrium settings. We extract the dynamical Lee-Yang zeros of a stochastic process involving Andreev tunneling between a normal-state island and two superconducting leads from measurements of the dynamical activity along a trajectory. From the short-time behavior of the Lee-Yang zeros, we predict the large-deviation statistics of the activity which is typically difficult to measure. Our method paves the way for further experiments on the statistical mechanics of many-body systems out of equilibrium.

Study of the upper-critical dimension of the East model through the breakdown of the Stokes-Einstein relation.

We investigate the dimensional dependence of dynamical fluctuations related to dynamic heterogeneity in supercooled liquid systems using kinetically constrained models. The d-dimensional spin-facilitated East model with embedded probe particles is used as a representative super-Arrhenius glass forming system. We examine the existence of an upper critical dimension in this model by considering decoupling of transport rates through an effective fractional Stokes-Einstein relation, D∼τ-1+ω, with D and τ the diffusion constant of the probe particle and the relaxation time of the model liquid, respectively, and where ω>0 encodes the breakdown of the standard Stokes-Einstein relation. To the extent that decoupling indicates non-mean-field behavior, our simulations suggest that the East model has an upper critical dimension at least above d = 10 and argue that it may actually be infinite. This result is due to the existence of hierarchical dynamics in the East model in any finite dimension. We discuss the relevance of these results for studies of decoupling in high dimensional atomistic models.

Phase Transition for Quenched Coupled Replicas in a Plaquette Spin Model of Glasses.

We study a three-dimensional plaquette spin model whose low temperature dynamics is glassy, due to localized defects and effective kinetic constraints. The thermodynamics of this system is smooth at all temperatures. We show that coupling it to a second system with a fixed (quenched) configuration leads to a phase transition, at finite coupling. The order parameter is the overlap between the copies, and the transition is between phases of low and high overlap. We find critical points whose properties are consistent with random-field Ising universality. We analyze the interfacial free energy cost between the high- and low-overlap states that coexist at (and below) the critical point, and we use this cost as the basis for a finite-size scaling analysis. We discuss these results in the context of mean-field and dynamical facilitation theories of the glass transition.

Towards a Theory of Metastability in Open Quantum Dynamics.

By generalizing concepts from classical stochastic dynamics, we establish the basis for a theory of metastability in Markovian open quantum systems. Partial relaxation into long-lived metastable states-distinct from the asymptotic stationary state-is a manifestation of a separation of time scales due to a splitting in the spectrum of the generator of the dynamics. We show here how to exploit this spectral structure to obtain a low dimensional approximation to the dynamics in terms of motion in a manifold of metastable states constructed from the low-lying eigenmatrices of the generator. We argue that the metastable manifold is in general composed of disjoint states, noiseless subsystems, and decoherence-free subspaces.

Robustness of Many-Body Localization in the Presence of Dissipation.

Many-body localization (MBL) has emerged as a novel paradigm for robust ergodicity breaking in closed quantum many-body systems. However, it is not yet clear to which extent MBL survives in the presence of dissipative processes induced by the coupling to an environment. Here we study heating and ergodicity for a paradigmatic MBL system-an interacting fermionic chain subject to quenched disorder-in the presence of dephasing. We find that, even though the system is eventually driven into an infinite-temperature state, heating as monitored by the von Neumann entropy can progress logarithmically slowly, implying exponentially large time scales for relaxation. This slow loss of memory of initial conditions makes signatures of nonergodicity visible over a long, but transient, time regime. We point out a potential controlled realization of the considered setup with cold atomic gases held in optical lattices.