The training process of many deep networks explores the same low-dimensional manifold.

We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories, but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold.

Bifurcation instructed design of multistate machines.

We propose a design paradigm for multistate machines where transitions from one state to another are organized by bifurcations of multiple equilibria of the energy landscape describing the collective interactions of the machine components. This design paradigm is attractive since, near bifurcations, small variations in a few control parameters can result in large changes to the system's state providing an emergent lever mechanism. Further, the topological configuration of transitions between states near such bifurcations ensures robust operation, making the machine less sensitive to fabrication errors and noise. To design such machines, we develop and implement a new efficient algorithm that searches for interactions between the machine components that give rise to energy landscapes with these bifurcation structures. We demonstrate a proof of concept for this approach by designing magnetoelastic machines whose motions are primarily guided by their magnetic energy landscapes and show that by operating near bifurcations we can achieve multiple transition pathways between states. This proof of concept demonstration illustrates the power of this approach, which could be especially useful for soft robotics and at the microscale where typical macroscale designs are difficult to implement.

Jamming and unusual charge density fluctuations of strange metals.

The strange metallic regime across a number of high-temperature superconducting materials presents numerous challenges to the classic theory of Fermi liquid metals. Recent measurements of the dynamical charge response of strange metals, including optimally doped cuprates, have revealed a broad, featureless continuum of excitations, extending over much of the Brillouin zone. The collective density oscillations of this strange metal decay into the continuum in a manner that is at odds with the expectations of Fermi liquid theory. Inspired by these observations, we investigate the phenomenology of bosonic collective modes and the particle-hole excitations in a class of strange metals by making an analogy to the phonons of classical lattices falling apart across an unconventional jamming-like transition associated with the onset of rigidity. By making comparisons to the experimentally measured dynamical response functions, we reproduce many of the qualitative features using the above framework. We conjecture that the dynamics of electronic charge density over an intermediate range of energy scales in a class of strongly correlated metals can be at the brink of a jamming-like transition.

Universal scaling for disordered viscoelastic matter near the onset of rigidity.

The onset of rigidity in interacting liquids, as they undergo a transition to a disordered solid, is associated with a rearrangement of the low-frequency vibrational spectrum. In this Letter, we derive scaling forms for the singular dynamical response of disordered viscoelastic networks near both jamming and rigidity percolation. Using effective-medium theory, we extract critical exponents, invariant scaling combinations, and analytical formulas for universal scaling functions near these transitions. Our scaling forms describe the behavior in space and time near the various onsets of rigidity, for rigid and floppy phases and the crossover region, including diverging length scales and timescales at the transitions.

Information geometry for multiparameter models: new perspectives on the origin of simplicity.

Complex models in physics, biology, economics, and engineering are oftensloppy, meaning that the model parameters are not well determined by the model predictions for collective behavior. Many parameter combinations can vary over decades without significant changes in the predictions. This review uses information geometry to explore sloppiness and its deep relation to emergent theories. We introduce themodel manifoldof predictions, whose coordinates are the model parameters. Itshyperribbonstructure explains why only a few parameter combinations matter for the behavior. We review recent rigorous results that connect the hierarchy of hyperribbon widths to approximation theory, and to the smoothness of model predictions under changes of the control variables. We discuss recent geodesic methods to find simpler models on nearby boundaries of the model manifold-emergent theories with fewer parameters that explain the behavior equally well. We discuss a Bayesian prior which optimizes the mutual information between model parameters and experimental data, naturally favoring points on the emergent boundary theories and thus simpler models. We introduce a 'projected maximum likelihood' prior that efficiently approximates this optimal prior, and contrast both to the poor behavior of the traditional Jeffreys prior. We discuss the way the renormalization group coarse-graining in statistical mechanics introduces a flow of the model manifold, and connect stiff and sloppy directions along the model manifold with relevant and irrelevant eigendirections of the renormalization group. Finally, we discuss recently developed 'intensive' embedding methods, allowing one to visualize the predictions of arbitrary probabilistic models as low-dimensional projections of an isometric embedding, and illustrate our method by generating the model manifold of the Ising model.

Yield Precursor Dislocation Avalanches in Small Crystals: The Irreversibility Transition.

The transition from elastic to plastic deformation in crystalline metals shares history dependence and scale-invariant avalanche signature with other nonequilibrium systems under external loading such as colloidal suspensions. These other systems exhibit transitions with clear analogies to work hardening and yield stress, with many typically undergoing purely elastic behavior only after "training" through repeated cyclic loading; studies in these other systems show a power-law scaling of the hysteresis loop extent and of the training time as the peak load approaches a so-called reversible-to-irreversible transition (RIT). We discover here that deformation of small crystals shares these key characteristics: yielding and hysteresis in uniaxial compression experiments of single-crystalline Cu nano- and micropillars decay under repeated cyclic loading. The amplitude and decay time of the yield precursor avalanches diverge as the peak stress approaches failure stress for each pillar, with a power-law scaling virtually equivalent to RITs in other nonequilibrium systems.

Visualizing probabilistic models and data with Intensive Principal Component Analysis.

Unsupervised learning makes manifest the underlying structure of data without curated training and specific problem definitions. However, the inference of relationships between data points is frustrated by the "curse of dimensionality" in high dimensions. Inspired by replica theory from statistical mechanics, we consider replicas of the system to tune the dimensionality and take the limit as the number of replicas goes to zero. The result is intensive embedding, which not only is isometric (preserving local distances) but also allows global structure to be more transparently visualized. We develop the Intensive Principal Component Analysis (InPCA) and demonstrate clear improvements in visualizations of the Ising model of magnetic spins, a neural network, and the dark energy cold dark matter ([Formula: see text]) model as applied to the cosmic microwave background.

Chebyshev Approximation and the Global Geometry of Model Predictions.

Complex nonlinear models are typically ill conditioned or sloppy; their predictions are significantly affected by only a small subset of parameter combinations, and parameters are difficult to reconstruct from model behavior. Despite forming an important universality class and arising frequently in practice when performing a nonlinear fit to data, formal and systematic explanations of sloppiness are lacking. By unifying geometric interpretations of sloppiness with Chebyshev approximation theory, we rigorously explain sloppiness as a consequence of model smoothness. Our approach results in universal bounds on model predictions for classes of smooth models, capturing global geometric features that are intrinsic to their model manifolds, and characterizing a universality class of models. We illustrate this universality using three models from disparate fields (physics, chemistry, biology): exponential curves, reaction rates from an enzyme-catalyzed chemical reaction, and an epidemiology model of an infected population.

Morphology of renormalization-group flow for the de Almeida-Thouless-Gardner universality class.

A replica-symmetry-breaking phase transition is predicted in a host of disordered media. The criticality of the transition has, however, long been questioned below its upper critical dimension, six, due to the absence of a critical fixed point in the renormalization-group flows at one-loop order. A recent two-loop analysis revealed a possible strong-coupling fixed point, but given the uncontrolled nature of perturbative analysis in the strong-coupling regime, debate persists. Here we examine the nature of the transition as a function of spatial dimension and show that the strong-coupling fixed point can go through a Hopf bifurcation, resulting in a critical limit cycle and a concomitant discrete scale invariance. We further investigate a different renormalization scheme and argue that the basin of attraction of the strong-coupling fixed point (or limit cycle) may stay finite for all dimensions.

Computation of a Theoretical Membrane Phase Diagram and the Role of Phase in Lipid-Raft-Mediated Protein Organization.

Lipid phase heterogeneity in the plasma membrane is thought to be crucial for many aspects of cell signaling, but the physical basis of participating membrane domains such as "lipid rafts" remains controversial. Here we consider a lattice model yielding a phase diagram that includes several states proposed to be relevant for the cell membrane, including microemulsion-which can be related to membrane curvature-and Ising critical behavior. Using a neural-network-based machine learning approach, we compute the full phase diagram of this lattice model. We analyze selected regions of this phase diagram in the context of a signaling initiation event in mast cells: recruitment of the membrane-anchored tyrosine kinase Lyn to a cluster of transmembrane IgE-FcεRI receptors. We find that model membrane systems in microemulsion and Ising critical states can mediate roughly equal levels of kinase recruitment (binding energy ∼ -0.6 kB T), whereas a membrane near a tricritical point can mediate a much stronger kinase recruitment (-1.7 kB T). By comparing several models for lipid heterogeneity within a single theoretical framework, this work points to testable differences between existing models. We also suggest the tricritical point as a new possibility for the basis of membrane domains that facilitate preferential partitioning of signaling components.

Measuring nonlinear stresses generated by defects in 3D colloidal crystals.

The mechanical, structural and functional properties of crystals are determined by their defects, and the distribution of stresses surrounding these defects has broad implications for the understanding of transport phenomena. When the defect density rises to levels routinely found in real-world materials, transport is governed by local stresses that are predominantly nonlinear. Such stress fields however, cannot be measured using conventional bulk and local measurement techniques. Here, we report direct and spatially resolved experimental measurements of the nonlinear stresses surrounding colloidal crystalline defect cores, and show that the stresses at vacancy cores generate attractive interactions between them. We also directly visualize the softening of crystalline regions surrounding dislocation cores, and find that stress fluctuations in quiescent polycrystals are uniformly distributed rather than localized at grain boundaries, as is the case in strained atomic polycrystals. Nonlinear stress measurements have important implications for strain hardening, yield and fatigue.

Scaling ansatz for the jamming transition.

We propose a Widom-like scaling ansatz for the critical jamming transition. Our ansatz for the elastic energy shows that the scaling of the energy, compressive strain, shear strain, system size, pressure, shear stress, bulk modulus, and shear modulus are all related to each other via scaling relations, with only three independent scaling exponents. We extract the values of these exponents from already known numerical or theoretical results, and we numerically verify the resulting predictions of the scaling theory for the energy and residual shear stress. We also derive a scaling relation between pressure and residual shear stress that yields insight into why the shear and bulk moduli scale differently. Our theory shows that the jamming transition exhibits an emergent scale invariance, setting the stage for the potential development of a renormalization group theory for jamming.

Pinning Susceptibility: The Effect of Dilute, Quenched Disorder on Jamming.

We study the effect of dilute pinning on the jamming transition. Pinning reduces the average contact number needed to jam unpinned particles and shifts the jamming threshold to lower densities, leading to a pinning susceptibility, χ_{p}. Our main results are that this susceptibility obeys scaling form and diverges in the thermodynamic limit as χ_{p}∝|ϕ-ϕ_{c}^{∞}|^{-γ_{p}} where ϕ_{c}^{∞} is the jamming threshold in the absence of pins. Finite-size scaling arguments yield these values with associated statistical (systematic) errors γ_{p}=1.018±0.026(0.291) in d=2 and γ_{p}=1.534±0.120(0.822) in d=3. Logarithmic corrections raise the exponent in d=2 to close to the d=3 value, although the systematic errors are very large.

Block copolymer self-assembly-directed synthesis of mesoporous gyroidal superconductors.

Superconductors with periodically ordered mesoporous structures are expected to have properties very different from those of their bulk counterparts. Systematic studies of such phenomena to date are sparse, however, because of a lack of versatile synthetic approaches to such materials. We demonstrate the formation of three-dimensionally continuous gyroidal mesoporous niobium nitride (NbN) superconductors from chiral ABC triblock terpolymer self-assembly-directed sol-gel-derived niobium oxide with subsequent thermal processing in air and ammonia gas. Superconducting materials exhibit a critical temperature (T c) of about 7 to 8 K, a flux exclusion of about 5% compared to a dense NbN solid, and an estimated critical current density (J c) of 440 A cm(-2) at 100 Oe and 2.5 K. We expect block copolymer self-assembly-directed mesoporous superconductors to provide interesting subjects for mesostructure-superconductivity correlation studies.

Weirdest Martensite: Smectic Liquid Crystal Microstructure and Weyl-Poincaré Invariance.

Smectic liquid crystals are remarkable, beautiful examples of materials microstructure, with ordered patterns of geometrically perfect ellipses and hyperbolas. The solution of the complex problem of filling three-dimensional space with domains of focal conics under constraining boundary conditions yields a set of strict rules, which are similar to the compatibility conditions in a martensitic crystal. Here we present the rules giving compatible conditions for the concentric circle domains found at two-dimensional smectic interfaces with planar boundary conditions. Using configurations generated by numerical simulations, we develop a clustering algorithm to decompose the planar boundaries into domains. The interfaces between different domains agree well with the smectic compatibility conditions. We also discuss generalizations of our approach to describe the full three-dimensional smectic domains, where the variant symmetry group is the Weyl-Poincaré group of Lorentz boosts, translations, rotations, and dilatations.

You can run, you can hide: The epidemiology and statistical mechanics of zombies.

We use a popular fictional disease, zombies, in order to introduce techniques used in modern epidemiology modeling, and ideas and techniques used in the numerical study of critical phenomena. We consider variants of zombie models, from fully connected continuous time dynamics to a full scale exact stochastic dynamic simulation of a zombie outbreak on the continental United States. Along the way, we offer a closed form analytical expression for the fully connected differential equation, and demonstrate that the single person per site two dimensional square lattice version of zombies lies in the percolation universality class. We end with a quantitative study of the full scale US outbreak, including the average susceptibility of different geographical regions.

Overshoot during phenotypic switching of cancer cell populations.

The dynamics of tumor cell populations is hotly debated: do populations derive hierarchically from a subpopulation of cancer stem cells (CSCs), or are stochastic transitions that mutate differentiated cancer cells to CSCs important? Here we argue that regulation must also be important. We sort human melanoma cells using three distinct cancer stem cell (CSC) markers - CXCR6, CD271 and ABCG2 - and observe that the fraction of non-CSC-marked cells first overshoots to a higher level and then returns to the level of unsorted cells. This clearly indicates that the CSC population is homeostatically regulated. Combining experimental measurements with theoretical modeling and numerical simulations, we show that the population dynamics of cancer cells is associated with a complex miRNA network regulating the Wnt and PI3K pathways. Hence phenotypic switching is not stochastic, but is tightly regulated by the balance between positive and negative cells in the population. Reducing the fraction of CSCs below a threshold triggers massive phenotypic switching, suggesting that a therapeutic strategy based on CSC eradication is unlikely to succeed.

Crossover behavior in interface depinning.

We study the crossover scaling behavior of the height-height correlation function in interface depinning in random media. We analyze experimental data from a fracture experiment and simulate an elastic line model with nonlinear couplings and disorder. Both exhibit a crossover between two different universality classes. For the experiment, we fit a functional form to the universal crossover scaling function. For the model, we vary the system size and the strength of the nonlinear term and describe the crossover between the two universality classes with a multiparameter scaling function. Our method provides a general strategy to extract scaling properties in depinning systems exhibiting crossover phenomena.

Perspective: Sloppiness and emergent theories in physics, biology, and beyond.

Large scale models of physical phenomena demand the development of new statistical and computational tools in order to be effective. Many such models are "sloppy," i.e., exhibit behavior controlled by a relatively small number of parameter combinations. We review an information theoretic framework for analyzing sloppy models. This formalism is based on the Fisher information matrix, which is interpreted as a Riemannian metric on a parameterized space of models. Distance in this space is a measure of how distinguishable two models are based on their predictions. Sloppy model manifolds are bounded with a hierarchy of widths and extrinsic curvatures. The manifold boundary approximation can extract the simple, hidden theory from complicated sloppy models. We attribute the success of simple effective models in physics as likewise emerging from complicated processes exhibiting a low effective dimensionality. We discuss the ramifications and consequences of sloppy models for biochemistry and science more generally. We suggest that the reason our complex world is understandable is due to the same fundamental reason: simple theories of macroscopic behavior are hidden inside complicated microscopic processes.

Visualization, coarsening, and flow dynamics of focal conic domains in simulated smectic-A liquid crystals.

Smectic liquid crystals vividly illustrate the subtle interplay of broken translational and orientational symmetries, by exhibiting defect structures forming geometrically perfect confocal ellipses and hyperbolas. Here, we develop and numerically implement an effective theory to study the dynamics of focal conic domains in smectic-A liquid crystals. We use the information about the smectic's structure and energy density provided by our simulations to develop several novel visualization tools for the focal conics. Our simulations accurately describe both simple and extensional shear, which we compare to experiments, and provide additional insight into the coarsening dynamics of focal conic domains.