Decomposing the Local Arrow of Time in Interacting Systems.

We show that the evidence for a local arrow of time, which is equivalent to the entropy production in thermodynamic systems, can be decomposed. In a system with many degrees of freedom, there is a term that arises from the irreversible dynamics of the individual variables, and then a series of non-negative terms contributed by correlations among pairs, triplets, and higher-order combinations of variables. We illustrate this decomposition on simple models of noisy logical computations, and then apply it to the analysis of patterns of neural activity in the retina as it responds to complex dynamic visual scenes. We find that neural activity breaks detailed balance even when the visual inputs do not, and that this irreversibility arises primarily from interactions between pairs of neurons.

Nonequilibrium Statistical Mechanics of Continuous Attractors.

Continuous attractors have been used to understand recent neuroscience experiments where persistent activity patterns encode internal representations of external attributes like head direction or spatial location. However, the conditions under which the emergent bump of neural activity in such networks can be manipulated by space and time-dependent external sensory or motor signals are not understood. Here, we find fundamental limits on how rapidly internal representations encoded along continuous attractors can be updated by an external signal. We apply these results to place cell networks to derive a velocity-dependent nonequilibrium memory capacity in neural networks.

A high-bias, low-variance introduction to Machine Learning for physicists.

Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, regularization, generalization, and gradient descent before moving on to more advanced topics in both supervised and unsupervised learning. Topics covered in the review include ensemble models, deep learning and neural networks, clustering and data visualization, energy-based models (including MaxEnt models and Restricted Boltzmann Machines), and variational methods. Throughout, we emphasize the many natural connections between ML and statistical physics. A notable aspect of the review is the use of Python Jupyter notebooks to introduce modern ML/statistical packages to readers using physics-inspired datasets (the Ising Model and Monte-Carlo simulations of supersymmetric decays of proton-proton collisions). We conclude with an extended outlook discussing possible uses of machine learning for furthering our understanding of the physical world as well as open problems in ML where physicists may be able to contribute.

The Information Bottleneck and Geometric Clustering.

The information bottleneck (IB) approach to clustering takes a joint distribution P(X,Y) and maps the data X to cluster labels T , which retain maximal information about Y (Tishby, Pereira, & Bialek, 1999 ). This objective results in an algorithm that clusters data points based on the similarity of their conditional distributions P(Y∣X) . This is in contrast to classic geometric clustering algorithms such as k -means and gaussian mixture models (GMMs), which take a set of observed data points {xi}i=1:N and cluster them based on their geometric (typically Euclidean) distance from one another. Here, we show how to use the deterministic information bottleneck (DIB) (Strouse & Schwab, 2017 ), a variant of IB, to perform geometric clustering by choosing cluster labels that preserve information about data point location on a smoothed data set. We also introduce a novel intuitive method to choose the number of clusters via kinks in the information curve. We apply this approach to a variety of simple clustering problems, showing that DIB with our model selection procedure recovers the generative cluster labels. We also show that, in particular limits of our model parameters, clustering with DIB and IB is equivalent to k -means and EM fitting of a GMM with hard and soft assignments, respectively. Thus, clustering with (D)IB generalizes and provides an information-theoretic perspective on these classic algorithms.

Coordination of size-control, reproduction and generational memory in freshwater planarians.

Uncovering the mechanisms that control size, growth, and division rates of organisms reproducing through binary division means understanding basic principles of their life cycle. Recent work has focused on how division rates are regulated in bacteria and yeast, but this question has not yet been addressed in more complex, multicellular organisms. We have, over the course of several years, assembled a unique large-scale data set on the growth and asexual reproduction of two freshwater planarian species, Dugesia japonica and Girardia tigrina, which reproduce by transverse fission and succeeding regeneration of head and tail pieces into new planarians. We show that generation-dependent memory effects in planarian reproduction need to be taken into account to accurately capture the experimental data. To achieve this, we developed a new additive model that mixes multiple size control strategies based on planarian size, growth, and time between divisions. Our model quantifies the proportions of each strategy in the mixed dynamics, revealing the ability of the two planarian species to utilize different strategies in a coordinated manner for size control. Additionally, we found that head and tail offspring of both species employ different mechanisms to monitor and trigger their reproduction cycles. Thus, we find a diversity of strategies not only between species but between heads and tails within species. Our additive model provides two advantages over existing 2D models that fit a multivariable splitting rate function to the data for size control: firstly, it can be fit to relatively small data sets and can thus be applied to systems where available data is limited. Secondly, it enables new biological insights because it explicitly shows the contributions of different size control strategies for each offspring type.

The Deterministic Information Bottleneck.

Lossy compression and clustering fundamentally involve a decision about which features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek ( 1999 ) formalized this notion as an information-theoretic optimization problem and proposed an optimal trade-off between throwing away as many bits as possible and selectively keeping those that are most important. In the IB, compression is measured by mutual information. Here, we introduce an alternative formulation that replaces mutual information with entropy, which we call the deterministic information bottleneck (DIB) and argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem turns out to be a deterministic encoder, or hard clustering, as opposed to the stochastic encoder, or soft clustering, that is optimal under the IB. We compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB significantly outperforms the IB in terms of the DIB cost function. We also empirically find that the DIB offers a considerable gain in computational efficiency over the IB, over a range of convergence parameters. Our derivation of the DIB also suggests a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.

From intracellular signaling to population oscillations: bridging size- and time-scales in collective behavior.

Collective behavior in cellular populations is coordinated by biochemical signaling networks within individual cells. Connecting the dynamics of these intracellular networks to the population phenomena they control poses a considerable challenge because of network complexity and our limited knowledge of kinetic parameters. However, from physical systems, we know that behavioral changes in the individual constituents of a collectively behaving system occur in a limited number of well-defined classes, and these can be described using simple models. Here, we apply such an approach to the emergence of collective oscillations in cellular populations of the social amoeba Dictyostelium discoideum. Through direct tests of our model with quantitative in vivo measurements of single-cell and population signaling dynamics, we show how a simple model can effectively describe a complex molecular signaling network at multiple size and temporal scales. The model predicts novel noise-driven single-cell and population-level signaling phenomena that we then experimentally observe. Our results suggest that like physical systems, collective behavior in biology may be universal and described using simple mathematical models.

Quantifying the role of population subdivision in evolution on rugged fitness landscapes.

Natural selection drives populations towards higher fitness, but crossing fitness valleys or plateaus may facilitate progress up a rugged fitness landscape involving epistasis. We investigate quantitatively the effect of subdividing an asexual population on the time it takes to cross a fitness valley or plateau. We focus on a generic and minimal model that includes only population subdivision into equivalent demes connected by global migration, and does not require significant size changes of the demes, environmental heterogeneity or specific geographic structure. We determine the optimal speedup of valley or plateau crossing that can be gained by subdivision, if the process is driven by the deme that crosses fastest. We show that isolated demes have to be in the sequential fixation regime for subdivision to significantly accelerate crossing. Using Markov chain theory, we obtain analytical expressions for the conditions under which optimal speedup is achieved: valley or plateau crossing by the subdivided population is then as fast as that of its fastest deme. We verify our analytical predictions through stochastic simulations. We demonstrate that subdivision can substantially accelerate the crossing of fitness valleys and plateaus in a wide range of parameters extending beyond the optimal window. We study the effect of varying the degree of subdivision of a population, and investigate the trade-off between the magnitude of the optimal speedup and the width of the parameter range over which it occurs. Our results, obtained for fitness valleys and plateaus, also hold for weakly beneficial intermediate mutations. Finally, we extend our work to the case of a population connected by migration to one or several smaller islands. Our results demonstrate that subdivision with migration alone can significantly accelerate the crossing of fitness valleys and plateaus, and shed light onto the quantitative conditions necessary for this to occur.

Energetic costs of cellular computation.

Cells often perform computations in order to respond to environmental cues. A simple example is the classic problem, first considered by Berg and Purcell, of determining the concentration of a chemical ligand in the surrounding media. On general theoretical grounds, it is expected that such computations require cells to consume energy. In particular, Landauer's principle states that energy must be consumed in order to erase the memory of past observations. Here, we explicitly calculate the energetic cost of steady-state computation of ligand concentration for a simple two-component cellular network that implements a noisy version of the Berg-Purcell strategy. We show that learning about external concentrations necessitates the breaking of detailed balance and consumption of energy, with greater learning requiring more energy. Our calculations suggest that the energetic costs of cellular computation may be an important constraint on networks designed to function in resource poor environments, such as the spore germination networks of bacteria.

Dynamic tuning of electrical and chemical synaptic transmission in a network of motion coding retinal neurons.

Recently, we demonstrated that gap junction coupling in the population of superior coding ON-OFF directionally selective ganglion cells (DSGCs) genetically labeled in the Hb9::eGFP mouse retina allows the passage of lateral anticipatory signals that help track moving stimuli. Here, we examine the properties of gap junctions in the DSGC network, and address how interactions between electrical and chemical synapses and intrinsic membrane properties contribute to the dynamic tuning of lateral anticipatory signals. When DSGC subtypes coding all four cardinal directions were individually loaded with the gap junction-permeable tracer Neurobiotin, only superior coding DSGCs exhibited homologous coupling. Consistent with these anatomical findings, gap junction-dependent feedback spikelets were only observed in Hb9(+) DSGCs. Recordings from pairs of neighboring Hb9(+) DSGCs revealed that coupling was reciprocal, non-inactivating, and relatively weak, and provided a substrate for an extensive subthreshold excitatory receptive field around each cell. This subthreshold activity appeared to boost coincident light-driven chemical synaptic responses. However, during responses to moving stimuli, gap junction-mediated boosting appeared to be dynamically modulated such that upstream DSGCs primed downstream cells, but not vice versa, giving rise to highly skewed responses in individual cells. We show that the asymmetry in priming arises from a combination of spatially offset GABAergic inhibition and activity-dependent changes in intrinsic membrane properties of DSGCs. Thus, dynamic interactions between electrical and chemical synapses and intrinsic membrane properties allow the network of DSGCs to propagate anticipatory responses most effectively along their preferred direction without leading to runaway excitation.

Zipf's law and criticality in multivariate data without fine-tuning.

The joint probability distribution of states of many degrees of freedom in biological systems, such as firing patterns in neural networks or antibody sequence compositions, often follows Zipf's law, where a power law is observed on a rank-frequency plot. This behavior has been shown to imply that these systems reside near a unique critical point where the extensive parts of the entropy and energy are exactly equal. Here, we show analytically, and via numerical simulations, that Zipf-like probability distributions arise naturally if there is a fluctuating unobserved variable (or variables) that affects the system, such as a common input stimulus that causes individual neurons to fire at time-varying rates. In statistics and machine learning, these are called latent-variable or mixture models. We show that Zipf's law arises generically for large systems, without fine-tuning parameters to a point. Our work gives insight into the ubiquity of Zipf's law in a wide range of systems.

α-Ketoglutarate coordinates carbon and nitrogen utilization via enzyme I inhibition.

Microbes survive in a variety of nutrient environments by modulating their intracellular metabolism. Balanced growth requires coordinated uptake of carbon and nitrogen, the primary substrates for biomass production. Yet the mechanisms that balance carbon and nitrogen uptake are poorly understood. We find in Escherichia coli that a sudden increase in nitrogen availability results in an almost immediate increase in glucose uptake. The concentrations of glycolytic intermediates and known regulators, however, remain homeostatic. Instead, we find that α-ketoglutarate, which accumulates in nitrogen limitation, directly blocks glucose uptake by inhibiting enzyme I, the first step of the sugar-phosphoenolpyruvate phosphotransferase system (PTS). This inhibition enables rapid modulation of glycolytic flux without marked changes in the concentrations of glycolytic intermediates by simultaneously altering import of glucose and consumption of the terminal glycolytic intermediate phosphoenolpyruvate. Quantitative modeling shows that this previously unidentified regulatory connection is, in principle, sufficient to coordinate carbon and nitrogen utilization.

Noise driven phase transitions in eco-evolutionary systems.

In complex ecosystems such as microbial communities, there is constant ecological and evolutionary feedback between the residing species and the environment occurring on concurrent timescales. Species respond and adapt to their surroundings by modifying their phenotypic traits, which in turn alters their environment and the resources available. To study this interplay between ecological and evolutionary mechanisms, we develop a consumer-resource model that incorporates phenotypic mutations. In the absence of noise, we find that phase transitions require finely-tuned interaction kernels. Additionally, we quantify the effects of noise on frequency dependent selection by defining a time-integrated mutation current, which accounts for the rate at which mutations and speciation occurs. We find three distinct phases: homogeneous, patterned, and patterned traveling waves. The last phase represents one way in which co-evolution of species can happen in a fluctuating environment. Our results highlight the principal roles that noise and non-reciprocal interactions between resources and consumers play in phase transitions within eco-evolutionary systems.

Extrinsic vs Intrinsic Criticality in Systems with Many Components.

Biological systems with many components often exhibit seemingly critical behaviors, characterized by atypically large correlated fluctuations. Yet the underlying causes remain unclear. Here we define and examine two types of criticality. Intrinsic criticality arises from interactions within the system which are fine-tuned to a critical point. Extrinsic criticality, in contrast, emerges without fine tuning when observable degrees of freedom are coupled to unobserved fluctuating variables. We unify both types of criticality using the language of learning and information theory. We show that critical correlations, intrinsic or extrinsic, lead to diverging mutual information between two halves of the system, and are a feature of learning problems, in which the unobserved fluctuations are inferred from the observable degrees of freedom. We argue that extrinsic criticality is equivalent to standard inference, whereas intrinsic criticality describes fractional learning, in which the amount to be learned depends on the system size. We show further that both types of criticality are on the same continuum, connected by a smooth crossover. In addition, we investigate the observability of Zipf's law, a power-law rank-frequency distribution often used as an empirical signature of criticality. We find that Zipf's law is a robust feature of extrinsic criticality but can be nontrivial to observe for some intrinsically critical systems, including critical mean-field models We further demonstrate that models with global dynamics, such as oscillatory models, can produce observable Zipf's law without relying on either external fluctuations or fine tuning. Our findings suggest that while possible in theory, fine tuning is not the only, nor the most likely, explanation for the apparent ubiquity of criticality in biological systems with many components. Our work offers an alternative interpretation in which criticality, specifically extrinsic criticality, results from the adaptation of collective behavior to external stimuli.

Kuramoto model with coupling through an external medium.

Synchronization of coupled oscillators is often described using the Kuramoto model. Here, we study a generalization of the Kuramoto model where oscillators communicate with each other through an external medium. This generalized model exhibits interesting new phenomena such as bistability between synchronization and incoherence and a qualitatively new form of synchronization where the external medium exhibits small-amplitude oscillations. We conclude by discussing the relationship of the model to other variations of the Kuramoto model including the Kuramoto model with a bimodal frequency distribution and the Millennium bridge problem.

Dynamical quorum-sensing in oscillators coupled through an external medium.

Many biological and physical systems exhibit population-density dependent transitions to synchronized oscillations in a process often termed "dynamical quorum sensing". Synchronization frequently arises through chemical communication via signaling molecules distributed through an external medium. We study a simple theoretical model for dynamical quorum sensing: a heterogenous population of limit-cycle oscillators diffusively coupled through a common medium. We show that this model exhibits a rich phase diagram with four qualitatively distinct physical mechanisms that can lead to a loss of coherent population-level oscillations, including a novel mechanism arising from effective time-delays introduced by the external medium. We derive a single pair of analytic equations that allow us to calculate phase boundaries as a function of population density and show that the model reproduces many of the qualitative features of recent experiments on BZ catalytic particles as well as synthetically engineered bacteria.

Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality.

Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.

Theory of Gating in Recurrent Neural Networks.

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However gating i.e., multiplicative interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: (i) timescales and (ii) dimensionality. The gate controlling timescales leads to a novel marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.

Emergence of local irreversibility in complex interacting systems.

Living systems are fundamentally irreversible, breaking detailed balance and establishing an arrow of time. But how does the evident arrow of time for a whole system arise from the interactions among its multiple elements? We show that the local evidence for the arrow of time, which is the entropy production for thermodynamic systems, can be decomposed. First, it can be split into two components: an independent term reflecting the dynamics of individual elements and an interaction term driven by the dependencies among elements. Adapting tools from nonequilibrium physics, we further decompose the interaction term into contributions from pairs of elements, triplets, and higher-order terms. We illustrate our methods on models of cellular sensing and logical computations, as well as on patterns of neural activity in the retina as it responds to visual inputs. We find that neural activity can define the arrow of time even when the visual inputs do not, and that the dominant contribution to this breaking of detailed balance comes from interactions among pairs of neurons.