Noise driven phase transitions in eco-evolutionary systems.

In complex ecosystems such as microbial communities, there is constant ecological and evolutionary feedback between the residing species and the environment occurring on concurrent timescales. Species respond and adapt to their surroundings by modifying their phenotypic traits, which in turn alters their environment and the resources available. To study this interplay between ecological and evolutionary mechanisms, we develop a consumer-resource model that incorporates phenotypic mutations. In the absence of noise, we find that phase transitions require finely-tuned interaction kernels. Additionally, we quantify the effects of noise on frequency dependent selection by defining a time-integrated mutation current, which accounts for the rate at which mutations and speciation occurs. We find three distinct phases: homogeneous, patterned, and patterned traveling waves. The last phase represents one way in which co-evolution of species can happen in a fluctuating environment. Our results highlight the principal roles that noise and non-reciprocal interactions between resources and consumers play in phase transitions within eco-evolutionary systems.

Extrinsic vs Intrinsic Criticality in Systems with Many Components.

Biological systems with many components often exhibit seemingly critical behaviors, characterized by atypically large correlated fluctuations. Yet the underlying causes remain unclear. Here we define and examine two types of criticality. Intrinsic criticality arises from interactions within the system which are fine-tuned to a critical point. Extrinsic criticality, in contrast, emerges without fine tuning when observable degrees of freedom are coupled to unobserved fluctuating variables. We unify both types of criticality using the language of learning and information theory. We show that critical correlations, intrinsic or extrinsic, lead to diverging mutual information between two halves of the system, and are a feature of learning problems, in which the unobserved fluctuations are inferred from the observable degrees of freedom. We argue that extrinsic criticality is equivalent to standard inference, whereas intrinsic criticality describes fractional learning, in which the amount to be learned depends on the system size. We show further that both types of criticality are on the same continuum, connected by a smooth crossover. In addition, we investigate the observability of Zipf's law, a power-law rank-frequency distribution often used as an empirical signature of criticality. We find that Zipf's law is a robust feature of extrinsic criticality but can be nontrivial to observe for some intrinsically critical systems, including critical mean-field models We further demonstrate that models with global dynamics, such as oscillatory models, can produce observable Zipf's law without relying on either external fluctuations or fine tuning. Our findings suggest that while possible in theory, fine tuning is not the only, nor the most likely, explanation for the apparent ubiquity of criticality in biological systems with many components. Our work offers an alternative interpretation in which criticality, specifically extrinsic criticality, results from the adaptation of collective behavior to external stimuli.

Theory of Gating in Recurrent Neural Networks.

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However gating i.e., multiplicative interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: (i) timescales and (ii) dimensionality. The gate controlling timescales leads to a novel marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.

Emergence of local irreversibility in complex interacting systems.

Living systems are fundamentally irreversible, breaking detailed balance and establishing an arrow of time. But how does the evident arrow of time for a whole system arise from the interactions among its multiple elements? We show that the local evidence for the arrow of time, which is the entropy production for thermodynamic systems, can be decomposed. First, it can be split into two components: an independent term reflecting the dynamics of individual elements and an interaction term driven by the dependencies among elements. Adapting tools from nonequilibrium physics, we further decompose the interaction term into contributions from pairs of elements, triplets, and higher-order terms. We illustrate our methods on models of cellular sensing and logical computations, as well as on patterns of neural activity in the retina as it responds to visual inputs. We find that neural activity can define the arrow of time even when the visual inputs do not, and that the dominant contribution to this breaking of detailed balance comes from interactions among pairs of neurons.

Decomposing the Local Arrow of Time in Interacting Systems.

We show that the evidence for a local arrow of time, which is equivalent to the entropy production in thermodynamic systems, can be decomposed. In a system with many degrees of freedom, there is a term that arises from the irreversible dynamics of the individual variables, and then a series of non-negative terms contributed by correlations among pairs, triplets, and higher-order combinations of variables. We illustrate this decomposition on simple models of noisy logical computations, and then apply it to the analysis of patterns of neural activity in the retina as it responds to complex dynamic visual scenes. We find that neural activity breaks detailed balance even when the visual inputs do not, and that this irreversibility arises primarily from interactions between pairs of neurons.

Inferring couplings in networks across order-disorder phase transitions.

Statistical inference is central to many scientific endeavors, yet how it works remains unresolved. Answering this requires a quantitative understanding of the intrinsic interplay between statistical models, inference methods, and the structure in the data. To this end, we characterize the efficacy of direct coupling analysis (DCA) - a highly successful method for analyzing amino acid sequence data-in inferring pairwise interactions from samples of ferromagnetic Ising models on random graphs. Our approach allows for physically motivated exploration of qualitatively distinct data regimes separated by phase transitions. We show that inference quality depends strongly on the nature of data-generating distributions: optimal accuracy occurs at an intermediate temperature where the detrimental effects from macroscopic order and thermal noise are minimal. Importantly our results indicate that DCA does not always outperform its local-statistics-based predecessors; while DCA excels at low temperatures, it becomes inferior to simple correlation thresholding at virtually all temperatures when data are limited. Our findings offer insights into the regime in which DCA operates so successfully, and more broadly, how inference interacts with the structure in the data.

Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality.

Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.

An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers.

Recent work demonstrates that deep neural networks trained using Empirical Risk Minimization (ERM) can generalize under distribution shift, outperforming specialized training algorithms for domain generalization. The goal of this paper is to further understand this phenomenon. In particular, we study the extent to which the seminal domain adaptation theory of Ben-David et al. (2007) explains the performance of ERMs. Perhaps surprisingly, we find that this theory does not provide a tight explanation of the out-of-domain generalization observed across a large number of ERM models trained on three popular domain generalization datasets. This motivates us to investigate other possible measures-that, however, lack theory-which could explain generalization in this setting. Our investigation reveals that measures relating to the Fisher information, predictive entropy, and maximum mean discrepancy are good predictors of the out-of-distribution generalization of ERM models. We hope that our work helps galvanize the community towards building a better understanding of when deep networks trained with ERM generalize out-of-distribution.

Perturbation Theory for the Information Bottleneck.

Extracting relevant information from data is crucial for all forms of learning. The information bottleneck (IB) method formalizes this, offering a mathematically precise and conceptually appealing framework for understanding learning phenomena. However the nonlinearity of the IB problem makes it computationally expensive and analytically intractable in general. Here we derive a perturbation theory for the IB method and report the first complete characterization of the learning onset-the limit of maximum relevant information per bit extracted from data. We test our results on synthetic probability distributions, finding good agreement with the exact numerical solution near the onset of learning. We explore the difference and subtleties in our derivation and previous attempts at deriving a perturbation theory for the learning onset and attribute the discrepancy to a flawed assumption. Our work also provides a fresh perspective on the intimate relationship between the IB method and the strong data processing inequality.

Spatial gradient sensing and chemotaxis via excitability in Dictyostelium discoideum.

The social amoeba Dictyostelium discoideum performs chemotaxis under starvation conditions, aggregating towards clusters of cells following waves of the signaling molecule cAMP. Cells sense extracellular cAMP and produce internal caches of cAMP to be released, relaying the signal. These events lead to traveling waves of cAMP washing over the population of cells. While much research has been performed to understand the functioning of the chemotaxis network in D. discoideum, limited work has been done to link the operation of the signal relay network with the chemotaxis network to provide a holistic view of the system. We take inspiration from D. discoideum and propose a model that directly links the relaying of a chemical message to the directional sensing of that signal. Utilizing an excitable dynamical systems model that has been previously validated experimentally, we show that it is possible to have both signal amplification and perfect adaptation in a single module. We show that noise plays a vital role in chemotaxing to static gradients, where stochastic tunneling of transient bursts biases the system towards accurate gradient sensing. Moreover, this model also automatically matches its internal time scale of adaptation to the naturally occurring periodicity of the traveling chemical waves generated in the population. Numerical simulations were performed to study the qualitative phenomenology of the system and explore how the system responds to diverse dynamic spatiotemporal stimuli. Finally, we address dynamical instabilities that impede chemotactic ability in a continuum version of the model.

Nonequilibrium Statistical Mechanics of Continuous Attractors.

Continuous attractors have been used to understand recent neuroscience experiments where persistent activity patterns encode internal representations of external attributes like head direction or spatial location. However, the conditions under which the emergent bump of neural activity in such networks can be manipulated by space and time-dependent external sensory or motor signals are not understood. Here, we find fundamental limits on how rapidly internal representations encoded along continuous attractors can be updated by an external signal. We apply these results to place cell networks to derive a velocity-dependent nonequilibrium memory capacity in neural networks.

Energy consumption and cooperation for optimal sensing.

The reliable detection of environmental molecules in the presence of noise is an important cellular function, yet the underlying computational mechanisms are not well understood. We introduce a model of two interacting sensors which allows for the principled exploration of signal statistics, cooperation strategies and the role of energy consumption in optimal sensing, quantified through the mutual information between the signal and the sensors. Here we report that in general the optimal sensing strategy depends both on the noise level and the statistics of the signals. For joint, correlated signals, energy consuming (nonequilibrium), asymmetric couplings result in maximum information gain in the low-noise, high-signal-correlation limit. Surprisingly we also find that energy consumption is not always required for optimal sensing. We generalise our model to incorporate time integration of the sensor state by a population of readout molecules, and demonstrate that sensor interaction and energy consumption remain important for optimal sensing.

A high-bias, low-variance introduction to Machine Learning for physicists.

Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, regularization, generalization, and gradient descent before moving on to more advanced topics in both supervised and unsupervised learning. Topics covered in the review include ensemble models, deep learning and neural networks, clustering and data visualization, energy-based models (including MaxEnt models and Restricted Boltzmann Machines), and variational methods. Throughout, we emphasize the many natural connections between ML and statistical physics. A notable aspect of the review is the use of Python Jupyter notebooks to introduce modern ML/statistical packages to readers using physics-inspired datasets (the Ising Model and Monte-Carlo simulations of supersymmetric decays of proton-proton collisions). We conclude with an extended outlook discussing possible uses of machine learning for furthering our understanding of the physical world as well as open problems in ML where physicists may be able to contribute.

The Information Bottleneck and Geometric Clustering.

The information bottleneck (IB) approach to clustering takes a joint distribution P(X,Y) and maps the data X to cluster labels T , which retain maximal information about Y (Tishby, Pereira, & Bialek, 1999 ). This objective results in an algorithm that clusters data points based on the similarity of their conditional distributions P(Y∣X) . This is in contrast to classic geometric clustering algorithms such as k -means and gaussian mixture models (GMMs), which take a set of observed data points {xi}i=1:N and cluster them based on their geometric (typically Euclidean) distance from one another. Here, we show how to use the deterministic information bottleneck (DIB) (Strouse & Schwab, 2017 ), a variant of IB, to perform geometric clustering by choosing cluster labels that preserve information about data point location on a smoothed data set. We also introduce a novel intuitive method to choose the number of clusters via kinks in the information curve. We apply this approach to a variety of simple clustering problems, showing that DIB with our model selection procedure recovers the generative cluster labels. We also show that, in particular limits of our model parameters, clustering with DIB and IB is equivalent to k -means and EM fitting of a GMM with hard and soft assignments, respectively. Thus, clustering with (D)IB generalizes and provides an information-theoretic perspective on these classic algorithms.

Coordination of size-control, reproduction and generational memory in freshwater planarians.

Uncovering the mechanisms that control size, growth, and division rates of organisms reproducing through binary division means understanding basic principles of their life cycle. Recent work has focused on how division rates are regulated in bacteria and yeast, but this question has not yet been addressed in more complex, multicellular organisms. We have, over the course of several years, assembled a unique large-scale data set on the growth and asexual reproduction of two freshwater planarian species, Dugesia japonica and Girardia tigrina, which reproduce by transverse fission and succeeding regeneration of head and tail pieces into new planarians. We show that generation-dependent memory effects in planarian reproduction need to be taken into account to accurately capture the experimental data. To achieve this, we developed a new additive model that mixes multiple size control strategies based on planarian size, growth, and time between divisions. Our model quantifies the proportions of each strategy in the mixed dynamics, revealing the ability of the two planarian species to utilize different strategies in a coordinated manner for size control. Additionally, we found that head and tail offspring of both species employ different mechanisms to monitor and trigger their reproduction cycles. Thus, we find a diversity of strategies not only between species but between heads and tails within species. Our additive model provides two advantages over existing 2D models that fit a multivariable splitting rate function to the data for size control: firstly, it can be fit to relatively small data sets and can thus be applied to systems where available data is limited. Secondly, it enables new biological insights because it explicitly shows the contributions of different size control strategies for each offspring type.

The Deterministic Information Bottleneck.

Lossy compression and clustering fundamentally involve a decision about which features are relevant and which are not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek ( 1999 ) formalized this notion as an information-theoretic optimization problem and proposed an optimal trade-off between throwing away as many bits as possible and selectively keeping those that are most important. In the IB, compression is measured by mutual information. Here, we introduce an alternative formulation that replaces mutual information with entropy, which we call the deterministic information bottleneck (DIB) and argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem turns out to be a deterministic encoder, or hard clustering, as opposed to the stochastic encoder, or soft clustering, that is optimal under the IB. We compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB significantly outperforms the IB in terms of the DIB cost function. We also empirically find that the DIB offers a considerable gain in computational efficiency over the IB, over a range of convergence parameters. Our derivation of the DIB also suggests a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.

Meteotsunamis in the Laurentian Great Lakes.

The generation mechanism of meteotsunamis, which are meteorologically induced water waves with spatial/temporal characteristics and behavior similar to seismic tsunamis, is poorly understood. We quantify meteotsunamis in terms of seasonality, causes, and occurrence frequency through the analysis of long-term water level records in the Laurentian Great Lakes. The majority of the observed meteotsunamis happen from late-spring to mid-summer and are associated primarily with convective storms. Meteotsunami events of potentially dangerous magnitude (height > 0.3 m) occur an average of 106 times per year throughout the region. These results reveal that meteotsunamis are much more frequent than follow from historic anecdotal reports. Future climate scenarios over the United States show a likely increase in the number of days favorable to severe convective storm formation over the Great Lakes, particularly in the spring season. This would suggest that the convectively associated meteotsunamis in these regions may experience an increase in occurrence frequency or a temporal shift in occurrence to earlier in the warm season. To date, meteotsunamis in the area of the Great Lakes have been an overlooked hazard.

A Central Role for Mixed Acetylcholine/GABA Transmission in Direction Coding in the Retina.

A surprisingly large number of neurons throughout the brain are endowed with the ability to co-release both a fast excitatory and inhibitory transmitter. The computational benefits of dual transmitter release, however, remain poorly understood. Here, we address the role of co-transmission of acetylcholine (ACh) and GABA from starburst amacrine cells (SACs) to direction-selective ganglion cells (DSGCs). Using a combination of pharmacology, optogenetics, and linear regression methods, we estimated the spatiotemporal profiles of GABA, ACh, and glutamate receptor-mediated synaptic activity in DSGCs evoked by motion. We found that ACh initiates responses to motion in natural scenes or under low-contrast conditions. In contrast, classical glutamatergic pathways play a secondary role, amplifying cholinergic responses via NMDA receptor activation. Furthermore, under these conditions, the network of SACs differentially transmits ACh and GABA to DSGCs in a directional manner. Thus, mixed transmission plays a central role in shaping directional responses of DSGCs.

Presynaptic Inhibition in the Striatum of the Basal Ganglia Improves Pattern Classification and Thus Promotes Superior Goal Selection.

This review article takes a multidisciplinary approach to understand how presynaptic inhibition in the striatum of the basal ganglia (BG) contributes to pattern classification and the selection of goals that control behavior. It is a difficult problem both because it is multidimensional and because it is has complex system dynamics. We focus on the striatum because, as the main site for input to the BG, it gets to decide what goals are important to consider.