Quantifying the stochasticity of policy parameters in reinforcement learning problems.

The stochastic dynamics of reinforcement learning is studied using a master equation formalism. We consider two different problems-Q learning for a two-agent game and the multiarmed bandit problem with policy gradient as the learning method. The master equation is constructed by introducing a probability distribution over continuous policy parameters or over both continuous policy parameters and discrete state variables (a more advanced case). We use a version of the moment closure approximation to solve for the stochastic dynamics of the models. Our method gives accurate estimates for the mean and the (co)variance of policy variables. For the case of the two-agent game, we find that the variance terms are finite at steady state and derive a system of algebraic equations for computing them directly.

Gene-influx-driven evolution.

Here we analyze the evolutionary process in the presence of continuous influx of genotypes with submaximum fitness from the outside to the given habitat with finite resources. We show that strong influx from the outside allows the low-fitness genotype to win the competition with the higher fitness genotype, and in a finite population, drive the latter to extinction. We analyze a mathematical model of this phenomenon and obtain the conditions for the transition from the high-fitness to the low-fitness genotype caused by the influx of the latter. We calculate the time to extinction of the high-fitness genotype in a finite population with two alleles and find the exact analytical dynamics of extinction for the case of many genes with epistasis. We solve a related quasispecies model for a single peak (random) fitness landscape as well as for a symmetric fitness landscape. In the symmetric landscape, a nonperturbative effect is observed such that even an extremely low influx of the low-fitness genotype drastically changes the steady state fitness distribution. A similar nonperturbative phenomenon is observed for the allele fixation time as well. The identified regime of influx-driven evolution appears to be relevant for a broad class of biological systems and could be central to the evolution of prokaryotes and viruses.

Weak mixed phase in the mutator model.

We consider the mutator model with unidirected transitions from the wild type to the mutator type, with different fitness functions for the wild types and mutator types. We calculate both the fraction of mutator types in the population and the surpluses, i.e., the mean number of mutations in the regular part of genomes for the wild type and mutator type, which have never been derived exactly. We identify the phase structure. Beside the mixed (ordinary evolution phase with finite fraction of wild types at large genome length) and the mutator phase (the absolute majority is mutators), we find another new phase as well-it has the mean fitness of the mixed phase but an exponentially small (in genome length) fraction of wild types. We identify the phase transition point and discuss its implications.

A simple statistical physics model for the epidemic with incubation period.

Based on the classical SIR model, we derive a simple modification for the dynamics of epidemics with a known incubation period of infection. The model is described by a system of integro-differential equations. Parameters of our model are directly related to epidemiological data. We derive some analytical results, as well as perform numerical simulations. We use the proposed model to analyze COVID-19 epidemic data in Armenia.

Solution of the Crow-Kimura model with a periodically changing (two-season) fitness function.

Since the origin of life, both evolutionary dynamics and rhythms have played a key role in the functioning of living systems. The Crow-Kimura model of periodically changing fitness function has been solved exactly, using integral equation with time-ordered exponent. We also found a simple approximate solution for the two-season case. The evolutionary dynamics accompanied by the rhythms provide important insights into the properties of certain biological systems and processes.

Key role of recombination in evolutionary processes with migration between two habitats.

Recombination is one of the leading forces of evolutionary dynamics. Although the importance of both recombination and migration in evolution is well recognized, there is currently no exact theory of evolutionary dynamics for large genome models that incorporates recombination, mutation, selection (quasispecies model with recombination), and spatial dynamics. To address this problem, we analyze the simplest spatial evolutionary process, namely, evolution of haploid populations with mutation, selection, recombination, and unidirectional migration, in its exact analytical form. This model is based on the quasispecies theory with recombination, but with replicators migrating from one habitat to another. In standard evolutionary models involving one habitat, the evolutionary processes depend on the ratios of fitness for different sequences. In the case of migration, we consider the absolute fitness values because there is no competition for resources between the population of different habitats. In the standard model without epistasis, recombination does not affect the mean fitness of the population. When migration is introduced, the situation changes drastically such that recombination can affect the mean fitness as strongly as mutation, as has been observed by Li and Nei for a few loci model without mutations. We have solved our model in the limit of large genome size for the fitness landscapes having different peaks in the first and second habitats and obtained the total population sizes for both habitats as well as the proportion of the population around two peak sequences in the second habitat. We identify four phases in the model and present the exact solutions for three of them.

Approximate perturbative solutions of quasispecies model with recombination.

Despite the major roles played by genetic recombination in ecoevolutionary processes, limited progress has been made in analyzing realistic recombination models to date, due largely to the complexity of the associated mechanisms and the strongly nonlinear nature of the dynamical differential systems. In this paper, we consider a many-loci genomic model with fitness dependent on the Hamming distance from a reference genome, and adopt a Hamilton-Jacobi formulation to derive perturbative solutions for general linear fitness landscapes. The horizontal gene transfer model is used to describe recombination processes. Cases of weak selection and weak recombination with simultaneous mutation and selection are examined, yielding semianalytical solutions for the distribution surplus of O(1/N) accuracy, where N is the number of nucleotides in the genome.

Allele fixation probability in a Moran model with fluctuating fitness landscapes.

Evolution on changing fitness landscapes (seascapes) is an important problem in evolutionary biology. We consider the Moran model of finite population evolution with selection in a randomly changing, dynamic environment. In the model, each individual has one of the two alleles, wild type or mutant. We calculate the fixation probability by making a proper ansatz for the logarithm of fixation probabilities. This method has been used previously to solve the analogous problem for the Wright-Fisher model. The fixation probability is related to the solution of a third-order algebraic equation (for the logarithm of fixation probability). We consider the strong interference of landscape fluctuations, sampling, and selection when the fixation process cannot be described by the mean fitness. Such an effect appears if the mutant allele has a higher fitness in one landscape and a lower fitness in another, compared with the wild type, and the product of effective population size and fitness is large. We provide a generalization of the Kimura formula for the fixation probability that applies to these cases. When the mutant allele has a fitness (dis-)advantage in both landscapes, the fixation probability is described by the mean fitness.

Solution of the Crow-Kimura model with changing population size and Allee effect.

The Crow-Kimura model is commonly used in the modeling of genetic evolution in the presence of mutations and associated selection pressures. We consider a modified version of the Crow-Kimura model, in which population sizes are not fixed and Allee saturation effects are present. We demonstrate the evolutionary dynamics in this system through an analytical approach, examining both symmetric and single-peak fitness landscape cases. Especially interesting are the dynamics of the populations near extinction. A special version of the model with saturation and degradation on the single-peak fitness landscape is investigated as a candidate of the Allee effect in evolution, revealing reduction tendencies of excessively large populations, and extinction tendencies for small populations. The analytical solutions for these dynamics are presented with accuracy O(1/N), where N is the number of nucleotides in the genome.

Quasispecies model of evolution with migration.

We consider the model of asexual evolution with migration, which was proposed by Waclaw et al. [Phys. Rev. Lett. 105, 268101 (2010)PRLTAO0031-900710.1103/PhysRevLett.105.268101]. This model setting is based on the standard mutation scheme from the quasispecies theory but with replicators moving from one habitat to another. The primary goal is to solve exactly the infinite population-genome length version of the model for the independent random distribution of fitnesses considered in the original paper. Moreover, we propose the analytical solution for the single peak and the symmetric fitness landscape. Our analytical solution is exact at the limit of large N. We found two phases-the correlated phase with the identical distributions of mutations in both habitats and the uncorrelated phase where the second habitat is choosing another peak of distribution in sequence space compared to the peak in the first habitat.

Looking for the optimal rate of recombination for evolutionary dynamics.

We consider many-site mutation-recombination models of evolution with selection. We are looking for situations where the recombination increases the mean fitness of the population, and there is an optimal recombination rate. We found two fitness landscapes supporting such nonmonotonic behavior of the mean fitness versus the recombination rate. The first case is related to the evolution near the error threshold on a neutral-network-like fitness landscape, for moderate genome lengths and large population. The more realistic case is the second one, in which we consider the evolutionary dynamics of a finite population on a rugged fitness landscape (the smooth fitness landscape plus some random contributions to the fitness). We also give the solution to the horizontal gene transfer model in the case of asymmetric mutations. To obtain nonmonotonic behavior for both mutation and recombination, we need a specially designed (ideal) fitness landscape.

Accurate analytic solution of chemical master equations for gene regulation networks in a single cell.

Studying gene regulation networks in a single cell is an important, interesting, and hot research topic of molecular biology. Such process can be described by chemical master equations (CMEs). We propose a Hamilton-Jacobi equation method with finite-size corrections to solve such CMEs accurately at the intermediate region of switching, where switching rate is comparable to fast protein production rate. We applied this approach to a model of self-regulating proteins [H. Ge et al., Phys. Rev. Lett. 114, 078101 (2015)PRLTAO0031-900710.1103/PhysRevLett.114.078101] and found that as a parameter related to inducer concentration increases the probability of protein production changes from unimodal to bimodal, then to unimodal, consistent with phenotype switching observed in a single cell.

Exact solution of a ratchet with switching sawtooth potential.

We consider the flashing potential ratchet model with general asymmetric potential. Using Bloch functions, we derive equations which allow for the calculation of both the ratchet's flux and higher moments of distribution for rather general potentials. We indicate how to derive the optimal transition rates for maximal velocity of the ratchet. We calculate explicitly the exact velocity of a ratchet with simple sawtooth potential from the solution of a system of 8 linear algebraic equations. Using Bloch functions, we derive the equations for the ratchet with potentials changing periodically with time. We also consider the case of the ratchet with evolution with two different potentials acting for some random periods of time.

Crossing fitness canyons by a finite population.

We consider the Wright-Fisher model of the finite population evolution on a fitness landscape defined in the sequence space by a path of nearly neutral mutations. We study a specific structure of the fitness landscape: One of the intermediate mutations on the mutation path results in either a large fitness value (climbing up a fitness hill) or a low fitness value (crossing a fitness canyon), the rest of the mutations besides the last one are neutral, and the last sequence has much higher fitness than any intermediate sequence. We derive analytical formulas for the first arrival time of the mutant with two point mutations. For the first arrival problem for the further mutants in the case of canyon crossing, we analytically deduce how the mean first arrival time scales with the population size and fitness difference. The location of the canyon on the path of sequences has a crucial role. If the canyon is at the beginning of the path, then it significantly prolongs the first arrival time; otherwise it just slightly changes it. Furthermore, the fitness hill at the beginning of the path strongly prolongs the arrival time period; however, the hill located near the end of the path shortens it. We optimize the first arrival time by applying a nonzero selection to the intermediate sequences. We extend our results and provide a scaling for the valley crossing time via the depth of the canyon and population size in the case of a fitness canyon at the first position. Our approach is useful for understanding some complex evolution systems, e.g., the evolution of cancer.

Allison mixture and the two-envelope problem.

In the present study, we have investigated the Allison mixture, a variant of the Parrondo's games where random mixing of two random sequences creates autocorrelation. We have obtained the autocorrelation function and mutual entropy of two elements. Our analysis shows that the mutual information is nonzero even if two distributions have identical average values. We have also considered the two-envelope problem and solved for its exact probability distribution.

Kinetics of biochemical sensing by single cells and populations of cells.

We investigate the collective stationary sensing using N communicative cells, which involves surface receptors, diffusive signaling molecules, and cell-cell communication messengers. We restrict the scenarios to the signal-to-noise ratios (SNRs) for both strong communication and extrinsic noise only. We modified a previous model [Bialek and Setayeshgar, Proc. Natl. Acad. Sci. USA 102, 10040 (2005)PNASA60027-842410.1073/pnas.0504321102] to eliminate the singularities in the fluctuation correlations by considering a uniform receptor distribution over the surface of each cell with a finite radius a. The modified model enables a simple and rigorous mathematical treatment of the collective sensing phenomenon. We then derive the scaling of the SNR for both juxtacrine and autocrine cases in all dimensions. For the optimal locations of the cells in the autocrine case, we find identical scaling for both two and three dimensions.

Exact solution of the hidden Markov processes.

We write a master equation for the distributions related to hidden Markov processes (HMPs) and solve it using a functional equation. Thus the solution of HMPs is mapped exactly to the solution of the functional equation. For a general case the latter can be solved only numerically. We derive an exact expression for the entropy of HMPs. Our expression for the entropy is an alternative to the ones given before by the solution of integral equations. The exact solution is possible because actually the model can be considered as a generalized random walk on a one-dimensional strip. While we give the solution for the two second-order matrices, our solution can be easily generalized for the L values of the Markov process and M values of observables: We should be able to solve a system of L functional equations in the space of dimension M-1.

Solution of classical evolutionary models in the limit when the diffusion approximation breaks down.

The discrete time mathematical models of evolution (the discrete time Eigen model, the Moran model, and the Wright-Fisher model) have many applications in complex biological systems. The discrete time Eigen model rather realistically describes the serial passage experiments in biology. Nevertheless, the dynamics of the discrete time Eigen model is solved in this paper. The 90% of results in population genetics are connected with the diffusion approximation of the Wright-Fisher and Moran models. We considered the discrete time Eigen model of asexual virus evolution and the Wright-Fisher model from population genetics. We look at the logarithm of probabilities and apply the Hamilton-Jacobi equation for the models. We define exact dynamics for the population distribution for the discrete time Eigen model. For the Wright-Fisher model, we express the exact steady state solution and fixation probability via the solution of some nonlocal equation then give the series expansion of the solution via degrees of selection and mutation rates. The diffusion theories result in the zeroth order approximation in our approach. The numeric confirms that our method works in the case of strong selection, whereas the diffusion method fails there. Although the diffusion method is exact for the mean first arrival time, it provides incorrect approximation for the dynamics of the tail of distribution.

The rich phase structure of a mutator model.

We propose a modification of the Crow-Kimura and Eigen models of biological molecular evolution to include a mutator gene that causes both an increase in the mutation rate and a change in the fitness landscape. This mutator effect relates to a wide range of biomedical problems. There are three possible phases: mutator phase, mixed phase and non-selective phase. We calculate the phase structure, the mean fitness and the fraction of the mutator allele in the population, which can be applied to describe cancer development and RNA viruses. We find that depending on the genome length, either the normal or the mutator allele dominates in the mixed phase. We analytically solve the model for a general fitness function. We conclude that the random fitness landscape is an appropriate choice for describing the observed mutator phenomenon in the case of a small fraction of mutators. It is shown that the increase in the mutation rates in the regular and the mutator parts of the genome should be set independently; only some combinations of these increases can push the complex biomedical system to the non-selective phase, potentially related to the eradication of tumors.

Exact probability distribution functions for Parrondo's games.

We study the discrete time dynamics of Brownian ratchet models and Parrondo's games. Using the Fourier transform, we calculate the exact probability distribution functions for both the capital dependent and history dependent Parrondo's games. In certain cases we find strong oscillations near the maximum of the probability distribution with two limiting distributions for odd and even number of rounds of the game. Indications of such oscillations first appeared in the analysis of real financial data, but now we have found this phenomenon in model systems and a theoretical understanding of the phenomenon. The method of our work can be applied to Brownian ratchets, molecular motors, and portfolio optimization.