Reinforcement learning informs optimal treatment strategies to limit antibiotic resistance.

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent, or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that it is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 [Formula: see text]-lactam antibiotics with which to treat the simulated Escherichia coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1,024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.

Antibiotic interactions shape short-term evolution of resistance in E. faecalis.

Antibiotic combinations are increasingly used to combat bacterial infections. Multidrug therapies are a particularly important treatment option for E. faecalis, an opportunistic pathogen that contributes to high-inoculum infections such as infective endocarditis. While numerous synergistic drug combinations for E. faecalis have been identified, much less is known about how different combinations impact the rate of resistance evolution. In this work, we use high-throughput laboratory evolution experiments to quantify adaptation in growth rate and drug resistance of E. faecalis exposed to drug combinations exhibiting different classes of interactions, ranging from synergistic to suppressive. We identify a wide range of evolutionary behavior, including both increased and decreased rates of growth adaptation, depending on the specific interplay between drug interaction and drug resistance profiles. For example, selection in a dual ß-lactam combination leads to accelerated growth adaptation compared to selection with the individual drugs, even though the resulting resistance profiles are nearly identical. On the other hand, populations evolved in an aminoglycoside and ß-lactam combination exhibit decreased growth adaptation and resistant profiles that depend on the specific drug concentrations. We show that the main qualitative features of these evolutionary trajectories can be explained by simple rescaling arguments that correspond to geometric transformations of the two-drug growth response surfaces measured in ancestral cells. The analysis also reveals multiple examples where resistance profiles selected by drug combinations are nearly growth-optimized along a contour connecting profiles selected by the component drugs. Our results highlight trade-offs between drug interactions and resistance profiles during the evolution of multi-drug resistance and emphasize evolutionary benefits and disadvantages of particular drug pairs targeting enterococci.

Pervasive and diverse collateral sensitivity profiles inform optimal strategies to limit antibiotic resistance.

Evolved resistance to one antibiotic may be associated with "collateral" sensitivity to other drugs. Here, we provide an extensive quantitative characterization of collateral effects in Enterococcus faecalis, a gram-positive opportunistic pathogen. By combining parallel experimental evolution with high-throughput dose-response measurements, we measure phenotypic profiles of collateral sensitivity and resistance for a total of 900 mutant-drug combinations. We find that collateral effects are pervasive but difficult to predict because independent populations selected by the same drug can exhibit qualitatively different profiles of collateral sensitivity as well as markedly different fitness costs. Using whole-genome sequencing of evolved populations, we identified mutations in a number of known resistance determinants, including mutations in several genes previously linked with collateral sensitivity in other species. Although phenotypic drug sensitivity profiles show significant diversity, they cluster into statistically similar groups characterized by selecting drugs with similar mechanisms. To exploit the statistical structure in these resistance profiles, we develop a simple mathematical model based on a stochastic control process and use it to design optimal drug policies that assign a unique drug to every possible resistance profile. Stochastic simulations reveal that these optimal drug policies outperform intuitive cycling protocols by maintaining long-term sensitivity at the expense of short-term periods of high resistance. The approach reveals a new conceptual strategy for mitigating resistance by balancing short-term inhibition of pathogen growth with infrequent use of drugs intended to steer pathogen populations to a more vulnerable future state. Experiments in laboratory populations confirm that model-inspired sequences of four drugs reduce growth and slow adaptation relative to naive protocols involving the drugs alone, in pairwise cycles, or in a four-drug uniform cycle.

Using Selection by Nonantibiotic Stressors to Sensitize Bacteria to Antibiotics.

Evolutionary adaptation of bacteria to nonantibiotic selective forces, such as osmotic stress, has been previously associated with increased antibiotic resistance, but much less is known about potentially sensitizing effects of nonantibiotic stressors. In this study, we use laboratory evolution to investigate adaptation of Enterococcus faecalis, an opportunistic bacterial pathogen, to a broad collection of environmental agents, ranging from antibiotics and biocides to extreme pH and osmotic stress. We find that nonantibiotic selection frequently leads to increased sensitivity to other conditions, including multiple antibiotics. Using population sequencing and whole-genome sequencing of single isolates from the evolved populations, we identify multiple mutations in genes previously linked with resistance to the selecting conditions, including genes corresponding to known drug targets or multidrug efflux systems previously tied to collateral sensitivity. Finally, we hypothesized based on the measured sensitivity profiles that sequential rounds of antibiotic and nonantibiotic selection may lead to hypersensitive populations by harnessing the orthogonal collateral effects of particular pairs of selective forces. To test this hypothesis, we show experimentally that populations evolved to a sequence of linezolid (an oxazolidinone antibiotic) and sodium benzoate (a common preservative) exhibit increased sensitivity to more stressors than adaptation to either condition alone. The results demonstrate how sequential adaptation to drug and nondrug environments can be used to sensitize bacteria to antibiotics and highlight new potential strategies for exploiting shared constraints governing adaptation to diverse environmental challenges.

Population Density Modulates Drug Inhibition and Gives Rise to Potential Bistability of Treatment Outcomes for Bacterial Infections.

The inoculum effect (IE) is an increase in the minimum inhibitory concentration (MIC) of an antibiotic as a function of the initial size of a microbial population. The IE has been observed in a wide range of bacteria, implying that antibiotic efficacy may depend on population density. Such density dependence could have dramatic effects on bacterial population dynamics and potential treatment strategies, but explicit measures of per capita growth as a function of density are generally not available. Instead, the IE measures MIC as a function of initial population size, and population density changes by many orders of magnitude on the timescale of the experiment. Therefore, the functional relationship between population density and antibiotic inhibition is generally not known, leaving many questions about the impact of the IE on different treatment strategies unanswered. To address these questions, here we directly measured real-time per capita growth of Enterococcus faecalis populations exposed to antibiotic at fixed population densities using multiplexed computer-automated culture devices. We show that density-dependent growth inhibition is pervasive for commonly used antibiotics, with some drugs showing increased inhibition and others decreased inhibition at high densities. For several drugs, the density dependence is mediated by changes in extracellular pH, a community-level phenomenon not previously linked with the IE. Using a simple mathematical model, we demonstrate how this density dependence can modulate population dynamics in constant drug environments. Then, we illustrate how time-dependent dosing strategies can mitigate the negative effects of density-dependence. Finally, we show that these density effects lead to bistable treatment outcomes for a wide range of antibiotic concentrations in a pharmacological model of antibiotic treatment. As a result, infections exceeding a critical density often survive otherwise effective treatments.

Autofluorescence from NADH Conformations Associated with Different Metabolic Pathways Monitored Using Nanosecond-Gated Spectroscopy and Spectral Phasor Analysis.

Cellular NADH conformation is increasingly recognized as an endogenous optical biomarker and metabolic indicator. Recently, we reported a real-time approach for tracking metabolism on the basis of the quantification of UV-excited autofluorescence spectrum shape. Here, we use nanosecond-gated spectral acquisition, combined with spectrum-shape quantification, to monitor the long excited-state lifetime autofluorescence (usually associated with protein-bound NADH conformations) separately from the autofluorescence signal as a whole. We observe that the autofluorescence response induced by two NADH-oxidation inhibitors­cyanide and ethanol­are similar in Saccharomyces cerevisiae when monitored using time-integrated detection but easily distinguished using time-gated detection. Results are consistent with the observation of multiple NADH conformations as assessed using spectral phasor analysis. Further, because well-known oxidation inhibitors are used, changes in spectrum shape can be associated with NADH conformations involved in the different metabolic pathways, giving bioanalytic utility to the spectral responses.

Frequency-dependent ecological interactions increase the prevalence, and shape the distribution, of pre-existing drug resistance.

The evolution of resistance remains one of the primary challenges for modern medicine from infectious diseases to cancers. Many of these resistance-conferring mutations often carry a substantial fitness cost in the absence of treatment. As a result, we would expect these mutants to undergo purifying selection and be rapidly driven to extinction. Nevertheless, pre-existing resistance is frequently observed from drug-resistant malaria to targeted cancer therapies in non-small cell lung cancer (NSCLC) and melanoma. Solutions to this apparent paradox have taken several forms from spatial rescue to simple mutation supply arguments. Recently, in an evolved resistant NSCLC cell line, we found that frequency-dependent ecological interactions between ancestor and resistant mutant ameliorate the cost of resistance in the absence of treatment. Here, we hypothesize that frequency-dependent ecological interactions in general play a major role in the prevalence of pre-existing resistance. We combine numerical simulations with robust analytical approximations to provide a rigorous mathematical framework for studying the effects of frequency-dependent ecological interactions on the evolutionary dynamics of pre-existing resistance. First, we find that ecological interactions significantly expand the parameter regime under which we expect to observe pre-existing resistance. Next, even when positive ecological interactions between mutants and ancestors are rare, these resistant clones provide the primary mode of evolved resistance because even weak positive interaction leads to significantly longer extinction times. We then find that even in the case where mutation supply alone is sufficient to predict pre-existing resistance, frequency-dependent ecological forces still contribute a strong evolutionary pressure that selects for increasingly positive ecological effects (negative frequency-dependent selection). Finally, we genetically engineer several of the most common clinically observed resistance mechanisms to targeted therapies in NSCLC, a treatment notorious for pre-existing resistance. We find that each engineered mutant displays a positive ecological interaction with their ancestor. As a whole, these results suggest that frequency-dependent ecological effects can play a crucial role in shaping the evolutionary dynamics of pre-existing resistance.

The evolution of diverse antimicrobial responses in vancomycin-intermediate Staphylococcus aureus and its therapeutic implications.

Staphylococcus aureus causes endocarditis, osteomyelitis, and bacteremia. Clinicians often prescribe vancomycin as an empiric therapy to account for methicillin-resistant S. aureus (MRSA) and narrow treatment based on culture susceptibility results. However, these results reflect a single time point before empiric treatment and represent a limited subset of the total bacterial population within the patient. Thus, while they may indicate that the infection is susceptible to a particular drug, this recommendation may no longer be accurate during therapy. Here, we addressed how antibiotic susceptibility changes over time by accounting for evolution. We evolved 18 methicillin-susceptible S. aureus (MSSA) populations under increasing vancomycin concentrations until they reached intermediate resistance levels. Sequencing revealed parallel mutations that affect cell membrane stress response and cell-wall biosynthesis. The populations exhibited repeated cross-resistance to daptomycin and varied responses to meropenem, gentamicin, and nafcillin. We accounted for this variability by deriving likelihood estimates that express a population's probability of exhibiting a drug response following vancomycin treatment. Our results suggest antistaphylococcal penicillins are preferable first-line treatments for MSSA infections but also highlight the inherent uncertainty that evolution poses to effective therapies. Infections may take varied evolutionary paths; therefore, considering evolution as a probabilistic process should inform our therapeutic choices.

Drug dependence in cancer is exploitable by optimally constructed treatment holidays.

Cancers with acquired resistance to targeted therapy can become simultaneously dependent on the presence of the targeted therapy drug for survival, suggesting that intermittent therapy may slow resistance. However, relatively little is known about which tumours are likely to become dependent and how to schedule intermittent therapy optimally. Here we characterized drug dependence across a panel of over 75 MAPK-inhibitor-resistant BRAFV600E mutant melanoma models at the population and single-clone levels. Melanocytic differentiated models exhibited a much greater tendency to give rise to drug-dependent progeny than their dedifferentiated counterparts. Mechanistically, acquired loss of microphthalmia-associated transcription factor in differentiated melanoma models drives ERK-JunB-p21 signalling to enforce drug dependence. We identified the optimal scheduling of 'drug holidays' using simple mathematical models that we validated across short and long timescales. Without detailed knowledge of tumour characteristics, we found that a simple adaptive therapy protocol can produce near-optimal outcomes using only measurements of total population size. Finally, a spatial agent-based model showed that optimal schedules derived from exponentially growing cells in culture remain nearly optimal in the context of tumour cell turnover and limited environmental carrying capacity. These findings may guide the implementation of improved evolution-inspired treatment strategies for drug-dependent cancers.

Reinforcement Learning informs optimal treatment strategies to limit antibiotic resistance.

Antimicrobial resistance was estimated to be associated with 4.95 million deaths worldwide in 2019. It is possible to frame the antimicrobial resistance problem as a feedback-control problem. If we could optimize this feedback-control problem and translate our findings to the clinic, we could slow, prevent or reverse the development of high-level drug resistance. Prior work on this topic has relied on systems where the exact dynamics and parameters were known a priori. In this study, we extend this work using a reinforcement learning (RL) approach capable of learning effective drug cycling policies in a system defined by empirically measured fitness landscapes. Crucially, we show that is possible to learn effective drug cycling policies despite the problems of noisy, limited, or delayed measurement. Given access to a panel of 15 ß-lactam antibiotics with which to treat the simulated E. coli population, we demonstrate that RL agents outperform two naive treatment paradigms at minimizing the population fitness over time. We also show that RL agents approach the performance of the optimal drug cycling policy. Even when stochastic noise is introduced to the measurements of population fitness, we show that RL agents are capable of maintaining evolving populations at lower growth rates compared to controls. We further tested our approach in arbitrary fitness landscapes of up to 1024 genotypes. We show that minimization of population fitness using drug cycles is not limited by increasing genome size. Our work represents a proof-of-concept for using AI to control complex evolutionary processes.