Model enhanced reinforcement learning to enable precision dosing: A theoretical case study with dosing of propofol.

ABSTRACT

Extending the potential of precision dosing requires evaluating methodologies offering more flexibility and higher degree of personalization. Reinforcement learning (RL) holds promise in its ability to integrate multidimensional data in an adaptive process built toward efficient decision making centered on sustainable value creation. For general anesthesia in intensive care units, RL is applied and automatically adjusts dosing through monitoring of patient's consciousness. We further explore the problem of optimal control of anesthesia with propofol by combining RL with state-of-the-art tools used to inform dosing in drug development. In particular, we used pharmacokinetic-pharmacodynamic (PK-PD) modeling as a simulation engine to generate experience from dosing scenarios, which cannot be tested experimentally. Through simulations, we show that, when learning from retrospective trial data, more than 100 patients are needed to reach an accuracy within the range of what is achieved with a standard dosing solution. However, embedding a model of drug effect within the RL algorithm improves accuracy by reducing errors to target by 90% through learning to take dosing actions maximizing long-term benefit. Data residual variability impacts accuracy while the algorithm efficiently coped with up to 50% interindividual variability in the PK and 25% in the PD model's parameters. We illustrate how extending the state definition of the RL agent with meaningful variables is key to achieve high accuracy of optimal dosing policy. These results suggest that RL constitutes an attractive approach for precision dosing when rich data are available or when complemented with synthetic data from model-based tools used in model-informed drug development.