Outracing champion Gran Turismo drivers with deep reinforcement learning.

Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits1. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world's best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing's important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world's best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.

Estimating naloxone need in the USA across fentanyl, heroin, and prescription opioid epidemics: a modelling study.

BACKGROUND: The US overdose crisis is driven by fentanyl, heroin, and prescription opioids. One evidence-based policy response has been to broaden naloxone distribution, but how much naloxone a community would need to reduce the incidence of fatal overdose is unclear. We aimed to estimate state-level US naloxone need in 2017 across three main naloxone access points (community-based programmes, provider prescription, and pharmacy-initiated distribution) and by dominant opioid epidemic type (fentanyl, heroin, and prescription opioid). METHODS: In this modelling study, we developed, parameterised, and applied a mechanistic model of risk of opioid overdose and used it to estimate the expected reduction in opioid overdose mortality after deployment of a given number of two-dose naloxone kits. We performed a literature review and used a modified-Delphi panel to inform parameter definitions. We refined an established model of the population at risk of overdose by incorporating changes in the toxicity of the illicit drug supply and in the naloxone access point, then calibrated the model to 2017 using data obtained from proprietary data sources, state health departments, and national surveys for 12 US states that were representative of each epidemic type. We used counterfactual modelling to project the effect of increased naloxone distribution on the estimated number of opioid overdose deaths averted with naloxone and the number of naloxone kits needed to be available for at least 80% of witnessed opioid overdoses, by US state and access point. FINDINGS: Need for naloxone differed by epidemic type, with fentanyl epidemics having the consistently highest probability of naloxone use during witnessed overdose events (range 58-76% across the three modelled states in this category) and prescription opioid-dominated epidemics having the lowest (range 0-20%). Overall, in 2017, community-based and pharmacy-initiated naloxone access points had higher probability of naloxone use in witnessed overdose and higher numbers of deaths averted per 100 000 people in state-specific results with these two access points than with provider-prescribed access only. To achieve a target of naloxone use in 80% of witnessed overdoses, need varied from no additional kits (estimated as sufficient) to 1270 kits needed per 100 000 population across the 12 modelled states annually. In 2017, only Arizona had sufficient kits to meet this target. INTERPRETATION: Opioid epidemic type and how naloxone is accessed have large effects on the number of naloxone kits that need to be distributed, the probability of naloxone use, and the number of deaths due to overdose averted. The extent of naloxone distribution, especially through community-based programmes and pharmacy-initiated access points, warrants substantial expansion in nearly every US state. FUNDING: National Institute of Health, National Institute on Drug Abuse.