Grandmaster level in StarCraft II using multi-agent reinforcement learning.

Many real-world applications require artificial agents to compete and coordinate with other agents in complex environments. As a stepping stone to this goal, the domain of StarCraft has emerged as an important challenge for artificial intelligence research, owing to its iconic and enduring status among the most difficult professional esports and its relevance to the real world in terms of its raw complexity and multi-agent challenges. Over the course of a decade and numerous competitions1-3, the strongest agents have simplified important aspects of the game, utilized superhuman capabilities, or employed hand-crafted sub-systems4. Despite these advantages, no previous agent has come close to matching the overall skill of top StarCraft players. We chose to address the challenge of StarCraft using general-purpose learning methods that are in principle applicable to other complex domains: a multi-agent reinforcement learning algorithm that uses data from both human and agent games within a diverse league of continually adapting strategies and counter-strategies, each represented by deep neural networks5,6. We evaluated our agent, AlphaStar, in the full game of StarCraft II, through a series of online games against human players. AlphaStar was rated at Grandmaster level for all three StarCraft races and above 99.8% of officially ranked human players.

Learning agents that acquire representations of social groups.

Humans are learning agents that acquire social group representations from experience. Here, we discuss how to construct artificial agents capable of this feat. One approach, based on deep reinforcement learning, allows the necessary representations to self-organize. This minimizes the need for hand-engineering, improving robustness and scalability. It also enables "virtual neuroscience" research on the learned representations.

What is the simplest model that can account for high-fidelity imitation?

What inductive biases must be incorporated into multi-agent artificial intelligence models to get them to capture high-fidelity imitation? We think very little is needed. In the right environments, both instrumental- and ritual-stance imitation can emerge from generic learning mechanisms operating on non-deliberative decision architectures. In this view, imitation emerges from trial-and-error learning and does not require explicit deliberation.

The synaptic representation of sound source location in auditory cortex.

A key function of the auditory system is to provide reliable information about the location of sound sources. Here, we describe how sound location is represented by synaptic input arriving onto pyramidal cells within auditory cortex by combining free-field acoustic stimulation in the frontal azimuthal plane with in vivo whole-cell recordings. We found that subthreshold activity was panoramic in that EPSPs could be evoked from all locations in all cells. Regardless of the sound location that evoked the largest EPSP, we observed a slowing in the EPSP slope along the contralateral-ipsilateral plane that was reflected in a temporal sequence of peak EPSP times. Contralateral sounds evoked EPSPs with earlier peak times and consequently generated action potential firing with shorter latencies than ipsilateral sounds. Thus, whereas spiking probability reflected the region of space evoking the largest EPSP, across the population, synaptic inputs enforced a gradient of spike latency and precision along the horizontal axis. Therefore, within auditory cortex and regardless of preferred location, the time window of synaptic integration reflects sound source location and ensures that spatial acoustic information is represented by relative timings of pyramidal cell output.

Low-frequency envelope sensitivity produces asymmetric binaural tuning curves.

Neurons in the auditory midbrain are sensitive to differences in the timing of sounds at the two ears--an important sound localization cue. We used broadband noise stimuli to investigate the interaural-delay sensitivity of low-frequency neurons in two midbrain nuclei: the inferior colliculus (IC) and the dorsal nucleus of the lateral lemniscus. Noise-delay functions showed asymmetries not predicted from a linear dependence on interaural correlation: a stretching along the firing-rate dimension (rate asymmetry), and a skewing along the interaural-delay dimension (delay asymmetry). These asymmetries were produced by an envelope-sensitive component to the response that could not entirely be accounted for by monaural or binaural nonlinearities, instead indicating an enhancement of envelope sensitivity at or after the level of the superior olivary complex. In IC, the skew-like asymmetry was consistent with intermediate-type responses produced by the convergence of ipsilateral peak-type inputs and contralateral trough-type inputs. This suggests a stereotyped pattern of input to the IC. In the course of this analysis, we were also able to determine the contribution of time and phase components to neurons' internal delays. These findings have important consequences for the neural representation of interaural timing differences and interaural correlation-cues critical to the perception of acoustic space.