Decision-Making With Speculative Opponent Models.

ABSTRACT

Opponent modeling has proven effective in enhancing the decision-making of the controlled agent by constructing models of opponent agents. However, existing methods often rely on access to the observations and actions of opponents, a requirement that is infeasible when such information is either unobservable or challenging to obtain. To address this issue, we introduce distributional opponent-aided multiagent actor-critic (DOMAC), the first speculative opponent modeling algorithm that relies solely on local information (i.e., the controlled agent's observations, actions, and rewards). Specifically, the actor maintains a speculated belief about the opponents using the tailored speculative opponent models that predict the opponents' actions using only local information. Moreover, DOMAC features distributional critic models that estimate the return distribution of the actor's policy, yielding a more fine-grained assessment of the actor's quality. This thus more effectively guides the training of the speculative opponent models that the actor depends upon. Furthermore, we formally derive a policy gradient theorem with the proposed opponent models. Extensive experiments under eight different challenging multiagent benchmark tasks within the MPE, Pommerman, and starcraft multiagent challenge (SMAC) demonstrate that our DOMAC successfully models opponents' behaviors and delivers superior performance against state-of-the-art (SOTA) methods with a faster convergence speed.