Informative Policy Representations in Multi-Agent Reinforcement Learning
  via Joint-Action Distributions
  In multi-agent reinforcement learning, the inherent non-stationarity of the
environment caused by other agents' actions posed significant difficulties for
an agent to learn a good policy independently. One way to deal with
non-stationarity is agent modeling, by which the agent takes into consideration
the influence of other agents' policies. Most existing work relies on
predicting other agents' actions or goals, or discriminating between their
policies. However, such modeling fails to capture the similarities and
differences between policies simultaneously and thus cannot provide useful
information when generalizing to unseen policies. To address this, we propose a
general method to learn representations of other agents' policies via the
joint-action distributions sampled in interactions. The similarities and
differences between policies are naturally captured by the policy distance
inferred from the joint-action distributions and deliberately reflected in the
learned representations. Agents conditioned on the policy representations can
well generalize to unseen agents. We empirically demonstrate that our method
outperforms existing work in multi-agent tasks when facing unseen agents.
