Conditional Imitation Learning for Multi-Agent Games
  While advances in multi-agent learning have enabled the training of
increasingly complex agents, most existing techniques produce a final policy
that is not designed to adapt to a new partner's strategy. However, we would
like our AI agents to adjust their strategy based on the strategies of those
around them. In this work, we study the problem of conditional multi-agent
imitation learning, where we have access to joint trajectory demonstrations at
training time, and we must interact with and adapt to new partners at test
time. This setting is challenging because we must infer a new partner's
strategy and adapt our policy to that strategy, all without knowledge of the
environment reward or dynamics. We formalize this problem of conditional
multi-agent imitation learning, and propose a novel approach to address the
difficulties of scalability and data scarcity. Our key insight is that
variations across partners in multi-agent games are often highly structured,
and can be represented via a low-rank subspace. Leveraging tools from tensor
decomposition, our model learns a low-rank subspace over ego and partner agent
strategies, then infers and adapts to a new partner strategy by interpolating
in the subspace. We experiments with a mix of collaborative tasks, including
bandits, particle, and Hanabi environments. Additionally, we test our
conditional policies against real human partners in a user study on the
Overcooked game. Our model adapts better to new partners compared to baselines,
and robustly handles diverse settings ranging from discrete/continuous actions
and static/online evaluation with AI/human partners.
