A New Formalism, Method and Open Issues for Zero-Shot Coordination
  In many coordination problems, independently reasoning humans are able to
discover mutually compatible policies. In contrast, independently trained
self-play policies are often mutually incompatible. Zero-shot coordination
(ZSC) has recently been proposed as a new frontier in multi-agent reinforcement
learning to address this fundamental issue. Prior work approaches the ZSC
problem by assuming players can agree on a shared learning algorithm but not on
labels for actions and observations, and proposes other-play as an optimal
solution. However, until now, this "label-free" problem has only been
informally defined. We formalize this setting as the label-free coordination
(LFC) problem by defining the label-free coordination game. We show that
other-play is not an optimal solution to the LFC problem as it fails to
consistently break ties between incompatible maximizers of the other-play
objective. We introduce an extension of the algorithm, other-play with
tie-breaking, and prove that it is optimal in the LFC problem and an
equilibrium in the LFC game. Since arbitrary tie-breaking is precisely what the
ZSC setting aims to prevent, we conclude that the LFC problem does not reflect
the aims of ZSC. To address this, we introduce an alternative informal
operationalization of ZSC as a starting point for future work.
