Coach-assisted Multi-Agent Reinforcement Learning Framework for
  Unexpected Crashed Agents
  Multi-agent reinforcement learning is difficult to be applied in practice,
which is partially due to the gap between the simulated and real-world
scenarios. One reason for the gap is that the simulated systems always assume
that the agents can work normally all the time, while in practice, one or more
agents may unexpectedly "crash" during the coordination process due to
inevitable hardware or software failures. Such crashes will destroy the
cooperation among agents, leading to performance degradation. In this work, we
present a formal formulation of a cooperative multi-agent reinforcement
learning system with unexpected crashes. To enhance the robustness of the
system to crashes, we propose a coach-assisted multi-agent reinforcement
learning framework, which introduces a virtual coach agent to adjust the crash
rate during training. We design three coaching strategies and the re-sampling
strategy for our coach agent. To the best of our knowledge, this work is the
first to study the unexpected crashes in the multi-agent system. Extensive
experiments on grid-world and StarCraft II micromanagement tasks demonstrate
the efficacy of adaptive strategy compared with the fixed crash rate strategy
and curriculum learning strategy. The ablation study further illustrates the
effectiveness of our re-sampling strategy.
