Correcting Experience Replay for Multi-Agent Communication
  We consider the problem of learning to communicate using multi-agent
reinforcement learning (MARL). A common approach is to learn off-policy, using
data sampled from a replay buffer. However, messages received in the past may
not accurately reflect the current communication policy of each agent, and this
complicates learning. We therefore introduce a 'communication correction' which
accounts for the non-stationarity of observed communication induced by
multi-agent learning. It works by relabelling the received message to make it
likely under the communicator's current policy, and thus be a better reflection
of the receiver's current environment. To account for cases in which agents are
both senders and receivers, we introduce an ordered relabelling scheme. Our
correction is computationally efficient and can be integrated with a range of
off-policy algorithms. We find in our experiments that it substantially
improves the ability of communicating MARL systems to learn across a variety of
cooperative and competitive tasks.
