On the interaction between supervision and self-play in emergent
  communication
  A promising approach for teaching artificial agents to use natural language
involves using human-in-the-loop training. However, recent work suggests that
current machine learning methods are too data inefficient to be trained in this
way from scratch. In this paper, we investigate the relationship between two
categories of learning signals with the ultimate goal of improving sample
efficiency: imitating human language data via supervised learning, and
maximizing reward in a simulated multi-agent environment via self-play (as done
in emergent communication), and introduce the term supervised self-play (S2P)
for algorithms using both of these signals. We find that first training agents
via supervised learning on human data followed by self-play outperforms the
converse, suggesting that it is not beneficial to emerge languages from
scratch. We then empirically investigate various S2P schedules that begin with
supervised learning in two environments: a Lewis signaling game with symbolic
inputs, and an image-based referential game with natural language descriptions.
Lastly, we introduce population based approaches to S2P, which further improves
the performance over single-agent methods.
