Emergent Communication Pretraining for Few-Shot Machine Translation
  While state-of-the-art models that rely upon massively multilingual
pretrained encoders achieve sample efficiency in downstream applications, they
still require abundant amounts of unlabelled text. Nevertheless, most of the
world's languages lack such resources. Hence, we investigate a more radical
form of unsupervised knowledge transfer in the absence of linguistic data. In
particular, for the first time we pretrain neural networks via emergent
communication from referential games. Our key assumption is that grounding
communication on images---as a crude approximation of real-world
environments---inductively biases the model towards learning natural languages.
On the one hand, we show that this substantially benefits machine translation
in few-shot settings. On the other hand, this also provides an extrinsic
evaluation protocol to probe the properties of emergent languages ex vitro.
Intuitively, the closer they are to natural languages, the higher the gains
from pretraining on them should be. For instance, in this work we measure the
influence of communication success and maximum sequence length on downstream
performances. Finally, we introduce a customised adapter layer and annealing
strategies for the regulariser of maximum-a-posteriori inference during
fine-tuning. These turn out to be crucial to facilitate knowledge transfer and
prevent catastrophic forgetting. Compared to a recurrent baseline, our method
yields gains of $59.0\%$$\sim$$147.6\%$ in BLEU score with only $500$ NMT
training instances and $65.1\%$$\sim$$196.7\%$ with $1,000$ NMT training
instances across four language pairs. These proof-of-concept results reveal the
potential of emergent communication pretraining for both natural language
processing tasks in resource-poor settings and extrinsic evaluation of
artificial languages.
