Obtaining Better Static Word Embeddings Using Contextual Embedding
  Models
  The advent of contextual word embeddings -- representations of words which
incorporate semantic and syntactic information from their context -- has led to
tremendous improvements on a wide variety of NLP tasks. However, recent
contextual models have prohibitively high computational cost in many use-cases
and are often hard to interpret. In this work, we demonstrate that our proposed
distillation method, which is a simple extension of CBOW-based training, allows
to significantly improve computational efficiency of NLP applications, while
outperforming the quality of existing static embeddings trained from scratch as
well as those distilled from previously proposed methods. As a side-effect, our
approach also allows a fair comparison of both contextual and static embeddings
via standard lexical evaluation tasks.
