Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word
  Embeddings to Dictionaries
  Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon
induction (BLI). Recent CLWE methods use linear projections, which underfit the
training dictionary, to generalize on BLI. However, underfitting can hinder
generalization to other downstream tasks that rely on words from the training
dictionary. We address this limitation by retrofitting CLWE to the training
dictionary, which pulls training translation pairs closer in the embedding
space and overfits the training dictionary. This simple post-processing step
often improves accuracy on two downstream tasks, despite lowering BLI test
accuracy. We also retrofit to both the training dictionary and a synthetic
dictionary induced from CLWE, which sometimes generalizes even better on
downstream tasks. Our results confirm the importance of fully exploiting
training dictionary in downstream tasks and explains why BLI is a flawed CLWE
evaluation.
