EigenNoise: A Contrastive Prior to Warm-Start Representations
  In this work, we present a naive initialization scheme for word vectors based
on a dense, independent co-occurrence model and provide preliminary results
that suggest it is competitive and warrants further investigation.
Specifically, we demonstrate through information-theoretic minimum description
length (MDL) probing that our model, EigenNoise, can approach the performance
of empirically trained GloVe despite the lack of any pre-training data (in the
case of EigenNoise). We present these preliminary results with interest to set
the stage for further investigations into how this competitive initialization
works without pre-training data, as well as to invite the exploration of more
intelligent initialization schemes informed by the theory of harmonic
linguistic structure. Our application of this theory likewise contributes a
novel (and effective) interpretation of recent discoveries which have
elucidated the underlying distributional information that linguistic
representations capture from data and contrast distributions.
