A Latent Variable Model Approach to PMI-based Word Embeddings
  Semantic word embeddings represent the meaning of a word via a vector, and
are created by diverse methods. Many use nonlinear operations on co-occurrence
statistics, and have hand-tuned hyperparameters and reweighting methods.
  This paper proposes a new generative model, a dynamic version of the
log-linear topic model of~\citet{mnih2007three}. The methodological novelty is
to use the prior to compute closed form expressions for word statistics. This
provides a theoretical justification for nonlinear models like PMI, word2vec,
and GloVe, as well as some hyperparameter choices. It also helps explain why
low-dimensional semantic embeddings contain linear algebraic structure that
allows solution of word analogies, as shown by~\citet{mikolov2013efficient} and
many subsequent papers.
  Experimental support is provided for the generative model assumptions, the
most important of which is that latent word vectors are fairly uniformly
dispersed in space.
