A Generative Word Embedding Model and its Low Rank Positive Semidefinite
  Solution
  Most existing word embedding methods can be categorized into Neural Embedding
Models and Matrix Factorization (MF)-based methods. However some models are
opaque to probabilistic interpretation, and MF-based methods, typically solved
using Singular Value Decomposition (SVD), may incur loss of corpus information.
In addition, it is desirable to incorporate global latent factors, such as
topics, sentiments or writing styles, into the word embedding model. Since
generative models provide a principled way to incorporate latent factors, we
propose a generative word embedding model, which is easy to interpret, and can
serve as a basis of more sophisticated latent factor models. The model
inference reduces to a low rank weighted positive semidefinite approximation
problem. Its optimization is approached by eigendecomposition on a submatrix,
followed by online blockwise regression, which is scalable and avoids the
information loss in SVD. In experiments on 7 common benchmark datasets, our
vectors are competitive to word2vec, and better than other MF-based methods.
