Gaussian Hierarchical Latent Dirichlet Allocation: Bringing Polysemy
  Back
  Topic models are widely used to discover the latent representation of a set
of documents. The two canonical models are latent Dirichlet allocation, and
Gaussian latent Dirichlet allocation, where the former uses multinomial
distributions over words, and the latter uses multivariate Gaussian
distributions over pre-trained word embedding vectors as the latent topic
representations, respectively. Compared with latent Dirichlet allocation,
Gaussian latent Dirichlet allocation is limited in the sense that it does not
capture the polysemy of a word such as ``bank.'' In this paper, we show that
Gaussian latent Dirichlet allocation could recover the ability to capture
polysemy by introducing a hierarchical structure in the set of topics that the
model can use to represent a given document. Our Gaussian hierarchical latent
Dirichlet allocation significantly improves polysemy detection compared with
Gaussian-based models and provides more parsimonious topic representations
compared with hierarchical latent Dirichlet allocation. Our extensive
quantitative experiments show that our model also achieves better topic
coherence and held-out document predictive accuracy over a wide range of corpus
and word embedding vectors.
