Representing Mixtures of Word Embeddings with Mixtures of Topic
  Embeddings
  A topic model is often formulated as a generative model that explains how
each word of a document is generated given a set of topics and
document-specific topic proportions. It is focused on capturing the word
co-occurrences in a document and hence often suffers from poor performance in
analyzing short documents. In addition, its parameter estimation often relies
on approximate posterior inference that is either not scalable or suffers from
large approximation error. This paper introduces a new topic-modeling framework
where each document is viewed as a set of word embedding vectors and each topic
is modeled as an embedding vector in the same embedding space. Embedding the
words and topics in the same vector space, we define a method to measure the
semantic difference between the embedding vectors of the words of a document
and these of the topics, and optimize the topic embeddings to minimize the
expected difference over all documents. Experiments on text analysis
demonstrate that the proposed method, which is amenable to mini-batch
stochastic gradient descent based optimization and hence scalable to big
corpora, provides competitive performance in discovering more coherent and
diverse topics and extracting better document representations.
