Learning Multi-Sense Word Distributions using Approximate
  Kullback-Leibler Divergence
  Learning word representations has garnered greater attention in the recent
past due to its diverse text applications. Word embeddings encapsulate the
syntactic and semantic regularities of sentences. Modelling word embedding as
multi-sense gaussian mixture distributions, will additionally capture
uncertainty and polysemy of words. We propose to learn the Gaussian mixture
representation of words using a Kullback-Leibler (KL) divergence based
objective function. The KL divergence based energy function provides a better
distance metric which can effectively capture entailment and distribution
similarity among the words. Due to the intractability of KL divergence for
Gaussian mixture, we go for a KL approximation between Gaussian mixtures. We
perform qualitative and quantitative experiments on benchmark word similarity
and entailment datasets which demonstrate the effectiveness of the proposed
approach.
