Dispersed Exponential Family Mixture VAEs for Interpretable Text
  Generation
  Deep generative models are commonly used for generating images and text.
Interpretability of these models is one important pursuit, other than the
generation quality. Variational auto-encoder (VAE) with Gaussian distribution
as prior has been successfully applied in text generation, but it is hard to
interpret the meaning of the latent variable. To enhance the controllability
and interpretability, one can replace the Gaussian prior with a mixture of
Gaussian distributions (GM-VAE), whose mixture components could be related to
hidden semantic aspects of data. In this paper, we generalize the practice and
introduce DEM-VAE, a class of models for text generation using VAEs with a
mixture distribution of exponential family. Unfortunately, a standard
variational training algorithm fails due to the mode-collapse problem. We
theoretically identify the root cause of the problem and propose an effective
algorithm to train DEM-VAE. Our method penalizes the training with an extra
dispersion term to induce a well-structured latent space. Experimental results
show that our approach does obtain a meaningful space, and it outperforms
strong baselines in text generation benchmarks. The code is available at
https://github.com/wenxianxian/demvae.
