Discrete Auto-regressive Variational Attention Models for Text Modeling
  Variational autoencoders (VAEs) have been widely applied for text modeling.
In practice, however, they are troubled by two challenges: information
underrepresentation and posterior collapse. The former arises as only the last
hidden state of LSTM encoder is transformed into the latent space, which is
generally insufficient to summarize the data. The latter is a long-standing
problem during the training of VAEs as the optimization is trapped to a
disastrous local optimum. In this paper, we propose Discrete Auto-regressive
Variational Attention Model (DAVAM) to address the challenges. Specifically, we
introduce an auto-regressive variational attention approach to enrich the
latent space by effectively capturing the semantic dependency from the input.
We further design discrete latent space for the variational attention and
mathematically show that our model is free from posterior collapse. Extensive
experiments on language modeling tasks demonstrate the superiority of DAVAM
against several VAE counterparts.
