A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
  When trained effectively, the Variational Autoencoder (VAE) is both a
powerful language model and an effective representation learning framework. In
practice, however, VAEs are trained with the evidence lower bound (ELBO) as a
surrogate objective to the intractable marginal data likelihood. This approach
to training yields unstable results, frequently leading to a disastrous local
optimum known as posterior collapse. In this paper, we investigate a simple fix
for posterior collapse which yields surprisingly effective results. The
combination of two known heuristics, previously considered only in isolation,
substantially improves held-out likelihood, reconstruction, and latent
representation learning when compared with previous state-of-the-art methods.
More interestingly, while our experiments demonstrate superiority on these
principle evaluations, our method obtains a worse ELBO. We use these results to
argue that the typical surrogate objective for VAEs may not be sufficient or
necessarily appropriate for balancing the goals of representation learning and
data distribution modeling.
