Avoiding Latent Variable Collapse With Generative Skip Models
  Variational autoencoders learn distributions of high-dimensional data. They
model data with a deep latent-variable model and then fit the model by
maximizing a lower bound of the log marginal likelihood. VAEs can capture
complex distributions, but they can also suffer from an issue known as "latent
variable collapse," especially if the likelihood model is powerful.
Specifically, the lower bound involves an approximate posterior of the latent
variables; this posterior "collapses" when it is set equal to the prior, i.e.,
when the approximate posterior is independent of the data. While VAEs learn
good generative models, latent variable collapse prevents them from learning
useful representations. In this paper, we propose a simple new way to avoid
latent variable collapse by including skip connections in our generative model;
these connections enforce strong links between the latent variables and the
likelihood function. We study generative skip models both theoretically and
empirically. Theoretically, we prove that skip models increase the mutual
information between the observations and the inferred latent variables.
Empirically, we study images (MNIST and Omniglot) and text (Yahoo). Compared to
existing VAE architectures, we show that generative skip models maintain
similar predictive performance but lead to less collapse and provide more
meaningful representations of the data.
