Preventing Posterior Collapse with delta-VAEs
  Due to the phenomenon of "posterior collapse," current latent variable
generative models pose a challenging design choice that either weakens the
capacity of the decoder or requires augmenting the objective so it does not
only maximize the likelihood of the data. In this paper, we propose an
alternative that utilizes the most powerful generative models as decoders,
whilst optimising the variational lower bound all while ensuring that the
latent variables preserve and encode useful information. Our proposed
$\delta$-VAEs achieve this by constraining the variational family for the
posterior to have a minimum distance to the prior. For sequential latent
variable models, our approach resembles the classic representation learning
approach of slow feature analysis. We demonstrate the efficacy of our approach
at modeling text on LM1B and modeling images: learning representations,
improving sample quality, and achieving state of the art log-likelihood on
CIFAR-10 and ImageNet $32\times 32$.
