DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from
  Low-Dimensional Latents
  Diffusion Probabilistic models have been shown to generate state-of-the-art
results on several competitive image synthesis benchmarks but lack a
low-dimensional, interpretable latent space, and are slow at generation. On the
other hand, Variational Autoencoders (VAEs) typically have access to a
low-dimensional latent space but exhibit poor sample quality. Despite recent
advances, VAEs usually require high-dimensional hierarchies of the latent codes
to generate high-quality samples. We present DiffuseVAE, a novel generative
framework that integrates VAE within a diffusion model framework, and leverage
this to design a novel conditional parameterization for diffusion models. We
show that the resulting model can improve upon the unconditional diffusion
model in terms of sampling efficiency while also equipping diffusion models
with the low-dimensional VAE inferred latent code. Furthermore, we show that
the proposed model can generate high-resolution samples and exhibits synthesis
quality comparable to state-of-the-art models on standard benchmarks. Lastly,
we show that the proposed method can be used for controllable image synthesis
and also exhibits out-of-the-box capabilities for downstream tasks like image
super-resolution and denoising. For reproducibility, our source code is
publicly available at \url{https://github.com/kpandey008/DiffuseVAE}.
