VAEBM: A Symbiosis between Variational Autoencoders and Energy-based
  Models
  Energy-based models (EBMs) have recently been successful in representing
complex distributions of small images. However, sampling from them requires
expensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high
dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate
samples quickly and are equipped with a latent space that enables fast
traversal of the data manifold. However, VAEs tend to assign high probability
density to regions in data space outside the actual data distribution and often
fail at generating sharp images. In this paper, we propose VAEBM, a symbiotic
composition of a VAE and an EBM that offers the best of both worlds. VAEBM
captures the overall mode structure of the data distribution using a
state-of-the-art VAE and it relies on its EBM component to explicitly exclude
non-data-like regions from the model and refine the image samples. Moreover,
the VAE component in VAEBM allows us to speed up MCMC updates by
reparameterizing them in the VAE's latent space. Our experimental results show
that VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on
several benchmark image datasets by a large margin. It can generate
high-quality images as large as 256$\times$256 pixels with short MCMC chains.
We also demonstrate that VAEBM provides complete mode coverage and performs
well in out-of-distribution detection. The source code is available at
https://github.com/NVlabs/VAEBM
