Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them
  on Images
  We present a hierarchical VAE that, for the first time, generates samples
quickly while outperforming the PixelCNN in log-likelihood on all natural image
benchmarks. We begin by observing that, in theory, VAEs can actually represent
autoregressive models, as well as faster, better models if they exist, when
made sufficiently deep. Despite this, autoregressive models have historically
outperformed VAEs in log-likelihood. We test if insufficient depth explains why
by scaling a VAE to greater stochastic depth than previously explored and
evaluating it CIFAR-10, ImageNet, and FFHQ. In comparison to the PixelCNN,
these very deep VAEs achieve higher likelihoods, use fewer parameters, generate
samples thousands of times faster, and are more easily applied to
high-resolution images. Qualitative studies suggest this is because the VAE
learns efficient hierarchical visual representations. We release our source
code and models at https://github.com/openai/vdvae.
