A Tutorial on VAEs: From Bayes' Rule to Lossless Compression
  The Variational Auto-Encoder (VAE) is a simple, efficient, and popular deep
maximum likelihood model. Though usage of VAEs is widespread, the derivation of
the VAE is not as widely understood. In this tutorial, we will provide an
overview of the VAE and a tour through various derivations and interpretations
of the VAE objective. From a probabilistic standpoint, we will examine the VAE
through the lens of Bayes' Rule, importance sampling, and the
change-of-variables formula. From an information theoretic standpoint, we will
examine the VAE through the lens of lossless compression and transmission
through a noisy channel. We will then identify two common misconceptions over
the VAE formulation and their practical consequences. Finally, we will
visualize the capabilities and limitations of VAEs using a code example (with
an accompanying Jupyter notebook) on toy 2D data.
