VAE Approximation Error: ELBO and Exponential Families
  The importance of Variational Autoencoders reaches far beyond standalone
generative models -- the approach is also used for learning latent
representations and can be generalized to semi-supervised learning. This
requires a thorough analysis of their commonly known shortcomings: posterior
collapse and approximation errors. This paper analyzes VAE approximation errors
caused by the combination of the ELBO objective and encoder models from
conditional exponential families, including, but not limited to, commonly used
conditionally independent discrete and continuous models. We characterize
subclasses of generative models consistent with these encoder families. We show
that the ELBO optimizer is pulled away from the likelihood optimizer towards
the consistent subset and study this effect experimentally. Importantly, this
subset can not be enlarged, and the respective error cannot be decreased, by
considering deeper encoder/decoder networks.
