The Evidence Lower Bound of Variational Autoencoders Converges to a Sum
  of Three Entropies
  The central objective function of a variational autoencoder (VAE) is its
variational lower bound. Here we show that for standard VAEs the variational
bound converges to a value given by the sum of three entropies: the (negative)
entropy of the latent distribution, the expected (negative) entropy of the
observable distribution, and the average entropy of the variational
distributions. Our derived analytical results are exact and apply for small as
well as complex neural networks for decoder and encoder. Furthermore, they
apply for finitely and infinitely many data points and at any stationary point
(including local and global maxima). As a consequence, we show that the
variance parameters of encoder and decoder play the key role in determining the
values of variational bounds at stationary points. Furthermore, the obtained
results can allow for closed-form analytical expressions at points of
convergence, which may be unexpected as neither variational lower bounds of
VAEs nor log-likelihoods of VAEs are closed-form during learning. As our main
contribution, we provide the proofs for convergence of standard VAEs to sums of
entropies. Furthermore, we numerically verify our analytical results and
discuss some potential applications. The obtained equality to entropy sums
provides novel information on those points in parameter space that variational
learning converges to. As such, we believe, they can contribute to our
understanding of established as well as novel VAE approaches.
