On Implicit Regularization in $\beta$-VAEs
  While the impact of variational inference (VI) on posterior inference in a
fixed generative model is well-characterized, its role in regularizing a
learned generative model when used in variational autoencoders (VAEs) is poorly
understood. We study the regularizing effects of variational distributions on
learning in generative models from two perspectives. First, we analyze the role
that the choice of variational family plays in imparting uniqueness to the
learned model by restricting the set of optimal generative models. Second, we
study the regularization effect of the variational family on the local geometry
of the decoding model. This analysis uncovers the regularizer implicit in the
$\beta$-VAE objective, and leads to an approximation consisting of a
deterministic autoencoding objective plus analytic regularizers that depend on
the Hessian or Jacobian of the decoding model, unifying VAEs with recent
heuristics proposed for training regularized autoencoders. We empirically
verify these findings, observing that the proposed deterministic objective
exhibits similar behavior to the $\beta$-VAE in terms of objective value and
sample quality.
