Rate-Regularization and Generalization in VAEs
  Variational autoencoders optimize an objective that combines a reconstruction
loss (the distortion) and a KL term (the rate). The rate is an upper bound on
the mutual information, which is often interpreted as a regularizer that
controls the degree of compression. We here examine whether inclusion of the
rate also acts as an inductive bias that improves generalization. We perform
rate-distortion analyses that control the strength of the rate term, the
network capacity, and the difficulty of the generalization problem. Decreasing
the strength of the rate paradoxically improves generalization in most
settings, and reducing the mutual information typically leads to underfitting.
Moreover, we show that generalization continues to improve even after the
mutual information saturates, indicating that the gap on the bound (i.e. the KL
divergence relative to the inference marginal) affects generalization. This
suggests that the standard Gaussian prior is not an inductive bias that
typically aids generalization, prompting work to understand what choices of
priors improve generalization in VAEs.
