To Regularize or Not To Regularize? The Bias Variance Trade-off in
  Regularized AEs
  Regularized Auto-Encoders (RAEs) form a rich class of neural generative
models. They effectively model the joint-distribution between the data and the
latent space using an Encoder-Decoder combination, with regularization imposed
in terms of a prior over the latent space. Despite their advantages, such as
stability in training, the performance of AE based models has not reached the
superior standards as that of the other generative models such as Generative
Adversarial Networks (GANs). Motivated by this, we examine the effect of the
latent prior on the generation quality of deterministic AE models in this
paper. Specifically, we consider the class of RAEs with deterministic
Encoder-Decoder pairs, Wasserstein Auto-Encoders (WAE), and show that having a
fixed prior distribution, \textit{a priori}, oblivious to the dimensionality of
the `true' latent space, will lead to the infeasibility of the optimization
problem considered. Further, we show that, in the finite data regime, despite
knowing the correct latent dimensionality, there exists a bias-variance
trade-off with any arbitrary prior imposition. As a remedy to both the issues
mentioned above, we introduce an additional state space in the form of flexibly
learnable latent priors, in the optimization objective of the WAEs. We
implicitly learn the distribution of the latent prior jointly with the AE
training, which not only makes the learning objective feasible but also
facilitates operation on different points of the bias-variance curve. We show
the efficacy of our model, called FlexAE, through several experiments on
multiple datasets, and demonstrate that it is the new state-of-the-art for the
AE based generative models.
