Deep Autoencoders: From Understanding to Generalization Guarantees
  A big mystery in deep learning continues to be the ability of methods to
generalize when the number of model parameters is larger than the number of
training examples. In this work, we take a step towards a better understanding
of the underlying phenomena of Deep Autoencoders (AEs), a mainstream deep
learning solution for learning compressed, interpretable, and structured data
representations. In particular, we interpret how AEs approximate the data
manifold by exploiting their continuous piecewise affine structure. Our
reformulation of AEs provides new insights into their mapping, reconstruction
guarantees, as well as an interpretation of commonly used regularization
techniques. We leverage these findings to derive two new regularizations that
enable AEs to capture the inherent symmetry in the data. Our regularizations
leverage recent advances in the group of transformation learning to enable AEs
to better approximate the data manifold without explicitly defining the group
underlying the manifold. Under the assumption that the symmetry of the data can
be explained by a Lie group, we prove that the regularizations ensure the
generalization of the corresponding AEs. A range of experimental evaluations
demonstrate that our methods outperform other state-of-the-art regularization
techniques.
