Benefits of Jointly Training Autoencoders: An Improved Neural Tangent
  Kernel Analysis
  A remarkable recent discovery in machine learning has been that deep neural
networks can achieve impressive performance (in terms of both lower training
error and higher generalization capacity) in the regime where they are
massively over-parameterized. Consequently, over the past year, the community
has devoted growing interest in analyzing optimization and generalization
properties of over-parameterized networks, and several breakthrough works have
led to important theoretical progress. However, the majority of existing work
only applies to supervised learning scenarios and hence are limited to settings
such as classification and regression. In contrast, the role of
over-parameterization in the unsupervised setting has gained far less
attention. In this paper, we study the gradient dynamics of two-layer
over-parameterized autoencoders with ReLU activation. We make very few
assumptions about the given training dataset (other than mild non-degeneracy
conditions). Starting from a randomly initialized autoencoder network, we
rigorously prove the linear convergence of gradient descent in two learning
regimes, namely: (i) the weakly-trained regime where only the encoder is
trained, and (ii) the jointly-trained regime where both the encoder and the
decoder are trained. Our results indicate the considerable benefits of joint
training over weak training for finding global optima, achieving a dramatic
decrease in the required level of over-parameterization. We also analyze the
case of weight-tied autoencoders (which is a commonly used architectural choice
in practical settings) and prove that in the over-parameterized setting,
training such networks from randomly initialized points leads to certain
unexpected degeneracies.
