The dynamics of representation learning in shallow, non-linear
  autoencoders
  Autoencoders are the simplest neural network for unsupervised learning, and
thus an ideal framework for studying feature learning. While a detailed
understanding of the dynamics of linear autoencoders has recently been
obtained, the study of non-linear autoencoders has been hindered by the
technical difficulty of handling training data with non-trivial correlations -
a fundamental prerequisite for feature extraction. Here, we study the dynamics
of feature learning in non-linear, shallow autoencoders. We derive a set of
asymptotically exact equations that describe the generalisation dynamics of
autoencoders trained with stochastic gradient descent (SGD) in the limit of
high-dimensional inputs. These equations reveal that autoencoders learn the
leading principal components of their inputs sequentially. An analysis of the
long-time dynamics explains the failure of sigmoidal autoencoders to learn with
tied weights, and highlights the importance of training the bias in ReLU
autoencoders. Building on previous results for linear networks, we analyse a
modification of the vanilla SGD algorithm which allows learning of the exact
principal components. Finally, we show that our equations accurately describe
the generalisation dynamics of non-linear autoencoders on realistic datasets
such as CIFAR10.
