Theory of Deep Learning III: explaining the non-overfitting puzzle
  A main puzzle of deep networks revolves around the absence of overfitting
despite large overparametrization and despite the large capacity demonstrated
by zero training error on randomly labeled data. In this note, we show that the
dynamics associated to gradient descent minimization of nonlinear networks is
topologically equivalent, near the asymptotically stable minima of the
empirical error, to linear gradient system in a quadratic potential with a
degenerate (for square loss) or almost degenerate (for logistic or crossentropy
loss) Hessian. The proposition depends on the qualitative theory of dynamical
systems and is supported by numerical results. Our main propositions extend to
deep nonlinear networks two properties of gradient descent for linear networks,
that have been recently established (1) to be key to their generalization
properties: 1. Gradient descent enforces a form of implicit regularization
controlled by the number of iterations, and asymptotically converges to the
minimum norm solution for appropriate initial conditions of gradient descent.
This implies that there is usually an optimum early stopping that avoids
overfitting of the loss. This property, valid for the square loss and many
other loss functions, is relevant especially for regression. 2. For
classification, the asymptotic convergence to the minimum norm solution implies
convergence to the maximum margin solution which guarantees good classification
error for "low noise" datasets. This property holds for loss functions such as
the logistic and cross-entropy loss independently of the initial conditions.
The robustness to overparametrization has suggestive implications for the
robustness of the architecture of deep convolutional networks with respect to
the curse of dimensionality.
