On generalization bounds for deep networks based on loss surface
  implicit regularization
  The classical statistical learning theory says that fitting too many
parameters leads to overfitting and poor performance. That modern deep neural
networks generalize well despite a large number of parameters contradicts this
finding and constitutes a major unsolved problem towards explaining the success
of deep learning. The implicit regularization induced by stochastic gradient
descent (SGD) has been regarded to be important, but its specific principle is
still unknown. In this work, we study how the local geometry of the energy
landscape around local minima affects the statistical properties of SGD with
Gaussian gradient noise. We argue that under reasonable assumptions, the local
geometry forces SGD to stay close to a low dimensional subspace and that this
induces implicit regularization and results in tighter bounds on the
generalization error for deep neural networks. To derive generalization error
bounds for neural networks, we first introduce a notion of stagnation sets
around the local minima and impose a local essential convexity property of the
population risk. Under these conditions, lower bounds for SGD to remain in
these stagnation sets are derived. If stagnation occurs, we derive a bound on
the generalization error of deep neural networks involving the spectral norms
of the weight matrices but not the number of network parameters. Technically,
our proofs are based on controlling the change of parameter values in the SGD
iterates and local uniform convergence of the empirical loss functions based on
the entropy of suitable neighborhoods around local minima. Our work attempts to
better connect non-convex optimization and generalization analysis with uniform
convergence.
