How noise affects the Hessian spectrum in overparameterized neural
  networks
  Stochastic gradient descent (SGD) forms the core optimization method for deep
neural networks. While some theoretical progress has been made, it still
remains unclear why SGD leads the learning dynamics in overparameterized
networks to solutions that generalize well. Here we show that for
overparameterized networks with a degenerate valley in their loss landscape,
SGD on average decreases the trace of the Hessian of the loss. We also
generalize this result to other noise structures and show that isotropic noise
in the non-degenerate subspace of the Hessian decreases its determinant. In
addition to explaining SGDs role in sculpting the Hessian spectrum, this opens
the door to new optimization approaches that may confer better generalization
performance. We test our results with experiments on toy models and deep neural
networks.
