The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD
  Training and Sample Size
  We apply state-of-the-art tools in modern high-dimensional numerical linear
algebra to approximate efficiently the spectrum of the Hessian of modern
deepnets, with tens of millions of parameters, trained on real data. Our
results corroborate previous findings, based on small-scale networks, that the
Hessian exhibits "spiked" behavior, with several outliers isolated from a
continuous bulk. We decompose the Hessian into different components and study
the dynamics with training and sample size of each term individually.
