Dissecting Hessian: Understanding Common Structure of Hessian in Neural
  Networks
  Hessian captures important properties of the deep neural network loss
landscape. Previous works have observed low rank structure in the Hessians of
neural networks. We make several new observations about the top eigenspace of
layer-wise Hessian: top eigenspaces for different models have surprisingly high
overlap, and top eigenvectors form low rank matrices when they are reshaped
into the same shape as the corresponding weight matrix. Towards formally
explaining such structures of the Hessian, we show that the new eigenspace
structure can be explained by approximating the Hessian using Kronecker
factorization; we also prove the low rank structure for random data at random
initialization for over-parametrized two-layer neural nets. Our new
understanding can explain why some of these structures become weaker when the
network is trained with batch normalization. The Kronecker factorization also
leads to better explicit generalization bounds.
