A Random Matrix Theory Approach to Damping in Deep Learning
  We conjecture that the inherent difference in generalisation between adaptive
and non-adaptive gradient methods in deep learning stems from the increased
estimation noise in the flattest directions of the true loss surface. We
demonstrate that typical schedules used for adaptive methods (with low
numerical stability or damping constants) serve to bias relative movement
towards flat directions relative to sharp directions, effectively amplifying
the noise-to-signal ratio and harming generalisation. We further demonstrate
that the numerical damping constant used in these methods can be decomposed
into a learning rate reduction and linear shrinkage of the estimated curvature
matrix. We then demonstrate significant generalisation improvements by
increasing the shrinkage coefficient, closing the generalisation gap entirely
in both logistic regression and several deep neural network experiments.
Extending this line further, we develop a novel random matrix theory based
damping learner for second order optimiser inspired by linear shrinkage
estimation. We experimentally demonstrate our learner to be very insensitive to
the initialised value and to allow for extremely fast convergence in
conjunction with continued stable training and competitive generalisation.
