Study on the Large Batch Size Training of Neural Networks Based on the
  Second Order Gradient
  Large batch size training in deep neural networks (DNNs) possesses a
well-known 'generalization gap' that remarkably induces generalization
performance degradation. However, it remains unclear how varying batch size
affects the structure of a NN. Here, we combine theory with experiments to
explore the evolution of the basic structural properties, including gradient,
parameter update step length, and loss update step length of NNs under varying
batch sizes. We provide new guidance to improve generalization, which is
further verified by two designed methods involving discarding small-loss
samples and scheduling batch size. A curvature-based learning rate (CBLR)
algorithm is proposed to better fit the curvature variation, a sensitive factor
affecting large batch size training, across layers in a NN. As an approximation
of CBLR, the median-curvature LR (MCLR) algorithm is found to gain comparable
performance to Layer-wise Adaptive Rate Scaling (LARS) algorithm. Our
theoretical results and algorithm offer geometry-based explanations to the
existing studies. Furthermore, we demonstrate that the layer wise LR
algorithms, for example LARS, can be regarded as special instances of CBLR.
Finally, we deduce a theoretical geometric picture of large batch size
training, and show that all the network parameters tend to center on their
related minima.
