The Multiscale Structure of Neural Network Loss Functions: The Effect on
  Optimization and Origin
  Local quadratic approximation has been extensively used to study the
optimization of neural network loss functions around the minimum. Though, it
usually holds in a very small neighborhood of the minimum, and cannot explain
many phenomena observed during the optimization process. In this work, we study
the structure of neural network loss functions and its implication on
optimization in a region beyond the reach of good quadratic approximation.
Numerically, we observe that neural network loss functions possesses a
multiscale structure, manifested in two ways: (1) in a neighborhood of minima,
the loss mixes a continuum of scales and grows subquadratically, and (2) in a
larger region, the loss shows several separate scales clearly. Using the
subquadratic growth, we are able to explain the Edge of Stability phenomenon[4]
observed for gradient descent (GD) method. Using the separate scales, we
explain the working mechanism of learning rate decay by simple examples.
Finally, we study the origin of the multiscale structure and propose that the
non-uniformity of training data is one of its cause. By constructing a
two-layer neural network problem we show that training data with different
magnitudes give rise to different scales of the loss function, producing
subquadratic growth or multiple separate scales.
