Towards Explaining the Regularization Effect of Initial Large Learning
  Rate in Training Neural Networks
  Stochastic gradient descent with a large initial learning rate is widely used
for training modern neural net architectures. Although a small initial learning
rate allows for faster training and better test performance initially, the
large learning rate achieves better generalization soon after the learning rate
is annealed. Towards explaining this phenomenon, we devise a setting in which
we can prove that a two layer network trained with large initial learning rate
and annealing provably generalizes better than the same network trained with a
small learning rate from the start. The key insight in our analysis is that the
order of learning different types of patterns is crucial: because the small
learning rate model first memorizes easy-to-generalize, hard-to-fit patterns,
it generalizes worse on hard-to-generalize, easier-to-fit patterns than its
large learning rate counterpart. This concept translates to a larger-scale
setting: we demonstrate that one can add a small patch to CIFAR-10 images that
is immediately memorizable by a model with small initial learning rate, but
ignored by the model with large learning rate until after annealing. Our
experiments show that this causes the small learning rate model's accuracy on
unmodified images to suffer, as it relies too much on the patch early on.
