Provable Super-Convergence with a Large Cyclical Learning Rate
  Conventional wisdom dictates that learning rate should be in the stable
regime so that gradient-based algorithms don't blow up. This letter introduces
a simple scenario where an unstably large learning rate scheme leads to a super
fast convergence, with the convergence rate depending only logarithmically on
the condition number of the problem. Our scheme uses a Cyclical Learning Rate
(CLR) where we periodically take one large unstable step and several small
stable steps to compensate for the instability. These findings also help
explain the empirical observations of [Smith and Topin, 2019] where they show
that CLR with a large maximum learning rate can dramatically accelerate
learning and lead to so-called "super-convergence". We prove that our scheme
excels in the problems where Hessian exhibits a bimodal spectrum and the
eigenvalues can be grouped into two clusters (small and large). The unstably
large step is the key to enabling fast convergence over the small
eigen-spectrum.
