Exploring loss function topology with cyclical learning rates
  We present observations and discussion of previously unreported phenomena
discovered while training residual networks. The goal of this work is to better
understand the nature of neural networks through the examination of these new
empirical results. These behaviors were identified through the application of
Cyclical Learning Rates (CLR) and linear network interpolation. Among these
behaviors are counterintuitive increases and decreases in training loss and
instances of rapid training. For example, we demonstrate how CLR can produce
greater testing accuracy than traditional training despite using large learning
rates. Files to replicate these results are available at
https://github.com/lnsmith54/exploring-loss
