Gradient-only line searches: An Alternative to Probabilistic Line
  Searches
  Step sizes in neural network training are largely determined using
predetermined rules such as fixed learning rates and learning rate schedules.
These require user input or expensive global optimization strategies to
determine their functional form and associated hyperparameters. Line searches
are capable of adaptively resolving learning rate schedules. However, due to
discontinuities induced by mini-batch sub-sampling, they have largely fallen
out of favour. Notwithstanding, probabilistic line searches, which use
statistical surrogates over a limited spatial domain, have recently
demonstrated viability in resolving learning rates for stochastic loss
functions.
  This paper introduces an alternative paradigm, Gradient-Only Line Searches
that are Inexact (GOLS-I), as an alternative strategy to automatically
determine learning rates in stochastic loss functions over a range of 15 orders
of magnitude without the use of surrogates. We show that GOLS-I is a
competitive strategy to reliably determine step sizes, adding high value in
terms of performance, while being easy to implement.
