A straightforward line search approach on the expected empirical loss
  for stochastic deep learning problems
  A fundamental challenge in deep learning is that the optimal step sizes for
update steps of stochastic gradient descent are unknown. In traditional
optimization, line searches are used to determine good step sizes, however, in
deep learning, it is too costly to search for good step sizes on the expected
empirical loss due to noisy losses. This empirical work shows that it is
possible to approximate the expected empirical loss on vertical cross sections
for common deep learning tasks considerably cheaply. This is achieved by
applying traditional one-dimensional function fitting to measured noisy losses
of such cross sections. The step to a minimum of the resulting approximation is
then used as step size for the optimization. This approach leads to a robust
and straightforward optimization method which performs well across datasets and
architectures without the need of hyperparameter tuning.
