Parabolic Approximation Line Search for DNNs
  A major challenge in current optimization research for deep learning is to
automatically find optimal step sizes for each update step. The optimal step
size is closely related to the shape of the loss in the update step direction.
However, this shape has not yet been examined in detail. This work shows
empirically that the batch loss over lines in negative gradient direction is
mostly convex locally and well suited for one-dimensional parabolic
approximations. By exploiting this parabolic property we introduce a simple and
robust line search approach, which performs loss-shape dependent update steps.
Our approach combines well-known methods such as parabolic approximation, line
search and conjugate gradient, to perform efficiently. It surpasses other step
size estimating methods and competes with common optimization methods on a
large variety of experiments without the need of hand-designed step size
schedules. Thus, it is of interest for objectives where step-size schedules are
unknown or do not perform well. Our extensive evaluation includes multiple
comprehensive hyperparameter grid searches on several datasets and
architectures. Finally, we provide a general investigation of exact line
searches in the context of batch losses and exact losses, including their
relation to our line search approach.
