Painless Stochastic Gradient: Interpolation, Line-Search, and
  Convergence Rates
  Recent works have shown that stochastic gradient descent (SGD) achieves the
fast convergence rates of full-batch gradient descent for over-parameterized
models satisfying certain interpolation conditions. However, the step-size used
in these works depends on unknown quantities and SGD's practical performance
heavily relies on the choice of this step-size. We propose to use line-search
techniques to automatically set the step-size when training models that can
interpolate the data. In the interpolation setting, we prove that SGD with a
stochastic variant of the classic Armijo line-search attains the deterministic
convergence rates for both convex and strongly-convex functions. Under
additional assumptions, SGD with Armijo line-search is shown to achieve fast
convergence for non-convex functions. Furthermore, we show that stochastic
extra-gradient with a Lipschitz line-search attains linear convergence for an
important class of non-convex functions and saddle-point problems satisfying
interpolation. To improve the proposed methods' practical performance, we give
heuristics to use larger step-sizes and acceleration. We compare the proposed
algorithms against numerous optimization methods on standard classification
tasks using both kernel methods and deep networks. The proposed methods result
in competitive performance across all models and datasets, while being robust
to the precise choices of hyper-parameters. For multi-class classification
using deep networks, SGD with Armijo line-search results in both faster
convergence and better generalization.
