Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
  Tuning the step size of stochastic gradient descent is tedious and error
prone. This has motivated the development of methods that automatically adapt
the step size using readily available information. In this paper, we consider
the family of SPS (Stochastic gradient with a Polyak Stepsize) adaptive
methods. These are methods that make use of gradient and loss value at the
sampled points to adaptively adjust the step size. We first show that SPS and
its recent variants can all be seen as extensions of the Passive-Aggressive
methods applied to nonlinear problems. We use this insight to develop new
variants of the SPS method that are better suited to nonlinear models. Our new
variants are based on introducing a slack variable into the interpolation
equations. This single slack variable tracks the loss function across
iterations and is used in setting a stable step size. We provide extensive
numerical results supporting our new methods and a convergence theory.
