Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation
  Regime
  Stochastic gradient descent (SGD) has achieved great success due to its
superior performance in both optimization and generalization. Most of existing
generalization analyses are made for single-pass SGD, which is a less practical
variant compared to the commonly-used multi-pass SGD. Besides, theoretical
analyses for multi-pass SGD often concern a worst-case instance in a class of
problems, which may be pessimistic to explain the superior generalization
ability for some particular problem instance. The goal of this paper is to
sharply characterize the generalization of multi-pass SGD, by developing an
instance-dependent excess risk bound for least squares in the interpolation
regime, which is expressed as a function of the iteration number, stepsize, and
data covariance. We show that the excess risk of SGD can be exactly decomposed
into the excess risk of GD and a positive fluctuation error, suggesting that
SGD always performs worse, instance-wisely, than GD, in generalization. On the
other hand, we show that although SGD needs more iterations than GD to achieve
the same level of excess risk, it saves the number of stochastic gradient
evaluations, and therefore is preferable in terms of computational time.
