Does Momentum Help? A Sample Complexity Analysis
  Heavy ball momentum is a popular acceleration idea in stochastic
optimization. There have been several attempts to understand its perceived
benefits, but the complete picture is still unclear. Specifically, the error
expression in the presence of noise has two separate terms: the bias and the
variance, but most existing works only focus on bias and show that momentum
accelerates its decay. Such analyses overlook the interplay between bias and
variance and, therefore, miss important implications. In this work, we analyze
a sample complexity bound of stochastic approximation algorithms with
heavy-ball momentum that accounts for both bias and variance. We find that for
the same step size, which is small enough, the iterates with momentum have
improved sample complexity compared to the ones without. However, by using a
different step-size sequence, the non-momentum version can nullify this
benefit. Subsequently, we show that our sample complexity bounds are indeed
tight for a small enough neighborhood around the solution and large enough
noise variance. Our analysis also sheds some light on the finite-time behavior
of these algorithms. This explains the perceived benefit in the initial phase
of momentum-based schemes.
