Mixing of Stochastic Accelerated Gradient Descent
  We study the mixing properties for stochastic accelerated gradient descent
(SAGD) on least-squares regression. First, we show that stochastic gradient
descent (SGD) and SAGD are simulating the same invariant distribution.
Motivated by this, we then establish mixing rate for SAGD-iterates and compare
it with those of SGD-iterates. Theoretically, we prove that the chain of SAGD
iterates is geometrically ergodic --using a proper choice of parameters and
under regularity assumptions on the input distribution. More specifically, we
derive an explicit mixing rate depending on the first 4 moments of the data
distribution. By means of illustrative examples, we prove that SAGD-iterate
chain mixes faster than the chain of iterates obtained by SGD. Furthermore, we
highlight applications of the established mixing rate in the convergence
analysis of SAGD on realizable objectives. The proposed analysis is based on a
novel non-asymptotic analysis of products of random matrices. This theoretical
result is substantiated and validated by experiments.
