On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and
  Non-Asymptotic Concentration
  We undertake a precise study of the asymptotic and non-asymptotic properties
of stochastic approximation procedures with Polyak-Ruppert averaging for
solving a linear system $\bar{A} \theta = \bar{b}$. When the matrix $\bar{A}$
is Hurwitz, we prove a central limit theorem (CLT) for the averaged iterates
with fixed step size and number of iterations going to infinity. The CLT
characterizes the exact asymptotic covariance matrix, which is the sum of the
classical Polyak-Ruppert covariance and a correction term that scales with the
step size. Under assumptions on the tail of the noise distribution, we prove a
non-asymptotic concentration inequality whose main term matches the covariance
in CLT in any direction, up to universal constants. When the matrix $\bar{A}$
is not Hurwitz but only has non-negative real parts in its eigenvalues, we
prove that the averaged LSA procedure actually achieves an $O(1/T)$ rate in
mean-squared error. Our results provide a more refined understanding of linear
stochastic approximation in both the asymptotic and non-asymptotic settings. We
also show various applications of the main results, including the study of
momentum-based stochastic gradient methods as well as temporal difference
algorithms in reinforcement learning.
