Tight High Probability Bounds for Linear Stochastic Approximation with
  Fixed Stepsize
  This paper provides a non-asymptotic analysis of linear stochastic
approximation (LSA) algorithms with fixed stepsize. This family of methods
arises in many machine learning tasks and is used to obtain approximate
solutions of a linear system $\bar{A}\theta = \bar{b}$ for which $\bar{A}$ and
$\bar{b}$ can only be accessed through random estimates $\{({\bf A}_n, {\bf
b}_n): n \in \mathbb{N}^*\}$. Our analysis is based on new results regarding
moments and high probability bounds for products of matrices which are shown to
be tight. We derive high probability bounds on the performance of LSA under
weaker conditions on the sequence $\{({\bf A}_n, {\bf b}_n): n \in
\mathbb{N}^*\}$ than previous works. However, in contrast, we establish
polynomial concentration bounds with order depending on the stepsize. We show
that our conclusions cannot be improved without additional assumptions on the
sequence of random matrices $\{{\bf A}_n: n \in \mathbb{N}^*\}$, and in
particular that no Gaussian or exponential high probability bounds can hold.
Finally, we pay a particular attention to establishing bounds with sharp order
with respect to the number of iterations and the stepsize and whose leading
terms contain the covariance matrices appearing in the central limit theorems.
