Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale
  Stochastic Approximation
  Motivated by their broad applications in reinforcement learning, we study the
linear two-time-scale stochastic approximation, an iterative method using two
different step sizes for finding the solutions of a system of two equations.
Our main focus is to characterize the finite-time complexity of this method
under time-varying step sizes and Markovian noise. In particular, we show that
the mean square errors of the variables generated by the method converge to
zero at a sublinear rate $\Ocal(k^{2/3})$, where $k$ is the number of
iterations. We then improve the performance of this method by considering the
restarting scheme, where we restart the algorithm after every predetermined
number of iterations. We show that using this restarting method the complexity
of the algorithm under time-varying step sizes is as good as the one using
constant step sizes, but still achieving an exact converge to the desired
solution. Moreover, the restarting scheme also helps to prevent the step sizes
from getting too small, which is useful for the practical implementation of the
linear two-time-scale stochastic approximation.
