Finite-Time Convergence Rates of Nonlinear Two-Time-Scale Stochastic
  Approximation under Markovian Noise
  We study the so-called two-time-scale stochastic approximation, a
simulation-based approach for finding the roots of two coupled nonlinear
operators. Our focus is to characterize its finite-time performance in a Markov
setting, which often arises in stochastic control and reinforcement learning
problems. In particular, we consider the scenario where the data in the method
are generated by Markov processes, therefore, they are dependent. Such
dependent data result to biased observations of the underlying operators. Under
some fairly standard assumptions on the operators and the Markov processes, we
provide a formula that characterizes the convergence rate of the mean square
errors generated by the method to zero. Our result shows that the method
achieves a convergence in expectation at a rate $\mathcal{O}(1/k^{2/3})$, where
$k$ is the number of iterations. Our analysis is mainly motivated by the
classic singular perturbation theory for studying the asymptotic convergence of
two-time-scale systems, that is, we consider a Lyapunov function that carefully
characterizes the coupling between the two iterates. In addition, we utilize
the geometric mixing time of the underlying Markov process to handle the bias
and dependence in the data. Our theoretical result complements for the existing
literature, where the rate of nonlinear two-time-scale stochastic approximation
under Markovian noise is unknown.
