A Tale of Two-Timescale Reinforcement Learning with the Tightest
  Finite-Time Bound
  Policy evaluation in reinforcement learning is often conducted using
two-timescale stochastic approximation, which results in various gradient
temporal difference methods such as GTD(0), GTD2, and TDC. Here, we provide
convergence rate bounds for this suite of algorithms. Algorithms such as these
have two iterates, $\theta_n$ and $w_n,$ which are updated using two distinct
stepsize sequences, $\alpha_n$ and $\beta_n,$ respectively. Assuming $\alpha_n
= n^{-\alpha}$ and $\beta_n = n^{-\beta}$ with $1 > \alpha > \beta > 0,$ we
show that, with high probability, the two iterates converge to their respective
solutions $\theta^*$ and $w^*$ at rates given by $\|\theta_n - \theta^*\| =
\tilde{O}( n^{-\alpha/2})$ and $\|w_n - w^*\| = \tilde{O}(n^{-\beta/2});$ here,
$\tilde{O}$ hides logarithmic terms. Via comparable lower bounds, we show that
these bounds are, in fact, tight. To the best of our knowledge, ours is the
first finite-time analysis which achieves these rates. While it was known that
the two timescale components decouple asymptotically, our results depict this
phenomenon more explicitly by showing that it in fact happens from some finite
time onwards. Lastly, compared to existing works, our result applies to a
broader family of stepsizes, including non-square summable ones.
