Variance-Reduced Off-Policy TDC Learning: Non-Asymptotic Convergence
  Analysis
  Variance reduction techniques have been successfully applied to
temporal-difference (TD) learning and help to improve the sample complexity in
policy evaluation. However, the existing work applied variance reduction to
either the less popular one time-scale TD algorithm or the two time-scale GTD
algorithm but with a finite number of i.i.d.\ samples, and both algorithms
apply to only the on-policy setting. In this work, we develop a variance
reduction scheme for the two time-scale TDC algorithm in the off-policy setting
and analyze its non-asymptotic convergence rate over both i.i.d.\ and Markovian
samples. In the i.i.d.\ setting, our algorithm achieves a sample complexity
$O(\epsilon^{-\frac{3}{5}} \log{\epsilon}^{-1})$ that is lower than the
state-of-the-art result $O(\epsilon^{-1} \log {\epsilon}^{-1})$. In the
Markovian setting, our algorithm achieves the state-of-the-art sample
complexity $O(\epsilon^{-1} \log {\epsilon}^{-1})$ that is near-optimal.
Experiments demonstrate that the proposed variance-reduced TDC achieves a
smaller asymptotic convergence error than both the conventional TDC and the
variance-reduced TD.
