A Multistep Lyapunov Approach for Finite-Time Analysis of Biased
  Stochastic Approximation
  Motivated by the widespread use of temporal-difference (TD-) and Q-learning
algorithms in reinforcement learning, this paper studies a class of biased
stochastic approximation (SA) procedures under a mild "ergodic-like" assumption
on the underlying stochastic noise sequence. Building upon a carefully designed
multistep Lyapunov function that looks ahead to several future updates to
accommodate the stochastic perturbations (for control of the gradient bias), we
prove a general result on the convergence of the iterates, and use it to derive
non-asymptotic bounds on the mean-square error in the case of constant
stepsizes. This novel looking-ahead viewpoint renders finite-time analysis of
biased SA algorithms under a large family of stochastic perturbations possible.
For direct comparison with existing contributions, we also demonstrate these
bounds by applying them to TD- and Q-learning with linear function
approximation, under the practical Markov chain observation model. The
resultant finite-time error bound for both the TD- as well as the Q-learning
algorithms is the first of its kind, in the sense that it holds i) for the
unmodified versions (i.e., without making any modifications to the parameter
updates) using even nonlinear function approximators; as well as for Markov
chains ii) under general mixing conditions and iii) starting from any initial
distribution, at least one of which has to be violated for existing results to
be applicable.
