A Unified View of TD Algorithms; Introducing Full-Gradient TD and
  Equi-Gradient Descent TD
  This paper addresses the issue of policy evaluation in Markov Decision
Processes, using linear function approximation. It provides a unified view of
algorithms such as TD(lambda), LSTD(lambda), iLSTD, residual-gradient TD. It is
asserted that they all consist in minimizing a gradient function and differ by
the form of this function and their means of minimizing it. Two new schemes are
introduced in that framework: Full-gradient TD which uses a generalization of
the principle introduced in iLSTD, and EGD TD, which reduces the gradient by
successive equi-gradient descents. These three algorithms form a new
intermediate family with the interesting property of making much better use of
the samples than TD while keeping a gradient descent scheme, which is useful
for complexity issues and optimistic policy iteration.
