Incrementally Learning Functions of the Return
  Temporal difference methods enable efficient estimation of value functions in
reinforcement learning in an incremental fashion, and are of broader interest
because they correspond learning as observed in biological systems. Standard
value functions correspond to the expected value of a sum of discounted
returns. While this formulation is often sufficient for many purposes, it would
often be useful to be able to represent functions of the return as well.
Unfortunately, most such functions cannot be estimated directly using TD
methods. We propose a means of estimating functions of the return using its
moments, which can be learned online using a modified TD algorithm. The moments
of the return are then used as part of a Taylor expansion to approximate
analytic functions of the return.
