Reward-estimation variance elimination in sequential decision processes
  Policy gradient methods are very attractive in reinforcement learning due to
their model-free nature and convergence guarantees. These methods, however,
suffer from high variance in gradient estimation, resulting in poor sample
efficiency. To mitigate this issue, a number of variance-reduction approaches
have been proposed. Unfortunately, in the challenging problems with delayed
rewards, these approaches either bring a relatively modest improvement or do
reduce variance at expense of introducing a bias and undermining convergence.
The unbiased methods of gradient estimation, in general, only partially reduce
variance, without eliminating it completely even in the limit of exact
knowledge of the value functions and problem dynamics, as one might have
wished. In this work we propose an unbiased method that does completely
eliminate variance under some, commonly encountered, conditions. Of practical
interest is the limit of deterministic dynamics and small policy stochasticity.
In the case of a quadratic value function, as in linear quadratic Gaussian
models, the policy randomness need not be small. We use such a model to analyze
performance of the proposed variance-elimination approach and compare it with
standard variance-reduction methods. The core idea behind the approach is to
use control variates at all future times down the trajectory. We present both a
model-based and model-free formulations.
