An Analysis of Measure-Valued Derivatives for Policy Gradients
  Reinforcement learning methods for robotics are increasingly successful due
to the constant development of better policy gradient techniques. A precise
(low variance) and accurate (low bias) gradient estimator is crucial to face
increasingly complex tasks. Traditional policy gradient algorithms use the
likelihood-ratio trick, which is known to produce unbiased but high variance
estimates. More modern approaches exploit the reparametrization trick, which
gives lower variance gradient estimates but requires differentiable value
function approximators. In this work, we study a different type of stochastic
gradient estimator - the Measure-Valued Derivative. This estimator is unbiased,
has low variance, and can be used with differentiable and non-differentiable
function approximators. We empirically evaluate this estimator in the
actor-critic policy gradient setting and show that it can reach comparable
performance with methods based on the likelihood-ratio or reparametrization
tricks, both in low and high-dimensional action spaces. With this work, we want
to show that the Measure-Valued Derivative estimator can be a useful
alternative to other policy gradient estimators.
