Action-depedent Control Variates for Policy Optimization via Stein's
  Identity
  Policy gradient methods have achieved remarkable successes in solving
challenging reinforcement learning problems. However, it still often suffers
from the large variance issue on policy gradient estimation, which leads to
poor sample efficiency during training. In this work, we propose a control
variate method to effectively reduce variance for policy gradient methods.
Motivated by the Stein's identity, our method extends the previous control
variate methods used in REINFORCE and advantage actor-critic by introducing
more general action-dependent baseline functions. Empirical studies show that
our method significantly improves the sample efficiency of the state-of-the-art
policy gradient approaches.
