Efficient Sample Reuse in Policy Gradients with Parameter-based
  Exploration
  The policy gradient approach is a flexible and powerful reinforcement
learning method particularly for problems with continuous actions such as robot
control. A common challenge in this scenario is how to reduce the variance of
policy gradient estimates for reliable policy updates. In this paper, we
combine the following three ideas and give a highly effective policy gradient
method: (a) the policy gradients with parameter based exploration, which is a
recently proposed policy search method with low variance of gradient estimates,
(b) an importance sampling technique, which allows us to reuse previously
gathered data in a consistent way, and (c) an optimal baseline, which minimizes
the variance of gradient estimates with their unbiasedness being maintained.
For the proposed method, we give theoretical analysis of the variance of
gradient estimates and show its usefulness through extensive experiments.
