PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method
  with Probabilistic Gradient Estimation
  Despite their success, policy gradient methods suffer from high variance of
the gradient estimate, which can result in unsatisfactory sample complexity.
Recently, numerous variance-reduced extensions of policy gradient methods with
provably better sample complexity and competitive numerical performance have
been proposed. After a compact survey on some of the main variance-reduced
REINFORCE-type methods, we propose ProbAbilistic Gradient Estimation for Policy
Gradient (PAGE-PG), a novel loopless variance-reduced policy gradient method
based on a probabilistic switch between two types of updates. Our method is
inspired by the PAGE estimator for supervised learning and leverages importance
sampling to obtain an unbiased gradient estimator. We show that PAGE-PG enjoys
a $\mathcal{O}\left( \epsilon^{-3} \right)$ average sample complexity to reach
an $\epsilon$-stationary solution, which matches the sample complexity of its
most competitive counterparts under the same setting. A numerical evaluation
confirms the competitive performance of our method on classical control tasks.
