Statistically Efficient Off-Policy Policy Gradients
  Policy gradient methods in reinforcement learning update policy parameters by
taking steps in the direction of an estimated gradient of policy value. In this
paper, we consider the statistically efficient estimation of policy gradients
from off-policy data, where the estimation is particularly non-trivial. We
derive the asymptotic lower bound on the feasible mean-squared error in both
Markov and non-Markov decision processes and show that existing estimators fail
to achieve it in general settings. We propose a meta-algorithm that achieves
the lower bound without any parametric assumptions and exhibits a unique 3-way
double robustness property. We discuss how to estimate nuisances that the
algorithm relies on. Finally, we establish guarantees on the rate at which we
approach a stationary point when we take steps in the direction of our new
estimated policy gradient.
