Optimal Estimation of Off-Policy Policy Gradient via Double Fitted
  Iteration
  Policy gradient (PG) estimation becomes a challenge when we are not allowed
to sample with the target policy but only have access to a dataset generated by
some unknown behavior policy. Conventional methods for off-policy PG estimation
often suffer from either significant bias or exponentially large variance. In
this paper, we propose the double Fitted PG estimation (FPG) algorithm. FPG can
work with an arbitrary policy parameterization, assuming access to a
Bellman-complete value function class. In the case of linear value function
approximation, we provide a tight finite-sample upper bound on policy gradient
estimation error, that is governed by the amount of distribution mismatch
measured in feature space. We also establish the asymptotic normality of FPG
estimation error with a precise covariance characterization, which is further
shown to be statistically optimal with a matching Cramer-Rao lower bound.
Empirically, we evaluate the performance of FPG on both policy gradient
estimation and policy optimization, using either softmax tabular or ReLU policy
networks. Under various metrics, our results show that FPG significantly
outperforms existing off-policy PG estimation methods based on importance
sampling and variance reduction techniques.
