Off-Policy Policy Gradient with State Distribution Correction
  We study the problem of off-policy policy optimization in Markov decision
processes, and develop a novel off-policy policy gradient method. Prior
off-policy policy gradient approaches have generally ignored the mismatch
between the distribution of states visited under the behavior policy used to
collect data, and what would be the distribution of states under the learned
policy. Here we build on recent progress for estimating the ratio of the state
distributions under behavior and evaluation policies for policy evaluation, and
present an off-policy policy gradient optimization technique that can account
for this mismatch in distributions. We present an illustrative example of why
this is important and a theoretical convergence guarantee for our approach.
Empirically, we compare our method in simulations to several strong baselines
which do not correct for this mismatch, significantly improving in the quality
of the policy discovered.
