Policy Optimization Through Approximate Importance Sampling
  Recent policy optimization approaches (Schulman et al., 2015a; 2017) have
achieved substantial empirical successes by constructing new proxy optimization
objectives. These proxy objectives allow stable and low variance policy
learning, but require small policy updates to ensure that the proxy objective
remains an accurate approximation of the target policy value. In this paper we
derive an alternative objective that obtains the value of the target policy by
applying importance sampling (IS). However, the basic importance sampled
objective is not suitable for policy optimization, as it incurs too high
variance in policy updates. We therefore introduce an approximation that allows
us to directly trade-off the bias of approximation with the variance in policy
updates. We show that our approximation unifies previously developed approaches
and allows us to interpolate between them. We develop a practical algorithm by
optimizing the introduced objective with proximal policy optimization
techniques (Schulman et al., 2017). We also provide a theoretical analysis of
the introduced policy optimization objective demonstrating bias-variance
trade-off. We empirically demonstrate that the resulting algorithm improves
upon state of the art on-policy policy optimization on continuous control
benchmarks.
