Stable Policy Optimization via Off-Policy Divergence Regularization
  Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization
(PPO) are among the most successful policy gradient approaches in deep
reinforcement learning (RL). While these methods achieve state-of-the-art
performance across a wide range of challenging tasks, there is room for
improvement in the stabilization of the policy learning and how the off-policy
data are used. In this paper we revisit the theoretical foundations of these
algorithms and propose a new algorithm which stabilizes the policy improvement
through a proximity term that constrains the discounted state-action visitation
distribution induced by consecutive policies to be close to one another. This
proximity term, expressed in terms of the divergence between the visitation
distributions, is learned in an off-policy and adversarial manner. We
empirically show that our proposed method can have a beneficial effect on
stability and improve final performance in benchmark high-dimensional control
tasks.
