Proximal Policy Optimization with Relative Pearson Divergence
  The recent remarkable progress of deep reinforcement learning (DRL) stands on
regularization of policy for stable and efficient learning. A popular method,
named proximal policy optimization (PPO), has been introduced for this purpose.
PPO clips density ratio of the latest and baseline policies with a threshold,
while its minimization target is unclear. As another problem of PPO, the
symmetric threshold is given numerically while the density ratio itself is in
asymmetric domain, thereby causing unbalanced regularization of the policy.
This paper therefore proposes a new variant of PPO by considering a
regularization problem of relative Pearson (RPE) divergence, so-called PPO-RPE.
This regularization yields the clear minimization target, which constrains the
latest policy to the baseline one. Through its analysis, the intuitive
threshold-based design consistent with the asymmetry of the threshold and the
domain of density ratio can be derived. Through four benchmark tasks, PPO-RPE
performed as well as or better than the conventional methods in terms of the
task performance by the learned policy.
