Proximal Policy Optimization with Adaptive Threshold for Symmetric
  Relative Density Ratio
  Deep reinforcement learning (DRL) is one of the promising approaches for
introducing robots into complicated environments. The recent remarkable
progress of DRL stands on regularization of policy, which allows the policy to
improve stably and efficiently. A popular method, so-called proximal policy
optimization (PPO), and its variants constrain density ratio of the latest and
baseline policies when the density ratio exceeds a given threshold. This
threshold can be designed relatively intuitively, and in fact its recommended
value range has been suggested. However, the density ratio is asymmetric for
its center, and the possible error scale from its center, which should be close
to the threshold, would depend on how the baseline policy is given. In order to
maximize the values of regularization of policy, this paper proposes a new PPO
derived using relative Pearson (RPE) divergence, therefore so-called PPO-RPE,
to design the threshold adaptively. In PPO-RPE, the relative density ratio,
which can be formed with symmetry, replaces the raw density ratio. Thanks to
this symmetry, its error scale from center can easily be estimated, hence, the
threshold can be adapted for the estimated error scale. From three simple
benchmark simulations, the importance of algorithm-dependent threshold design
is revealed. By simulating additional four locomotion tasks, it is verified
that the proposed method statistically contributes to task accomplishment by
appropriately restricting the policy updates.
