Merging Deterministic Policy Gradient Estimations with Varied
  Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
  Deep reinforcement learning (DRL) on Markov decision processes (MDPs) with
continuous action spaces is often approached by directly training parametric
policies along the direction of estimated policy gradients (PGs). Previous
research revealed that the performance of these PG algorithms depends heavily
on the bias-variance tradeoffs involved in estimating and using PGs. A notable
approach towards balancing this tradeoff is to merge both on-policy and
off-policy gradient estimations. However existing PG merging methods can be
sample inefficient and are not suitable to train deterministic policies
directly. To address these issues, this paper introduces elite PGs and
strengthens their variance reduction effect by adopting elitism and policy
consolidation techniques to regularize policy training based on policy
behavioral knowledge extracted from elite trajectories. Meanwhile, we propose a
two-step method to merge elite PGs and conventional PGs as a new extension of
the conventional interpolation merging method. At both the theoretical and
experimental levels, we show that both two-step merging and interpolation
merging can induce varied bias-variance tradeoffs during policy training. They
enable us to effectively use elite PGs and mitigate their performance impact on
trained policies. Our experiments also show that two-step merging can
outperform interpolation merging and several state-of-the-art algorithms on six
benchmark control tasks.
