{'fraction_non_alphanumeric': 0.04065231878944686, 'fraction_numerical': 0.013603042063585402, 'mean_word_length': 4.64505421553441, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning

Deep reinforcement learning (DRL) on Markov decision processes (MDPs) with continuous action spaces is often approached by directly training parametric policies along the direction of estimated policy gradients (PGs). Previous research revealed that the performance of these PG algorithms depends heavily on the bias-variance tradeoffs involved in estimating and using PGs. A notable approach towards balancing this tradeoff is to merge both on-policy and off-policy gradient estimations. However existing PG merging methods can be sample inefficient and are not suitable to train deterministic policies directly. To address these issues, this paper introduces elite PGs and strengthens their variance reduction effect by adopting elitism and policy consolidation techniques to regularize policy training based on policy behavioral knowledge extracted from elite trajectories. Meanwhile, we propose a two-step method to merge elite PGs and conventional PGs as a new extension of the conventional interpolation merging method. At both the theoretical and experimental levels, we show that both two-step merging and interpolation merging can induce varied bias-variance tradeoffs during policy training. They enable us to effectively use elite PGs and mitigate their performance impact on trained policies. Our experiments also show that two-step merging can outperform interpolation merging and several state-of-the-art algorithms on six benchmark control tasks.


Introduction
Research on deep reinforcement learning (DRL) algorithms has made remarkable progress in solving many challenging Markov Decision Processes (MDPs) with continuous state spaces and continuous action spaces [4,22,24,23,10,41,42]. Among all the state-of-the-art DRL algorithms, an important family aims at directly learning action-selection policies modelled as deep neural networks (DNNs) by iteratively applying policy gradient (PG) updates to the DNNs based on either (or both) on-policy or (or and) off-policy samples collected from the learning environment. In comparison to DRL algorithms that indirectly derive policies from learned value functions, PG algorithms have wellunderstood convergence properties and are naturally suited to handle continuous action spaces while achieving high sample efficiency on large-scale MDPs [6].
Previous research revealed that PG algorithms can exhibit highly unstable learning behavior due to to several major issues [9,1,13,14,27,28,26,18,20]. In particular, PG algorithms can be seriously affected by the bias-variance tradeoffs involved in estimating and using PGs [40,33,29,35]. A notable approach towards tackling this problem is to merge multiple gradient estimations, as demonstrated in [12,11,41]. In fact researchers have explored the benefits of merging on-policy estimation of PG with off-policy estimation to unleash the advantages of both [12]. However, existing PG merging techniques can only train stochastic policies and may also suffer from poor data efficiency due to one-time use of environment samples collected in the on-policy fashion. They are hence unsuitable for pure off-policy PG algorithms such as Deep Deterministic Policy Gradient (DDPG) [22] and twin-delayed DDPG (TD3) [8], which are designed to train deterministic policies. In view of this, it is important to develop new algorithms to merge multiple off-policy estimations of deterministic PGs (DPG) (i.e., PGs of deterministic policies) for enhanced learning efficiency and effectiveness. This research question has not attracted sufficient attention in the past.
We seek to investigate two inter-related questions: (Q1) how to estimate DPG in an off-policy fashion with varied bias-variance tradeoffs; and (Q2) how to merge multiple DPG estimations to effectively train a policy network. To answer Q1, we propose to estimate conventional DPG and elite DPG respectively based on two separate experience replay buffers (ERBs), i.e., the full ERB and the elite ERB. The full ERB keeps track of all state samples obtained since the beginning of the learning process. The elite ERB maintains an elite group of sampled trajectories with the highest cumulative rewards. Hence, conventional DPG is expected to have lower bias but higher variance than elite DPG. On the other hand, the newly introduced elite DPG aims at exploiting and improving best trajectories sampled so far. To further reduce the estimation variance associated with elite DPG, inspired by [20], we develop a policy consolidation technique to extract policy behavioral knowledge from the elite ERB with the help of a generative model such as the variational autoencoder (VAE) [7]. Elite DPG is then constrained by the behavioral deviation of trained policies from the VAE model.
To answer Q2, we study two different DPG merging methods. The first interpolation merging method linearly combines conventional DPG with elite DPG [12,11]. The second two-step merging method is newly developed in this paper to amplify the variance reduction effect of elite DPG through a two-step iterative process. Theoretically, we show that both interpolation merging and two-step merging enable us to reduce the variance of trained policy parameters with controllable bias, contributing to improved learning reliability and performance. Meanwhile, the performance impact of using biased estimation of elite DPG can be effectively bounded. On six difficult benchmark control tasks, we show that TD3 enhanced by two-step merging (TD3-2M) can noticeably outperform TD3 enhanced by interpolation merging (TD3-IM), TD3 and other cutting-edge DRL algorithms, including Soft Actor-Critic (SAC) [14], Proximal Policy Optimization (PPO) [34] and Interpolated Policy Gradient (IPG), which is designed to merge on-policy and off-policy PG estimations [12].

Related Work
Merging multiple DRL methods has been frequently attempted in the literature. Q-Prop [11], PGQ [27], ACER [41] and IPG [12] are some of the recent examples with the aim to combine on-policy and off-policy learning. Different from these research works, this paper studies the task of learning deterministic policies by merging multiple off-policy DPGs without involving on-policy learning. Our proposed use of the elite ERB is related to the prioritized experience replay (PER) technique [17]. However, this paper aims to balance the bias-variance tradeoff via DPG merging rather than PER since PER introduces additional bias to all estimated DPGs, potentially affecting learning stability. Hence our research is more closely related to IPG than PER.
Huge efforts have been made to control noise in estimated PGs. For example, stochastic policies can be trained with reduced variance by using smoothed and target value function networks [25,2,15]. In this paper, we rely on time-tested variance reduction techniques introduced by TD3 to learn value functions. Meanwhile, the variance involved in policy training can be noticeably reduced with the help of some variance control techniques [35]. For example, the stochastic variance-reduced policy gradient (SVRPG) algorithm reuses past gradient information to reduce variance of newly estimated PGs [29]. All PGs are estimated through G(PO)MDP [3] based on on-policy samples 1 .
KL-divergence is commonly used to regularize behavioral deviation of stochastic policies [20]. Since KL-divergence does not apply to deterministic policies, we propose to measure the deviation in action spaces as a new regularizer. We also build a VAE-based generative model to extract essential policy behavioral knowledge from the elite ERB and use the model to regularize policy training. VAE has been frequently explored to support meta-RL, transfer learning and multi-task learning [30,19,38]. We propose a new use of VAE for estimating elite DPGs in this paper.

Markov Decision Process and Actor-Critic Reinforcement Learning
We are interested in MDPs with continuous action spaces. At any time t, an RL agent has access to the current state of its learning environment, denoted as s t ∈ S. Based on this, the agent performs an action a t selected from an m-dimensional continuous action space A ⊂ R m . This causes an environment transition to a new state s t+1 at time t + 1, governed by unknown statetransition probability Pr(s t , s t+1 , a t ). The environment also produces a scalar and bounded reward r(s t , a t ) as its immediate feedback to the agent. Guided by a parametric deterministic policy π θ : S → A that specifies the action a to be performed in any state s, the agent can generate a trajectory τ = {(s τ t , s τ t+1 , a τ t , r τ t )} ∞ t=0 involving a sequence of consecutive state transitions over time, starting from an initial state s 0 . The goal for RL is to identify the optimal policy parameters θ * that maximizes the expected long-term cumulative reward over all trajectories, as defined below: with γ ∈ (0, 1) being a discount factor. According to the actor-critic (AC) framework, the critic in an AC algorithm for DRL is responsible for learning the value functions of π θ . Specifically, off-policy training of the Q-function can be performed by the critic to minimize the Bellman loss below over a batch of environment samples B f collected from the full ERB: (2) Q π θ ω is a parametric approximation of the Q-function with trainable parameters ω.Q π θ ω in (2) represents the target Q-network [8]. Guided by the value functions learned by the critic, the actor in the AC algorithm proceeds to train policy π θ by using DPGs estimated according to: where β refers to the stochastic behavioral policy for environment exploration, as defined below: with µ a ∼ N (0, Σ a ) and Σ a is an m × m diagonal covariance matrix that controls the scale of environment exploration. ρ β in (3) stands for the discounted state visitation distribution induced by following policy β [36]. In practice, DDPG and TD3 use environment samples stored in the full ERB to estimate ∇ θ J(π θ ). Policy π θ is hence trained with conventional DPGs alone in the two algorithms.

Twin-Delayed Deep Deterministic Policy Gradient
The failure of DDPG is commonly associated with overestimation of the Q-function, which may cause divergence of the actor [16,8]. To improve learning stability of DDPG, TD3 introduces three key innovations: (1) Clipped double Q-learning: TD3 learns two Q-functions in parallel and uses the smaller of the two to determine the target Q-value in the Bellman loss defined in (2). (2) Delayed policy update: TD3 trains policy less frequently than Q-function. As a rule of thumb, every two updates of Q-function will be followed by one update of policy. According to [8], the less frequent policy updates can benefit from low-variance estimation of Q-function, thereby improving the quality of policy training. (3) Target policy smoothing: TD3 adds noise to the target action π θ (s ) in (2) to enforce the notion that any two similar actions a and a should have similar Q-values in any state s, thereby mitigating the impact of critic error on policy training. The added noise is also clipped to keep the target action close to the original action.

Merge Deterministic Policy Gradient with Varied Bias-Variance Tradeoff
In this section, we first study the techniques to estimate conventional DPG and elite DPG. Eligible methods for merging conventional DPG and elite DPG will be proposed subsequently. Finally theoretical analysis will be performed to bound the performance impact of using elite DPGs under certain conditions and assumptions. We will also show that both interpolation merging and two-step merging methods can reduce variance of trained policies with controllable bias.

Estimate Deterministic Policy Gradient
We study two different approaches to estimate DPG. Similar to DDPG and TD3, the first approach aims at estimating DPG with low bias but potentially high variance. For this purpose, we randomly sample a batch of records B f from the full ERB and use the batch to estimate conventional DPG, denoted as ∇ c θ J(π θ ). Because these samples were obtained by using many different policies, we can effectively mitigate the possible bias induced by any specific policies. Hence, DDPG and TD3 can effectively train a policy network by using ∇ c θ J(π θ ) alone [22,8]. On the other hand, since only a small collection B f is sampled from thousands or millions of records stored in the full ERB 2 , the estimation of ∇ c θ J(π θ ) can vary substantially, depending on the actual samples contained in B f . Such high variance may noticeably slow down the learning progress.
To address this issue, we propose a new way of estimating DPG with low variance at the cost of increased bias. Guided by the aim for the newly estimated DPG to increase the chance for an RL agent to generate trajectories with high cumulative rewards, we decide to maintain a separate elite ERB and sample the elite batch, denoted as B e , from the elite ERB to estimate DPG. The elite ERB always contain the top κ 3 trajectories with the highest cumulative rewards ever observed.
Using (3) to compute the elite DPG ∇ e θ J(π θ ) does not necessarily increase the chance for an RL agent to produce high-reward trajectories. Note that in (3) the action a in any sampled state s t is determined completely by policy π θ and π θ (s t ) may deviate substantially from the sampled action a t recorded as part of the elite trajectories. Such behavioral deviation poses difficulties for the RL agent to progressively improve on its past success (i.e., elite trajectories). To tackle this problem and to further reduce the variance involved in estimating the elite DPG, we design a new regularizer based on the policy consolidation mechanism to control behavioral deviation of trained policies.
Specifically, the regularized ∇ e θ J(π θ ) can be estimated as below: which will be used in the i-th learning iteration to train policy π θi . By studying the asymptotic trend of θ i across a long sequence of learning iterations in Subsection 4.3, we can deepen our understanding of the interpolation merging and two-step merging methods.
Driven by the learning rules in (9) for updating θ IM and θ 2M , we develop two new DRL algorithms based on TD3, i.e., TD3-IM and TD3-2M. Algorithm 1 in Appendix A describes the policy training procedure of the two algorithms as well as TD3. TD3, TD3-IM and TD3-2M adopt the same method to train Q-networks. Meanwhile, only TD3-IM and TD3-2M require to maintain the elite ERB for the purpose of estimating the elite DPG. The two algorithms differ by the DPG merging methods adopted. Other parts of the three algorithms remain identical.
measures the probability differences between π θ and β 2 in selecting the initial state s 0 . It can be reduced by increasing the size of the elite ERB. Upon using the full ERB to estimate conventional DPG ∇ c θ J(π θ ), D T V (ρ π θ 0 , ρ β1 0 ) ≈ 0. This is because the distribution of initial states recorded in the full ERB is very close to ρ π θ 0 , which is determined completely by the learning environment. Moreover, (11) indicates that the bias ofJ(π θ , β 1 , β 2 , υ) is less than that ofJ(π θ , β 2 ) since υ < 1. Upon using small α,J(π θ , π θ , β 1 , β 2 , υ) can also keep its bias at a low level. As explained in Appendix B, theoretically Proposition 1 produces three new learning rules with guaranteed monotonic policy improvements, which can be realized approximately and efficiently by training policy π θ via ∇ e θ J(π θ ), ∇ IM θ J(π θ ) and ∇ 2M θ J(π θ ). Besides Proposition 1, additional analysis has also be performed in Appendix C to study the bias introduced by imprecise approximations of the value function and to bound directly the bias involved in estimating and merging DPGs. The analysis provides theoretical clues regarding the importance of adopting the regularizer in (5) and (6). To further analyze the bias-variance tradeoffs achievable by using ∇ IM θ J(π θ ) and ∇ 2M θ J(π θ ), we perform a noisy quadratic analysis below [44]. Assuming that the policy parameters θ under training are close to the optimal policy parameters θ * , J(π θ ) andJ(π θ , β 2 ) can be approximated respectively by the following noisy quadratic models: Assume without loss of generality that θ * = 0, similar to [43,31]. In (12), c 1 ∼ N (0, Σ 1 ) 5 and c 2 ∼ N ( , Σ 2 ) with = 0 that reflects the estimation bias associated with ∇ e θ J(π θ ). Both Σ 1 and Σ 2 are diagonal matrices with positive diagonal elements and Diag(Σ 1 ) Diag(Σ 2 ). In line with where the diagonal matrix A with positive diagonal elements controls the sensitivity of J(π θ ) with respect to θ. Despite of being simple, (12) remains as challenging optimization targets in the literature [44,31]. Our noisy quadratic analysis produces Proposition 2 below. Its proof is presented in Appendix D.
Proposition 2 Based on the noisy quadratic models in (12) and provided that α is sufficiently small such that αDiag(A) < Diag(I), policy parameters trained by using the three learning rules in (9) have the following convergence properties in expectation: Moreover, the variances of trained policy parameters converge to the following fixed points: Consistent with Proposition 1, Proposition 2 indicates that the bias involved training θ IM and θ 2M according to (9) can be controlled via υ. Meanwhile, θ IM enjoys less bias than θ 2M at the expense of relatively higher variances. In fact, because Diag(Σ 2 ) Diag(Σ 1 ), it can be easily verified that Therefore the three learning rules in (9) provide varied bias-variance tradeoffs while training policy π θ .

Experiments
Experiments have been performed on six benchmark control tasks, including Ant, Half Cheetah, Hopper, Lunar Lander, Walker2D and Bipedal Walker. We adopt a popular implementation of these benchmarks provided by OpenAI GYM 6 and powered by the PyBullet physics engine [37]. Many previous studies utilized a different version of these benchmarks that rely on the MuJoCo physics engine with varied system dynamics and reward schemes [39]. We prefer to use the PyBullet version since the PyBullet software package is publicly available with increasing popularity. PyBullet benchmarks are also widely considered to be tougher to solve than their MuJoCo counterparts [37].      (5) With regularizer in (6) (a) Performance difference of TD3-2M upon using (5), (6) and "no regularizer" to estimate elite DPGs.  slightly better performance than SAC and TD3. Since TD3-IM and TD3-2M achieved overall better performance than TD3, it can be confirmed that merging conventional and elite DPGs can improve policy training effectiveness. The fact that TD3-2M performed more effectively than TD3-IM also verifies the usefulness of the newly developed two-step merging method. Besides TD3-IM, interpolation merging has also been exploited to merge on-policy PG with off-policy PG in IPG [12]. Despite of using the same merging method, TD3-IM can significantly outperform IPG on most of the benchmarks.
We also examined other settings of the two hyper-parameters. As demonstrated in Figure 2, on Half Cheetah, the performance of TD3-2M is not heavily influenced by υ. Meanwhile, λ seems to have stronger influence than υ. With λ = 0.2, TD3-2M achieved clearly better performance than other settings of λ. However, setting λ to 0.1 appears to produce more reliable learning behavior. While we reported only the influence of λ and υ on Half Cheetah, similar observations have been witnessed on other benchmarks. It is also suitable for TD3-IM to adopt identical settings of υ and λ as TD3-2M. Figure 3(a) compares the performance of TD3-2M upon using either (5) or (6) to estimate elite DPGs. Our experiments clearly show the effectiveness of using (6). In particular, with (6), steady performance improvement can be realized throughout the entire learning process. In comparison, learning slows down noticeably after the initial phase when elite DPG is estimated through (5) or without using the regularizer term in (6). Figure 3(b) further shows that TD3-2M can substantially reduce the variance of estimated DPGs for policy training than TD3, explaining empirically why it can outperform TD3 on most of the benchmark problems.

Conclusion
In this paper we studied an important research question on how to estimate and merge DPGs with varied bias-variance tradeoffs for effective DRL. Driven by this question, we introduced the elite DPG to be estimated by using both the elitism and policy consolidation techniques. Meanwhile, we have theoretically and experimentally studied two DPG merging methods, i.e., interpolation merging and two-step merging. Both merging methods can be adopted by TD3 to balance the bias-variance tradeoffs during policy training. They noticeably improved the learning performance of TD3, significantly outperforming several state-of-the-art DRL algorithms. Additionally, we showed analytically that the performance impact of using biased estimation of elite DPGs can be effectively bounded and also mitigated through DPG merging.

Appendix A
This appendix presents the pseudo code of TD3, TD3-IM and TD3-2M, as shown in Algorithm 1.

Appendix B
To prove Proposition 1, we need to utilize Lemmas 1, which will be introduced and proved first.
Based on Lemma 1, we can proceed to prove Proposition 1.
It is interesting to note that Proposition 1 produces, in theory, a DRL algorithm with monotonic policy improvement guarantees. Specifically, define A new policy updating rule can be established. Particularly, given policy πi at the i-th learning iteration (i ≥ 0), policy πi+1 for the next learning iteration can be determined as Based on Proposition 1, it is straightforward to see that: (14) hence guarantees monotonic policy improvement. Consequently, we can prove the convergence of the policy training process upon using (14) [32]. Since (14) is not practically feasible, we choose to continuously update the elite ERB with elite trajectories newly generated by trained policies. Because the elite ERB embodies policy β2 inJ(π, β2), the updating process can be considered as approximating (14) in an efficient but imprecise manner. In fact, the elite ERB keeps track of those "good" initial states on which the train policies have achieved the highest cumulative rewards. Accordingly, the probability density of ρ β 2 0 is expected to concentrate mainly on such "good" states. This will approximately maximize E s∼ρ β 2 0 [Q π ω (s, π(s))] and hence M (π, β2). Meanwhile, with the help of the VAE generative model V and the regularizer in (6), we can effectively train πi+1 to approximate β2, thereby roughly fulfilling the optimization requirement in (14). Following the same idea, we can also construct the following learning rules with guaranteed monotonic policy improvement: which can be approximately realized by interpolation merging and two-step merging respectively, providing theoretical evidence regarding why ∇ IM J(π) and ∇ 2M J(π) can be utilized to train policy π effectively.
Proposition 3 shows that the error involved in approximating the Q-function byQ π ω , as measured by χ3, introduces additional bias for policy training. Meanwhile, the extra bias caused by using two-step merging, as reflected by the difference between J(π) andĴ(π, π , β1, β2, υ), can be controlled via the learning rate α. Besides Proposition 3, we can also bound directly the bias involved in estimating and merging DPGs. Consider that represent respectively the precise as well as imprecise estimations of the DPG (the latter for elite DPG). In order to bound the difference between the two, we need to utilize an important assumption: the state-transition probability of an MDP is continuous with respect to its actions, meaning that small differences in actions performed by an RL agent will lead to small differences in state-transition probabilities, as formulated below: s ∈S s∈S ρ π t−1 (s) Pr(s, s , π(s)) − s∈S ρ π t−1 (s) Pr(s, s , β(s))+ s∈S ρ π t−1 (s) Pr(s, s , β(s)) − s∈S ρ β t−1 (s) Pr(s, s , β(s)) ≤ 1 2 s ∈S s∈S ρ π t−1 (s)(Pr(s, s , π(s)) − Pr(s, s , β(s))) s ∈S s∈S ρ π t−1 (s) (Pr(s, s , π(s)) − Pr(s, s , β(s))) (Pr(s, s , π(s)) − Pr(s, s , β(s))) In view of the above, it can be shown further that This proves Lemma 2.
By leveraging on Lemma 2, Proposition 4 can be established to bound the difference between ∇J(π) and ∇ eĴ (π).
Proposition 4 indicates that the bias introduced by ∇ eĴ (π) can be mitigated by reducing either ∆ π,β 2 or DT V (ρ π 0 , ρ β 2 0 ). This is consistent with the findings captured by Proposition 1. Particularly, Proposition 4 provides theoretical clues regarding the importance of adopting the regularizer in (5) and (6) that helps to keep ∆ π,β 2 at a low level. As a special case of Proposition 4 when Q π ω =Q π ω and hence ψ1 ≈ 0, which again confirms the importance of controlling ∆ π,β 2 via the regularizer in (5) and (6), even when the Q-function can be approximated precisely. In line with the above analysis, define which corresponds to the DPG obtained via interpolation merging. Its difference from ∇J(π) can be bounded below: (1 − γ) 2 ∆ π,β 1 +
Hence, similar to ∇ IM J(π), ∇ 2M J(π) can also mitigate the performance impact of using elite DPGs for policy training. Moreover, noisy quadratic analysis in Proposition 2 suggests that ∇ 2M J(π) can effectively reduce the variance of trained policy parameters.
Appendix D