{'fraction_non_alphanumeric': 0.040050527567246245, 'fraction_numerical': 0.01214890771288453, 'mean_word_length': 4.746584116140051, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Representation Gap in Deep Reinforcement Learning

Deep reinforcement learning gives the promise that an agent learns good policy from high-dimensional information. Whereas representation learning removes irrelevant and redundant information and retains pertinent information. We consider the representation capacity of action value function and theoretically reveal its inherent property, \textit{representation gap} with its target action value function. This representation gap is favorable. However, through illustrative experiments, we show that the representation of action value function grows similarly compared with its target value function, i.e. the undesirable inactivity of the representation gap (\textit{representation overlap}). Representation overlap results in a loss of representation capacity, which further leads to sub-optimal learning performance. To activate the representation gap, we propose a simple but effective framework \underline{P}olicy \underline{O}ptimization from \underline{P}reventing \underline{R}epresentation \underline{O}verlaps (POPRO), which regularizes the policy evaluation phase through differing the representation of action value function from its target. We also provide the convergence rate guarantee of POPRO. We evaluate POPRO on gym continuous control suites. The empirical results show that POPRO using pixel inputs outperforms or parallels the sample-efficiency of methods that use state-based features.


Introduction
By combining representation capabilities of deep neural network (DNN) with the credit assignment capabilities of reinforcement learning (RL), Deep RL (DRL) is able to develop a self-control agent that can perform complex control tasks from high-dimensional observations such as image pixels and sensors information [1,2,3], where DNN is utilized to parameterize the policy function or value function. DRL gives the promise that the agent learns good policy by tackling the high-dimensional information. However, this naturally needs to remove irrelevant and redundant information and retain pertinent information, which is the job of representation learning. Thus, representation learning in DRL has attracted much attention from researchers [4,5].
Representation learning methods in DRL focus on how to obtain good task-related representations [6,5,7,8,9,10,11]. Some of them borrow insights from other areas such as computer vision [10,11]. Some works attempt to use auxiliary tasks (e.g. predicting the future conditioned on the past observations or actions [12,13]) to improve the representational capacity of RL and thus improve the empirical performance. These works somehow did not attach the importance of the inherent representation property of the core instrument in DRL, the action value function. In this work, we investigate the representation capacity in action value network and theoretically show that a good action value function representation should have the inherent property of representation gap.
Typical DRL methods utilize a target value function to stabilize the policy evaluation phase [2,14,15,16,17]. We consider the representation capacity of action value function under this setting. Following the commonly used definition of representation of action value function [18,19,20,21], we separate the action value network into a nonlinear encoder and a linear layer. The representation can be considered as the output of the nonlinear encoder. We start by investigating the Bellman equation [22] from the perspective of representation. We then theoretically develop our notion of 'good' representation from the Bellman update of the action value function, i.e., a good representation of action value function should have an inherent representation gap from its target value network. We then experimentally check the representation of action value function and its target, finding that the representation of value function and its target grow similarly as training, resulting in an undesirable phenomenon, which we call representation overlap. This similarity leads to the collapse of the representation gap. The similarity we catch is not natural, because the input of the corresponding neural network is different state-action pairs, and the action value network is different from its target.
The aforementioned similarity of representation between value function and its target inspired us to improve the representation capacity of the corresponding network by activating the representation gap. Thus, we propose an easy-to-implement and effective framework, Policy Optimization from Preventing Representation Overlaps (POPRO), to activate representation gap and prevent the representation overlap for boosting the performance of DRL algorithms. Specifically, POPRO regularizes the policy evaluation phase by pushing the representation of the value function away from its target, while keeps the policy optimization phase unchanged. We study the representations of the POPRO framework on PyBullet continuous control suite [23], and we find that the representation similarity between the value function and its target has been significantly alleviated. Meanwhile, the empirical performance of POPRO outperforms other tested algorithms such as TD3 [15], METD3 [24], etc. We then extend POPRO to high dimensional input scenarios, DMControl suite [25]. The empirical performance of POPRO outperforms or matches the compared baselines such as CURL [5], DREAMER [7], etc. POPRO using pixel inputs outperforms or parallels the sample-efficiency of methods that use state-based features. We show that the DRL algorithm can be significantly improved by activating the representation gap between the action value function and its target.
In this work, we make the following contributions. (i) We theoretically show that there should exist a representation gap between action value function and its target. (ii) We define representation overlap phenomenon as representations (of the value function and its target) tend to grow similar when training the value function. (iii) To activate the representation gap, we propose an easy-toimplement and effective framework POPRO which adds a regularizer to penalize the representation overlap phenomenon. In addition, we also provide the convergence rate guarantee of POPRO. (iv) To demonstrate the effectiveness of the POPRO framework, we evaluate it against PyBullet and DMControl suites. The empirical results show that POPRO outperforms or matches the state-of-the-art representation learning RL methods.
Most prior works of representation learning use auxiliary tasks or mutual-information based representation learning. The UNREAL algorithm [39] added unsupervised auxiliary tasks to conventional deep RL methods. The PBL agent [40] achieved good performance in simulation settings by adding an auxiliary optimization term to objectives that utilize the forward and backward prediction history information of neural networks. The CPC [41] algorithm introduced contrastive loss in time level and achieve good performance in the simulator. CURL algorithm [5] introduces multiple auxiliary tasks including augmentation, contrastive learning.
Recently, there emerged several works studying representation learning from a geometric view [43,44,38]. [43,44] considered geometry of value functions, while [38] studies the geometry of policy functions.
However, there are limited prior works that explicitly consider representation capacity loss [45], which imposes regularization to force the network to converge to the original weights. Our work differentiates from previous works from the following three perspectives. First, our work origins from analyzing the inherent representation capacity of action value function and its target. Our insights are from activating the representation gap. Our proposed framework POPRO explicitly consider the representation capacity of action value network instead of considering how to learn good representations with the help of auxiliary tasks. We also provide the convergence rate guarantee of POPRO. Second, POPRO can couple with other algorithms adopting complex auxiliary unsupervised, self-supervised, and contrastive learning tasks. What's more, the experimental results show that POPRO is parallel to the method utilizing auxiliary tasks. The proposed framework POPRO is markedly clear and easy to implement. Third, our algorithm is also suitable for pixel input and vector input, which is verified in experiments. In this paper, we formulate RL as a Markov Decision Process (MDP) with a six-tuple < S, A, R, , 0 , >, where S is state space, A is action space, R : S × A → R is a scalar reward function, ( | , ) is transition probability function, 0 is the initial state distribution, and ∈ (0, 1) is the discount factor determining the rate of decay of importance rewards. At each time step , the agent encounters state and chooses an action w.r.t. its policy function , deterministic or stochastic, then encounters a new state +1 and a reward . RL aims to optimize the policy through return, which is defined as = = − ( , ). Action value function ( , ) represents the quality of a specific action in a state . Formally, action value (Q) function is defined as where trajectory is a state-action sequence ( 0 , 0 , 1 , 1 , 2 , 2 · · · ) induced by policy and transition probability function . A fourtuple ( , , , +1 ) is called a transition. The Q value can be recursively computed by Bellman equation [22] ( , where ∼ (·| , ) and ∼ ( ). The process of evaluating value function is known as policy evaluation phase.
When the action space is very large or continuous, indirectly obtaining the policy by action value function is intractable as DQN does [46]. Thus, Policy Gradient theorem [47] is introduced to optimize the policy directly  In this section, we first theoretically define the representation gap. Then we experimentally show the phenomenon that the representation of action value function would grow similar to that of its target. To activate the representation gap, we propose Policy Optimization from Preventing Representation Overlaps (POPRO) framework, which regularizes the policy evaluation phase through differing the representation of action value function from its target. We also provide the convergence rate guarantee of POPRO.

Representation gap
We define the representation of value function to facilitate subsequent discussions. Definition 4.1 (Representation of action value function). Given a multilayer neural network representing Q function parameterized by Θ, Θ represents the parameters of th layer, Θ −1 represents the parameters of the last layer, and Θ + represents the parameters of the neural networks except for those of the last layer. Then the representation Φ of Q function is defined as An intuitive way to understand the representation of action value function is that we can split the action value network as a nonlinear encoder and a linear part. The representation of action value function is the output of the nonlinear encoder. Definition 4.2 (Representation Gap). Given a multi-layer neural network representing Q function parameterized by Θ, Θ represents the parameters of th layer, Θ −1 represents the parameters of the last layer, and Θ + represents the parameters of the neural networks except for those of the last layer. Then the representation gap is defined as Proof. Check the Appendix section 7 for the proof.
The theorem 4.3 shows that there exists an inherent representation gap between the action value network and its target after the policy evaluation process converges. The Representation gap is natural, because the input of the corresponding neural network is different state-action pairs, and the action value network is different from its target.  The theorem 4.3 shows that there exists a representation gap between the action value function and its target. Thus, we experimentally check the representation gap between action value function and its target in two algorithm TD3 and CURL.

Collapse of representation gap
Firstly, we choose three computationally easy similarity measures to evaluate the representation gap. Specifically, we take normalized where n is the dimension of Φ 1 , which can measure the distance in each dimension. We also choose cosine similarity , which measures the similarity of the representation as a whole. We also measure the dot product of two representations, defined by Dot We show the experimental results in fig. 2, which shows that the representation of action value network grows similarly with its target when training. This similarity results in representation overlap. The similarity we catch is not natural, because the input of the corresponding neural network is different state-action pairs, and the action value network is different from its target. Representation overlap leads to the inactivity of the representation gap. The two inputs pairs ( , ), ( , ) of action value network and its target satisfy the dynamics of environments ( | , ) and policy (·|·). The two input pairs are contiguous in time. The replay buffer setting used by Deep Q-networks [46] is believed to break the correlation of the data at the trajectory level. However, we feed two slightly different neural networks with two adjacent inputs related ( , ) and ( , ) respectively. Thus, the correlation at transition level is naturally kept, which is hard to be broken up. This is a potential reason why the representation overlap happens.

Activating representation gap
Thus, we propose an easy-to-implement and effective framework, Policy Optimization from Preventing Representation Overlaps (POPRO), to activate representation gap and prevent the representation overlap in order to improve the performance of deep RL algorithms. Specifically, POPRO regularizes the policy evaluation phase by keeping the representation of the value function different from its target. For the policy optimization phase, POPRO keeps the conventional policy optimization method.
Section 4.2 shows that in experiments, there exists representation overlap between action value function and its target. The representation of action value function grow similarly compared with the target action value function. However, section 4.1 shows that the representation gap between action value function and its target is inherent. Thus, to improve the representation capacity of action value function, the agent needs to activate the representation gap. Thus, to activate the representation gap, we propose Policy Optimization from Preventing Representation Overlaps (POPRO) framework.
In the policy evaluation phase, POPRO regularizes the action value network by keeping the its representation different from its target network. For conventional policy evaluation phase [14, 15, 17], the optimization objective is For the policy evaluation phase, the optimization objective of POPRO is where is a hyper-parameter controlling the regularization effect of activating representation gap and simultaneously preventing representation overlap. For the policy improvement phase, POPRO can adopt any conventional policy gradient method such as deterministic policy gradient [49] We also provide a theoretical guarantee of the convergence of our algorithm by theorem 4.5. Assumption 4.4. The 2 -norm of is uniformly bounded by the square of some positive constant , i.e. Φ( ; Θ + ) 2 ≤ 2 for any ∈ S × A and network weights Θ.
Let be the Bellman Operator. We have the following convergence result for the core update step in POPRO. Theorem 4.5 (One-step Approximation Error of POPRO Update). Suppose assumption 4.4 hold, let F ⊂ B (S × A) be a class of measurable function on × that are bounded by max = max /(1 − ), and let be a probability distribution on S × A. Also, let {( , )} ∈ [ ] be i.i.d. random variables in S × A following . For each ∈ [ ], let and be the reward and the next state corresponding to ( , ). In addition, for ∈ F , we define we defineˆas the solution to the lease-square with regularization problem, Meanwhile, for any > 0, let N ( , F , · ∞ ) be the minimal -covering set of F with respect to ∞ -norm, and we denote by its cardinality. Then for any ∈ (0, 1] and any > 0, we have where and are two absolute constants and is defined as We defer the proof of this theorem to Appendix section 7. We follow the proof style of [50].  [23] and DMControl suites [25]. The proposed algorithm POPRO is simple and easy to implement. Considering the recent concerns of reproducing crisis [51,52], we do not add any engineering tricks to the implementation of POPRO so that POPRO achieves the purpose as we initially designed.
The reason why we did not conduct an ablation analysis is because the POPRO framework only adds a regularization term to its backbone algorithm. Thus, the comparison with its backbone algorithm naturally becomes an ablation experiment.

Experimental settings
Random seeds. For the random seeds, if not otherwise specified, we evaluate each tested algorithm over 10 random seeds to ensure the reproducibility of our experiments. Also, we set all seeds fixed in our experiments including but not limited to those used in PyTorch, NumPy, Gym, and CUDA.
Environments. For the experiment environments, we use state-based PyBullet [23] and pixel-based DMControl [25] suites to measure the performance and sample-efficiency of POPRO. We can check the representation capacity of POPRO on state-based and pixel-based suites. Control tasks in PyBullet are generally considered harder than that of MuJoCo [53] suite [54]. For the interactive protocol, we utilize Gym [48]

Results
Throughout the paper, our proposed framework POPRO uses only one hyperparameter , controlling the magnitude of the regularization effect. And all the experiments are reported based on = 5 − 4. The reader can obtain better performance improvement by choosing a hyper-parameter beta that is more suitable for a specific environment.
PyBullet Suite. To validate the empirical performance of the POPRO framework, we firstly evaluate our proposed algorithm in the state-based suite PyBullet. We show the performance curves of experimental results in fig. 3, and the final 10 evaluation average return in

Limitations and Conclusion
We identify three limitations of this paper. First, we did not reveal the relationship between the representation gap and other contrastive unsupervised representations for RL. The experiments suggest that the two classes of methods might be orthogonal but in-depth investigations are required. Second, the performance of our POPRO framework matches that of State SAC. However, it is of great intellectual interest to further analyze why this happens. Third, in this work, we only carried out experiments to couple POPRO with TD3 and CURL algorithms. It remains to be checked how POPRO performs when coupled with more DRL algorithms that utilizing auxiliary tasks. Addressing these limitations are meaningful future directions.
In this work, we started by investigating the representations of the action value function and its target and found that there should exist an inherent representation gap between them. We then experimentally check the similarity between action value network and its target for TD3 and CURL methods. The results show that the representation of the action value network grows similar to its target, resulting in representation overlap. To activate the representation gap, we proposed the POPRO framework and provide the convergence rate guarantee of POPRO. We conduct extensive experiments to measure the performance and sample-efficiency of the POPRO framework, where POPRO shows superior performance compared to the tested algorithms. We believe our work sheds light on the nature of the inherent representation gap resulting from combining parameterization tool deep neural network and reinforcement learning.  Proof. Following eq. (4), the Bellman Equation eq. (2) can be rewritten as After the policy evaluation converges, the Θ, Θ satisfy EΘ = EΘ .
Let 1 , · · · , be the centers of minimal -covering the of F . By the definition of -covering, there exists * ∈ [ ] such that ˆ− * ∞ ≤ . Notice that * is a random variable sinceˆis obtained from data. Theorem 7.2 (One-step Approximation Error of POPRO Update). Suppose assumption 4.4 hold, let F ⊆ B (S × A) be a class of measurable function on S × A that are bounded by max = max /(1 − ), and let be a probability distribution on S × A. Also, let {( , )} ∈ [ ] be i.i.d. random variables in following . Based on {( , , )} ∈ [ ] , we defineˆas the solution to the lease-square with regularization problem, At the same time, for any > 0, let N ( , F , · ∞ ) be the where and are two absolute constants and is defined as Proof.
Step (i): We relate E[ ˆ− 2 ] with its empirical counterpart ˆ− 2 . Since = + max ∈A ( +1 , ) for each ∈ [ ]. By the definition ofˆ, for any ∈ F we have For each ∈ [ ], we define = − ( ) ( ). Then eq. (17) can be rewritten as We start by bounding the value of Φ ( ; Θ )EΦ( +1 ; Θ ) − Φ ( ; Θˆ)EΦ( +1 ; Θ ˆ) . First, by Cauchy-Schwartz Inequality, we have where we used assumption 4.4 for the second inequality. Thus, by triangle inequality, we have And eq. (18) reduces to Then we bound the rest in the right side of eq. (18). Since both and are deterministic, we have Moreover, since E( | ) = 0 by definition, we have E[ · ( )] = 0 for any bounded and measurable function . Thus it holds that In addition, by triangle inequality and eq. (22) we have where * satisfies * ≤ . In the following, we upper bound the two terms on the right side of eq. (23) respectively. For the first term, by applying the Cauchy-Schwarz inequality twice, we have where we use the fact that { } ∈ [ ] have the same marginal distributions and ˆ− * ≤ . Since both and are bounded by max , is a bounded random variable by its definition. Thus, there exists a constant > 0 depending on such that E( 2 ) ≤ 2 · 2 max . Then eq. (24) implies It remains to upper bound the second term on the right side of eq. (23). We define self-normalized random variables where the first inequality follows from triangle inequality and the second follows from the fact that ≤ eq. (27), we obtain where the last inequality follows from Moreover, since is centered conditioning on { }, is a sub-Gaussian random variable. Specifically, there exists an absolute constant > 0 such that 2 ≤ · max for each ∈ [ ]. Here the 2 -norm of a random variable is defined as Moreover, since is a summation of independent sub-Guassian random variables, by Lemma 5.9 of [61], the 2 -norm of satisfies where > 0 is an absolute constant. Furthermore, by Lemma 5.14 and 5.15 of [61], 2 is a sub-exponential random variable, and its moment-generating function is bounded by for any satisfying · | | · 2 · 2 max ≤ 1, where and are two positive absolute constants. Moreover, by Jensen's Inequality, we bound the moment-generating function of max ∈ [ ] 2 by Combining eq. (33) and eq. (34), we have where > 0 is an absolute constant. Hence, plugging eq. (35) into eq. (27) and eq. (28), we upper bound the second term of eq. (22) by Finally, combining eq. (21), eq. (25) and eq. (36), we obtain the following inequality where and are two constants. Here in the first inequality we take the infimum over F because eq. (17) holds for any ∈ F , and the second inequality holds because log ≤ .
Finally, combining steps (i) and together, i.e., eq. (39) and eq. (52), we conclude that where 1 and 2 are two absolute constants. Moreover, since ∈ F which concludes the proof of theorem 4.5.

Appendix B: Experimental Settings
In this section, we provide the experimental settings in detail.

Training details
Computational resources. All experiments are conducted on two GPU servers. The first one has 3 Titan XP GPUs and Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz. The second one has 4 Titan RTX GPUs and an Intel(R) Xeon(R) Gold 6137 CPU @ 3.90GHz. Each random seed for DMControl takes 2 days to finish. For PyBullet and MuJoCo tasks, it takes 5 hours to finish a random seed. For PyBullet and MuJoCo suite, we simultaneously launch 70 seeds. For the DMControl suite, we simultaneously run 15 random seeds.
Random seeds. If not otherwise specified, we evaluate each tested algorithm over 10 random seeds to ensure the reproducibility of our experiments. Also, we set all seeds fixed in our experiments.
PyBullet. When we train the agent in the Pybullet suite, the agent starts by randomly collecting 25,000 states and actions for better exploration. Then we evaluate the agent for ten episodes every 5,000 timesteps. We take the average return of ten episodes as a key evaluation metric. Note that for a more fair evaluation of the algorithms, at the evaluation phase, we do not apply any exploration tricks in the algorithms (e.g. injecting noise into actions in TD3), because these exploration tricks may harm the performance of tested algorithms. The complete timesteps are 1 million. The results are reported over ten random seeds. For the hyper-parameter of POPRO, we take 5 − 4 for every task and never change it. Note that statistics in Table table 2

Missing Algorithms
Our POPRO implementations are based on TD3 and CURL respectively. We present POPRO based on TD3 in algorithm 2. Algorithm 2 POPRO (based on TD3) 1: Initialize actor network , and critic network for = 1, 2 with random parameters , 1 , 2 2: Initialize target networks ← , ← 3: Initialize replay buffer B 4: Initialize , , ,˜, , total steps , and = 0 5: Reset the environment and receive initial state 6: while < do  In this section, we provide additional experimental results.

Similarity measures
We present experiments under Manhattan distance and cosine similarity measures in fig. 5  The shaded area stands for a standard deviation. Column: various environments. Row: different algorithm. The representations of action value networks of TD3 and CURL agents grow similar as training processing, which results in the collapse of representation gap. But POPRO framework does not.

Experiments on MuJoCo suite
We present the performance of POPRO on MuJoCo suite in table 5. The results show that our proposed framework POPRO outperforms or matches the compared algorithms in 5 out 7 MuJoCo environments. Compared with its backbone algorithm TD3, the POPRO framework outperforms it in 6 out of 7 environments.

Experiments on 16 DMControl tasks
We present the performance curves of POPRO on a total on a total of 16 DMControl environments in fig. 6. We run 4 seeds in each environments. The POPRO framework always achieves good performance in the environments tested.  Figure 6: Performance curves for DMControl suite. The shaded region represents a standard deviation of the average evaluation over 4 seeds. The curves are smoothed by moving average. The POPRO framework always achieves good performance in the environments tested.