{'fraction_non_alphanumeric': 0.03001588742623695, 'fraction_numerical': 0.009589196550158875, 'mean_word_length': 5.075491209927611, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback-Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.


Introduction
Deep reinforcement learning (RL) has been successfully applied to various tasks, including Go [22], Atari [13] and robot control tasks [21], etc. However, a growing body of research has shown that it remains challenging to learn deep RL agents that are able to generalise to new environments [5]. Agents can 'overfit' to the training environments visual appearance [25,5] physical dynamics [14,29] or even specific training seeds [27]. If there is a new environment, or domain-shift, between training and testing, RL tend to under-perform significantly. This problem has longer been appreciated, and is particularly salient, in the field of robotics, where there is an inevitable reality gap [25,11] between the simulated environments used for training and deployment in the real-world. These issues have motivated an important line of research in improving zero-shot domain transfer [5,1], i.e., to learn generalised policies from training domains that can be directly and successfully deployed in unseen testing domains without further learning or adaptation.
One common strategy to improve policy generalisation is to apply domain randomisation during training. With domain randomisation, we first generate a random set of, or distribution over, domains with different properties such as observation function [25], dynamics [16], or both [1]. Then a policy is trained with this distribution, for example by drawing a new random sample from the domain distribution during each learning episode. If this distribution of training domains is rich enough to include within its support properties of the real-world or target environment -and if a policy can be successfully trained to fit the entire training domain distribution -then that policy can be successfully deployed in the target environment without adaptation. A challenge with domain randomisation is that training a single model on a wide distribution of domains usually leads to high variance gradient estimates, making the RL policy optimisation problem challenging [12]. To ameliorate this issue, several studies have proposed distillation techniques where an ensemble of workers are each trained locally in a different domain -where gradients are lower variance, and hence RL is more stable. The local workers are then distilled into a single global policy that should perform across all domains ( LG] 9 Dec 2020 [7,24,18,15,6]. Distilling local policies into a global policy with a supervised objective provides a more stable way to learn from a broad distribution over domains and ultimately enable better generalisation to held-out testing domains -albeit at some additional computational cost [7]. This work continues this line of investigation. Our main contribution is to show that this centralised distillation is unnecessarily, and indeed sub-optimal. In particular, we propose an online distillation framework, where each worker both learns to optimise performance in a local domain and also mimics its peers from other domains with peer-to-peer distillation. More specifically, inspired by deep mutual learning [28,2], we train each worker with two losses: a conventional RL loss and a distillation loss that measures the similarity in predictions between the local worker and the others. In this work, we use expected Kullback-Leibler (KL) divergence between predicted distributions as the objective to optimise for information exchange. This online distillation framework is named Peer-to-Peer Distillation Reinforcement Learning (P2PDRL). We empirically show that, compared to conventional baselines, and offline distillation alternatives, e.g. divide-and-conquer (DnC) reinforcement learning [7], P2PDRL achieves more competitive learning performance without the additional cost of a centralised distillation step. Overall P2PDRL learns more quickly, succeeds to learn on a wider distribution of domains, and transfers better to novel testing environments compared to competitors.
The key contributions of this work are summarised as: • We propose P2PDRL, an online distillation framework for deep RL agents that uses an ensemble of local workers that learn with local experience only, while sharing knowledge via regularisation on the statistical distance of their prediction distribution; • Our experiments show that P2PDRL can achieve more effective learning on a wider distribution of training environments compared to competitors, without distilling to a centralised policy. All workers show increased robustness to the domain shift of novel environments.

Related Work
Domain Generalisation in Deep RL A growing body of work has discussed generalisation problems in reinforcement learning. In the context of tasks with discrete and continuous observation/action spaces, [5] and [27] concluded that deep RL policies could easily overfit to random seeds used during training and underperform during testing. In the context of domain-shift between training and testing, [29] and [14] further highlighted the problem that deep RL agents often fail to generalise, e.g., across different friction coefficients or wind conditions.
To improve RL agents' performance in testing environments, several adaptation methods have been proposed [8,19]. These use data from testing domains and aim to perform sample efficient adaptation. Recent advances [4,17] perform meta-learning on training environments to learn how to adapt efficiently to testing environments. Where direct deployment to target domains without adaptation is desired or necessary, a common approach is domain randomisation. Here the training simulator is setup to provide a diverse array of environments for learning in terms of observations [20,25,1] or dynamic perturbations [23]. A key challenge here is that, in absence of a known and precisely-specified model of the testing domain, one must define a rather wide distribution of training domains to be confident that its span includes likely testing domains. However, learning from such a diverse training signal is extremely challenging in RL, and often leads to unstable learning and poor convergence [12,26], as we also illustrate here in Fig. 1.
In this paper we address this dichotomy between overfitting of agents to individual domains, and inability of agents to fit to a wide distribution of domains. Our framework enables state of the art on-policy methods [21] to effectively and stably learn a wide distribution of domains by performing local RL and global knowledge exchange via distillation.
Policy Distillation The notion of model distillation [10] has increasingly been applied to policy distillation in reinforcement learning. It is often used for learning from multiple source tasks, where a student agent is trained to mimic the behaviour of one or multiple teacher policies [6]. Several early studies [18,15] assumed pre-trained teacher policies and trained one student to match the state-dependent probability of the teacher's predicted actions. Later methods [24] learned a global policy in parallel with learning multiple local policies: the global policy is trained to distil from local individuals while local policies learning is regularised by the global policy. Finally, [7] proposed to Compared to these methods, we propose a online peer-to-peer distillation framework that avoids explicitly distilling to a global policy. Instead, we learn a cohort of workers collaboratively by performing local RL and global peer-to-peer knowledge distillation. Peers use their own local states for distillation, thus significantly simplifying data exchange between workers compared to alternatives such as [7]. This idea of online distillation has also been explored in supervised learning to achieve learning better performance in single-task learning [28] or to scale single-task supervised learning to large datasets through distributed computing [2]. To our knowledge, it has not been explored in RL, or as a tool to enable learning on a wider distribution of training domains, and hence improved robustness and cross-domain generalisation.

Preliminaries
An episodic Markov decision process (MDP) is defined by S, A, R, T, γ , where S is the state space, A is the action space, R : S ×A → R is the reward function, T : S ×A → S is the transition function, and γ is the discount factor. The objective of a learning agent is to learn a stochastic policy π * : S × A → R that maximises the expected cumulative return: π * = arg max π E π ∞ t=0 γ t R(s t , a t ). We denote the parameters that describe a domain as ξ. In domain randomisation, each training domain is randomly sampled with domain parameters ξ from a pre-defined set Ξ. Thus, the modified objective function becomes Policies trained with a diverse set of domains should generalise better to unseen testing domains [1]. However, in practice, diverse training domains lead to high variance in gradients [12], thus leading to poor learning behaviour. In practice, this manifests as a significant drop in training performance as diversity of training domains increases. In such cases, the effect of the policy's inability to fit the training task at all dominates its greater exposure to diverse environments due to domain randomisation. Thus performance in both training and testing domains is poor.
This challenge is illustrated in Fig. 1, which shows the performance of an agent during training as well as its performance when tested on a held-out domain -all plotted as a function of training set diversity. As the diversity of training domains increases from zero, we see that training performance decreases continually as it becomes harder for a single policy to fit an increasingly diverse distribution of environments (Fig. 1, below). More interestingly, we also observe that testing performance initially increases, as the increased domain randomisation benefits robustness to the held out domain (Fig. 1  above). However as training diversity continues to increase, testing performance starts to drop, as the stronger domain randomisation challenge makes it difficult for the policy to learn the task at all.

Online Distillation
To avoid training with data sampled from all domains, we propose to assign each worker to a domain, and train it with data from its local domain only. Alongside the conventional RL loss, workers are regularised by each other through an online distillation loss, which encourages all workers to act similarly across all domains given the same state input.

Local Optimisation
We formulate the proposed peer-to-peer distillation method with a cohort of K workers, as illustrated in Fig. 2. At each iteration, each worker randomly samples a domain and generates trajectory data by following its local policy. The objective function includes a conventional RL loss and a KL divergence based distillation loss that matches predicted distributions across all workers. In particular, we use proximal policy optimisation (PPO) [21], a state of the art on-policy deep reinforcement learning algorithm, as the base optimiser. A surrogate loss is used in PPO: where is the hyperparameter, and A t,ξ is the estimated advantage in domain ξ at timestep t. The critic networks, parameterised by φ i , are trained locally with conventional MSE loss, where target values are computed through Bellman equations using local trajectory data.
Peer-to-Peer Distillation For online distillation across different workers, we minimise KL divergence between workers' predictions. More specifically, for the i-th worker, where E πi,ξi (·) is the expectation over trajectory data generated in domain ξ i with policy π i .
Discussion Our overall algorithm is summarised in Fig. 2  Collect trajectory data τ i following policy π i for T timesteps in domain ξ i . for i = 1 to K do 12: Sample minibatch B i from τ i 13: 14: Compute target value V targ with Bellman equation. 15:  Fig. 3. To generate different domains, we change wind condition, gravity constant, friction coefficient, robot mass and initial state distribution in simulation. We summarise the diversity of randomised distribution as a scalar ∈ [0, 1]. Higher means more diverse distributions. Given a specific , dynamic parameters are randomly sampled from a uniform distribution defined by . For example, robot mass in Walker task m is sampled from m ∼ U(m 0 (1 − 0.5 ), m 0 (1 + 0.5 )). Detailed task descriptions are listed in the Appendix.
Our experiments are designed to address the following questions: Q1: How does P2PDRL compare to baselines in sample efficiency and asymptotic return?
Q2: Does P2PDRL learn policies that are more robust against domain shifts than competing methods?
We compare our P2PDRL, with the following prior methods as baselines: PPO [21]. PPO provides the state of the art on-policy Deep RL method, which is also used as the basic learning algorithms across all settings. At each iteration, agents are optimised with the union of datasets from all sampled domains.
Distributed PPO [9]. In contrast to PPO, where one agent is optimised by the rich data set, in this setting, workers gather gradient information with local data and a central learner updates a global policy with the average of gradients across all domains.
Distral [24]. Originally focused on knowledge transferring when learning multiple tasks, Distral trains an ensemble of task-specific policies, constrained against a single global policy at each gradient step. The global policy is trained with supervised learning, to distil the common behaviours from task-specific policies.
DnC [7]. Originally focused on providing robustness to choice of initial state in single domain learning, DnC partitions the initial state distribution into multiple sub-domains, and alternates between local policy optimisation steps and global distillation steps. In this work, we go beyond initial states, and train agents in the domains with various sets of dynamics parameters. Moreover, we do not require domain knowledge to partition the distribution. Instead, domains are sampled from the same distribution for all workers.
Implementation Details To demonstrate the performance of our proposed method, we take K = 2 workers in parallel, regularised by each other. Except where noted otherwise, we control the total number of timesteps to compare sample efficiency fairly: At each iteration, the vanilla PPO samples 4096 steps; and each worker in other algorithms samples 2048 steps. In DnC and P2PDRL, half the number of samples are used by each actor to train locally. For the implementation of actor and critic networks, we use two separate MLP networks with two hidden layers, 64 units in each layer. For each task, we run a hyperparameter sweep for each method, and report the best performing agents. All results are averaged over 8 random seeds. Full implementation details are listed in the Appendix.

Training and Generalisation Performance
Training Efficiency Under Domain Randomisation As training distribution, we take tr = 0.2 for all five tasks. From the learning curves shown in Fig. 3(middle), we can see that P2PDRL generally outperforms the baseline methods in terms of asymptotic performance across all 5 tasks, and is able to learn more sample-efficiently in some tasks, such as Ant and HalfCheetah.
Generalisation to Novel Environments We next evaluate the generalisation performance of the policy learned above, by evaluating it on novel testing domain with different randomisation strengths te . Fig. 3(right) shows that P2PDRL learned policies are able generally to generalise better against domain shifts to novel environments.

Dependence of Testing Performance on Randomisation Strength in Training
We also demonstrate asymptotic training return of a policy and its generalisation performance as a function of range of training domains, as initially motivated in Fig. 1. We train our policy with different levels of randomisation tr and test all learned policies in a pre-defined target domain where te = 0.5. As shown in Fig. 4, P2PDRL generally outperforms vanilla PPO, but their performance levels decrease in tandem as the training distribution becomes more diverse and harder to fit with a single model. Crucially, as discussed earlier, the generalisation performance of PPO in terms of testing return drops rapidly after around tr = 0.5. Meanwhile the testing return of P2PDRL is stable up to a much higher level of training domain randomisation.

Task Description
For our experiments, we modify five continuous control tasks from OpenAI gym, including Walker2d, Ant, Humanoid, Hopper and HalfCheetah. We use a scalar ∈ [0, 1] to control the diversity of randomised domains. The pre-defined distributions are listed in Table. 1.  We report the best performing set of hyperparameters.