{'fraction_non_alphanumeric': 0.035425078635643696, 'fraction_numerical': 0.01097552814152185, 'mean_word_length': 4.6522506619594, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.


Introduction
Recent progress in reinforcement learning (RL) has achieved very impressive results ranging from playing games [1,2], to applications in dialogue systems [3], to robotics [4]. Despite the progress, the learning algorithms for solving many of these tasks are designed to deal with stationary environments. On the other hand, real-world is often nonstationary either due to complexity [5], changes in the dynamics or the objectives in the environment over the life-time of a system [6], or presence of multiple learning actors [7]. Nonstationarity breaks the standard assumptions and requires agents to continuously adapt, both at training and execution time, in order to succeed.
Learning under nonstationary conditions is challenging. The classical approaches to dealing with nonstationarity are usually based on context detection [8] and tracking [5], i.e., reacting to the already happened changes in the environment by continuously fine-tuning the policy. Unfortunately, modern deep RL algorithms, while able to achieve super-human performance on certain tasks, are known to be sample inefficient. At the same time, nonstationarity allows only for limited interaction before the properties of the environment change. Thus, it immediately puts learning into the few-shot regime and often renders simple fine-tuning methods impractical.
A nonstationary environment can be seen as a sequence of stationary tasks, and hence we propose to tackle it as a multi-task learning problem [9]. The learning-to-learn (or meta-learning) approaches [10,11] are particularly appealing in the few-shot regime, as they produce flexible learning rules that can generalize from only a handful of examples. Meta-learning has shown promising results in the supervised domain and have gained a lot of attention from the research community recently [12][13][14][15]. In this paper, we develop a gradient-based meta-learning algorithm similar to [15] and suitable for continuous adaptation of RL agents in nonstationary environments. More concretely, our agents meta-learn to anticipate the changes in the environment and update their policies accordingly.

Related Work
The problem of continuous adaptation considered in this work is related to continual or life-long [11,22] and never-ending [23] learning problems. Life-long learning systems aim at solving multiple tasks sequentially by efficiently transferring and utilizing knowledge from already learned tasks to the new ones while minimizing the effect of catastrophic forgetting [24]. Never-ending learning is concerned with mastering a fixed set of tasks in iterations such that the set keeps growing and the performance on all the tasks in the set keeps improving from iteration to iteration.
The scope of continuous adaptation is much narrower and more precise. While life-long and neverending learning settings are defined as general multi-task problems [22,23], continuous adaptation targets to solve a single but nonstationary task or environment. The nonstationarity in the former two problems exists and is dictated by the selected sequence of tasks. In the latter case, we assume that nonstationarity is caused by some underlying dynamics in the properties of a given task in the first place (e.g., changes in the behavior of other agents in a multi-agent setting). Finally, in the life-long and never-ending scenarios the boundary between training and execution is blurred as such systems constantly operate in the training regime. Continuous adaptation, on the other hand, expects a (potentially trained) agent to adapt to the changes in the environment at execution time under the pressure of limited data or interaction experience with the environment between the changes 2 .
Nonstationarity of multi-agent environments is a well known issue that has been extensively studied in the context of learning in simple multi-player iterated games (such as rock-paper-scissors) where each episode is one-shot interaction [26][27][28]. In such games, discovering and converging to a Nash equilibrium strategy is a success for the learning agents. Modeling and exploiting opponents [29,30] or event their learning processes [31] is advantageous as it improves convergence or helps to discover equilibria of certain properties (e.g., leads to cooperative behavior). In contrast, each episode in RoboSumo consists of multiple steps, happens in continuous time, and requires learning a good intra-episodic controller. Finding Nash equilibria in such setting is hard. Thus, fast adaptation becomes one of the few viable strategies against changing opponents. (c) Figure 1: (a) A probabilistic model for MAML in a multi-task RL setting. The task, T , the policies, π, and the trajectories, τ , are all random variables with dependencies encoded in the edges of the given graph. (b) Our extended model suitable for continuous adaptation to a task changing dynamically due to non-stationarity of the environment. Policy and trajectories at a previous step are used to construct a new policy for the current step. (c) Computation graph for the meta-update from φi to φi+1. Boxes represent replicas of the policy graphs with the specified parameters. The model is optimized via truncated backpropagation through time starting from LT i+1 .
Our proposed method for continuous adaptation follows the general meta-learning paradigm [10,11], i.e., it learns a high-level procedure that can be used to generate a good policy each time the environment changes. There is a wealth of work on meta-learning, including methods for learning update rules for neural models that were explored in the past [32][33][34], and more recent approaches that focused on learning optimizers for deep networks [13,[35][36][37], generating model parameters [38][39][40], learning task embeddings [14,41] including memory-based approaches [12], learning to learn implicitly via RL [42,43], or simply learning a good initialization in a model-agnostic fashion [15].

Method
The problem of continuous adaptation in nonstationary environments immediately puts learning into the few-shot regime: the agent must learn from only limited amount of experience that it can collect before its environment changes. Therefore, we build our method upon the previous work on gradient-based model-agnostic meta-learning (MAML) that has been shown successful in the few-shot settings [15]. In this section, we re-derive MAML for multi-task reinforcement learning from a probabilistic perspective, and then extend it to the case of dynamically changing tasks.

A probabilistic view of model-agnostic meta-learning (MAML)
Assume that we are given a distribution over tasks, D(T ), where each task, T , is a tuple: (1) L T is a task-specific loss function that maps a trajectory, τ := (x 0 , a 1 , x 1 , r 1 , . . . , a H , x H , r H ) ∈ T , to a loss value, i.e., L T : T → R; P T (x) and P T (x t+1 | x t , a t ) define the Markovian dynamics of the environment in task T ; H denotes the horizon; observations, x t , and actions, a t , are elements (typically, vectors) of the observation space, X , and action space, A, respectively. The loss of a trajectory, τ , is the negative cumulative reward, L T (τ ) := − Algorithm 1 Meta-learning at training time.

Continuous adaptation via meta-learning
In the classical multi-task setting, we make no assumptions about the distribution of tasks, D(T ). When the environment is nonstationary, we can see it as a sequence of stationary tasks on a certain timescale where the tasks correspond to different dynamics of the environment. Then, D(T ) is defined by the environment changes, and the tasks become sequentially dependent. Hence, we would like to exploit this dependence between consecutive tasks and meta-learn a rule that keeps updating the policy in a way that minimizes the total expected loss encountered during the interaction with the changing environment. For instance, in the multi-agent setting, when playing against an opponent that changes its strategy incrementally (e.g., due to learning), our agent should ideally meta-learn to anticipate the changes and update its policy accordingly.
In the probabilistic language, our nonstationary environment is equivalent to a distribution of tasks represented by a Markov chain (Fig. 1b). The goal is to minimize the expected loss over the chain of tasks of some length L: Here, P(T 0 ) and P(T i+1 | T i ) denote the initial and the transition probabilities in the Markov chain of tasks. Note that (i) we deal with Markovian dynamics on two levels of hierarchy, where the upper level is the dynamics of the tasks and the lower level is the MDPs that represent particular tasks, and (ii) the objectives, R Ti,Ti+1 , will depend on the way the meta-learning process is defined. Since we are interested in meta-learning updates that are optimal with respect to the Markovian transitions between the tasks, we define the expected loss on a pair of consecutive tasks as follows: The principal difference between the loss in (3) and (6) is that trajectories τ 1:K i,θ come from the current task, T i , and are used to construct a policy, π φ , that is good for the upcoming task, T i+1 .
Note that even though the policy parameters, φ i , are sequentially dependent (Fig. 1b), in (6) we always start from the initial parameters, θ 3 . Hence, optimizing R Ti,Ti+1 (θ) is equivalent to truncated backpropagation through time with a unit lag in the chain of tasks.
To construct parameters of the policy for task T i+1 , we start from θ and do multiple 4 meta-gradient steps with adaptive step sizes as follows (assuming the number of steps is M ): where {α m } M m=1 is a set of meta-gradient step sizes that are optimized jointly with θ. The computation graph for the meta-update is given in Fig. 1c. The expression for the policy gradient is the same as in (4) but with the expectation is now taken w.r.t. to both T i and T i+1 : More details and the analog of the policy gradient theorem for our setting are given in Appendix A.
Note that computing meta-updates requires interacting with the environment under π θ while computing the loss, R Ti,Ti+1 , requires using π φ , and hence, interacting with each task in the sequence twice. This is often impossible at execution time, and hence we use slightly different algorithms at training and execution times.
Meta-learning at training time. Once we have access to a distribution over pairs of consecutive tasks 5 , P(T i−1 , T i ), we can train the meta-updates by optimizing θ and α jointly with a gradient method, as given in Algorithm 1. We use π θ to collect trajectories from T i and π φ when interacting with T i+1 . Intuitively, the algorithm is searching for θ and α such that the meta-update (7) computed on the trajectories from T i brings us to a policy π φ that is good for solving T i+1 . The main assumption here is that the trajectories from T i contain some information about T i+1 . Note that we treat metaupdates as part of the computation graph ( Fig. 1c) and optimize θ and α via backpropagation through the entire graph, which requires computing second order derivatives.
Adaptation at execution time. Note that to compute unbiased meta-gradient updates at training time, we have to collect experience in T i using π θ . At test time, due to environment nonstationarity, we usually do not have the luxury to access to the same task multiple times. Thus, we keep acting according to π φ and re-use past experience to for computing updates of φ for each new incoming task (see Algorithm 2). To adjust for the fact that the past experience was collected under a policy different from π θ , we use importance weight correction. In case of single step meta-update, we have: where π φi−1 and π φi are used to rollout from T i−1 and T i , respectively. Extending importance weight correction to multi-step updates is straightforward and requires simply adding importance weights to each of the intermediate steps in (7). underlying dynamics, and (ii) a competitive multi-agent environment, RoboSumo, that allows different agents to play sequences of games against each other and keep adapting to incremental changes in each other's policies. All our environments are based on MuJoCo physics simulator [47], and all agents are simple multi-leg robots, as shown in Fig. 2a.

Dynamic
First, we consider the problem of robotic locomotion in a changing environment. We use a six-leg agent ( Fig. 2b) that observes the absolute position and velocity of its body, the angles and velocities of its legs, and it acts by applying torques to its joints. The agent is rewarded proportionally to its moving speed in a fixed direction. To induce nonstationarity, we select a pair of legs of the agent and scale down the torques applied to the corresponding joints by a factor that linearly changes from 1 to 0 over the course of 7 episodes. In other words, during the first episode all legs are fully functional, while during the last episode the agent has two legs fully paralyzed (even though the policy can generate torques, they are multiplied by 0 before being passed to the environment). The goal of the agent is to learn to adapt from episode to episode by changing its gait so that it is able to move with a maximal speed in a given direction despite the changes in the environment [cf. 48,49].. Also, there are 15 ways to select a pair of legs of a six-leg creature which gives us 15 different nonstationary environments. This allows us to use a subset of these environments for training and a separate held out set for testing. The training and testing procedures are described in the next section.

Competitive
Our multi-agent environment, RoboSumo, allows agents to compete in the 1-vs-1 regime following the standard sumo rules 6 . We introduce three types of agents, Ant, Bug, and Spider, with different anatomies (Fig. 2a). During the game, each agent observes positions of itself and the opponent, its own joint angles, the corresponding velocities, and the forces exerted on its own body (i.e., equivalent of tactile senses). The action spaces are continuous.  Iterated adaptation games. To test adaptation, we define the iterated adaptation game (Fig. 3)-a game between a pair of agents that consists of K rounds each of which consists of one or more fixed length episodes (500 time steps each). The outcome of each round is either win, loss, or draw. The agent that wins the majority of rounds (with at least 5% margin) is declared the winner of the game. There are two distinguishing aspects of our setup: First, the agents are trained either via pure self-play or versus opponents from a fixed training collection. At test time, they face a new opponent from a testing collection. Second, the agents are allowed to learn (or adapt) at test time. In particular, an agent should exploit the fact that it plays against the same opponent multiple consecutive rounds and try to adjust its behavior accordingly. Since the opponent may also be adapting, the setup allows to test different continuous adaptation strategies, one versus the other.
Reward shaping. In RoboSumo, rewards are naturally sparse: the winner gets +2000, the loser is penalized for -2000, and in case of a draw both opponents receive -1000 points. To encourage fast learning at the early stages of training, we shape the rewards given to agents in the following way: the agent (i) gets reward for staying closer to the center of the ring, for moving towards the opponent, and for exerting forces on the opponent's body, and (ii) gets penalty inversely proportional to the opponent's distance to the center of the ring. At test time, the agents continue having access to the shaped reward as well and may use it to update their policies. Throughout our experiments, we use discounted rewards with the discount factor, γ = 0.995. More details on rewards are in Appendix D.2.
Calibration. To study adaptation, we need a well-calibrated environment in which none of the agents has an initial advantage. To ensure the balance, we increased the mass of the weaker agents (Ant and Spider) such that the win rates in games between one agent type versus the other type in the non-adaptation regime became almost equal (for details on calibration see Appendix D.3).

Experiments
Our goal is to test different adaptation strategies in the proposed nonstationary RL settings. However, it is known that the test-time behavior of an agent may highly depend on a variety of factors besides the chosen adaptation method, including training curriculum, training algorithm, policy class, etc. Hence, we first describe the precise setup that we use in our experiments to eliminate irrelevant factors and focus on the effects of adaptation. Most of the low-level details are deferred to appendices. Video highlights of our experiments are available at https://goo.gl/tboqaN.

The setup
Policies. We consider 3 types of policy networks: (i) a 2-layer MLP, (ii) embedding (i.e., 1 fullyconnected layer replicated across the time dimension) followed by a 1-layer LSTM, and (iii) RL 2 [43] of the same architecture as (ii) which additionally takes previous reward and done signals as inputs at each step, keeps the recurrent state throughout the entire interaction with a given environment (or an opponent), and resets the state once the latter changes. For advantage functions, we use networks of the same structure as for the corresponding policies and have no parameter sharing between the two. Our meta-learning agents use the same policy and advantage function structures as the baselines and learn a 3-step meta-update with adaptive step sizes as given in (7). Illustrations and details on the architectures are given in Appendix B.
Meta-learning. We compute meta-updates via gradients of the negative discounted rewards received during a number of previous interactions with the environment. At training time, meta-learners interact with the environment twice, first using the initial policy, π θ , and then the meta-updated policy, π φ . At test time, the agents are limited to interacting with the environment only once, and hence always act according to π φ and compute meta-updates using importance-weight correction (see Sec. 3.2 and Algorithm 2). Additionally, to reduce the variance of the meta-updates at test time, the agents store the experience collected during the interaction with the test environment (and the corresponding importance weights) into the experience buffer and keep re-using that experience to update π φ as in (7). The size of the experience buffer is fixed to 3 episodes for nonstationary locomotion and 75 episodes for RoboSumo. More details are given in Appendix C.1.
Adaptation baselines. Besides our proposed method for continuous adaptation via meta-learning, we consider three baseline strategies: (i) naive (or no adaptation), (ii) implicit adaptation via RL 2 , and (iii) adaptation via tracking [5] that uses PPO for online updates at execution time.
Training in nonstationary locomotion. We train all methods on the same collection of nonstationary locomotion environments constructed by choosing all possible pairs of legs whose joint torques are To evaluate adaptation strategies, we ran each of them in each environment for 7 episodes followed by a full reset of the environment, policy, and meta-updates (repeated 50 times). Shaded regions are 95% confidence intervals. Best viewed in color.
scaled except 3 pairs that are held out for testing (i.e., 12 training and 3 testing environments for the six-leg creature). The agents are trained on the environments concurrently, i.e., to compute a policy update, we rollout from all environments in parallel and then compute, aggregate, and average the gradients (for details, see Appendix C.2). LSTM policies retain their state over the course of 7 episodes in each environment. Meta-learning agents compute meta-updates for each nonstationary environment separately.
Training in RoboSumo. To ensure consistency of the training curriculum for all agents, we first pre-train a number of policies of each type for every agent type via pure self-play with the PPO algorithm [46]. We snapshot and save versions of the pre-trained policies at each iteration. This lets us train other agents to play against versions of the pre-trained opponents at various stages of mastery. Next, we train the baselines and the meta-learning agents against the pool of pre-trained opponents 7 concurrently. At each iteration k we (a) randomly select an opponent from the training pool, (b) sample a version of the opponent's policy to be in [1, k] (this ensures that even when the opponent is strong, sometimes an undertrained version is selected which allows the agent learn to win at early stages), and (c) rollout against that opponent. All baseline policies are trained with PPO; meta-learners also used PPO as the outer loop for optimizing θ and α parameters. We retain the states of the LSTM policies over the course of interaction with the same version of the same opponent and reset it each time the opponent version is updated. Similarly to the locomotion setup, meta-learners compute meta-updates for each opponent in the training pool separately. A more detailed description of the distributed training is given in Appendix C.2.
Experimental design. We design our experiments to answer the following questions: • When the interaction with the environment before it changes is strictly limited to one or very few episodes, what is the behavior of different adaptation methods in nonstationary locomotion and competitive multi-agent environments? • What is the sample complexity of different methods, i.e., how many episodes is required for a method to successfully adapt to the changes? We test this by controlling the amount of experience the agent is allowed to get form the same environment before it changes.
Additionally, we ask the following questions specific to the competitive multi-agent setting: • Given a diverse population of agents that have been trained under the same curriculum, how do different adaptation methods rank in a competition versus each other? • When the population of agents is evolved for several generations-such that the agents interact with each other via iterated adaptation games, and those that lose disappear while the winners get duplicated-what happens with the proportions of different agents in the population?
Finally, we conclude with discussing limitations of our meta-learning algorithm for continuous adaptation and potential avenues for future research.  At test time, both agents and opponents started from versions 700. Opponents' versions were increasing with each consecutive round as if they were learning via self-play, while agents were allowed to adapt only from the limited experience with a given opponent. Each round consisted of 3 episodes. Each iterated game was repeated 100 times; shaded regions denote bootstrapped 95% confidence intervals; no smoothing. Best viewed in color.

Adaptation in the few-shot regime and sample complexity
Few-shot adaptation in nonstationary locomotion environments. Having trained baselines and meta-learning policies as described in Sec. 5.1, we selected 3 testing environments that corresponded to disabling 3 different pairs of legs of the six-leg agent: back, middle, and front legs. The results are presented on Fig. 4. Three observations: First, during the very first episode, the meta-learned initial policy, π θ , turns out to be suboptimal for the task (it underperforms compared to other policies). However, after 1-2 episodes (and environment changes), it starts performing on par with other policies. Second, by the 6th and 7th episodes, meta-updated policies perform much better than the rest. Note that we use 3 gradient meta-updates for the adaptation of the meta-learners; the meta-updates are computed based on experience collected during the previous 2 episodes. Finally, tracking is not able to improve upon the baseline without adaptation and sometimes leads to even worse results.

Adaptation in
RoboSumo under the few-shot constraint. To evaluate different adaptation methods in the competitive multi-agent setting consistently, we consider a variation of the iterated adaptation game, where changes in the opponent's policies at test time are pre-determined but unknown to the agents. In particular, we pre-train 3 opponents (1 of each type, Fig. 2a) with LSTM policies with PPO via self-play (the same way as we pre-train the training pool of opponents, see Sec. 5.1) and snapshot their policies at each iteration. Next, we run iterated games between our trained agents that use different adaptation algorithms versus policy snapshots of the pre-trained opponents. Crucially, the policy version of the opponent keeps increasing from round to round as if it was training via self-play 8  (see Fig. 3). This setup allows us to test different adaptation strategies consistently against the same learning opponents.
The results are given on Fig. 5. We note that meta-learned adaptation strategies, in most cases, are able to adapt and improve their win-rates within about 100 episodes of interaction with constantly improving opponents. On the other hand, performance of the baselines often deteriorates during the rounds of iterated games. Note that the pre-trained opponents were observing 90 episodes of self-play per iteration, while the agents have access to only 3 episodes per round.
Sample complexity of adaptation in RoboSumo. Meta-learning helps to find an update suitable for fast or few-shot adaptation. However, how do different adaptation methods behave when more experience is available? To answer this question, we employ the same setup as previously and vary the number of episodes per round in the iterated game from 3 to 90. Each iterated game is repeated 20 times, and we measure the win-rates during the last 25 rounds of the game.
The results are presented on Fig. 6. When the number of episodes per round goes above 50, adaptation via tracking technically turns into "learning at test time," and it is able to learn to compete against the self-trained opponents that it has never seen at training time. The meta-learned adaptation strategy performed near constantly the same in both few-shot and standard regimes. This suggests that it meta-learned a particular bias at training time that allowed it perform better from limited experience but limited its capacity of utilizing more data. Note that, by design, the meta-updates are fixed to only 3 gradient steps from θ with step-sizes α (learned at training), while tracking keeps updating the policy with PPO throughout the iterated game. Allowing for meta-updates that become more flexible with the availability of data can help to overcome this limitation. We leave this to future work.

Evaluation on the population-level
Combining different adaptation strategies with different policies and agents of different anatomies puts us in a situation where we have a diverse population of agents which we would like to rank according to the level of their mastery in adaptation (or find the "fittest"). To do so, we employ TrueSkill [21]-a metric similar to the ELO rating, but more popular in 1-vs-1 competitive video-games. In this experiment, we consider a population of 105 trained agents: 3 agent types, 7 different policy and adaptation combinations, and 5 different stages of training (from 500 to 2000 training iterations). First, we assume that the initial distribution of any agent's skill is N (25, 25/3) and the default distance that guarantees about 76% of winning, β = 4.1667. Next, we randomly generate 1000 matches between pairs of opponents and let them adapt while competing with each other in 100-round iterated adaptation games (states of the agents are reset before each game). After each game, we record the outcome and updated our belief about the skill of the corresponding agents using the TrueSkill algorithm 9 . The distributions of the skill for the agents of each type after 1000 iterated adaptation games between randomly selected players from the pool are visualized in Fig. 7.
There are a few observations we can make: First, recurrent policies were dominant. Second, adaptation via RL 2 tended to perform equally or a little worse than plain LSTM with or without tracking in this setup. Finally, agents that meta-learned adaptation rules at training time, consistently demonstrated higher skill scores in each of the categories corresponding to different policies and agent types. * The rank is a conservative estimate of the skill, r = µ − 3σ, to ensure that the actual skill of the agent is higher with 99% confidence. Bug+LSTM-meta -1

Ant+LSTM-meta -2
Bug+LSTM-track -3 Ant+LSTM-RL2 -4 Ant+LSTM -5 Bug+MLP-meta -6 Ant+MLP-meta -7 Spd+MLP-meta -8 Spd+MLP -9 Bug+MLP Since TrueSkill represents the belief about the skill of an agent as a normal distribution (i.e., with two parameters, µ and σ), we can use it to infer a priori probability of an agent, a, winning against its opponent, o, as follows [21]: The ranking of the top-5 agents with MLP and LSTM policies according to their TrueSkill is given in Tab. 1 and the a priori win rates in Fig. 8. Note that within the LSTM and MLP categories, the best meta-learners are 10 to 25% more likely to win the best agents that use other adaptation strategies. Finally, we enlarge the population from 105 to 1050 agents by duplicating each of them 10 times. We evolve the population for several generations as follows. Initially, we start with a balanced population of different creatures. Next, we randomly match 1000 pairs of agents, make them play iterated adaptation games, remove the losers from the population and duplicate the winners. The same process is repeated 10 times. The result is presented in Fig 9. We see that many agents quickly disappear from the population while the meta-learners end up dominating.

Conclusion and Future Directions
In this work, we proposed a simple gradient-based meta-learning approach suitable for continuous adaptation in nonstationary environments. The key idea of the method is to regard nonstationarity as a sequence of stationary tasks and train agents to exploit the dependencies between consecutive tasks such that they can handle similar nonstationarities at execution time. We applied our method to nonstationary locomotion and within a competitive multi-agent setting. For the latter, we designed the RoboSumo environment and defined iterated adaptation games that allowed us to test various aspects of adaptation strategies. In both cases, meta-learned adaptation rules were more efficient than the baselines in the few-shot regime. Additionally, agents that meta-learned to adapt demonstrated the highest level of skill when competing in iterated games against each other.
The problem of continuous adaptation in nonstationary and competitive environments is far from being solved, and this work is the first attempt to use meta-learning in such setup. Indeed, our meta-learning algorithm has a few limiting assumptions and design choices that we have made mainly due to computational considerations. First, our meta-learning rule is to one-step-ahead update of the policy and is computationally similar to backpropagation through time with a unit time lag. This could potentially be extended to fully recurrent meta-updates that take into account the full history of interaction with the changing environment. Additionally, our meta-updates were based on the gradients of a surrogate loss function. While such updates explicitly optimized the loss, they required computing second order derivatives at training time, slowing down the training process by an order of magnitude compared to baselines. Utilizing information provided by the loss but avoiding explicit backpropagation through the gradients would be more appealing and scalable. Finally, our approach is unlikely to work with sparse rewards as the meta-updates use policy gradients and heavily rely on the reward signal. Introducing auxiliary dense rewards designed to enable meta-learning is a potential way to overcome this issue that would like to explore in the future work.

A Derivations and the policy gradient theorem
In this section, we derive the policy gradient update for MAML as give in (4) as well as formulate and equivalent of the policy gradient theorem [50] in the learning-to-learn setting.
Our derivation is not bound to a particular form of the meta-update. In general, we are interested in meta-learning a procedure, f θ , parametrized by θ, which, given access to a limited experience on a task, can produce a good policy for solving it. Note that f θ is responsible for both collecting the initial experience and constructing the final policy for the given task. For example, in case of MAML [15], f θ is represented by the initial policy, π θ , and the meta-update rule (4) that produces π φ with φ = θ − α∇ θ L(τ 1:K ).
More formally, after querying K trajectories, τ 1:K , we want to produce π φ that minimizes the expected loss w.r.t. the distribution over tasks: Note that the inner-most expectation is conditional on the experience, τ 1:K , which our meta-learning procedure, f θ , collects to produce a task-specific policy, π φ . At training time, we are given a finite sample of tasks from the distribution D(T ) and can search forθ close to optimal by optimizing over the empirical distribution: We re-write the objective function for task T i in (12) more explicitly by expanding the expectations: Trajectories, τ and τ 1:K , and parameters φ of the policy π φ can be thought as random variables that we marginalize out to construct the objective that depends on θ only. The meta-update rule (4) assumes the following P Ti (φ | θ, τ 1:k ): Note that by specifying P Ti (φ | θ, τ 1:K ) differently, we may arrive at different meta-learning algorithms. After plugging (14) into (13) and integrating out φ, we get the following expected loss for task T i as a function of θ: The gradient of (15) will take the following form: where φ = φ(θ, τ 1:K ) as given in (15). Note that the expression consists of two terms: the first term is the standard policy gradient w.r.t. the meta-updated policy, π φ , while the second one is the policy gradient w.r.t. the original policy, π φ , that is used to collect τ 1:K . If we were to omit marginalization of τ 1:K (as it was done in the original paper [15]), the terms would disappear. Finally, the gradient can be re-written in a more succinct form: The update given in (17) is an unbiased estimate of the gradient as long as the loss L Ti is simply the sum of discounted rewards (i.e., it extends the classical REINFORCE algorithm [44] to metalearning). Similarly, we can define L Ti that uses a value or advantage function and extend the policy gradient theorem [50] to make it suitable for meta-learning. Theorem 1 (Meta policy gradient theorem). For any MDP, gradient of the value function w.r.t. θ takes the following form: where d φ T (x) is the stationary distribution under policy π φ .
Proof. We define task-specific value functions under the generated policy, π φ , as follows: where the expectations are taken w.r.t. the dynamics of the environment of the given task, T , and the policy, π φ . Next, we need to marginalize out τ 1:K : and after the gradient w.r.t. θ, we arrive at: where the first term is similar to the expression used in the original policy gradient theorem [50] while the second one comes from differentiating trajectories τ 1:K that depend on θ. Following Sutton et al. [50], we unroll the derivative of the Q-function in the first term and arrive at the following final expression for the policy gradient: Remark 1. The same theorem is applicable to the continuous setting with the only changes in the distributions used to compute expectations in (18) and (19). In particular, the outer expectation in (18) should be taken w.r.t. p Ti (τ | θ) while the inner expectation w.r.t. p Ti+1 (τ | φ).

A.1 Multiple meta-gradient steps
All our derivations so far assumed single step gradient-based meta-update. Experimentally, we found that the multi-step version of the update often leads to a more stable training and better test time performance. In particular, we construct φ via intermediate M gradient steps: Since P T (θ m+1 | θ m , τ m 1:K ) at each intermediate steps are delta functions, the final expression for the multi-step MAML objective has the same form as (15), with integration taken w.r.t. all intermediate trajectories. Similarly, an unbiased estimate of the gradient of the objective gets M additional terms:

B Additional details on the architectures
The neural architectures used for our policies and value functions are illustrated in Fig. 10. Our MLP architectures were memory-less and reactive. The LSTM architectures had used a fully connected embedding layer (with 64 hidden units) followed by a recurrent layer (also with 64 units). The state in LSTM-based architectures was kept throughout each episode and reset to zeros at the beginning of each new episode. The RL 2 architecture additionally took reward and done signals from the previous time step and kept the state throughout the whole interactions with a given environment (or opponent). The recurrent architectures were unrolled for T = 10 time steps and optimized with PPO via backprop through time.
C Additional details on meta-learning and optimization C.1 Meta-updates for continuous adaptation Our meta-learned adaptation methods were used with MLP and LSTM policies (Fig. 10). The meta-updates were based on 3 gradient steps with adaptive step sizes α were initialized with 0.001. There are a few additional details to note: 1. θ and φ parameters were a concatenation of the policy and the value function parameters. 2. At the initial stages of optimization, meta-gradient steps often tended to "explode", hence we clipped them by values norms to be between -0.1 and 0.1. 3. We used different surrogate loss functions for the meta-updates and for the outer optimization.
For meta-updates, we used the vanilla policy gradients computed on the negative discounted rewards, while for the outer optimization loop we used the PPO objective.

C.2 On PPO and its distributed implementation
As mentioned in the main text and similar to [20], large batch sizes were used to ensure enough exploration throughout policy optimization and were critical for learning in the competitive setting of RoboSumo. In our experiments, the epoch size of the PPO was set 32,000 episodes and the batch size was set to 8,000. The PPO clipping hyperparameter was set to = 0.2 and the KL penalty was set to 0. In all our experiments, the learning rate (for meta-learning, the learning rate for θ and α) was set to 0.0003. The generalized advantage function estimator (GAE) [51] was optimized jointly with the policy (we used γ = 0.995 and λ = 0.95).
To train our agents in reasonable time, we used a distributed implementation of the PPO algorithm. To do so, we versioned the agent's parameters (i.e., kept parameters after each update and assigned it a version number) and used a versioned queue for rollouts. Multiple worker machines were generating rollouts in parallel for the most recent available version of the agent parameters and were pushing them into the versioned rollout queue. The optimizer machine collected rollouts from the queue and made a PPO optimization steps (see [46] for details) as soon as enough rollouts were available.
We trained agents on multiple environments simultaneously. In nonstationary locomotion, each environment corresponded to a different pair of legs of the creature becoming dysfunctional. In RoboSumo, each environment corresponded to a different opponent in the training pool. Simultaneous training was achieved via assigning these environments to rollout workers uniformly at random, so that the rollouts in each mini-batch were guaranteed to come from all training environments.

D Additional details on the environments D.1 Observation and action spaces
Both observation and action spaces in RoboSumo continuous. The observations of each agent consist of the position of its own body (7 dimensions that include 3 absolute coordinates in the global cartesian frame and 4 quaternions), position of the opponent's body (7 dimensions), its own joint angles and velocities (2 angles and 2 velocities per leg), and forces exerted on each part of its own body (6 dimensions for torso and 18 for each leg) and forces exerted on the opponent's torso (6 dimensions). All forces were squared and clipped at 100. Additionally, we normalized observations using a running mean and clipped their values between -5 and 5. The action spaces had 2 dimensions per joint. Table 2 summarizes the observation and action spaces for each agent type. Note that the agents observe neither any of the opponents velocities, nor positions of the opponent's limbs. This allows us to keep the observation spaces consistent regardless of the type of the opponent. However, even though the agents are blind to the opponent's limbs, they can sense them via the forces applied to the agents' bodies when in contact with the opponent.

D.2 Shaped rewards
In RoboSumo, the winner gets 2000 reward, the loser is penalized for -2000, and in case of draw both agents get -1000. In addition to the sparse win/lose rewards, we used the following dense rewards to encourage fast learning at the early training stages: • Quickly push the opponent outside. The agent got penalty at each time step proportional to exp{−d opp } where d opp was the distance of the opponent from the center of the ring. • Moving towards the opponent. Reward at each time step proportional to magnitude of the velocity component towards the opponent. • Hit the opponent. Reward proportional to the square of the total forces exerted on the opponent's torso. • Control penalty. The l 2 penalty on the actions to prevent jittery/unnatural movements.

D.3 RoboSumo calibration
To calibrate the RoboSumo environment we used the following procedure. First, we trained each agent via pure self-play with LSTM policy using PPO for the same number of iterations, tested them one against the other (without adaptation), and recorded the win rates (Table 3). To ensure the balance, we kept increasing the mass of the weaker agents and repeated the calibration procedure until the win rates equilibrated.  Table 4 gives average win rates for the last 25 rounds of iterated adaptation games played by different agents with different adaptation methods (win rates for each episode are visualized in Figure 5).

E.2 Intolerance to large distributional shifts
Continuous adaptation via meta-learning assumes consistency in the changes of the environment or the opponent. What happens if the changes are drastic? Unfortunately, the training process of our meta-learning procedure turns out to be sensitive to such shifts and can diverge when the distributional shifts from iteration to iteration are large. Fig. 11 shows the training curves for a meta-learning agent with MLP policy trained against versions of an MLP opponent pre-trained via self-play. At each iteration, we kept updating the opponent policy by 1 to 10 steps. The meta-learned policy was able to achieve non-negative rewards by the end of training only when the opponent was changing up to 4 steps per iteration.  Figure 11: Reward curves for a meta-learning agent trained against a learning opponent. Both agents were Ants with MLP policies. At each iteration, the opponent was updating its policy for a given number of steps using self-play, while the meta-learning agent attempted to learn to adapt to the distributional shifts. For each setting, the training process was repeated 15 times; shaded regions denote 90% confidence intervals.