{'fraction_non_alphanumeric': 0.030905435205603838, 'fraction_numerical': 0.01608509534310546, 'mean_word_length': 4.658165137614679, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots

Legged locomotion is commonly studied and expressed as a discrete set of gait patterns, like walk, trot, gallop, which are usually treated as given and pre-programmed in legged robots for efficient locomotion at different speeds. However, fixing a set of pre-programmed gaits limits the generality of locomotion. Recent animal motor studies show that these conventional gaits are only prevalent in ideal flat terrain conditions while real-world locomotion is unstructured and more like bouts of intermittent steps. What principles could lead to both structured and unstructured patterns across mammals and how to synthesize them in robots? In this work, we take an analysis-by-synthesis approach and learn to move by minimizing mechanical energy. We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots. The emergent gaits are structured in ideal terrains and look similar to that of horses and sheep. The same approach leads to unstructured gaits in rough terrains which is consistent with the findings in animal motor control. We validate our hypothesis in both simulation and real hardware across natural terrains. Videos at https://energy-locomotion.github.io


Introduction
A common approach to understanding mammalian locomotion is to study it in terms of a set of discrete gaits. Seminal works include Muybridge [1]'s motion video of a galloping horse and Hildebrand [2]'s study of different gaits observed at different speeds. This has inspired research on replicating these gaits in robots by pre-programming them to achieve effective locomotion. This approach of pre-programming fixed gait patterns has lead to immense progress in robotic locomotion [3,4,5,6].
However, pre-programmed gaits limit the ability of legged systems to perform general locomotion in diverse terrains and at different speeds. Studies of motor control in both animals and human infants [7] in the last few decades have shown that general locomotion in natural settings is irregular and does not fit predefined fixed gaits. It is better defined in terms of bouts of steps than periodic cycles [8,9]. That being said, we do see consistent regular patterns in ideal conditions when an animal is moving in a straight line over flat terrain at different speeds, e.g., walk, trot, canter, gallop, etc. [2,10,11,12] and these patterns are shared across species of diverse animals [13]. How do we reconcile these seemingly contradictory arguments about unstructured locomotion patterns on complex terrains with that of regular patterns seen in flat terrains under the same rubric?
One way to make sense of these differences is to see them from the lens of minimizing energy consumption. Locomotion consumes a significant fraction of an animal's metabolic energy budget which suggests that there would be evolutionary selection pressure for locomotion strategies that minimize energy consumption. Indeed, this has been a focus of several studies in biomechanics and energetics. Hoyt and Taylor [12] used energy minimization to explain why animals switch from one gait to another as their speed increases. Similar studies explain gaits across different animals including humans [14,15]. However, most of the biomechanics studies are limited to the straight-line motion of animals in ideal flat terrains. Furthermore, these studies take gaits as given and offer an explanation for them from an energy perspective without any statement about their synthesis.
In this work, we take an analysis-by-synthesis approach to show how energy minimization leads to the emergence of structured locomotion gait patterns in flat terrains as well as unstructured gaits in 5th Conference on Robot Learning (CoRL 2021), London, UK. Figure 1: We demonstrate via analysis-by-synthesis approach that learning to move forward by minimizing energy consumption plays a key role in the emergence of natural locomotion patterns in quadruped robots. We do not pre-program any primitives for leg motions and the policy directly output desired joint angles. Top: Bio-energetics driven learning on flat terrain leads to different gaits namely walk, trot and bounce (similar to gallop) as the speed increases. Our high-speed bounce gait displays an emergent flight phase despite low energy usage. This corresponds to the nearest animals (sheep/horse) with similar Froude numbers. Bottom: The same pipeline on diverse uneven terrains leads to unstructured gait as is true in general animal locomotion. Please see videos on the website. complex terrains. We use energetics to design an end-to-end learning framework and display the resulting gait patterns in a real quadruped robot. We employ model-free reinforcement learning as the optimizer to learn controllers to make the quadruped move forward while simultaneously minimizing the bio-energetics constraints of minimizing work [16]. We train our policies on simple fractal terrains with varying frequency of terrain heights instead of perfectly flat terrain. This mimics the real world more closely and achieves efficient and robust gaits without the need for artificial rewards for foot clearance or periodic external push during training. At low frequency, it resembles the simple flat terrain settings and at high frequency, it simulates complex terrains. In addition, we present a learning pipeline that enables smooth gait transition given changing target speeds by combining RL and knowledge distillation from trained expert gait policies. We then transfer the policies onto the real robot by performing online adaptation [17].
We empirically demonstrate that minimizing energy consumption plays a key role in the emergence of natural gait patterns in a quadruped legged robot. Our quadruped robot is closest to sheep and horse in terms of physicality as determined by the Froude number [18] which is a scalar metric characterizing gaits of quadrupeds (discussed more in Section 4.2). Therefore, energetics-based training leads to similar three gaits in our robot as found in horses and sheep, namely, walk, trot and gallop as the speed is gradually increased in flat terrains Figure 1. Interestingly, we show that the gait selection is inherently linked with the speed of the robot. For instance, trotting is only energy efficient around the medium speed of 0.9m/s. If we increase or decrease the speed of the robot, it takes more energy for trot than to gallop (similar to bouncing gait) or walk as shown in Figure 2. Finally, when the same pipeline is run in uneven complex terrains, it leads to unstructured gaits as consistent with animal locomotion in usual diverse conditions [9,8].
Prior works have used energy penalty to obtain energy-efficient gaits [19,20] but have not shown the correspondence to different gaits at different speeds. Furthermore, out of these works, the ones which deploy these gaits on a real robot either use a predefined hand-coded gait library, use reference motions [19], or only learn high-level contact sequences for a predefined swing leg motion [21]. These approaches have some fundamental limitations in contrast to what we propose. Firstly, the use of predefined foot motions never optimize for efficient swing leg motions, and using reference trajectories gives gaits that look realistic, but are not necessarily energy-efficient for the specific robot we have. Second, learning controllers end to end without the use of any priors (reference motions or predefined motions) allows for the possibility of learning behaviors that don't follow any predefined gaits. We show the robustness of this complex terrain policy by successfully deploying it in complex terrains in the real world.
The main contributions of this paper include: • Show that minimizing energy consumption plays a key role in the emergence of natural locomotion patterns in both flat as well as complex terrains at different speeds without relying on demonstrations or predefined motion heuristics. • Show that the emergent gaits at different target speeds correspond to conventional animals in the similar Froude number range (sheep/horse) without any sort of pre-programming. • Present a distillation-based learning pipeline to obtain velocity-conditioned policy that displays smooth gait transition as the target speed is changed. • Demonstrate the emergent behaviors, robustness analysis, and gait patterns in simulation as well as a real-world budget quadruped robot.  We would like to validate that indeed certain gaits will emerge at certain speeds if we minimize the energy consumption. To do so, on the right, we plot the energy consumption vs speed of the A1 robot in simulation for walk, trot and bounce (modification of gallop/canter) gaits. We use [6] to achieve the different gaits by hard-coding the contact sequences for walk, trot and bounce. This plot is qualitatively similar to the plot in [12], which shows the energy consumption of different horse gaits across different speeds. We additionally plot the energy consumption of the three policies we get at different speeds (0.375 m/s, 0.9 m/s, 1.5 m/s). This plot shows two things. First, it is observed that certain gaits are only efficient at certain speeds and using them for other speeds is suboptimal. For instance, walk is optimal at low speeds with transition to trot and bounce. Secondly, we observe that our learning framework converges to the optimal gait at each of the speeds, and beats the efficiency of MPC gaits from [6], which does not optimize the swing foot trajectory for energy. Details of MPC method are in the supplementary.

Learning Locomotion Policy
We learn the locomotion policy π which takes as input the current state x t ∈ R 30 , previous action a t−1 ∈ R 12 to predict the next action a t (Equation 1). The predicted action a t is the desired joint position for the 12 robot joints which is converted to torque using a PD controller.
We implement π as MLPs (details in Section A) and train the base policy π end to end using modelfree reinforcement learning. At time step t, π takes the current state x t , previous action a t−1 to predict an action a t . RL maximizes the following expected return of the policy π: where τ = {(x 0 , a 0 , r 0 ), (x 1 , a 1 , r 1 )...} is the trajectory of the agent when executing policy π, and p(τ |π) represents the likelihood of the trajectory under π. Bottom: Foot contact plots and key frames on unstable moving planks. Videos on the website.

Stable Gait through Natural Constraints
Instead of adding artificial simulation noise, we train our agent under the following natural constraints. First, the reward function is motivated from bio-energetic constraints of minimizing work [16]. We found these reward functions to be critical for learning realistic gaits in simulation. Second, we train our policies on uneven terrain ( Figure 1) as a substitute for additional rewards used by [22] for foot clearance and robustness to external push. We see that these constraints are enough to achieve all the gaits and results we demonstrate in this paper.

Energy Consumption-Based Reward
Let's denote the linear velocity as v and the angular velocity as ω, both in the robot's base frame. We additionally define joint torques as τ and joint velocities asq. We define our reward as sum of the following three terms: where, r forward rewards the agent for walking straight at the specified speed, r energy penalizes energy consumption and r alive is the survival bonus.
We use simple rules to set the hyper-parameters. At a given target linear speed v target x , the survival bonus c is set to 20 * v target x . We set α 1 = 0.04, and α 2 = 20 across all settings.
Notice that the actual energy consumption on the robot depending on the low-level hardware design is not directly measurable. We estimate the unit energy consumption per time step by summing the instantaneous power of the 12 motors by multiplying the torque and the joint velocity at each motor. Also notice that this reward is much more concise than those used in prior works. We discuss the role of minimizing energy consumption in Section 4.4.

Sim to Real Transfer
For simulation to real transfer, we use RMA [17] which consists of the an adaptation module on top of the base policy. During deployment, the adaptation module uses the state history to estimate the vector of extrinsics (which contains environment information) online. This adaptation module is trained in simulation using supervised learning after the base policy is learned using RL with the true extrinsics parameters. The environment perturbations we train on is listed in Table 1.

Experimental Setup and Training Details
Hardware and Simulation: We use Unitree's A1 robot as our hardware platform [23] and use its A1 URDF [23] to simulate the A1 robot in RaiSim simulator [24]. The full details of terrain and environment setup are provided in supplementary due to space constraints. The action space is 12 dimensional corresponding to the target joint position for the 12 robot joints. The predicted target joint angles a =q ∈ R 12 is converted to torques τ using a PD controller with target joint velocities set to 0. Environmental Variations: All environmental variations with their ranges are listed in Table 1. Policies with simple terrain and mild environment variations achieve walking, trotting and bouncing gaits. To deploy our robot in the complex terrain, we use aggressive environment randomization with sharper fractal terrains which gives us the unstructured walking terrain.
Policy Learning: The policy is a multi-layer perceptron with 3 layers which takes in the current state x t ∈ R 30 , previous action a t−1 ∈ R 12 and outputs 12-dim target joint angles. The hidden layers have 128 units. We train the policy and the environment encoder network using PPO [25]. The training runs for 15, 000 iterations with a batch size of 100, 000 split into 4 minibatches and learning rate = 5e−4. We train for a total of 1.5 billion samples which takes roughly 24 hours on a desktop machine with 1 GPU.  We analyze the performance of the walking policy on complex terrain with aggressive perturbations in the real world (videos on website). We show two uneven terrain deployments of our method in Figure 3. During the deployment of the robot in the rocky terrain, we see that the robot roughly follows the walking gait which repeatedly gets obstructed by the rocks and has to step over rocks of different heights, prematurely terminating the swing motion of the leg. Consequently, the gait we observe is an unstructured walking gait. We further analyse this in another deployment on unstable planks which move as walk trot Figure 5: Trotting with 1 kg payload (two bottles of 500 ml water are strapped onto the robot). The 1 kg payload is out of the normal perturbation of environment parameters used in simulation training shown in Table  1. We show qualitative and gait patterns for other two gaits in the appendix. We find that the gait patters are robust and remain the same even in the presence of disturbances.
the robot tries to cross it (see video). The foot contact plot for this setup is not periodic and changes depending on the complexity of the terrain the policy needs to react to. In the video, this can been seen when the front right foot of the robot steps on an unstable plank which moves as a result, but the robot maintains stability and successfully crosses it.

Emergence of Walking, Trotting and Bouncing
Simulation Gaits: We train 3 separate policies for 3 target speeds -0.375 m/s (low), 0.9 m/s (median) and 1.5 m/s (high) in simulation. We use the same hyper-parameters across all target speeds. We observe that the walking gait emerges at target speed 0.375 m/s, trotting emerges at 0.9 m/s and hopping emerges at 1.5 m/s in simulation. Note that we use RMA [17] (Section 2.5) to transfer our policies to the real world without any fine-tuning. We observe the same foot contact sequence in the real world as in simulation. In Table 2, we compare the performance of our policies with MPC [6] and observe that our walking gait can achieve accurate speed tracking while being 50% more energy efficient than the MPC baseline. We test the robustness of the walking policies by adding a 1 Kg payload on the robot by strapping two bottles with 500 ml water. In Figure 5, we show 4 key frames of trotting gait. Note that the 1 kg payload is outside the range of training perturbations (Table 1), but our policies with the Sim-to-Real Adaptation in Section 2.5 learn to compensate the increase in weight by implicitly predicting changes in dynamics via latent extrinsics. We didn't test bouncing with payload, because the extra 1 kg prevents the human operator from swiftly preempting the robot at high speed by lifting.  Froude Number Analysis: In Biomechanics, Froude number F is a scalar metric characterizing gaits of quadrupeds and bipeds [18]. Concretely, F = v 2 /gh, where v is the linear speed, g is the gravitational acceleration, and h is the height of the hip joint from the ground. Existing research shows that animals with a similar morphology but with different sizes tend to use the same gait when walking with equal Froude numbers [26,10]. We calculate the Froude numbers of our robot in the real world for the low, median and high speeds, and compare them with the Froude numbers of quadrupeds in nature -dogs, horses and sheep [12,10]. We compare the Froude number of animals at the gait transition boundaries and observe that our robot is closest to the sheep in Table 3.  Figure 7: Energy consumption plot during training. Even though our reward function does not include the hand-designed locomotion penalties that are commonly used in prior works (torque, delta torque, foot slip, joint speed, and action), we show that all these penalties are minimized in our training framework as a byproduct of minimizing energy consumption.

Gait Transition via a Velocity-Conditioned Policy
We have shown walking, trotting and bouncing emerged at three different target speeds. To demonstrate foot patterns at a continuous range of velocities, we propose a learning scheme ( Figure 6) for a velocity-conditioned policy that enables smooth gait transition when the command velocity changes. Naive Multi-Task Training Fails: We first tried the naive approach to train the velocityconditioned policy in a multi-task fashion by randomly sampling desired velocities and using the corresponding velocity-conditioned reward (Section 2.4). However, this did not work and the resulting emerged gaits collapse to only two modes: walking and trotting, but trotting at high speeds is not energy-efficient ( Figure 7). We believe the reason for failure is difficulty in optimization as the robot is now tasked not only to learn to move forward but also do it by learning different gaits which causes it to collapse.
Learning via Distillation and RL: We sidestep this issue via stage-wise distillation approach. We first train our fixed-velocity policies as described in the paper which lead to walking, trotting and

Ablation Studies
The two key components we use during policy training to get reliable walking gaits is an energy based reward function, and a fractal terrain. We posit that these ingredients are minimal, and we demonstrate this by ablating each of them below. Minimizing Energy Consumption Also Minimizes Artificial Penalties: To demonstrate the sufficiency of minimizing energy consumption, we plot the decreasing trend of the artificial penalties during training in Figure 7, even though these artificial penalties are not included in our reward function. These include Torque penalty, delta torque penalty, foot slip penalty, joint speed penalty, and the action regularization [21,27,22,28,29]. A high positive correlation between energy consumption and artificial penalties can be observed. All these artificial penalties are minimized in our training framework as a byproduct of minimizing energy consumption. During training, the robot first learns to stand still with a minimal energy consumption (around epoch 500), and then starts moving forward with a slightly higher energy consumption. Minimizing Energy Consumption: In Figure 8, we show the the resulting gaits of policies trained without the unit energy consumption term in the reward function. All policies show high-frequency tapping behaviors that are not natural and inefficient. The high-frequency joint movements can burn the joint motors in the robot and are not transferable to the real hardware. Fractal Terrain during Training: We train the policies with energy minimization (Section 2.4) but on a flat terrain without fractal perturbations (Section 2.3). All training trials converge to unnatural and unstable gaits. We found that adding fractal terrain also facilitates a larger foot clearance and robustness which improves real-world hardware deployment. We show the key frames of 2 example unnatural gaits in the supplementary.

Related Works
Model-Based Optimization for Gaits: Model-based controllers to generate gaits [6] optimize for the stance legs for a given foot contact sequence. They use predefined swing leg motion and show stable gait generation on a real robot system. One class of method is contact-implicit optimization [30,31,32], optimizing contact forces along with sequences. However, these gaits do not emerge as an energy-minimizing solution to moving at different speeds. [33] analyzes biped gaits by optimizing energy consumption, but uses a minimal mechanical model assuming point-mass body and massless legs. In contrast, our swing leg motion, stance leg force and contact sequence, all emerge from minimizing energy consumption of moving of a real robot. We take a functional approach to gaits where we achieve them not as predefined movements, but as a consequence of energy minimization. Learning for Legged Locomotion and Gaits: Data-driven learning for legged locomotion has shown robust controllers for quadrupedal robots [22,34,35,36,37]. Of these, [34,22] focus on a controller on complex terrains but do not synthesize and analyze leg patterns of the robot across diverse gaits at different target speeds. [35] demonstrates a learning method to select gait controllers, but the low-level gait controllers are predefined primitives. Concurrent work [21] shows the learned gaits and gait transition, but their method relies on many human priors (e.g. predefined cyclic motion priors), shows trotting at high speed with short periods of flight phase, and requires powerful on-board computation power (Apple M1 chip) to solve online MPC optimization. In graphics, [38] shows low energy also plays a role in generating natural locomotion animations.

Conclusion
This work demonstrates that energy consumption plays a key role in the emergence of natural locomotion patterns in animals by following an analysis-by-synthesis approach of showing the result in real quadruped robots. The allows generation of the straight line lab gaits, as well as unstructured complex terrain gaits in animals. We perform thorough analysis of gait patterns, show their robustness under disturbances, and propose a learning pipeline for smooth gait transition . Interestingly, the gaits obtained by our robot are most similar to those in horses and sheep -the animals which are close to it in terms of the Froude number metric space.

A Experimental Setup and Training Details
Hardware Details: We use Unitree's A1 robot as our hardware platform [23]. A1 is a medium sized quadruped with 18 DoFs out of which 12 are actuated. Its mass is about 12 kg. We measure the joint positions and velocities of the 12 joints from the motor encoders and roll and pitch angles from the IMU sensor. To sense the foot contacts with ground in the real world, the foot sensor readings are taken on-board. The feet of the robot are deformable rubber membranes with filled air and air pressure sensors attached. When the foot membranes are deformed, readings from the pressure sensor will increase. We binarize the pressure reading by setting a threshold. Whenever the pressure reading from a foot is above that threshold, we treat it as a foot contact. The deployed policy uses position control for the joints of the robots. We use a PD controller to convert target joint position to torque with fixed gains (K p = 55 and K d = 0.8).
Simulation Setup: We use the A1 URDF [23] to simulate the A1 robot in the RaiSim simulator [24]. We generate complex terrains using the inbuilt fractal terrain generator (flat terrain for structured gait emergence: number of octaves = 2, fractal lacunarity = 2.0, fractal gain = 0.25, frequency = 10Hz, amplitude = 0.23m; uneven terrain for unstructured gait emergence: number of octaves = 2, fractal lacunarity = 2.0, fractal gain = 0.25, frequency = 20Hz, amplitude = 0.27m). We simulate each episode for a maximum of 1000 steps and terminate the episode earlier if the height of the robot drops below 0.28m, magnitude of the body roll exceeds 0.4 radians or the pitch exceeds 0.2 radians. The control frequency of the policy is 100Hz, and the simulation time step is 0.0025s.
Environmental Variations: All environmental variations with their ranges are listed in Table 1  is converted to torques τ using a PD controller: τ = K p (q − q) + K d q −q . K p and K d are manually-specified gains, and the target joint velocitiesq are set to 0.
Reward: We use the energy consumption-based reward throughout all settings. On uneven terrain for unstructured gaits, we can add the extra penalties from [17] like minimizing ground impacts to explicitly reduce the risks of hardware wearing because complex terrain can otherwise damage the hardware quickly. However, these extra penalties are not responsible for the unstructured gait emergence in complex uneven terrain.
Hyperparameters of Policy Learning: PPO [25] and Adam optimizer [39] are used to train the base policy and the environmental factor encoder in RMA [17]. The total number of training epoch is 15, 000. During each epoch, a batch of 100, 000 state-action transitions, which is split into 4 mini-batches, is sampled from the current policy. Each mini-batch is used for 4 times to compute the loss and backpropagation. The total loss is the surrogate policy loss plus half the value loss. The action log probability ratio is clipped between 0.8 and 1.2, and the target value is clipped to 0.8 − 1.2 times the range of the value that is computed in previous iteration. We lower bound the standard deviation of the parameterized Gaussian action space to 0.2 to encourage exploration. We set λ and γ in the generalization advantage estimation [40] to 0.95 and 0.998 respectively. We set the learning rate of the optimizer to 5e−4, β to (0.9, 0.999), and to 1e−8.  Table 1 in the main paper.

B.1 Emergence of Walking, Trotting and Bouncing
In Figure 9 of the Supplementary, we show four key frames of the walking at low target speed (0.375 m/s), trotting at median target speed (0.9 m/s) and bouncing (galloping) gaits at high target speed (1.5 m/s). At high speed, the robot gallops for the first few seconds, where the front two legs leave the ground first and also contact the ground first after the flight phase. Then, it switches to using bouncing gait where the four legs hit the ground at approximately the same time.  Table 1 of the main paper. For more qualitative results on robustness, please refer to videos at https://energy-locomotion. github.io.

C Additional Ablation Studies
Fractal Terrain during Training In Figure 11, we show the key frames of 2 gaits at low and median speeds when the policies are trained with energy minimization (Section 2.4) but on a flat terrain without fractal perturbations (Section 2.3). All 10 training trials converge to unnatural and unstable gaits. We found that adding fractal terrain also facilitates a larger foot clearance and robustness which improves real-world hardware deployment.

D Details of the MPC baseline
We use a reference implementation [41] of convex MPC [6] for Unitree A1. The parameters for gait generation to enforce constraints on ground reaction forces are fine-tuned for walking gait, trotting gait and bouncing gait. For the walking gait, we test at target speeds v target in the range from 0.1m/s to 0.7m/s every 0.1m/s. We set the duty factors of 4 legs to [0.8, 0.8, 0.8, 0.8], initial leg phases to [0, 0.25, 0.5, 0], and the stance duration to −0.5v target + 0.75 seconds. Only the Rear-Left foot is initialized in swing phase. For the trotting gait, we test at target speeds v target in the range from 0.5m/s to 1.5m/s every 0.1m/s. We set the duty factors of 4 legs to [0.6, 0.6, 0.6, 0.6], initial leg phases to [0.9, 0, 0, 0.9], and the stance duration to −0.15v target + 0.325 seconds. The Front-Right and Rear-Left feet are initialized in swing phase. For the bouncing gait, we test at target speeds v target in the range from 1.0m/s to 2.0m/s every 0.1m/s. We set the duty factors of 4 legs to

E Details of Gait Transition
We first train our fixed-velocity policies as described in the paper which lead to a walking policy π walking , a trotting policy π trotting and a bouncing (galloping) policy π bouncing . Each gait policy has its own encoder µ to embed the environmental parameters e t which are only available during simulation. We then treat these policies as experts and collect demonstration data from them to self-supervise and bootstrap the initial training phase of the velocity-conditioned policy Π. Illustration in Figure 6. We represent the command velocity inputs at these three velocity modes as three one-hot vectors of length three. Notice that these three policies serve as experts at three different velocity modes at low (0.375 m/s), median (0.9 m/s) and high ( ). The velocity-conditioned policy Π cannot rely on environmental parameters, since these environmental parameters are not available when the policy is deployed on the robot. Instead, the velocity-conditioned policy Π relies on 20 history steps of observations (x t−1 to x t−20 ) and actions (a t−2 to a t−21 ) which implicitly contain information about environmental parameters and are encoded by the adaptation module φ into a 8-dimensional vector fed to Π. The velocity-conditioned policy Π is trained by minimizing self-supervised L2 loss at the three velocity modes minus expected returns at randomly sampled velocities from the continuous range of 0.375 m/s -1.5 m/s. We linearly anneal the L2 loss to zero during initial 2500 training epochs, and then velocity-conditioned policy is optimized with only RL loss at randomly sampled velocities for additional 2500 training epochs.