{'fraction_non_alphanumeric': 0.041458528115400024, 'fraction_numerical': 0.011299585948978228, 'mean_word_length': 4.413738250180766, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Safe Reinforcement Learning for Legged Locomotion

Designing control policies for legged locomotion is complex due to the under-actuated and non-continuous robot dynamics. Model-free reinforcement learning provides promising tools to tackle this challenge. However, a major bottleneck of applying model-free reinforcement learning in real world is safety. In this paper, we propose a safe reinforcement learning framework that switches between a safe recovery policy that prevents the robot from entering unsafe states, and a learner policy that is optimized to complete the task. The safe recovery policy takes over the control when the learner policy violates safety constraints, and hands over the control back when there are no future safety violations. We design the safe recovery policy so that it ensures safety of legged locomotion while minimally intervening in the learning process. Furthermore, we theoretically analyze the proposed framework and provide an upper bound on the task performance. We verify the proposed framework in four locomotion tasks on a simulated and real quadrupedal robot: efficient gait, catwalk, two-leg balance, and pacing. On average, our method achieves 48.6% fewer falls and comparable or better rewards than the baseline methods in simulation. When deployed it on real-world quadruped robot, our training pipeline enables 34% improvement in energy efficiency for the efficient gait, 40.9% narrower of the feet placement in the catwalk, and two times more jumping duration in the two-leg balance. Our method achieves less than five falls over the duration of 115 minutes of hardware time.


I. INTRODUCTION
The promise of deep reinforcement learning (RL) in solving complex and high-dimensional problems autonomously has attracted much interest among robotics researchers. However, effectively training an RL policy requires exploring a large set of robot states and actions, including many that are not safe for the robot. This is especially true for systems that are inherently unstable such as legged robots. One way to leverage RL for robotics problems is to learn the policy in computer simulation and then deploy it in the real world [1], [2]. However, this requires addressing the challenging sim-to-real gap. Another approach to tackle this issue is to directly learn or fine-tune a control policy in the real-world [3], [4], with the main challenge being ensuring safety during learning. Our work falls into this category by introducing a safe RL framework for learning legged locomotion while satisfying the safety constraints during training.
We formulate the problem of safe locomotion learning in the context of safe RL. Inspired by prior work [5], [6], [7], 1 TY is with Princeton University ty3@princeton.edu 2 TY, TZ, LL, SH, JT, and WY are with Google Research {jimmyyang, tingnan, luulinda, sehoonha, jietan, magicmelon}@google.com 3 SH is with Georgia Institute of Technology sehoonha@gatech.edu 1 Video is included in the submission and the project website: https: //sites.google.com/view/saferlleggedlocomotion/ Catwalk Two-leg balance Fig. 1: We evaluate our algorithm in legged locomotion tasks: catwalk and two-leg balance. The narrow feet placement and standing with two legs lead to falling easily during the learning process. Our algorithm overcomes these challengesoutperforming prior methods in terms of safety violations and learning efficiency both in simulation and the real world.
our learning framework adopts a two-policy structure: a safe recovery policy that recovers robots from near-unsafe states, and a learner policy that is optimized to perform the desired control task. Our safe learning framework switches between the safe recovery policy and the learner policy to prevent the learning agent from safety constraint violations (e.g., falls).
Different from prior methods that learn a safety critic function which predicts the possibility of safe violations [5], [6], [7], we propose a model-based approach to determine when to switch between the two policies based on the knowledge about the system dynamics. In reality, it is often the case that the designer has some knowledge of the system dynamics at hand. Our goal is to exploit this knowledge to design a safety mechanism without relying on a black-box approach (e.g., neural networks). More specifically, we first define a safety trigger set that includes states where the robot is close to violating safety constraints but can still be saved by a safe recovery policy. When the learner policy takes the robot to the safety trigger set, we switch to the safe recovery policy, which drives the robot back to safe states. We then determine when to switch back to the learner policy by leveraging an approximate dynamics model of the robot (e.g., centroidal dynamics model for legged robots) to rollout the planned future robot trajectory: if the predicted future states are all in the safe states, we will hand the control back to learner policy, otherwise we will keep using the safe recovery policy (see Fig. 2 for illustration). Such switch criteria allow the learning agent to explore near safety-violation regions while minimally intervening in the learning process.
The contributions are as follows.
(1) We provide a the-oretical analysis of the proposed framework and derive performance bounds under system dynamics error since perfect knowledge about the system dynamics is hard to obtain. Specifically, we construct a regret-type of bound that compares the learning outcome of the agent to the one with full knowledge about the system dynamics. This helps us to quantify the effect of dynamics error on learning performance.
(2) To evaluate the performance of the framework, we compare the proposed safe learning framework with state-ofthe-art safe RL algorithms on four simulated and real legged locomotion tasks: efficient gait, catwalk, two-leg balance, and pacing (see Fig. 1). In all cases, our algorithm achieves comparable or superior performance to prior approaches, averaging 48.6% fewer falls in simulation and zero or nearzero falls in the real world. In addition, we show that this framework can effectively reduce the number of uses of the safe recovery policy over prior approaches [5], which improves learning speed.

II. RELATED WORK
Safe RL. Learning constraint-satisfying policies has been explored in the context of safe RL [8], [9], [10], [11], [12], [7], [13]. In model-free RL, in which we do not know the system dynamics, prior work either designs policy update approaches [14], [15], adds a weighted copy of the cost function in the reward objective [16], [17], [18], [19], modifies output actions [20], [21], or predict the probability of being unsafe [22], [23]. Despite requiring little prior knowledge about the robotics system, these methods suffer from safety constraint violations during the learning process, especially during the initial learning stage where random exploration of the neural network policy is performed. In contrast, our approach exploits the knowledge of the system dynamics to do safe planning. Many prior works in model-based RL use a Gaussian process or neural networks to learn the system dynamics [24], [25], [26], and ensures robot safety by formulating an MPC problem [27], [28], or constructing a barrier or Lyapunov function for safety certificates [29], [17]. Despite being more general, these methods suffer from the curse of dimensionality. As we focus on the locomotion problem, we choose to leverage an approximate dynamics model widely used in locomotion control. RL for Legged Locomotion. RL has been applied in legged locomotion to acquire complex locomotion skills [30], [31], [32], [33], [4], [34], [3], [35]. However, these approaches often directly learn the policy or fine-tune the learned policy in the real world without considering safety and thus require humans to recover robots when falling. In contrast, we use the safe recovery policy to ensure safety during the learning process. Without manually resetting robots from falls, our algorithm achieves better wall-clock time learning efficiency. Recovery RL. This work [5] also uses a safe recovery policy to take over the control when violating safety constraints. However, there are a few key differences. (1) Algorithm. First, Recovery RL uses safety critics to predict the probability of future cost violations given the current state. When the prediction value is above the pre-defined threshold, the Fig. 2: An illustration of safety trigger set C tri (orange), safe set C safe (green), and failure set C failure (red) with two policies π learner , and π safe . On the left, without the reachability criteria, we encounter a frequent switch between the safe recovery policy π safe and the learner policy π learner . On the right, by applying the reachability criteria, we push the learning agent away from C tri , which reduces the risk of violating the safety constraints.
safe recovery policy kicks in. However, the safety critics are pre-trained from the policy that maximizes cost violations (i.e., considering the worst-case policy) in simulation. Such safety critics do not accurately reflect the safety performance of the learner policy when deploying in the real world. This mismatch can potentially prevent the agent from exploring the environment (i.e., too conservative), or ensuring safety (i.e., too aggressive) as seen in Section VI.
(2) Generalization. Second, the safety critics, which are parametrized by deep neural networks, do not guarantee correct verification of safety when visiting states unseen during training.

III. BACKGROUND
We frame our problem as a constrained Markov Decision Process (CMDP) [36], defined as a tuple < S, A, T, R, C > . Here S is the set of states, A is the set of actions, and T is the transition function that specifies the conditional probability T (s |s, a) or the deterministic function s = T (s, a) that the next state is s given the current state s and action a. In addition, R : S × A → R is a reward function, and C : S → R is a constraint cost function that indicates the failure state (e.g., falling down). The reward function encodes the benefit of using action a in state s, while the cost function encodes the corresponding constraint violation penalty (we assume the value of C is non-negative).
A policy π(a|s) is a map from states to probability distributions on A : a ∼ π(a|s). The state then transits from s to s according to T. Finally, the agent receives a reward R(s, a) and incurs a cost C(s).
, while keeping the cumulative discounted cost below a pre-defined threshold h : J C (π) .
Note that we refer to systems with J C (π) > h as cost violations or failure. One difference between conventional RL and safe RL is that we want the entire training process to satisfy the safety constraints, not just at the end of training.

IV. ALGORITHM
In this section, we first provide the necessary definitions, followed by proposing the safe control approach using ap-proximate knowledge about system dynamics. Finally, we conclude the section by outlining the proposed framework.

A. Setup
Definition 1: We define: (1.1) Safety trigger set C tri ⊆ S, a set of states that triggers π safe . (1.2) Safe set C safe = C tri \ S in which the state does not belong to the safety trigger set; (1.3) Failure set C failure = {s ∈ S : C(s) > 0} in which an non-negative cost penalty (i.e., falls) incurred for certain s. Note that C failure ⊆ C tri . Definition (1.3) defines unsafe states that we want to avoid. For example, when the height of a robot is too low (i.e., indication of falls), the learning agent is in C failure . Fig. 2 shows these three sets. To ensure safety of the learning agent, we will also need the following assumptions.
Assumption 1: An approximate knowledge of the true system dynamics T, denoted byT , in which their difference is bounded by T −T 2 ≤ . Approximate dynamics models are often leveraged to achieve efficient planning of robot motions. For example, the centroidal dynamics model (CDM) or spring-loaded inverted pendulum model (SLIP) are commonly used to design locomotion controllers, while a unicycle model is popular for modeling wheeled robots. In this work, we utilize a centroidal dynamics model to predict the future state trajectory given the current state and π learner 's control inputs to certify the learning agent's safety.

B. The Safe Control Approach
Based on these definitions and assumptions, the proposed algorithm uses two policies, a learner policy π learner , which maximizes the task reward, and a safe recovery policy π safe , which tries to bring the agent back to safe states when reaching the safety trigger set. A naive way to select which policy to query is as follows.
This control approach implies that whenever the state is in the safety trigger set, the safe recovery policy takes over the control from the learner, and returns the control back when the learner is in the safe set. However, this poses a potential issue: while in principle the control approach in Eq. (1) can ensure safety, it may cause frequent switch between π safe and π learner , which hinders policy exploration and impacts learning efficiency. Fig. 2 illustrates this observation. To address this issue, when the learning agent is in safety trigger set, the safe recovery policy takes over the control and should return the control to the learning agent only when π learner can guarantee that there would be no further switches for a specific future horizon. This reduces the potential use of the safe recovery policy. Formally, we define the following planning criteria for determining when should π safe hand control back to π learner . Definition 2: Reachability Planning Criteria.
∈ π learner with s τ / ∈ C tri where w ∈ R + is the planning step. Definition (2) says that we want to find a minimal step i (i.e., the minimal intervention) such that all the remaining actions are from π learner after a minimal initial sequence of actions from π safe . This ensures that when π safe hands the control back to π learner , π learner can maximally reduce future use of the safe recovery policy. However, Definition (2) is computationally intensive and non-convex, which can limit the practicality of the proposed approach for real-time validation of safety in some applications such as robots with onboard processing. An alternative is to remove the planning component in reachability planning criteria and only verify whether future states are in the safety trigger set given all actions from π learner . Formally, π safe returns the control back to π learner if the following criteria are satisfied.
The criteria say that at time t, we check whether the future states are in C tri for w steps. Hence, the final proposed safe control approach becomes π safe handing control back criterion π learner (·|s), otherwise. ( There are two conditions that we use a safe recovery policy. The first condition (i.e., safe recovery policy trigger criterion) is when the current state is in the safety trigger set (i.e., s t ∈ C tri ). The second condition (i.e., safe recovery policy handing control back criterion) is that when the previous action is from safe recovery policy and there does not exist a sequence of actions produced by the learner policy such that the resulting states are not in the safety trigger set (i.e., . Note that the reachability criteria (i.e., the second condition) are only checked if the previous action is from π safe , which determines when to hand the control back to the learner. This is different from Recovery RL [5], in which π safe hands the control back to the learner whenever the prediction of future cost constraint violations are below a certain threshold at any state in the environment. Our approach allows the agent to have enough freedom to explore the states that are near C tri while avoiding constant use of π safe . π safe is triggered only when needed. Finally, the approximate dynamicsT is used to unroll the future states given the current state and π learner for the reachability criteria.

C. Reinforcement Learning with the Safe Control Approach
Algorithm 1 illustrates the proposed framework with the safe control approach. In addition, when storing the data Algorithm 1 Safe RL with Approximate System Dynamics 1: Given: (1) Knowledge about system dynamics s = T (s, a) or s =T (·|s, a), (2) an initial learner policy π 0 learner = π(·|θ 0 learner ) parameterized by θ 0 learner , (3) a safe recovery policy π safe (·|s), (4) safety trigger set C tri , and (5) a trajectory buffer B 2: Parameter: (1) the number of policy updates K, (2) the number of interactions W, (3) planning horizon w 3: for k = 0, 1, · · · , K do 4: for t = 1, · · · , W do 5: Use control approach in Eq. (2) 6: π safe , and zero otherwise. Update θ k learner using B 10: end for trajectory in the replay buffer (line 7), we always use the action proposed by π learner with the negative penalty for indicating the use of π safe . This implies that we treat π safe as part of the environment, and we want the learning agent to reduce the use of π safe over the learning process to embed safety to the agent. Note that one can also terminate the episode when triggering π safe to discourage the use of π safe .

V. THEORETICAL ANALYSIS FOR DYNAMICS ERROR
The proposed reachability criteria rely on the knowledge about the system dynamics T to predict the future state trajectory. Hence, we want to quantify the effect of the approximation error of the system dynamics on the performance of the task objective. Let {y t } ∈ S denote a target safe trajectory that the learning agent wants to track. Then, we consider the following simplified problem in which we want to find a sequence of actions that minimize the tracking errors for W steps between the safe and the planned trajectories under the proposed control approach.
arg min Assumption 2: We assume:  is the diameter of the set (i.e., a 1 − a 2 2 ≤ D, for every a 1 , a 2 ∈ A). Asm. (2.1) and (2.2) define the structure of the system dynamics, which has been applied to a wide range of tasks such as legged locomotion [37]. In addition, Asm. (2.3) says that the function value does not change abruptly given two points. Asm. (2.4) implies that the size of A is bounded.
To provide a performance guarantee of Problem (3), we compare the value of the task objective of the proposed algorithm running online (denoted by cost(T )) against the optimal offline algorithm, which makes the optimal decision with full knowledge of the system dynamics T (denoted by cost(T )). Hence, we have the following Theorem.
Theorem 5.1: (Performance Guarantee.) We have where σ 1 (·) is the largest singular value of a matrix, s 1 is the initial state, and scalar value a max := max t∈{1,··· ,W } a t 2 .
Proof: Please check our project website for a proof. We now make several observations for Theorem 5.1.
, which implies that the online decision planning performs as good as the offline algorithm. However, when there is a dynamics error, the leading singular value of the system matrices difference will potentially lower the performance of the task objective with the cost violations.
(2) If the system dynamics is less smoother, i.e., α is large, then the performance of the proposed algorithm degrades. This implies that when the system is unstable and unpredictable, we have a large tracking error.
(3) It shows the performance drop to the offline algorithm grows linearly w.r.t the horizon W. In addition, the smoothness value G represents how predicable the dynamics is-the smaller the value is, the more predicable the dynamics are. (4) In practice for the locomotion tasks in the paper, their system dynamics are non-linear. However, we can treat it locally linear when the time step is small enough. We leave the non-linear version of the bound as the future work. In summary, Thm 5.1 provides an intuition about the effect of horizon W and model errors.

VI. EXPERIMENTS
We study the following questions: (1) How does our algorithm perform compared to other baselines in terms of constraint satisfaction and final reward? (2) What is the effect of the planning horizon and the safety trigger set?

A. Setup
Robots. We evaluate our approach on a simulated Unitree Laikago [38], a quadrupedal robot that weighs 24kg and has 12 actuated joints. For the real-world experiment, we use a Unitree A1 [39], a quadrupedal robot that weighs 12.7kg with better hardware reliability and agility.
Hierarchical Policies. We use a hierarchical policy framework that combines RL and optimal control for π learner and π safe . This framework consists of a high-level RL policy that produces gait parameters and feet placements, and a low-level model predictive control (MPC) controller [37] that takes in these parameters to compute desired torque for each motor in the robot. Instead of directly commanding the motor's angle, this approach offers stable operation and streamlines the policy training due to a smaller action space and a robust MPC controller. The gait parameters include stepping frequency (Ω ∈ R, the frequency of completing one swing-stance cycle of each leg), swing ratio (p swing ∈ R, the portion of swing duration), and phase offsets relative to the front-right leg (θ 1 , θ 2 , θ 3 ∈ R, the angle added to the front-right leg's phase angle for the other three leg's phase angles). The feet placements include p FR,y , p FL,y , p RR,y and p RL,y ∈ R, which are the desired feet positions in the sideway (y) direction (FR: front-right; FL: front-left; RR: rearright; RL: rear-left). Such policy parameterization allows us to take advantage of the MPC controller while being expressive enough to produce a diverse set of gaits.
The MPC controller uses a centroidal dynamics model (CDM) proposed in [37] to serve asT . The dynamics can be expressed as a linear dynamic model s = As + Ba, where A and B are state and control matrices that depend on the gait parameters and feet positions. When rolling out the states usingT , we use the action from π learner and keep the other MPC parameters fixed. The state in CDM is s t = [x, y, z,ẋ,ẏ,ż, φ, θ, ψ,φ,θ,ψ] T , where x, y, z ∈ R are the robot's position,ẋ,ẏ,ż ∈ R are the velocity, φ, θ, ψ are the roll, pitch, yaw, andφ,θ,ψ are the angular velocity. Tasks. We compare our algorithm with existing approaches on four legged locomotion tasks shown in Fig. 1 and Fig. 3.
(1) Efficient Gait. The robot learns how to walk with low energy consumption. The robot is rewarded for consuming less energy. The actions of the policy network produce the delta change of the gait parameters at each step, including the delta change of the stepping frequency Ω, the swing ratio p swing , and the phase offsets θ 1 , θ 2 , θ 3 .
(2) Catwalk. The robot learns a catwalk gait pattern, in which left and right two feet are close to each other. This is challenging because by narrowing the support polygon, the robot becomes less stable. The robot is rewarded for using narrow foot placement: where e ∈ R + is a positive survival bonus to make the reward non-negative. The actions of the policy network produce the gait parameters in the efficient gait task and the feet placement p FR,y , p FL,y , p RR,y and p RL,y .
(3) Two-leg Balance. The robot learns a two-leg balance policy, in which the front-right and rear-left feet are in stance, and the other two are lifted. The robot can easily fall without delicate balance control because the contact polygon degenerates into a line segment. The robot can only control these two stance feet, and is rewarded for staying at a target height of 0.45m: R := e − (z − 0.45) 2 . The action includes the delta change of the stepping frequency Ω, the swing ratio p swing , and the phase offset θ 3 .
(4) Pacing. We want to produce a desired stepping frequency and the swing ratio for performing the pacing behavior under different desired speeds. The robot is rewarded for matching target speed: , wherė x target ,ẏ target ,ż target are the target velocities. The action includes the delta change of the stepping frequency Ω, and the swing ratio p swing . Note that here we fixed the phase offsets to be θ 1 = π, θ 2 = 0, θ 3 = π to enforce the pacing gait: legs on the same side move in sync. Baselines. The goal is to demonstrate that the proposed approach can enable safe training using the knowledge of the system dynamics to determine switching timing. In addition, we position our paper in the field of RL. Hence we compare our method with the following state-of-the-art RL or safe RL baselines.
(1) TRPO. Trust region policy optimization (TRPO [40]) only optimizes the reward objective. This will serve as a cost constraint performance lower bound.
(2) PCPO. Projection-based constrained policy optimization (PCPO, [15]) optimizes the reward while using projection to satisfy the cost constraint. In addition, PCPO is the state-of-the-art model-free approach over CPO [14].
(3) TRPO w/ negative penalty (TRPO-N). It optimizes the reward objective with an added penalty for entering C failure : R(s, a) − C(s). This is to show that the penalty-based approach would fail when the dynamics are complex. (4) Recovery RL. We are inspired by Recovery RL [5], which also consists of two policies: a safe recovery policy and a learner policy. This is a direct comparison of the proposed switch criteria with the value function-based method. We use hierarchical policies for all the baselines. Implementation Details. We implement these tasks using Pybullet [41] simulator. We use a two-layer multi-layer perceptron (MLP)-based policy network with a tanh activation function for π learner and π safe . The low-level MPC controller runs at the frequency of 250Hz, and the policy network predicts the gait parameters with 125Hz.
(1) Observation Space. The observation space of the policy network includes the previous gait parameters, the height of the robot, base orientation, linear and angular velocities.
(2) Safe Recovery Policy π safe . We want to learn a policy that can safely bring the agent to a balanced and stationary state. We use the above hierarchical approach to train the policy, and add random perturbation to the robot to simulate the unstable and unsafe behavior encountered during the learning process. This allows us to train a policy that can stabilize the robots robustly. The robot is rewarded for being stable with zero velocity and maintaining the desired height: R := e − [z,ẋ,ẏ,ż, φ, θ, ψ,φ,θ,ψ] T − [0.45, 0, 0, 0, 0, 0, 0, 0, 0, 0] T 2 2 . The action includes the gait parameters used in the efficient gait. Note that all π safe in the simulation are learned first before learning π learner in the proposed method and Catwalk Two-leg balance Pacing Fig. 3: Simulation Results. We report the total number of falls versus final rewards for the tested algorithms and task pairs over five runs. We observe that the proposed approach achieves the fewest number of falls while having comparable reward performance (i.e., in the top-left corner) (best viewed in color).
(a)  Recovery RL. We stop training π safe during learning π learner . For the real-world experiments, we compare two types of π safe : a learned MLP-based π safe and a simulation-tuned MPC controller. We find that the optimized MPC controller works better in the real world, possibly because the smaller number of tunable parameters leads to a smaller sim-to-real gap. As such, we use this controller in our real-world experiments.
(3) Constraint Cost Function C. For all the tasks, C(·) outputs 1 when the height is below 0.1m and zero otherwise. (4) Safety Trigger Set C tri . To ensure safety, we want the robot to maintain a certain height, stay upright with smaller side-tilting velocities. Hence we use the following safety trigger set for Laikago: C tri = {s ∈ S : z < 0.4 ∨ z > 0.55 ∨ |φ| > 0.26 ∨ |θ| > 0.26 ∨ |ẏ| > 0.5 ∨ |φ| > 0.5}, where the unit for angle is in radian. For A1 robot, we use the same safety trigger set except for the height bounds to be z < 0.2 and z > 0.3 since the robot is smaller. We design our C tri inspired by the capturability theory [42]. In particular, we choose an initial tight C tri such that the corresponding capture point for the robot does not exceed the task space (0.15m for Laikago and 0.1m for A1). We then fine-tune C tri on the real A1 robot with a random policy to identify C tri that can prevent the robot from falling. In addition, we also conduct an ablation on the effect of the safety trigger set where we use a stricter criterion: C 2 tri = {s ∈ S : z < 0.4, z > 0.55, |φ| > 0.26, |θ| > 0.26, |ẏ| > 0.375, |φ| > 0.375}, and C 3 tri = {s ∈ S : z < 0.4, z > 0.55, |φ| > 0.26, |θ| > 0.26, |ẏ| > 0.25, |φ| > 0.25}. Note that C 1 tri := C tri (default criteria). (5) Planning Horizon w. We also ablate the planning horizon w (one step is 0.016 seconds) to see its effect on the number of uses of the safe recovery policy and falling events. Specifically, we use w = 0, 5, 10, 15, 20. Other than explicitly specified, we use w = 10 as the default in simulation. For the real-world experiment, we use w = 0 for π safe that lasts 1 second due to computation budget in planning and we find this approach empirically works well.

B. Results
Simulation Results. The total number of falls versus the final reward value are shown for all tested algorithms and tasks in Fig. 3. The learning performance for baseline oracle, TRPO, indicates the reward and the constraint performance when the safety constraint is ignored. Ideally, we want the algorithm to be in the top-left corner (more rewards and fewer falls). Overall, we find that our algorithm is able to improve the reward while achieving the fewest number of falls in all tasks (in the top-left corner). In addition, we observe that (1) Recovery RL has more falls and cannot improve reward effectively in the catwalk tasks. This is because the safety critic cannot predict the cost violation well; (2) TRPO-N requires a significant effort to select a good value of C to balance between the reward improvement and the constraint satisfaction; (3) PCPO has the moderate reward and constraint performance due to the projection step.
We also observe that the learning algorithm in the twoleg balance (7.19% for our algorithm of the total of 13,600 training trajectories) and pacing tasks (9.50%) tend to have more falls than that of the catwalk task (2.46%). This is because the robot with only two stance legs is highly unstable, and the pacing gait is different from the gait used in the safe recovery policy, which creates a sudden change of the gait and hence causes a wiggling movement. Ablation Studies. To demonstrate the efficacy of reachability criteria in the proposed framework, we ablate the planning horizon w. Fig. 4(a) shows the results with w = 0, 5, 10, 15, 20 in the catwalk task in terms of the total number of falls, the number of uses of π safe , and the reward.

Efficient gait Catwalk
Two-leg balance Fig. 5: Real-world Experiments. We report the real-world experiment results of reward learning curves and the percentage of uses of π safe on the efficient gait, catwalk, and two-leg balance tasks. We observe that our algorithm is able to improve the reward while avoiding triggering π safe over the learning process. The learned policies are shown in Fig. 1.
Note that when w = 0, it is equivalent to the removal of the reachability criteria in our algorithm, and similar to a shielding-based MPC approach in [43]. In addition, w = 0 does not mean our method is similar to recovery RL since it can trigger π safe before reaching the safety boundary.
Overall, we observe that our approach outperforms Recovery RL in terms of the number of uses of π safe and the reward, no matter how long the horizon is. This is because the safety critic in Recovery RL is too conservative in predicting the cost violation, which hinders the exploration of the agent (reducing the reward). In addition, the safety critic is pretrained based on the worst-case policy that maximizes the cost violation. This leads to a conservative learning strategy. Furthermore, as we keep increasing w to 20, the number of falls reduces with more uses of π safe and fewer rewards. This is expected since π safe intervenes the learning agent more often, which reduces the reward performance. This shows a trade-off between safety and reward performance, and the horizon w gives users a knob to tune based on the problems. Furthermore, Fig. 4(b) ablates the design of the safety trigger set C tri in the catwalk task. We see that the stricter C tri is, the fewer the falls and the more uses of π safe are. This observation implies one needs to design a good C tri based on the recovery capability of π safe . Real-world Experiments. Fig. 5 shows the reward and the percentage of uses of π safe among the trajectory steps collected for one policy update on the efficient gait, catwalk, and two-leg balance tasks. For the efficient gait and the catwalk tasks, one policy update corresponds to 10 trajectories of a total of 4,000 steps, and for the two-leg balance task, it corresponds to 5 trajectories of a total of 2,000 steps. Here, all the policies are trained from scratch, except for the two-leg balance task with the policy pre-trained in the simulation since it requires more samples to train. Note that we do not include the other baselines since they cannot even collect 10 trajectories of data without falling to perform one policy gradient update. And Recovery RL requires much more steps to learn since its safety critic, pre-trained with the objective function to maximize the cost values in simulation, is too conservative in predicting the cost violations in the real world. For instance, in the catwalk task our approach achieves zero falls and uses π safe 6.8% of total steps among first policy update, compared to Recovery RL with zero falls but 75.6% of total steps of use of π safe . This makes these baselines laborious to train in practice.
Overall, we see that on these tasks, the reward increases and the percentage of uses of π safe decreases over policy updates. For instance, the percentage of uses of π safe decreases from 20% to near 0% in the efficient gait task. For the twoleg balance task, the percentage drops from near 82.5% to 67.5%, suggesting that the two-leg balance is substantially harder than the previous two tasks. Still, the policy does improve the reward. This observation implies that the learner can gradually learn the task while avoiding triggering π safe . In addition, this suggests that we can design a C tri and π safe that do not hinder the exploration of the policy as the performance increases.
Finally, Fig. 1 shows the results of learned locomotion skills (please visit our project website for more videos). First, in the efficient gait task, the robot learns to use a smaller stepping frequency and achieves 34% less energy than the nominal trotting gait. Second, in the catwalk task, the distance between two sides of the legs is 0.09m, which is 40.9% smaller than the nominal distance. Third, in the two-leg balance task, the robot can maintain balance by jumping up to four times via two legs, compared to two jumps from the policy pre-trained from simulation. Without π safe , learning such locomotion skills would damage the robot and require manually re-positioning the robot when falling. Note that there is no single fall nor a manual reset during the entire learning in the efficient gait (45mins of realworld training data collection, excluding automatic position reset or battery replacement) and catwalk tasks (29mins), and less than 5 falls in the two-leg balance task (28mins). Our results suggest that learning legged locomotion skills autonomously is possible in the real world.

VII. CONCLUSIONS
We studied the problem of safe reinforcement learning for acquiring locomotion skills for quadruped robots by combining a safe recovery policy and a learner policy. The safe recovery policy takes over the control when the robot is close to a safety violation, and returns the control back when the robot stays safe in the estimated near future. We provided a theoretical analysis of the algorithm and quantified the effect on the model errors for the learning performance. We evaluate our algorithm on both simulated and real quadruped robots for a variety of challenging locomotion tasks and demonstrate the effectiveness of the proposed framework compared to baseline methods.
No model is without limitation. Although uncommon, our current π safe can still fail to save the robot from unsafe states (see supplementary video for failure case) due to imperfections in π safe and the safety trigger set C tri . Further investigations in improving these would enable the learner policy to explore the environment more effectively and thus improve the learning efficiency. In addition, we currently do not consider the model uncertainty from the environment and non-linear dynamics in our theoretical analysis. Including these would further improve the generality of our approach. Finally, designing an appropriate reward when incorporating the π safe can impact our learning performance. We use a penalty-based approach that obtained reasonable results in our experiments. We plan to investigate this in future work to further improve the learning performance.