{'fraction_non_alphanumeric': 0.030242709970316046, 'fraction_numerical': 0.007543216343635411, 'mean_word_length': 4.6794922649742166, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sim-to-Real Learning of Compliant Bipedal Locomotion on Torque Sensor-Less Gear-Driven Humanoid

Sim-to-real is a mainstream method to cope with the large number of trials needed by typical deep reinforcement learning. However, transferring a policy trained in simulation to actual hardware remains challenging due to the reality gap. In particular, the characteristics of actuators in legged robots have a considerable influence on sim-to-real transfer. High reduction ratio gears are widely used in actuators, and the reality gap issue becomes especially pronounced when even the utilization of backdrivability is considered to control joints compliantly. We propose a new simulation model of gears to address this gap. Additionally, the difficulty in achieving stable bipedal locomotion causes typical methods to fail to tune physical parameters in simulation with the behavior of transferred policy. Thus, we propose a method for system identification that can utilize failed attempts. The method's effectiveness is verified using a biped robot, the ROBOTIS-OP3, and the sim-to-real transferred policy can stabilize the robot under severe disturbances and walk on uneven surfaces without force and torque sensors.


I. INTRODUCTION
If robots can achieve stable bipedal walking, they can traverse various environments, such as slopes, uneven roads, and stairs. To achieve this mission, robots need to be robust against disturbances and changes in environments. As approaches to improve the robustness of legged robots, research based on torque control and the use of compliance has been popular in recent years [1]- [3]. On the other hand, robots that require high-power joints, such as full-scale humanoids, are often equipped with gears with high reduction ratios to provide output torque while downsizing the actuators. In addition, since installing a torque sensor increases the cost of the actuator, a configuration that uses a high reduction ratio gear without a joint torque sensor is still a popular option [4]- [6]. Such actuators have elements that are difficult to model, primarily represented by gear friction. Due to this friction, the ideal torque output may not always be achieved if using the joint-level feed-forward compensation. Generally, it is not easy to manually design a controller that can work with these considerations.
Deep reinforcement learning (DRL) is being established as a method for developing high-performance robot controllers [7]- [9]. Since DRL requires many trials, the sim-toreal approach, in which policies trained using simulations are transferred to a real-world robot, has been attracting atten- tion [10]- [12]. In addition, simulation training is considered more promising because it can add various disturbances and try unusual cases without the risk of breaking robots. However, there are differences in behavior between the simulation and the real environment, commonly referred to as reality gap, which prevents the policies trained on the simulation from working in the real environment. Especially for legged robots, it has been reported that the model of the actuators has a considerable influence on the reality gap [12]- [14]. To address this challenge, methods to construct a realistic simulation by identifying the simulation parameters of dynamics reproducing actual robot behavior are studied. However, the instability of the bipedal locomotion causes the sim-to-real transfer to fail catastrophically. It makes system identification with the behavior of transferred policy challenging. In this study, we propose 1) an actuator model featuring the high reduction ratio gears for physics simulation and 2) a new system identification method that utilizes the failure experience when a sim-to-real transfer fails and realizes robust bipedal locomotion, including balancing and walking, using DRL.

A. Joint control in RL for legged robots
In recent years, many studies on reinforcement learning for legged robots have been reported, and many of them have achieved high performance on real robots [10], [14], [15]. These studies often employed a position controller in each joint, and RL policy commands target angles. One of the reasons is that position control improves learning efficiency and task performance compared to torque or velocity control [16]. On the other hand, some studies have been conducted to improve robustness to disturbances and environmental recognition errors by exploiting passivity (in other words, utilizing the joint backdrivability) through compliant control [1]- [3]. Currently, these studies mainly focus on the robot with torque-controlled joints. So, utilizing backdrivability by sim-to-real RL is not a well-explored area, especially in robots with gear-driven and torque sensorless joints. In this study, we focus on it and address some difficulties described following subsections.

B. Sim-to-real RL for Robotics
It is widely known that controllers trained by RL in simulations do not work well in actual robots due to the reality gap, and various approaches have been developed to overcome this challenge. Domain randomization [17] is a frequently used approach for the reality gap. The method trains RL policy in many simulations that randomized dynamics or sensory appearances, such as joint friction or textures of the object. Domain randomization can increase the robustness of some domain gaps, and it tends to successful sim-to-real transfer. However, specifying appropriate randomization distribution is a difficult problem. Too large randomization often makes RL policy too conservative, and performance becomes low. Another approach to address the reality gap is precisely reproducing the real behavior on the simulation [11], [18]. Construct a realistic simulation, then train RL on that. Moreover, it often combines small domain randomization. Assuming that simulation is characterized by some dynamics model parameters, such as the size of joint friction, reproducing the real behavior corresponds to finding parameter values or distributions. In this paper, we call this system identification. Researches along this approach, we can roughly categorize them into 1) manual system identification [10], [14], 2) system identification based on collected task irrelevant data [12], [13], 3) system identification based on rolled out RL policy behavior [11], [18]. Since focusing on frictional gear-driven joint and utilization of backdrivability, we combine 2) and 3). On the focus, the behavior of interaction with the environment is important, so 1) or 2) approach may require a large amount of data. Moreover, tasks with instability, such as bipedal walking, where a slight difference in action can easily make a big difference in the future compared to tasks with arms fixed to a base. In such case, precise identification is challenging by 3) with trajectory comparison [18] or parameter prediction [11]. In this paper, we use both 2) and 3) complementary approaches. It allows relatively small data collection and system identification using the failing behavior of the rolledout RL policy.

C. Modeling Actuators in Sim-to-Real Learning
The reality gap is especially critical for actuators in legged robots. Several kinds of research have been conducted to improve the actuator models in the simulation, broadly classified into two categories to address the challenge.
1) Use a neural network for the actuator model and train with largely collected data [12], [13] (Network training corresponds to system identification).
2) Design a detailed model of the actuator and implement it in the simulation [10], [19]  The first method has the potential to build highly accurate models but requires large amounts of data. In this study, we extend category 2) to model actuators with high reduction ratio gears with low-dimensional parameters. It is expected that parameters can be obtained with relatively small amounts of data. In addition, since high reduction ratio gears are a major characteristic, the model can be applied to many actuators.

III. PRELIMINARY OF RL
Let (S, A, T, R, γ, s 0 ) be a Markov decision process with state s ∈ S, action a ∈ A, transition probability T (s t+1 |s, a), reward function R(s, a) ∈ R, discount factor γ ∈ [0, 1], and initial state distribution s 0 . RL typically learns a policy π : S → A to maximize the expected discounted accu- . This time, the state transition is a transition from control timing t to t + 1 in a physical simulation or the real world. S corresponds to the sensor data and A to the commands to the joint actuators. Although various RL algorithms have been proposed, in this study, we use Soft-Actor-Critic [20], which supports continuous action spaces and has been reported to have high performance.

IV. METHOD
The proposed sim-to-real learning method consists of two main components, the first is improved robot simulation featuring an actuator model, and the second is a system identification method to specify the simulation parameters for RL training. The method consists of the following three phases. Phase 1: Perform first system identification with exciting motion, such as squatting motion. Phase 2: Using the identified parameters, train the RL policy using simulation, then run the trained policy on the actual robot. Phase 3: If the robot does not work well, re-identify the simulation parameters using the failure experience. Phase 2 is then executed again.
We first describe the proposed actuator model in section IV-A. Then, the first system identification is described in section IV-B.1, followed by the system re-identification in section IV-B.2.

A. Actuator Modeling
This section describes our actuator model. The actuator consists mainly of a DC motor (Section IV-A.1) and gears (Section IV-A.2), as illustrated in Fig. 2. The actuator is controlled by variable gain PD, described in Section IV-A. 3. In this study, we implement this actuator model on the physics engine Mujoco [21], and some descriptions assume that, but note that it can be implemented in other physics engines.

1) DC Motor Model:
For the DC motor model, we use the commonly used model [10], which is represented by the following equation: where τ is the exerted torque by the motor, I is the current, K t is the torque constant of the motor, R is the terminal resistance, V pwm is the voltage applied to the motor, V back−emf is the back electromotive force (EMF) voltage, andq is the angular velocity of the shaft.
A current controller controls voltage to drive the DC motor with the target current. The applied voltage V pwm at each simulation time step can be formulated by the following equation: where I targ is the target current and V battery is battery voltage.
2) Gear Model with Directional Transmission Efficiency: The gear model considers the transmission efficiency and load-independent friction loss. The former depends on the force, while the latter mainly depends on the angular velocity.
For high reduction ratio gears, there is an asymmetry in transmission efficiency depending on the drive direction [22], [23]. This work introduces this directional transmission efficiency (DTE) and proposes an implementation method for existing physics engines.
The method treats the force loss due to transmission efficiency as an additional brake torque applied to the joint in each simulation time step. The brake torque is calculated from the generated torque by the motor, load torque, and the forward and backward transmission efficiencies. Assuming that the simulation time step is small enough for the target dynamics, the brake torque is calculated by the following equation: where τ m is the generated torque from the motor, τ a is the load torque on the joint, and η f w and η bw are the forward and backward transmission efficiencies, respectively. τ m = r gear τ where r gear is gear reduction ratio. When τ m is clearly greater than τ a or the directions of τ m and τ a are the same, it is regarded as a forward drive state, and the brake torque based on the η f w is generated. If the directions of τ m and τ a are opposite and τ a is larger even after considering the η bw , it is considered to be in the backward drive state, and the brake torque based on the η bw is generated. A state that does not meet either conditions is an antagonistic state with no apparent difference between τ m and τ a , and thus generates a brake torque, such that τ m + τ a + τ brake = 0. Since τ a is approximated using the value of the current simulation state, there is a concern that the simulation may become unstable. However, it is expected to be stable when used in conjunction with the load-independent friction as below. The load-independent friction loss generates torque that cancels joint torque in the absolute upper bound. In particular, Mujoco will handle it as a constraint and allow the constant value. So we calculate the friction loss value at each time step to generate approximated static and viscous friction. For the calculation, Stribeck function τ f ric is used: where each parameter f s , f c , k v , andq static are the static friction force, the Coulomb friction coefficient, the viscous friction coefficient, and the angular velocity value regarded as non-static, respectively.
3) Variable Gain PD: The actuator model described so far is controlled by the PD controller. We adopted this because previous research has reported that placing the position control at the lower level of the RL policy improves learning efficiency and performance. Generally, the PD controller is used with fixed gains, but in this study, the gains are also commanded with the target position. It allows a strategy to increase backdrivability while reducing target position tracking performance by specifying a smaller gain, depending on the task situation.

B. System Identification for Sim-to-real
This section describes the system identification method of the simulation parameters, such as the motor torque constant or frictions of the actuator model shown in Section IV-A.
1) First System Identification: Computing simulation parameters with black-box optimization [12], [24] to match the observed results on the actual robot with the observed results on the simulation in similar motions is suitable for sim-to-real learning due to the following advantages: 1) The simulation can be improved from the results of the actual robot alone without detaching the actuator from the robot, and 2) Since optimizing the various parameters simultaneously, the behavior of the combination of each parameter can be matched to the actual robot.
First, a suitable exciting motion for system identification is designed as time series data θ targ of the target angle of each joint. The exciting motion used in this study will be described in section V-B. A simple position control feedback system is provided to track θ targ in the actual robot. We then collect a set of sensor information O real for the robot in motion. This sensor information includes the angle of each joint, angular velocity, current, and the tilt of the upper body. Similarly, on the simulation built based on the parameters φ that includes parameters of body inertial and actuator model (actually used in this study are shown in table I), the same position control feedback system is used to perform the motion to track θ targ and obtain O sim . The parameter φ that minimizes the difference between O real and O sim is calculated by sampling-based black-box optimization as follows: min where L exc is an error function calculates distance between O sim and O real .
2) System Re-Identification from Failure Experience: Even if we use the simulation improved by the identification described in section IV-B.1 for RL training, the transferred policy sometimes does not work. For example, the actual robot may not even be able to stand on flat ground if a policy has been trained to balance against disturbances in a simulation. In such cases, task-relevant data is lacking in first system identification, and a possible solution is using data while transferred policy performing the task. The goal here is to find simulation parameters that reproduce the behavior of an actual robot not only for the excitation motion but also for the results of the policy rollout. However, in such a failure case, control commands by the policy become oscillatory, and the sensor information, such as joint angle trajectory, widely diverges. So, it is challenging to calculate the distance between simulation and actual policy behavior (policy gap) by comparison of sensor information for each time series in the same way as equation (5). Therefore, we introduce multi-objective optimization with the minimization of the excitation motion gap to achieve accurate identification while allowing for a rough calculation of the policy gap. min L policy is the policy gap function that is described later, O sim policy and O real policy are the sensor information observed in rollouts with the policy on the sim and real. We use the difference between cumulative rewards acquired in policy rollouts for the policy gap function L policy . L policy = | 1 where K is the number of rollouts, t is the time index and R p is the reward function for re-identification. The maximum length is set for t (we use 4 sec), and subsequent observations are not considered to prevent the variance of cumulative reward becomes too large. In this study, we use a proprioceptive reward function for R p , which can be calculated using only the robot's sensor information and is extracted from the task's reward function. R p includes characteristics of the behavior, such as the time until fall based on survival rewards or joint angular velocity penalties, so we can assume that cumulative reward can be one feature.
Then, the parameters of the simulation in which the policy fails can be obtained while keeping the error in the exciting motion as small as possible. Note that if the problem is ideal, L exc and L policy can be optimal simultaneously, and multiobjective optimization is not needed. However, in the real problem, L exc and L policy are in a bit trade-off relationship, so multi-objective is suited for this method. Multi-objective optimization returns the Pareto-front solutions as a result. To decide the best from Pareto-fronts, we use simple heuristics that select the solution (simulation parameters) with a minimum L exc value while L policy is under the threshold based on the average of L policy values in whole searched solutions.

A. Robot Setup: ROBOTIS-OP3
In this study, we use ROBOTIS-OP3 [25]. ROBOTIS-OP3 has a length of 51 cm, a weight of about 3.5 Kg, 6-DOFs in each leg, and 20-DOFs in total. The control is executed at 125 Hz.
All the joints of the robot are composed of Dynamixel servo motors XM430-W350. The stall torque is 4.1 Nm at 12.0 V and 2.3 A. The gears consist of metal spur gears and have a reduction ratio of 353.5. XM430-W350 equips a mode that allows the command of the target motor current directly, and this mode is used.

B. System Identification
In this study, simulation parameters for system identification are shown in Table I. Note that the motor armature is the additional link inertia caused by the rotor inertia of the DC motor and was amplified with the square of the gear reduction ratio from the rotor inertia itself. The simulation was implemented using Mujoco [21]. The size of the simulation time step was set to 1 ms.
For the equation (5) of the black-box optimization for first system identification, we use the Tree-structured Parzen Estimator algorithm implemented in Optuna [26] since it gives better results than CMAES in our experimental setup. For the multi-objective optimization equation (6), we used the NSGA-II algorithm with also Optuna. We set the number of trials for each optimization in the experiment to 2000.
For system identification, a simple squatting motion is used for the exciting motion θ targ . While keeping a forwardleaning posture, execute a flexion/extension movement such that the knee joint angle flexes from 1.47 rad to 0.6 rad at a speed of approximately 0.5 Hz. To evaluate the error L exc from the squatting motion, equation (7) is used.
Where r is upper body orientation,ṙ is body angular velocity, θ is the joint angle, and I is joint current. N is the number of focusing joints.
C. Task for Evaluation 1) Balancing task: The objective of the balancing task is to maintain balance during standing upright on a board with a dynamically changing tilt.
The action of the policy in this RL is the target angle and P gain of PD control for each joint. The target joints are five for each leg except for the hip-yaw axis, for 10 joints for both legs, and the action space has 20 dimensions. The observation of RL policy in this task includes the following elements: • The position and velocity of the six key points placed on the three corners of each leg soles with the base link as the origin (36 dims) • Command in the previous step (20 dims) • Body orientation in the Euler angle (3 dims) • Body angular velocity (3 dims) To be aware of the time series of the state, we concatenate the above observation obtained in the previous step with the observation obtained in the latest step, then feed it to the policy. The reward function is calculated by subtracting scores from the survival reward by each penalty term, including the following elements: the difference from the reference posture, the magnitude of the joint angular velocity, the distance of the base link position from the reference point, the tilt of the base link, the angular velocity of the base link, and the magnitude of the joint current.
For evaluation on the actual robot, we use a board that tilts max of about 6 • for the roll axis. Disturbances are given by 1 Kg weight falling on the board's edge from two fixed heights.
2) Walking task: The bipedal walking task is an advanced task that requires more active motion than the balancing task. The objective is to walk forward on a flat surface at a constant speed. The action and observation spaces are the same as in the balancing task, but the periodic signals of the walking phase encoded in two dimensions are added to observations. The reward function is designed similarly to the prior study by Siekmann et al. [15], so refer to their paper for details. The penalty terms of the reward included the difference from the target velocity, the floor reaction force in the swing leg and the toe velocity in the supporting leg, the base link posture difference from the target posture, the acceleration and angular velocity of the base link, the magnitude of the joint current, and the distance to the action of the previous step. The target walking cycle was set at 1.0 sec, and the ratio of the support leg phase to the swing leg phase was set at 0.7 : 0.3.

VI. RESULTS
The following three points are evaluated to verify the effectiveness of the proposed method.

A. Evaluation of Sim-to-real Learning Method
For the balancing task, we performed the sim-to-real learning process described in section IV. The whole learning process was conducted three times, with different seed values, and the results are shown in Table II. The left column lists the names of the set of simulation parameters φ to train the policy, and the right column shows the number of successful attempts. "Excitation only" corresponds to the identified parameter values with the first system identification at phase 1. As shown in Table II, the policy failed to work on actual robot. Then re-identification was processed, and the result simulation parameters are labeled as "Re-Identified" on Table II. Policies that trained with "Re-Identified", succeeded in balancing on the tilting board in all attempts. Despite the small amount of actual robot operation time for data collection (only 5 seconds of squat motion and five failure rollouts), sim-to-real was achieved without domain randomization.
In addition, we trained a balancing policy with naive domain randomization as a baseline. During training, each simulation parameter is uniformly sampled within a range shown in Table I. The transferred policy could not succeed in the balancing task in any attempt.
We processed the sim-to-real learning in two attempts with different seed values for the walking task. To evaluate the performance of each policy rollout, we use the average velocity (distance forward in the target direction divided by time). Table III shows the results, the unit of value is cm/s, and zero is put when the policy could not go forward. The values in the brackets show the policy performance in the simulation used for training. Attempt 2 succeeded in sim-to-real. The attempt 1, the walking policy trained with "Excitation only" seems conservative (relatively stable, but slow), which would have made it difficult to evaluate the policy gap with L policy in equation (6) and it tends to miss identification. Especially the challenging RL task, such as walking, the simulation parameters affect the learning performance. "Re-Identified" on attempt 1 that miss identified might make RL difficult. In addition, we trained a walking policy with naive domain randomization same as the balancing task with two seeds. Both policies failed to walk on the actual robot.
We also found that the re-identified parameters with the balancing task can generalize to the walking task. We trained a walking policy with that parameters, the policy performs better than the re-identified with walking tasks.

B. Evaluation of Actuator Modeling
First, we describe the effectiveness of the DTE model in the sim-to-real process of the balancing task. In the Table II, parameter "Excitation only (w/o DTE)" is the result of the first system identification without DTE model, also "Re-Identified (w/o DTE)" is the result of the re-identification without DTE. As shown in the table, it failed to sim-to-real without the DTE model. From this result, we can say that DTE is one of the critical components of the actuator model.
Next, we show the effect of the DTE model in the first system identification with the squatting motion. The error scores of the first system identification (Equation (5) and (7)), the best of three attempts was 0.037 for "Excitation only" and 0.070 for "Excitation only (w/o DTE)". Fig. 3 shows the graphs of the sensor information of the actual and the simulated robot. With DTE model reproduced sensor information well. In particular, the DTE's ability to represent changes in a frictional loss in static and driving conditions is thought to have contributed to the reproducibility of the overall behavior.

C. Evaluation of Policy Performance
To verify the performance of the policy acquired with the proposed method, we demonstrate and evaluate the tasks on an actual robot. Each motion of the policy can be viewed in the supplementary video.
1) Balancing Task: Fig. 4 shows the snapshot of balance task experiment. It was confirmed that the policy could maintain balance under impulsive disturbances, and the robot could maintain an upright posture.
2) Walking Task: To realize the robust policy utilizing the backdrivability that is the aiming of this research, we trained a walking policy on the simulation with random uneven terrains. For training, the re-identified simulation parameters with the balancing task were used. The trained policy passed three uneven surfaces for testing; the surface and snapshot of walking are shown in Fig. 5. We used some toy brick parts for unevenness. Each brick has a 5 mm height on top. Fig. 6 shows the command for the left ankle roll joint by the policy while walking. The P gain is reduced as the leg transitions into the support phase, indicating a compliant landing. In addition, we tested an existing walking module of ROBOTIS-OP3 with the default settings as a baseline using stiff position-controlled joints. The walking module failed on these uneven surfaces. The fact that the policy was able to traverse uneven surfaces, which is difficult with a general walking controller, without force or contact sensors indicates that the policy can take advantage of backdrivability.

VII. CONCLUSION
In this paper, we propose a method to acquire a robust control policy of the actual robot by RL, using a model of high reduction ratio gears and a system identification method that utilizes the experience of failed sim-to-real transfers. The method was verified on the actual robot, ROBOTIS-OP3, and achieved balancing control under disturbance and walking on uneven surfaces without any force/torque sensor.