Bridging Model-based Safety and Model-free Reinforcement Learning
  through System Identification of Low Dimensional Linear Models
  Bridging model-based safety and model-free reinforcement learning (RL) for
dynamic robots is appealing since model-based methods are able to provide
formal safety guarantees, while RL-based methods are able to exploit the robot
agility by learning from the full-order system dynamics. However, current
approaches to tackle this problem are mostly restricted to simple systems. In
this paper, we propose a new method to combine model-based safety with
model-free reinforcement learning by explicitly finding a low-dimensional model
of the system controlled by a RL policy and applying stability and safety
guarantees on that simple model. We use a complex bipedal robot Cassie, which
is a high dimensional nonlinear system with hybrid dynamics and underactuation,
and its RL-based walking controller as an example. We show that a
low-dimensional dynamical model is sufficient to capture the dynamics of the
closed-loop system. We demonstrate that this model is linear, asymptotically
stable, and is decoupled across control input in all dimensions. We further
exemplify that such linearity exists even when using different RL control
policies. Such results point out an interesting direction to understand the
relationship between RL and optimal control: whether RL tends to linearize the
nonlinear system during training in some cases. Furthermore, we illustrate that
the found linear model is able to provide guarantees by safety-critical optimal
control framework, e.g., Model Predictive Control with Control Barrier
Functions, on an example of autonomous navigation using Cassie while taking
advantage of the agility provided by the RL-based controller.
