Protective Policy Transfer
  Being able to transfer existing skills to new situations is a key capability
when training robots to operate in unpredictable real-world environments. A
successful transfer algorithm should not only minimize the number of samples
that the robot needs to collect in the new environment, but also prevent the
robot from damaging itself or the surrounding environment during the transfer
process. In this work, we introduce a policy transfer algorithm for adapting
robot motor skills to novel scenarios while minimizing serious failures. Our
algorithm trains two control policies in the training environment: a task
policy that is optimized to complete the task of interest, and a protective
policy that is dedicated to keep the robot from unsafe events (e.g. falling to
the ground). To decide which policy to use during execution, we learn a safety
estimator model in the training environment that estimates a continuous safety
level of the robot. When used with a set of thresholds, the safety estimator
becomes a classifier for switching between the protective policy and the task
policy. We evaluate our approach on four simulated robot locomotion problems
and a 2D navigation problem and show that our method can achieve successful
transfer to notably different environments while taking the robot's safety into
consideration.
