Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and
  Generalization Guarantees
  Safety is a critical component of autonomous systems and remains a challenge
for learning-based policies to be utilized in the real world. In particular,
policies learned using reinforcement learning often fail to generalize to novel
environments due to unsafe behavior. In this paper, we propose
Sim-to-Lab-to-Real to safely close the reality gap. To improve safety, we apply
a dual policy setup where a performance policy is trained using the cumulative
task reward and a backup (safety) policy is trained by solving the reach-avoid
Bellman Equation based on Hamilton-Jacobi reachability analysis. In Sim-to-Lab
transfer, we apply a supervisory control scheme to shield unsafe actions during
exploration; in Lab-to-Real transfer, we leverage the Probably Approximately
Correct (PAC)-Bayes framework to provide lower bounds on the expected
performance and safety of policies in unseen environments. We empirically study
the proposed framework for ego-vision navigation in two types of indoor
environments including a photo-realistic one. We also demonstrate strong
generalization performance through hardware experiments in real indoor spaces
with a quadrupedal robot. See
https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary
material.
