Conservative Safety Critics for Exploration
  Safe exploration presents a major challenge in reinforcement learning (RL):
when active data collection requires deploying partially trained policies, we
must ensure that these policies avoid catastrophically unsafe regions, while
still enabling trial and error learning. In this paper, we target the problem
of safe exploration in RL by learning a conservative safety estimate of
environment states through a critic, and provably upper bound the likelihood of
catastrophic failures at every training iteration. We theoretically
characterize the tradeoff between safety and policy improvement, show that the
safety constraints are likely to be satisfied with high probability during
training, derive provable convergence guarantees for our approach, which is no
worse asymptotically than standard RL, and demonstrate the efficacy of the
proposed approach on a suite of challenging navigation, manipulation, and
locomotion tasks. Empirically, we show that the proposed approach can achieve
competitive task performance while incurring significantly lower catastrophic
failure rates during training than prior methods. Videos are at this url
https://sites.google.com/view/conservative-safety-critics/home
