Learning Barrier Certificates: Towards Safe Reinforcement Learning with
  Zero Training-time Violations
  Training-time safety violations have been a major concern when we deploy
reinforcement learning algorithms in the real world. This paper explores the
possibility of safe RL algorithms with zero training-time safety violations in
the challenging setting where we are only given a safe but trivial-reward
initial policy without any prior knowledge of the dynamics model and additional
offline data. We propose an algorithm, Co-trained Barrier Certificate for Safe
RL (CRABS), which iteratively learns barrier certificates, dynamics models, and
policies. The barrier certificates, learned via adversarial training, ensure
the policy's safety assuming calibrated learned dynamics model. We also add a
regularization term to encourage larger certified regions to enable better
exploration. Empirical simulations show that zero safety violations are already
challenging for a suite of simple environments with only 2-4 dimensional state
space, especially if high-reward policies have to visit regions near the safety
boundary. Prior methods require hundreds of violations to achieve decent
rewards on these tasks, whereas our proposed algorithms incur zero violations.
