Learning Policies with Zero or Bounded Constraint Violation for
  Constrained MDPs
  We address the issue of safety in reinforcement learning. We pose the problem
in an episodic framework of a constrained Markov decision process. Existing
results have shown that it is possible to achieve a reward regret of
$\tilde{\mathcal{O}}(\sqrt{K})$ while allowing an
$\tilde{\mathcal{O}}(\sqrt{K})$ constraint violation in $K$ episodes. A
critical question that arises is whether it is possible to keep the constraint
violation even smaller. We show that when a strictly safe policy is known, then
one can confine the system to zero constraint violation with arbitrarily high
probability while keeping the reward regret of order
$\tilde{\mathcal{O}}(\sqrt{K})$. The algorithm which does so employs the
principle of optimistic pessimism in the face of uncertainty to achieve safe
exploration. When no strictly safe policy is known, though one is known to
exist, then it is possible to restrict the system to bounded constraint
violation with arbitrarily high probability. This is shown to be realized by a
primal-dual algorithm with an optimistic primal estimate and a pessimistic dual
update.
