Learning with Safety Constraints: Sample Complexity of Reinforcement
  Learning for Constrained MDPs
  Many physical systems have underlying safety considerations that require that
the policy employed ensures the satisfaction of a set of constraints. The
analytical formulation usually takes the form of a Constrained Markov Decision
Process (CMDP). We focus on the case where the CMDP is unknown, and RL
algorithms obtain samples to discover the model and compute an optimal
constrained policy. Our goal is to characterize the relationship between safety
constraints and the number of samples needed to ensure a desired level of
accuracy -- both objective maximization and constraint satisfaction -- in a PAC
sense. We explore two classes of RL algorithms, namely, (i) a generative model
based approach, wherein samples are taken initially to estimate a model, and
(ii) an online approach, wherein the model is updated as samples are obtained.
Our main finding is that compared to the best known bounds of the unconstrained
regime, the sample complexity of constrained RL algorithms are increased by a
factor that is logarithmic in the number of constraints, which suggests that
the approach may be easily utilized in real systems.
