Achieving Zero Constraint Violation for Concave Utility Constrained
  Reinforcement Learning via Primal-Dual Approach
  Reinforcement learning (RL) is widely used in applications where one needs to
perform sequential decision-making while interacting with the environment. The
standard RL problem with safety constraints can be formulated via constrained
Markov Decision Processes (CMDP), which is linear in objective and constraints
in occupancy measure space, where the problem becomes challenging in the case
where model is unknown apriori. The problem further becomes challenging when
the decision requirement includes optimizing a concave utility while satisfying
some nonlinear safety constraints (we call this problem concave CMDP (CCMDP)).
In the literature, various algorithms are available to solve CMDP problems in a
model-free manner to achieve $\epsilon$-optimal cumulative reward with
$\epsilon$ feasible policies. An $\epsilon$-feasible policy implies that it
suffers from constraint violation. An important question here is whether we can
achieve $\epsilon$-optimal objective with zero constraint violations or not. To
answer that in affirmative, we advocate the use of randomized primal-dual
approach to solve the CCMDP problems and propose a conservative stochastic
primal-dual algorithm (CSPDA). We prove that CSPDA exhibit
$\tilde{\mathcal{O}}\left(1/\epsilon^2\right)$ sample complexity to achieve
$\epsilon$-optimal concave utility with zero constraint violations for the
CCMDP, and CMDP as well which is a special case. In the prior works, the best
available sample complexity for the $\epsilon$-optimal policy for CMDP with
zero constraint violation is $\tilde{\mathcal{O}}\left(1/\epsilon^5\right)$.
Hence, the proposed algorithm provides a significant improvement as compared to
the state of the art.
