Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization
  under Model Uncertainty
  In this paper, we focus on the problem of robustifying reinforcement learning
(RL) algorithms with respect to model uncertainties. Indeed, in the framework
of model-based RL, we propose to merge the theory of constrained Markov
decision process (CMDP), with the theory of robust Markov decision process
(RMDP), leading to a formulation of robust constrained-MDPs (RCMDP). This
formulation, simple in essence, allows us to design RL algorithms that are
robust in performance, and provides constraint satisfaction guarantees, with
respect to uncertainties in the system's states transition probabilities. The
need for RCMPDs is important for real-life applications of RL. For instance,
such formulation can play an important role for policy transfer from simulation
to real world (Sim2Real) in safety critical applications, which would benefit
from performance and safety guarantees which are robust w.r.t model
uncertainty. We first propose the general problem formulation under the concept
of RCMDP, and then propose a Lagrangian formulation of the optimal problem,
leading to a robust-constrained policy gradient RL algorithm. We finally
validate this concept on the inventory management problem.
