Towards Painless Policy Optimization for Constrained MDPs
  We study policy optimization in an infinite horizon, $\gamma$-discounted
constrained Markov decision process (CMDP). Our objective is to return a policy
that achieves large expected reward with a small constraint violation. We
consider the online setting with linear function approximation and assume
global access to the corresponding features. We propose a generic primal-dual
framework that allows us to bound the reward sub-optimality and constraint
violation for arbitrary algorithms in terms of their primal and dual regret on
online linear optimization problems. We instantiate this framework to use
coin-betting algorithms and propose the Coin Betting Politex (CBP) algorithm.
Assuming that the action-value functions are $\varepsilon_b$-close to the span
of the $d$-dimensional state-action features and no sampling errors, we prove
that $T$ iterations of CBP result in an $O\left(\frac{1}{(1 - \gamma)^3
\sqrt{T}} + \frac{\varepsilon_b\sqrt{d}}{(1 - \gamma)^2} \right)$ reward
sub-optimality and an $O\left(\frac{1}{(1 - \gamma)^2 \sqrt{T}} +
\frac{\varepsilon_b \sqrt{d}}{1 - \gamma} \right)$ constraint violation.
Importantly, unlike gradient descent-ascent and other recent methods, CBP does
not require extensive hyperparameter tuning. Via experiments on synthetic and
Cartpole environments, we demonstrate the effectiveness and robustness of CBP.
