Reinforcement Learning of Markov Decision Processes with Peak
  Constraints
  In this paper, we consider reinforcement learning of Markov Decision
Processes (MDP) with peak constraints, where an agent chooses a policy to
optimize an objective and at the same time satisfy additional constraints. The
agent has to take actions based on the observed states, reward outputs, and
constraint-outputs, without any knowledge about the dynamics, reward functions,
and/or the knowledge of the constraint-functions. We introduce a game theoretic
approach to construct reinforcement learning algorithms where the agent
maximizes an unconstrained objective that depends on the simulated action of
the minimizing opponent which acts on a finite set of actions and the output
data of the constraint functions (rewards). We show that the policies obtained
from maximin Q-learning converge to the optimal policies. To the best of our
knowledge, this is the first time learning algorithms guarantee convergence to
optimal stationary policies for the MDP problem with peak constraints for both
discounted and expected average rewards.
