A Lyapunov-based Approach to Safe Reinforcement Learning
  In many real-world reinforcement learning (RL) problems, besides optimizing
the main objective function, an agent must concurrently avoid violating a
number of constraints. In particular, besides optimizing performance it is
crucial to guarantee the safety of an agent during training as well as
deployment (e.g. a robot should avoid taking actions - exploratory or not -
which irrevocably harm its hardware). To incorporate safety in RL, we derive
algorithms under the framework of constrained Markov decision problems (CMDPs),
an extension of the standard Markov decision problems (MDPs) augmented with
constraints on expected cumulative costs. Our approach hinges on a novel
\emph{Lyapunov} method. We define and present a method for constructing
Lyapunov functions, which provide an effective way to guarantee the global
safety of a behavior policy during training via a set of local, linear
constraints. Leveraging these theoretical underpinnings, we show how to use the
Lyapunov approach to systematically transform dynamic programming (DP) and RL
algorithms into their safe counterparts. To illustrate their effectiveness, we
evaluate these algorithms in several CMDP planning and decision-making tasks on
a safety benchmark domain. Our results show that our proposed method
significantly outperforms existing baselines in balancing constraint
satisfaction and performance.
