Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve
  Optimality?
  For all its successes, Reinforcement Learning (RL) still struggles to deliver
formal guarantees on the closed-loop behavior of the learned policy. Among
other things, guaranteeing the safety of RL with respect to safety-critical
systems is a very active research topic. Some recent contributions propose to
rely on projections of the inputs delivered by the learned policy into a safe
set, ensuring that the system safety is never jeopardized. Unfortunately, it is
unclear whether this operation can be performed without disrupting the learning
process. This paper addresses this issue. The problem is analysed in the
context of $Q$-learning and policy gradient techniques. We show that the
projection approach is generally disruptive in the context of $Q$-learning
though a simple alternative solves the issue, while simple corrections can be
used in the context of policy gradient methods in order to ensure that the
policy gradients are unbiased. The proposed results extend to safe projections
based on robust MPC techniques.
