Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  We study the problem of safe offline reinforcement learning (RL), the goal is
to learn a policy that maximizes long-term reward while satisfying safety
constraints given only offline data, without further interaction with the
environment. This problem is more appealing for real world RL applications, in
which data collection is costly or dangerous. Enforcing constraint satisfaction
is non-trivial, especially in offline settings, as there is a potential large
discrepancy between the policy distribution and the data distribution, causing
errors in estimating the value of safety constraints. We show that na\"ive
approaches that combine techniques from safe RL and offline RL can only learn
sub-optimal solutions. We thus develop a simple yet effective algorithm,
Constraints Penalized Q-Learning (CPQ), to solve the problem. Our method admits
the use of data generated by mixed behavior policies. We present a theoretical
analysis and demonstrate empirically that our approach can learn robustly
across a variety of benchmark control tasks, outperforming several baselines.
