Separated Proportional-Integral Lagrangian for Chance Constrained
  Reinforcement Learning
  Safety is essential for reinforcement learning (RL) applied in real-world
tasks like autonomous driving. Chance constraints which guarantee the
satisfaction of state constraints at a high probability are suitable to
represent the requirements in real-world environment with uncertainty. Existing
chance constrained RL methods like the penalty method and the Lagrangian method
either exhibit periodic oscillations or cannot satisfy the constraints. In this
paper, we address these shortcomings by proposing a separated
proportional-integral Lagrangian (SPIL) algorithm. Taking a control
perspective, we first interpret the penalty method and the Lagrangian method as
proportional feedback and integral feedback control, respectively. Then, a
proportional-integral Lagrangian method is proposed to steady learning process
while improving safety. To prevent integral overshooting and reduce
conservatism, we introduce the integral separation technique inspired by PID
control. Finally, an analytical gradient of the chance constraint is utilized
for model-based policy optimization. The effectiveness of SPIL is demonstrated
by a narrow car-following task. Experiments indicate that compared with
previous methods, SPIL improves the performance while guaranteeing safety, with
a steady learning process.
