Safe Reinforcement Learning for Autonomous Vehicles through Parallel
  Constrained Policy Optimization
  Reinforcement learning (RL) is attracting increasing interests in autonomous
driving due to its potential to solve complex classification and control
problems. However, existing RL algorithms are rarely applied to real vehicles
for two predominant problems: behaviours are unexplainable, and they cannot
guarantee safety under new scenarios. This paper presents a safe RL algorithm,
called Parallel Constrained Policy Optimization (PCPO), for two autonomous
driving tasks. PCPO extends today's common actor-critic architecture to a
three-component learning framework, in which three neural networks are used to
approximate the policy function, value function and a newly added risk
function, respectively. Meanwhile, a trust region constraint is added to allow
large update steps without breaking the monotonic improvement condition. To
ensure the feasibility of safety constrained problems, synchronized parallel
learners are employed to explore different state spaces, which accelerates
learning and policy-update. The simulations of two scenarios for autonomous
vehicles confirm we can ensure safety while achieving fast learning.
