Deep Surrogate Q-Learning for Autonomous Driving
  Challenging problems of deep reinforcement learning systems with regard to
the application on real systems are their adaptivity to changing environments
and their efficiency w.r.t. computational resources and data. In the
application of learning lane-change behavior for autonomous driving, agents
have to deal with a varying number of surrounding vehicles. Furthermore, the
number of required transitions imposes a bottleneck, since test drivers cannot
perform an arbitrary amount of lane changes in the real world. In the
off-policy setting, additional information on solving the task can be gained by
observing actions from others. While in the classical RL setup this knowledge
remains unused, we use other drivers as surrogates to learn the agent's value
function more efficiently. We propose Surrogate Q-learning that deals with the
aforementioned problems and reduces the required driving time drastically. We
further propose an efficient implementation based on a permutation-equivariant
deep neural network architecture of the Q-function to estimate action-values
for a variable number of vehicles in sensor range. We show that the
architecture leads to a novel replay sampling technique we call Scene-centric
Experience Replay and evaluate the performance of Surrogate Q-learning and
Scene-centric Experience Replay in the open traffic simulator SUMO.
Additionally, we show that our methods enhance real-world applicability of RL
systems by learning policies on the real highD dataset.
