Deep Inverse Q-learning with Constraints
  Popular Maximum Entropy Inverse Reinforcement Learning approaches require the
computation of expected state visitation frequencies for the optimal policy
under an estimate of the reward function. This usually requires intermediate
value estimation in the inner loop of the algorithm, slowing down convergence
considerably. In this work, we introduce a novel class of algorithms that only
needs to solve the MDP underlying the demonstrated behavior once to recover the
expert policy. This is possible through a formulation that exploits a
probabilistic behavior assumption for the demonstrations within the structure
of Q-learning. We propose Inverse Action-value Iteration which is able to fully
recover an underlying reward of an external agent in closed-form analytically.
We further provide an accompanying class of sampling-based variants which do
not depend on a model of the environment. We show how to extend this class of
algorithms to continuous state-spaces via function approximation and how to
estimate a corresponding action-value function, leading to a policy as close as
possible to the policy of the external agent, while optionally satisfying a
list of predefined hard constraints. We evaluate the resulting algorithms
called Inverse Action-value Iteration, Inverse Q-learning and Deep Inverse
Q-learning on the Objectworld benchmark, showing a speedup of up to several
orders of magnitude compared to (Deep) Max-Entropy algorithms. We further apply
Deep Constrained Inverse Q-learning on the task of learning autonomous
lane-changes in the open-source simulator SUMO achieving competent driving
after training on data corresponding to 30 minutes of demonstrations.
