Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal
  Demonstrations
  This paper considers the problem of inverse reinforcement learning in
zero-sum stochastic games when expert demonstrations are known to be not
optimal. Compared to previous works that decouple agents in the game by
assuming optimality in expert strategies, we introduce a new objective function
that directly pits experts against Nash Equilibrium strategies, and we design
an algorithm to solve for the reward function in the context of inverse
reinforcement learning with deep neural networks as model approximations. In
our setting the model and algorithm do not decouple by agent. In order to find
Nash Equilibrium in large-scale games, we also propose an adversarial training
algorithm for zero-sum stochastic games, and show the theoretical appeal of
non-existence of local optima in its objective function. In our numerical
experiments, we demonstrate that our Nash Equilibrium and inverse reinforcement
learning algorithms address games that are not amenable to previous approaches
using tabular representations. Moreover, with sub-optimal expert demonstrations
our algorithms recover both reward functions and strategies with good quality.
