What Matters for Adversarial Imitation Learning?
  Adversarial imitation learning has become a popular framework for imitation
in continuous control. Over the years, several variations of its components
were proposed to enhance the performance of the learned policies as well as the
sample complexity of the algorithm. In practice, these choices are rarely
tested all together in rigorous empirical studies. It is therefore difficult to
discuss and understand what choices, among the high-level algorithmic options
as well as low-level implementation details, matter. To tackle this issue, we
implement more than 50 of these choices in a generic adversarial imitation
learning framework and investigate their impacts in a large-scale study (>500k
trained agents) with both synthetic and human-generated demonstrations. While
many of our findings confirm common practices, some of them are surprising or
even contradict prior work. In particular, our results suggest that artificial
demonstrations are not a good proxy for human data and that the very common
practice of evaluating imitation algorithms only with synthetic demonstrations
may lead to algorithms which perform poorly in the more realistic scenarios
with human demonstrations.
