f-IRL: Inverse Reinforcement Learning via State Marginal Matching
  Imitation learning is well-suited for robotic tasks where it is difficult to
directly program the behavior or specify a cost for optimal control. In this
work, we propose a method for learning the reward function (and the
corresponding policy) to match the expert state density. Our main result is the
analytic gradient of any f-divergence between the agent and expert state
distribution w.r.t. reward parameters. Based on the derived gradient, we
present an algorithm, f-IRL, that recovers a stationary reward function from
the expert density by gradient descent. We show that f-IRL can learn behaviors
from a hand-designed target state density or implicitly through expert
observations. Our method outperforms adversarial imitation learning methods in
terms of sample efficiency and the required number of expert trajectories on
IRL benchmarks. Moreover, we show that the recovered reward function can be
used to quickly solve downstream tasks, and empirically demonstrate its utility
on hard-to-explore tasks and for behavior transfer across changes in dynamics.
