On Computation and Generalization of Generative Adversarial Imitation
  Learning
  Generative Adversarial Imitation Learning (GAIL) is a powerful and practical
approach for learning sequential decision-making policies. Different from
Reinforcement Learning (RL), GAIL takes advantage of demonstration data by
experts (e.g., human), and learns both the policy and reward function of the
unknown environment. Despite the significant empirical progresses, the theory
behind GAIL is still largely unknown. The major difficulty comes from the
underlying temporal dependency of the demonstration data and the minimax
computational formulation of GAIL without convex-concave structure. To bridge
such a gap between theory and practice, this paper investigates the theoretical
properties of GAIL. Specifically, we show: (1) For GAIL with general reward
parameterization, the generalization can be guaranteed as long as the class of
the reward functions is properly controlled; (2) For GAIL, where the reward is
parameterized as a reproducing kernel function, GAIL can be efficiently solved
by stochastic first order optimization algorithms, which attain sublinear
convergence to a stationary solution. To the best of our knowledge, these are
the first results on statistical and computational guarantees of imitation
learning with reward/policy function approximation. Numerical experiments are
provided to support our analysis.
