Robust Imitation of Diverse Behaviors
  Deep generative models have recently shown great promise in imitation
learning for motor control. Given enough data, even supervised approaches can
do one-shot imitation learning; however, they are vulnerable to cascading
failures when the agent trajectory diverges from the demonstrations. Compared
to purely supervised methods, Generative Adversarial Imitation Learning (GAIL)
can learn more robust controllers from fewer demonstrations, but is inherently
mode-seeking and more difficult to train. In this paper, we show how to combine
the favourable aspects of these two approaches. The base of our model is a new
type of variational autoencoder on demonstration trajectories that learns
semantic policy embeddings. We show that these embeddings can be learned on a 9
DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a
resulting smooth interpolation of reaching behavior. Leveraging these policy
representations, we develop a new version of GAIL that (1) is much more robust
than the purely-supervised controller, especially with few demonstrations, and
(2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own
does not. We demonstrate our approach on learning diverse gaits from
demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics
environment.
