Imitation Learning by State-Only Distribution Matching
  Imitation Learning from observation describes policy learning in a similar
way to human learning. An agent's policy is trained by observing an expert
performing a task. While many state-only imitation learning approaches are
based on adversarial imitation learning, one main drawback is that adversarial
training is often unstable and lacks a reliable convergence estimator. If the
true environment reward is unknown and cannot be used to select the
best-performing model, this can result in bad real-world policy performance. We
propose a non-adversarial learning-from-observations approach, together with an
interpretable convergence and performance metric.
  Our training objective minimizes the Kulback-Leibler divergence (KLD) between
the policy and expert state transition trajectories which can be optimized in a
non-adversarial fashion. Such methods demonstrate improved robustness when
learned density models guide the optimization. We further improve the sample
efficiency by rewriting the KLD minimization as the Soft Actor Critic objective
based on a modified reward using additional density models that estimate the
environment's forward and backward dynamics. Finally, we evaluate the
effectiveness of our approach on well-known continuous control environments and
show state-of-the-art performance while having a reliable performance estimator
compared to several recent learning-from-observation methods.
