Sample Efficient Imitation Learning via Reward Function Trained in
  Advance
  Imitation learning (IL) is a framework that learns to imitate expert behavior
from demonstrations. Recently, IL shows promising results on high dimensional
and control tasks. However, IL typically suffers from sample inefficiency in
terms of environment interaction, which severely limits their application to
simulated domains. In industrial applications, learner usually have a high
interaction cost, the more interactions with environment, the more damage it
causes to the environment and the learner itself. In this article, we make an
effort to improve sample efficiency by introducing a novel scheme of inverse
reinforcement learning. Our method, which we call \textit{Model Reward Function
Based Imitation Learning} (MRFIL), uses an ensemble dynamic model as a reward
function, what is trained with expert demonstrations. The key idea is to
provide the agent with an incentive to match the demonstrations over a long
horizon, by providing a positive reward upon encountering states in line with
the expert demonstration distribution. In addition, we demonstrate the
convergence guarantee for new objective function. Experimental results show
that our algorithm reaches the competitive performance and significantly
reducing the environment interactions compared to IL methods.
