State-only Imitation with Transition Dynamics Mismatch
  Imitation Learning (IL) is a popular paradigm for training agents to achieve
complicated goals by leveraging expert behavior, rather than dealing with the
hardships of designing a correct reward function. With the environment modeled
as a Markov Decision Process (MDP), most of the existing IL algorithms are
contingent on the availability of expert demonstrations in the same MDP as the
one in which a new imitator policy is to be learned. This is uncharacteristic
of many real-life scenarios where discrepancies between the expert and the
imitator MDPs are common, especially in the transition dynamics function.
Furthermore, obtaining expert actions may be costly or infeasible, making the
recent trend towards state-only IL (where expert demonstrations constitute only
states or observations) ever so promising. Building on recent adversarial
imitation approaches that are motivated by the idea of divergence minimization,
we present a new state-only IL algorithm in this paper. It divides the overall
optimization objective into two subproblems by introducing an indirection step
and solves the subproblems iteratively. We show that our algorithm is
particularly effective when there is a transition dynamics mismatch between the
expert and imitator MDPs, while the baseline IL methods suffer from performance
degradation. To analyze this, we construct several interesting MDPs by
modifying the configuration parameters for the MuJoCo locomotion tasks from
OpenAI Gym.
