Energy-Based Imitation Learning
  We tackle a common scenario in imitation learning (IL), where agents try to
recover the optimal policy from expert demonstrations without further access to
the expert or environment reward signals. Except the simple Behavior Cloning
(BC) that adopts supervised learning followed by the problem of compounding
error, previous solutions like inverse reinforcement learning (IRL) and recent
generative adversarial methods involve a bi-level or alternating optimization
for updating the reward function and the policy, suffering from high
computational cost and training instability. Inspired by recent progress in
energy-based model (EBM), in this paper, we propose a simplified IL framework
named Energy-Based Imitation Learning (EBIL). Instead of updating the reward
and policy iteratively, EBIL breaks out of the traditional IRL paradigm by a
simple and flexible two-stage solution: first estimating the expert energy as
the surrogate reward function through score matching, then utilizing such a
reward for learning the policy by reinforcement learning algorithms. EBIL
combines the idea of both EBM and occupancy measure matching, and via theoretic
analysis we reveal that EBIL and Max-Entropy IRL (MaxEnt IRL) approaches are
two sides of the same coin, and thus EBIL could be an alternative of
adversarial IRL methods. Extensive experiments on qualitative and quantitative
evaluations indicate that EBIL is able to recover meaningful and interpretative
reward signals while achieving effective and comparable performance against
existing algorithms on IL benchmarks.
