SoftDICE for Imitation Learning: Rethinking Off-policy Distribution
  Matching
  We present SoftDICE, which achieves state-of-the-art performance for
imitation learning. SoftDICE fixes several key problems in ValueDICE, an
off-policy distribution matching approach for sample-efficient imitation
learning. Specifically, the objective of ValueDICE contains logarithms and
exponentials of expectations, for which the mini-batch gradient estimate is
always biased. Second, ValueDICE regularizes the objective with replay buffer
samples when expert demonstrations are limited in number, which however changes
the original distribution matching problem. Third, the re-parametrization trick
used to derive the off-policy objective relies on an implicit assumption that
rarely holds in training. We leverage a novel formulation of distribution
matching and consider an entropy-regularized off-policy objective, which yields
a completely offline algorithm called SoftDICE. Our empirical results show that
SoftDICE recovers the expert policy with only one demonstration trajectory and
no further on-policy/off-policy samples. SoftDICE also stably outperforms
ValueDICE and other baselines in terms of sample efficiency on Mujoco benchmark
tasks.
