Strictly Batch Imitation Learning by Energy-based Distribution Matching
  Consider learning a policy purely on the basis of demonstrated behavior --
that is, with no access to reinforcement signals, no knowledge of transition
dynamics, and no further interaction with the environment. This *strictly batch
imitation learning* problem arises wherever live experimentation is costly,
such as in healthcare. One solution is simply to retrofit existing algorithms
for apprenticeship learning to work in the offline setting. But such an
approach leans heavily on off-policy evaluation or offline model estimation,
and can be indirect and inefficient. We argue that a good solution should be
able to explicitly parameterize a policy (i.e. respecting action conditionals),
implicitly learn from rollout dynamics (i.e. leveraging state marginals), and
-- crucially -- operate in an entirely offline fashion. To address this
challenge, we propose a novel technique by *energy-based distribution matching*
(EDM): By identifying parameterizations of the (discriminative) model of a
policy with the (generative) energy function for state distributions, EDM
yields a simple but effective solution that equivalently minimizes a divergence
between the occupancy measure for the demonstrator and a model thereof for the
imitator. Through experiments with application to control and healthcare
settings, we illustrate consistent performance gains over existing algorithms
for strictly batch imitation learning.
