Feedback in Imitation Learning: The Three Regimes of Covariate Shift
  Imitation learning practitioners have often noted that conditioning policies
on previous actions leads to a dramatic divergence between "held out" error and
performance of the learner in situ. Interactive approaches can provably address
this divergence but require repeated querying of a demonstrator. Recent work
identifies this divergence as stemming from a "causal confound" in predicting
the current action, and seek to ablate causal aspects of current state using
tools from causal inference. In this work, we argue instead that this
divergence is simply another manifestation of covariate shift, exacerbated
particularly by settings of feedback between decisions and input features. The
learner often comes to rely on features that are strongly predictive of
decisions, but are subject to strong covariate shift.
  Our work demonstrates a broad class of problems where this shift can be
mitigated, both theoretically and practically, by taking advantage of a
simulator but without any further querying of expert demonstration. We analyze
existing benchmarks used to test imitation learning approaches and find that
these benchmarks are realizable and simple and thus insufficient for capturing
the harder regimes of error compounding seen in real-world decision making
problems. We find, in a surprising contrast with previous literature, but
consistent with our theory, that naive behavioral cloning provides excellent
results. We detail the need for new standardized benchmarks that capture the
phenomena seen in robotics problems.
