On Covariate Shift of Latent Confounders in Imitation and Reinforcement
  Learning
  We consider the problem of using expert data with unobserved confounders for
imitation and reinforcement learning. We begin by defining the problem of
learning from confounded expert data in a contextual MDP setup. We analyze the
limitations of learning from such data with and without external reward, and
propose an adjustment of standard imitation learning algorithms to fit this
setup. We then discuss the problem of distribution shift between the expert
data and the online environment when the data is only partially observable. We
prove possibility and impossibility results for imitation learning under
arbitrary distribution shift of the missing covariates. When additional
external reward is provided, we propose a sampling procedure that addresses the
unknown shift and prove convergence to an optimal solution. Finally, we
validate our claims empirically on challenging assistive healthcare and
recommender system simulation tasks.
