Transfer Reinforcement Learning under Unobserved Contextual Information
  In this paper, we study a transfer reinforcement learning problem where the
state transitions and rewards are affected by the environmental context.
Specifically, we consider a demonstrator agent that has access to a
context-aware policy and can generate transition and reward data based on that
policy. These data constitute the experience of the demonstrator. Then, the
goal is to transfer this experience, excluding the underlying contextual
information, to a learner agent that does not have access to the environmental
context, so that they can learn a control policy using fewer samples. It is
well known that, disregarding the causal effect of the contextual information,
can introduce bias in the transition and reward models estimated by the
learner, resulting in a learned suboptimal policy. To address this challenge,
in this paper, we develop a method to obtain causal bounds on the transition
and reward functions using the demonstrator's data, which we then use to obtain
causal bounds on the value functions. Using these value function bounds, we
propose new Q learning and UCB-Q learning algorithms that converge to the true
value function without bias. We provide numerical experiments for robot motion
planning problems that validate the proposed value function bounds and
demonstrate that the proposed algorithms can effectively make use of the data
from the demonstrator to accelerate the learning process of the learner.
