Learning without Knowing: Unobserved Context in Continuous Transfer
  Reinforcement Learning
  In this paper, we consider a transfer Reinforcement Learning (RL) problem in
continuous state and action spaces, under unobserved contextual information.
For example, the context can represent the mental view of the world that an
expert agent has formed through past interactions with this world. We assume
that this context is not accessible to a learner agent who can only observe the
expert data. Then, our goal is to use the context-aware expert data to learn an
optimal context-unaware policy for the learner using only a few new data
samples. Such problems are typically solved using imitation learning that
assumes that both the expert and learner agents have access to the same
information. However, if the learner does not know the expert context, using
the expert data alone will result in a biased learner policy and will require
many new data samples to improve. To address this challenge, in this paper, we
formulate the learning problem as a causal bound-constrained Multi-Armed-Bandit
(MAB) problem. The arms of this MAB correspond to a set of basis policy
functions that can be initialized in an unsupervised way using the expert data
and represent the different expert behaviors affected by the unobserved
context. On the other hand, the MAB constraints correspond to causal bounds on
the accumulated rewards of these basis policy functions that we also compute
from the expert data. The solution to this MAB allows the learner agent to
select the best basis policy and improve it online. And the use of causal
bounds reduces the exploration variance and, therefore, improves the learning
rate. We provide numerical experiments on an autonomous driving example that
show that our proposed transfer RL method improves the learner's policy faster
compared to existing imitation learning methods and enjoys much lower variance
during training.
