Learn Dynamic-Aware State Embedding for Transfer Learning
  Transfer reinforcement learning aims to improve the sample efficiency of
solving unseen new tasks by leveraging experiences obtained from previous
tasks. We consider the setting where all tasks (MDPs) share the same
environment dynamic except reward function. In this setting, the MDP dynamic is
a good knowledge to transfer, which can be inferred by uniformly random policy.
However, trajectories generated by uniform random policy are not useful for
policy improvement, which impairs the sample efficiency severely. Instead, we
observe that the binary MDP dynamic can be inferred from trajectories of any
policy which avoids the need of uniform random policy. As the binary MDP
dynamic contains the state structure shared over all tasks we believe it is
suitable to transfer. Built on this observation, we introduce a method to infer
the binary MDP dynamic on-line and at the same time utilize it to guide state
embedding learning, which is then transferred to new tasks. We keep state
embedding learning and policy learning separately. As a result, the learned
state embedding is task and policy agnostic which makes it ideal for transfer
learning. In addition, to facilitate the exploration over the state space, we
propose a novel intrinsic reward based on the inferred binary MDP dynamic. Our
method can be used out-of-box in combination with model-free RL algorithms. We
show two instances on the basis of \algo{DQN} and \algo{A2C}. Empirical results
of intensive experiments show the advantage of our proposed method in various
transfer learning tasks.
