Unsupervised Domain Adaptation with Dynamics-Aware Rewards in
  Reinforcement Learning
  Unsupervised reinforcement learning aims to acquire skills without prior goal
representations, where an agent automatically explores an open-ended
environment to represent goals and learn the goal-conditioned policy. However,
this procedure is often time-consuming, limiting the rollout in some
potentially expensive target environments. The intuitive approach of training
in another interaction-rich environment disrupts the reproducibility of trained
skills in the target environment due to the dynamics shifts and thus inhibits
direct transferring. Assuming free access to a source environment, we propose
an unsupervised domain adaptation method to identify and acquire skills across
dynamics. Particularly, we introduce a KL regularized objective to encourage
emergence of skills, rewarding the agent for both discovering skills and
aligning its behaviors respecting dynamics shifts. This suggests that both
dynamics (source and target) shape the reward to facilitate the learning of
adaptive skills. We also conduct empirical experiments to demonstrate that our
method can effectively learn skills that can be smoothly deployed in target.
