PAnDR: Fast Adaptation to New Environments from Offline Experiences via
  Decoupling Policy and Environment Representations
  Deep Reinforcement Learning (DRL) has been a promising solution to many
complex decision-making problems. Nevertheless, the notorious weakness in
generalization among environments prevent widespread application of DRL agents
in real-world scenarios. Although advances have been made recently, most prior
works assume sufficient online interaction on training environments, which can
be costly in practical cases. To this end, we focus on an
offline-training-online-adaptation setting, in which the agent first learns
from offline experiences collected in environments with different dynamics and
then performs online policy adaptation in environments with new dynamics. In
this paper, we propose Policy Adaptation with Decoupled Representations (PAnDR)
for fast policy adaptation. In offline training phase, the environment
representation and policy representation are learned through contrastive
learning and policy recovery, respectively. The representations are further
refined by mutual information optimization to make them more decoupled and
complete. With learned representations, a Policy-Dynamics Value Function (PDVF)
[Raileanu et al., 2020] network is trained to approximate the values for
different combinations of policies and environments from offline experiences.
In online adaptation phase, with the environment context inferred from few
experiences collected in new environments, the policy is optimized by gradient
ascent with respect to the PDVF. Our experiments show that PAnDR outperforms
existing algorithms in several representative policy adaptation problems.
