Policy Transfer across Visual and Dynamics Domain Gaps via Iterative
  Grounding
  The ability to transfer a policy from one environment to another is a
promising avenue for efficient robot learning in realistic settings where task
supervision is not available. This can allow us to take advantage of
environments well suited for training, such as simulators or laboratories, to
learn a policy for a real robot in a home or office. To succeed, such policy
transfer must overcome both the visual domain gap (e.g. different illumination
or background) and the dynamics domain gap (e.g. different robot calibration or
modelling error) between source and target environments. However, prior policy
transfer approaches either cannot handle a large domain gap or can only address
one type of domain gap at a time. In this paper, we propose a novel policy
transfer method with iterative "environment grounding", IDAPT, that alternates
between (1) directly minimizing both visual and dynamics domain gaps by
grounding the source environment in the target environment domains, and (2)
training a policy on the grounded source environment. This iterative training
progressively aligns the domains between the two environments and adapts the
policy to the target environment. Once trained, the policy can be directly
executed on the target environment. The empirical results on locomotion and
robotic manipulation tasks demonstrate that our approach can effectively
transfer a policy across visual and dynamics domain gaps with minimal
supervision and interaction with the target environment. Videos and code are
available at https://clvrai.com/idapt .
