Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide
  Task Spaces
  In this paper, we study the problem of zero-shot sim-to-real when the task
requires both highly precise control with sub-millimetre error tolerance, and
wide task space generalisation. Our framework involves a coarse-to-fine
controller, where trajectories begin with classical motion planning using
ICP-based pose estimation, and transition to a learned end-to-end controller
which maps images to actions and is trained in simulation with domain
randomisation. In this way, we achieve precise control whilst also generalising
the controller across wide task spaces, and keeping the robustness of
vision-based, end-to-end control. Real-world experiments on a range of
different tasks show that, by exploiting the best of both worlds, our framework
significantly outperforms purely motion planning methods, and purely
learning-based methods. Furthermore, we answer a range of questions on best
practices for precise sim-to-real transfer, such as how different image sensor
modalities and image feature representations perform.
