Closing the Sim-to-Real Loop: Adapting Simulation Randomization with
  Real World Experience
  We consider the problem of transferring policies to the real world by
training on a distribution of simulated scenarios. Rather than manually tuning
the randomization of simulations, we adapt the simulation parameter
distribution using a few real world roll-outs interleaved with policy training.
In doing so, we are able to change the distribution of simulations to improve
the policy transfer by matching the policy behavior in simulation and the real
world. We show that policies trained with our method are able to reliably
transfer to different robots in two real world tasks: swing-peg-in-hole and
opening a cabinet drawer. The video of our experiments can be found at
https://sites.google.com/view/simopt
