Distributionally Robust Policy Learning via Adversarial Environment
  Generation
  Our goal is to train control policies that generalize well to unseen
environments. Inspired by the Distributionally Robust Optimization (DRO)
framework, we propose DRAGEN - Distributionally Robust policy learning via
Adversarial Generation of ENvironments - for iteratively improving robustness
of policies to realistic distribution shifts by generating adversarial
environments. The key idea is to learn a generative model for environments
whose latent variables capture cost-predictive and realistic variations in
environments. We perform DRO with respect to a Wasserstein ball around the
empirical distribution of environments by generating realistic adversarial
environments via gradient ascent on the latent space. We demonstrate strong
Out-of-Distribution (OoD) generalization in simulation for (i) swinging up a
pendulum with onboard vision and (ii) grasping realistic 3D objects. Grasping
experiments on hardware demonstrate better sim2real performance compared to
domain randomization.
