Robust Reinforcement Learning for Continuous Control with Model
  Misspecification
  We provide a framework for incorporating robustness -- to perturbations in
the transition dynamics which we refer to as model misspecification -- into
continuous control Reinforcement Learning (RL) algorithms. We specifically
focus on incorporating robustness into a state-of-the-art continuous control RL
algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve
this by learning a policy that optimizes for a worst case expected return
objective and derive a corresponding robust entropy-regularized Bellman
contraction operator. In addition, we introduce a less conservative,
soft-robust, entropy-regularized objective with a corresponding Bellman
operator. We show that both, robust and soft-robust policies, outperform their
non-robust counterparts in nine Mujoco domains with environment perturbations.
In addition, we show improved robust performance on a high-dimensional,
simulated, dexterous robotic hand. Finally, we present multiple investigative
experiments that provide a deeper insight into the robustness framework. This
includes an adaptation to another continuous control RL algorithm as well as
learning the uncertainty set from offline data. Performance videos can be found
online at https://sites.google.com/view/robust-rl.
