Provably Robust Blackbox Optimization for Reinforcement Learning
  Interest in derivative-free optimization (DFO) and "evolutionary strategies"
(ES) has recently surged in the Reinforcement Learning (RL) community, with
growing evidence that they can match state of the art methods for policy
optimization problems in Robotics. However, it is well known that DFO methods
suffer from prohibitively high sampling complexity. They can also be very
sensitive to noisy rewards and stochastic dynamics. In this paper, we propose a
new class of algorithms, called Robust Blackbox Optimization (RBO). Remarkably,
even if up to $23\%$ of all the measurements are arbitrarily corrupted, RBO can
provably recover gradients to high accuracy. RBO relies on learning gradient
flows using robust regression methods to enable off-policy updates. On several
MuJoCo robot control tasks, when all other RL approaches collapse in the
presence of adversarial noise, RBO is able to train policies effectively. We
also show that RBO can be applied to legged locomotion tasks including path
tracking for quadruped robots.
