Guided Policy Search as Approximate Mirror Descent
  Guided policy search algorithms can be used to optimize complex nonlinear
policies, such as deep neural networks, without directly computing policy
gradients in the high-dimensional parameter space. Instead, these methods use
supervised learning to train the policy to mimic a "teacher" algorithm, such as
a trajectory optimizer or a trajectory-centric reinforcement learning method.
Guided policy search methods provide asymptotic local convergence guarantees by
construction, but it is not clear how much the policy improves within a small,
finite number of iterations. We show that guided policy search algorithms can
be interpreted as an approximate variant of mirror descent, where the
projection onto the constraint manifold is not exact. We derive a new guided
policy search algorithm that is simpler and provides appealing improvement and
convergence guarantees in simplified convex and linear settings, and show that
in the more general nonlinear setting, the error in the projection step can be
bounded. We provide empirical results on several simulated robotic navigation
and manipulation tasks that show that our method is stable and achieves similar
or better performance when compared to prior guided policy search methods, with
a simpler formulation and fewer hyperparameters.
