Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy
  RL
  Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL
algorithms with model-based (Mb)-RL approaches to get the best from both:
asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired
by these works, we propose a hierarchical framework that integrates online
learning for the Mb-trajectory optimization with off-policy methods for the
Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent
based Model Predictive Control (DMD-MPC) is used as the inner loop to obtain an
optimal sequence of actions. These actions are in turn used to significantly
accelerate the outer loop Mf-RL. We show that our formulation is generic for a
broad class of MPC based policies and objectives, and includes some of the
well-known Mb-Mf approaches. Based on the framework we define two algorithms to
increase sample efficiency of Off Policy RL and to guide end to end RL
algorithms for online adaption respectively. Thus we finally introduce two
novel algorithms: Dynamic-Mirror Descent Model Predictive RL(DeMoRL), which
uses the method of elite fractions for the inner loop and Soft Actor-Critic
(SAC) as the off-policy RL for the outer loop and Dynamic-Mirror Descent Model
Predictive Layer(DeMo Layer), a special case of the hierarchical framework
which guides linear policies trained using Augmented Random Search(ARS). Our
experiments show faster convergence of the proposed DeMo RL, and better or
equal performance compared to other Mf-Mb approaches on benchmark MuJoCo
control tasks. The DeMo Layer was tested on classical Cartpole and custom-built
Quadruped trained using Linear Policy.
