How to Learn a Useful Critic? Model-based Action-Gradient-Estimator
  Policy Optimization
  Deterministic-policy actor-critic algorithms for continuous control improve
the actor by plugging its actions into the critic and ascending the
action-value gradient, which is obtained by chaining the actor's Jacobian
matrix with the gradient of the critic with respect to input actions. However,
instead of gradients, the critic is, typically, only trained to accurately
predict expected returns, which, on their own, are useless for policy
optimization. In this paper, we propose MAGE, a model-based actor-critic
algorithm, grounded in the theory of policy gradients, which explicitly learns
the action-value gradient. MAGE backpropagates through the learned dynamics to
compute gradient targets in temporal difference learning, leading to a critic
tailored for policy improvement. On a set of MuJoCo continuous-control tasks,
we demonstrate the efficiency of the algorithm in comparison to model-free and
model-based state-of-the-art baselines.
