Model-Augmented Actor-Critic: Backpropagating through Paths
  Current model-based reinforcement learning approaches use the model simply as
a learned black-box simulator to augment the data for policy optimization or
value function learning. In this paper, we show how to make more effective use
of the model by exploiting its differentiability. We construct a policy
optimization algorithm that uses the pathwise derivative of the learned model
and policy across future timesteps. Instabilities of learning across many
timesteps are prevented by using a terminal value function, learning the policy
in an actor-critic fashion. Furthermore, we present a derivation on the
monotonic improvement of our objective in terms of the gradient error in the
model and value function. We show that our approach (i) is consistently more
sample efficient than existing state-of-the-art model-based algorithms, (ii)
matches the asymptotic performance of model-free algorithms, and (iii) scales
to long horizons, a regime where typically past model-based approaches have
struggled.
