Control-Oriented Model-Based Reinforcement Learning with Implicit
  Differentiation
  The shortcomings of maximum likelihood estimation in the context of
model-based reinforcement learning have been highlighted by an increasing
number of papers. When the model class is misspecified or has a limited
representational capacity, model parameters with high likelihood might not
necessarily result in high performance of the agent on a downstream control
task. To alleviate this problem, we propose an end-to-end approach for model
learning which directly optimizes the expected returns using implicit
differentiation. We treat a value function that satisfies the Bellman
optimality operator induced by the model as an implicit function of model
parameters and show how to differentiate the function. We provide theoretical
and empirical evidence highlighting the benefits of our approach in the model
misspecification regime compared to likelihood-based methods.
