Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
  Many model-based reinforcement learning (RL) methods follow a similar
template: fit a model to previously observed data, and then use data from that
model for RL or planning. However, models that achieve better training
performance (e.g., lower MSE) are not necessarily better for control: an RL
agent may seek out the small fraction of states where an accurate model makes
mistakes, or it might act in ways that do not expose the errors of an
inaccurate model. As noted in prior work, there is an objective mismatch:
models are useful if they yield good policies, but they are trained to maximize
their accuracy, rather than the performance of the policies that result from
them. In this work, we propose a single objective for jointly training the
model and the policy, such that updates to either component increases a lower
bound on expected return. This joint optimization mends the objective mismatch
in prior work. Our objective is a global lower bound on expected return, and
this bound becomes tight under certain assumptions. The resulting algorithm
(MnM) is conceptually similar to a GAN: a classifier distinguishes between real
and fake transitions, the model is updated to produce transitions that look
realistic, and the policy is updated to avoid states where the model
predictions are unrealistic.
