A Game Theoretic Framework for Model Based Reinforcement Learning
  Model-based reinforcement learning (MBRL) has recently gained immense
interest due to its potential for sample efficiency and ability to incorporate
off-policy data. However, designing stable and efficient MBRL algorithms using
rich function approximators have remained challenging. To help expose the
practical challenges in MBRL and simplify algorithm design from the lens of
abstraction, we develop a new framework that casts MBRL as a game between: (1)
a policy player, which attempts to maximize rewards under the learned model;
(2) a model player, which attempts to fit the real-world data collected by the
policy player. For algorithm development, we construct a Stackelberg game
between the two players, and show that it can be solved with approximate
bi-level optimization. This gives rise to two natural families of algorithms
for MBRL based on which player is chosen as the leader in the Stackelberg game.
Together, they encapsulate, unify, and generalize many previous MBRL
algorithms. Furthermore, our framework is consistent with and provides a clear
basis for heuristics known to be important in practice from prior works.
Finally, through experiments we validate that our proposed algorithms are
highly sample efficient, match the asymptotic performance of model-free policy
gradient, and scale gracefully to high-dimensional tasks like dexterous hand
manipulation. Additional details and code can be obtained from the project page
at https://sites.google.com/view/mbrl-game
