Model-based Reinforcement Learning: A Survey
  Sequential decision making, commonly formalized as Markov Decision Process
(MDP) optimization, is a important challenge in artificial intelligence. Two
key approaches to this problem are reinforcement learning (RL) and planning.
This paper presents a survey of the integration of both fields, better known as
model-based reinforcement learning. Model-based RL has two main steps. First,
we systematically cover approaches to dynamics model learning, including
challenges like dealing with stochasticity, uncertainty, partial observability,
and temporal abstraction. Second, we present a systematic categorization of
planning-learning integration, including aspects like: where to start planning,
what budgets to allocate to planning and real data collection, how to plan, and
how to integrate planning in the learning and acting loop. After these two
sections, we also discuss implicit model-based RL as an end-to-end alternative
for model learning and planning, and we cover the potential benefits of
model-based RL. Along the way, the survey also draws connections to several
related RL fields, like hierarchical RL and transfer learning. Altogether, the
survey presents a broad conceptual overview of the combination of planning and
learning for MDP optimization.
