On the role of planning in model-based deep reinforcement learning
  Model-based planning is often thought to be necessary for deep, careful
reasoning and generalization in artificial agents. While recent successes of
model-based reinforcement learning (MBRL) with deep function approximation have
strengthened this hypothesis, the resulting diversity of model-based methods
has also made it difficult to track which components drive success and why. In
this paper, we seek to disentangle the contributions of recent methods by
focusing on three questions: (1) How does planning benefit MBRL agents? (2)
Within planning, what choices drive performance? (3) To what extent does
planning improve generalization? To answer these questions, we study the
performance of MuZero (Schrittwieser et al., 2019), a state-of-the-art MBRL
algorithm with strong connections and overlapping components with many other
MBRL algorithms. We perform a number of interventions and ablations of MuZero
across a wide range of environments, including control tasks, Atari, and 9x9
Go. Our results suggest the following: (1) Planning is most useful in the
learning process, both for policy updates and for providing a more useful data
distribution. (2) Using shallow trees with simple Monte-Carlo rollouts is as
performant as more complex methods, except in the most difficult reasoning
tasks. (3) Planning alone is insufficient to drive strong generalization. These
results indicate where and how to utilize planning in reinforcement learning
settings, and highlight a number of open questions for future MBRL research.
