A Unified Perspective on Value Backup and Exploration in Monte-Carlo
  Tree Search
  Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex
decision-making problems through the synergy of Monte-Carlo planning and
Reinforcement Learning (RL). The highly combinatorial nature of the problems
commonly addressed by MCTS requires the use of efficient exploration strategies
for navigating the planning tree and quickly convergent value backup methods.
These crucial problems are particularly evident in recent advances that combine
MCTS with deep neural networks for function approximation. In this work, we
propose two methods for improving the convergence rate and exploration based on
a newly introduced backup operator and entropy regularization. We provide
strong theoretical guarantees to bound convergence rate, approximation error,
and regret of our methods. Moreover, we introduce a mathematical framework
based on the use of the $\alpha$-divergence for backup and exploration in MCTS.
We show that this theoretical formulation unifies different approaches,
including our newly introduced ones, under the same mathematical framework,
allowing to obtain different methods by simply changing the value of $\alpha$.
In practice, our unified perspective offers a flexible way to balance between
exploration and exploitation by tuning the single $\alpha$ parameter according
to the problem at hand. We validate our methods through a rigorous empirical
study from basic toy problems to the complex Atari games, and including both
MDP and POMDP problems.
