Combining Q-Learning and Search with Amortized Value Estimates
  We introduce "Search with Amortized Value Estimates" (SAVE), an approach for
combining model-free Q-learning with model-based Monte-Carlo Tree Search
(MCTS). In SAVE, a learned prior over state-action values is used to guide
MCTS, which estimates an improved set of state-action values. The new
Q-estimates are then used in combination with real experience to update the
prior. This effectively amortizes the value computation performed by MCTS,
resulting in a cooperative relationship between model-free learning and
model-based search. SAVE can be implemented on top of any Q-learning agent with
access to a model, which we demonstrate by incorporating it into agents that
perform challenging physical reasoning tasks and Atari. SAVE consistently
achieves higher rewards with fewer training steps, and---in contrast to typical
model-based search approaches---yields strong performance with very small
search budgets. By combining real experience with information computed during
search, SAVE demonstrates that it is possible to improve on both the
performance of model-free learning and the computational cost of planning.
