Surprising Negative Results for Generative Adversarial Tree Search
  While many recent advances in deep reinforcement learning (RL) rely on
model-free methods, model-based approaches remain an alluring prospect for
their potential to exploit unsupervised data to learn environment model. In
this work, we provide an extensive study on the design of deep generative
models for RL environments and propose a sample efficient and robust method to
learn the model of Atari environments. We deploy this model and propose
generative adversarial tree search (GATS) a deep RL algorithm that learns the
environment model and implements Monte Carlo tree search (MCTS) on the learned
model for planning. While MCTS on the learned model is computationally
expensive, similar to AlphaGo, GATS follows depth limited MCTS. GATS employs
deep Q network (DQN) and learns a Q-function to assign values to the leaves of
the tree in MCTS. We theoretical analyze GATS vis-a-vis the bias-variance
trade-off and show GATS is able to mitigate the worst-case error in the
Q-estimate. While we were expecting GATS to enjoy a better sample complexity
and faster converges to better policies, surprisingly, GATS fails to outperform
DQN. We provide a study on which we show why depth limited MCTS fails to
perform desirably.
