Is Policy Learning Overrated?: Width-Based Planning and Active Learning
  for Atari
  Width-based planning has shown promising results on Atari 2600 games using
pixel input, while using substantially fewer environment interactions than
reinforcement learning. Recent width-based approaches have computed feature
vectors for each screen using a hand designed feature set or a variational
autoencoder trained on game screens (VAE-IW), and prune screens that do not
have novel features during the search. We propose Olive (Online-VAE-IW), which
updates the VAE features online using active learning to maximize the utility
of screens observed during planning. Experimental results in 55 Atari games
demonstrate that it outperforms Rollout-IW by 42-to-11 and VAE-IW by 32-to-20.
Moreover, Olive outperforms existing work based on policy-learning ($\pi$-IW,
DQN) trained with 100x training budget by 30-to-22 and 31-to-17, and a state of
the art data-efficient reinforcement learning (EfficientZero) trained with the
same training budget and ran with 1.8x planning budget by 18-to-7 in Atari 100k
benchmark, with no policy learning at all. The source code is available at
github.com/ibm/atari-active-learning .
