Neural Algorithmic Reasoners are Implicit Planners
  Implicit planning has emerged as an elegant technique for combining learned
models of the world with end-to-end model-free reinforcement learning. We study
the class of implicit planners inspired by value iteration, an algorithm that
is guaranteed to yield perfect policies in fully-specified tabular
environments. We find that prior approaches either assume that the environment
is provided in such a tabular form -- which is highly restrictive -- or infer
"local neighbourhoods" of states to run value iteration over -- for which we
discover an algorithmic bottleneck effect. This effect is caused by explicitly
running the planning algorithm based on scalar predictions in every state,
which can be harmful to data efficiency if such scalars are improperly
predicted. We propose eXecuted Latent Value Iteration Networks (XLVINs), which
alleviate the above limitations. Our method performs all planning computations
in a high-dimensional latent space, breaking the algorithmic bottleneck. It
maintains alignment with value iteration by carefully leveraging neural
graph-algorithmic reasoning and contrastive self-supervised learning. Across
eight low-data settings -- including classical control, navigation and Atari --
XLVINs provide significant improvements to data efficiency against value
iteration-based implicit planners, as well as relevant model-free baselines.
Lastly, we empirically verify that XLVINs can closely align with value
iteration.
