Differentiable Random Access Memory using Lattices
  We introduce a differentiable random access memory module with $O(1)$
performance regardless of size, scaling to billions of entries. The design
stores entries on points of a chosen lattice to calculate nearest neighbours of
arbitrary points efficiently by exploiting symmetries. Augmenting a standard
neural network architecture with a single memory layer based on this, we can
scale the parameter count up to memory limits with negligible computational
overhead, giving better accuracy at similar cost. On large language modelling
tasks, these enhanced models with larger capacity significantly outperform the
unmodified transformer baseline. We found continued scaling with memory size up
to the limits tested.
