Lie-Access Neural Turing Machines
  External neural memory structures have recently become a popular tool for
algorithmic deep learning (Graves et al. 2014, Weston et al. 2014). These
models generally utilize differentiable versions of traditional discrete
memory-access structures (random access, stacks, tapes) to provide the storage
necessary for computational tasks. In this work, we argue that these neural
memory systems lack specific structure important for relative indexing, and
propose an alternative model, Lie-access memory, that is explicitly designed
for the neural setting. In this paradigm, memory is accessed using a continuous
head in a key-space manifold. The head is moved via Lie group actions, such as
shifts or rotations, generated by a controller, and memory access is performed
by linear smoothing in key space. We argue that Lie groups provide a natural
generalization of discrete memory structures, such as Turing machines, as they
provide inverse and identity operators while maintaining differentiability. To
experiment with this approach, we implement a simplified Lie-access neural
Turing machine (LANTM) with different Lie groups. We find that this approach is
able to perform well on a range of algorithmic tasks.
