Self-Attentive Associative Memory
  Heretofore, neural networks with external memory are restricted to single
memory with lossy representations of memory interactions. A rich representation
of relationships between memory pieces urges a high-order and segregated
relational memory. In this paper, we propose to separate the storage of
individual experiences (item memory) and their occurring relationships
(relational memory). The idea is implemented through a novel Self-attentive
Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of
associative memories that represent the hypothetical high-order relationships
between arbitrary pairs of memory elements, through which a relational memory
is constructed from an item memory. The two memories are wired into a single
sequential model capable of both memorization and relational reasoning. We
achieve competitive results with our proposed two-memory model in a diversity
of machine learning tasks, from challenging synthetic problems to practical
testbeds such as geometry, graph, reinforcement learning, and question
answering.
