Variational Memory Addressing in Generative Models
  Aiming to augment generative models with external memory, we interpret the
output of a memory module with stochastic addressing as a conditional mixture
distribution, where a read operation corresponds to sampling a discrete memory
address and retrieving the corresponding content from memory. This perspective
allows us to apply variational inference to memory addressing, which enables
effective training of the memory module by using the target information to
guide memory lookups. Stochastic addressing is particularly well-suited for
generative models as it naturally encourages multimodality which is a prominent
aspect of most high-dimensional datasets. Treating the chosen address as a
latent variable also allows us to quantify the amount of information gained
with a memory lookup and measure the contribution of the memory module to the
generative process. To illustrate the advantages of this approach we
incorporate it into a variational autoencoder and apply the resulting model to
the task of generative few-shot learning. The intuition behind this
architecture is that the memory module can pick a relevant template from memory
and the continuous part of the model can concentrate on modeling remaining
variations. We demonstrate empirically that our model is able to identify and
access the relevant memory contents even with hundreds of unseen Omniglot
characters in memory
