Memory Augmented Optimizers for Deep Learning
  Popular approaches for minimizing loss in data-driven learning often involve
an abstraction or an explicit retention of the history of gradients for
efficient parameter updates. The aggregated history of gradients nudges the
parameter updates in the right direction even when the gradients at any given
step are not informative. Although the history of gradients summarized in
meta-parameters or explicitly stored in memory has been shown effective in
theory and practice, the question of whether $all$ or only a subset of the
gradients in the history are sufficient in deciding the parameter updates
remains unanswered. In this paper, we propose a framework of memory-augmented
gradient descent optimizers that retain a limited view of their gradient
history in their internal memory. Such optimizers scale well to large real-life
datasets, and our experiments show that the memory augmented extensions of
standard optimizers enjoy accelerated convergence and improved performance on a
majority of computer vision and language tasks that we considered.
Additionally, we prove that the proposed class of optimizers with fixed-size
memory converge under assumptions of strong convexity, regardless of which
gradients are selected or how they are linearly combined to form the update
step.
