A Biologically Inspired Visual Working Memory for Deep Networks
  The ability to look multiple times through a series of pose-adjusted glimpses
is fundamental to human vision. This critical faculty allows us to understand
highly complex visual scenes. Short term memory plays an integral role in
aggregating the information obtained from these glimpses and informing our
interpretation of the scene. Computational models have attempted to address
glimpsing and visual attention but have failed to incorporate the notion of
memory. We introduce a novel, biologically inspired visual working memory
architecture that we term the Hebb-Rosenblatt memory. We subsequently introduce
a fully differentiable Short Term Attentive Working Memory model (STAWM) which
uses transformational attention to learn a memory over each image it sees. The
state of our Hebb-Rosenblatt memory is embedded in STAWM as the weights space
of a layer. By projecting different queries through this layer we can obtain
goal-oriented latent representations for tasks including classification and
visual reconstruction. Our model obtains highly competitive classification
performance on MNIST and CIFAR-10. As demonstrated through the CelebA dataset,
to perform reconstruction the model learns to make a sequence of updates to a
canvas which constitute a parts-based representation. Classification with the
self supervised representation obtained from MNIST is shown to be in line with
the state of the art models (none of which use a visual attention mechanism).
Finally, we show that STAWM can be trained under the dual constraints of
classification and reconstruction to provide an interpretable visual sketchpad
which helps open the 'black-box' of deep learning.
