Multimodal Continuous Visual Attention Mechanisms
  Visual attention mechanisms are a key component of neural network models for
computer vision. By focusing on a discrete set of objects or image regions,
these mechanisms identify the most relevant features and use them to build more
powerful representations. Recently, continuous-domain alternatives to discrete
attention models have been proposed, which exploit the continuity of images.
These approaches model attention as simple unimodal densities (e.g. a
Gaussian), making them less suitable to deal with images whose region of
interest has a complex shape or is composed of multiple non-contiguous patches.
In this paper, we introduce a new continuous attention mechanism that produces
multimodal densities, in the form of mixtures of Gaussians. We use the EM
algorithm to obtain a clustering of relevant regions in the image, and a
description length penalty to select the number of components in the mixture.
Our densities decompose as a linear combination of unimodal attention
mechanisms, enabling closed-form Jacobians for the backpropagation step.
Experiments on visual question answering in the VQA-v2 dataset show competitive
accuracies and a selection of regions that mimics human attention more closely
in VQA-HAT. We present several examples that suggest how multimodal attention
maps are naturally more interpretable than their unimodal counterparts, showing
the ability of our model to automatically segregate objects from ground in
complex scenes.
