Saccader: Improving Accuracy of Hard Attention Models for Vision
  Although deep convolutional neural networks achieve state-of-the-art
performance across nearly all image classification tasks, their decisions are
difficult to interpret. One approach that offers some level of interpretability
by design is \textit{hard attention}, which uses only relevant portions of the
image. However, training hard attention models with only class label
supervision is challenging, and hard attention has proved difficult to scale to
complex datasets. Here, we propose a novel hard attention model, which we term
Saccader. Key to Saccader is a pretraining step that requires only class labels
and provides initial attention locations for policy gradient optimization. Our
best models narrow the gap to common ImageNet baselines, achieving $75\%$ top-1
and $91\%$ top-5 while attending to less than one-third of the image.
