Attention, please! A survey of Neural Attention Models in Deep Learning
  In humans, Attention is a core property of all perceptual and cognitive
operations. Given our limited ability to process competing sources, attention
mechanisms select, modulate, and focus on the information most relevant to
behavior. For decades, concepts and functions of attention have been studied in
philosophy, psychology, neuroscience, and computing. For the last six years,
this property has been widely explored in deep neural networks. Currently, the
state-of-the-art in Deep Learning is represented by neural attention models in
several application domains. This survey provides a comprehensive overview and
analysis of developments in neural attention models. We systematically reviewed
hundreds of architectures in the area, identifying and discussing those in
which attention has shown a significant impact. We also developed and made
public an automated methodology to facilitate the development of reviews in the
area. By critically analyzing 650 works, we describe the primary uses of
attention in convolutional, recurrent networks and generative models,
identifying common subgroups of uses and applications. Furthermore, we describe
the impact of attention in different application domains and their impact on
neural networks' interpretability. Finally, we list possible trends and
opportunities for further research, hoping that this review will provide a
succinct overview of the main attentional models in the area and guide
researchers in developing future approaches that will drive further
improvements.
