Visual Attention Methods in Deep Learning: An In-Depth Survey
  Inspired by the human cognitive system, attention is a mechanism that
imitates the human cognitive awareness about specific information, amplifying
critical details to focus more on the essential aspects of data. Deep learning
has employed attention to boost performance for many applications.
Interestingly, the same attention design can suit processing different data
modalities and can easily be incorporated into large networks. Furthermore,
multiple complementary attention mechanisms can be incorporated in one network.
Hence, attention techniques have become extremely attractive. However, the
literature lacks a comprehensive survey specific to attention techniques to
guide researchers in employing attention in their deep models. Note that,
besides being demanding in terms of training data and computational resources,
transformers only cover a single category in self-attention out of the many
categories available. We fill this gap and provide an in-depth survey of 50
attention techniques categorizing them by their most prominent features. We
initiate our discussion by introducing the fundamental concepts behind the
success of attention mechanism. Next, we furnish some essentials such as the
strengths and limitations of each attention category, describe their
fundamental building blocks, basic formulations with primary usage, and
applications specifically for computer vision. We also discuss the challenges
and open questions related to attention mechanism in general. Finally, we
recommend possible future research directions for deep attention.
