Neural Attention Models in Deep Learning: Survey and Taxonomy
  Attention is a state of arousal capable of dealing with limited processing
bottlenecks in human beings by focusing selectively on one piece of information
while ignoring other perceptible information. For decades, concepts and
functions of attention have been studied in philosophy, psychology,
neuroscience, and computing. Currently, this property has been widely explored
in deep neural networks. Many different neural attention models are now
available and have been a very active research area over the past six years.
From the theoretical standpoint of attention, this survey provides a critical
analysis of major neural attention models. Here we propose a taxonomy that
corroborates with theoretical aspects that predate Deep Learning. Our taxonomy
provides an organizational structure that asks new questions and structures the
understanding of existing attentional mechanisms. In particular, 17 criteria
derived from psychology and neuroscience classic studies are formulated for
qualitative comparison and critical analysis on the 51 main models found on a
set of more than 650 papers analyzed. Also, we highlight several theoretical
issues that have not yet been explored, including discussions about biological
plausibility, highlight current research trends, and provide insights for the
future.
