Attention in Natural Language Processing
  Attention is an increasingly popular mechanism used in a wide range of neural
architectures. The mechanism itself has been realized in a variety of formats.
However, because of the fast-paced advances in this domain, a systematic
overview of attention is still missing. In this article, we define a unified
model for attention architectures in natural language processing, with a focus
on those designed to work with vector representations of the textual data. We
propose a taxonomy of attention models according to four dimensions: the
representation of the input, the compatibility function, the distribution
function, and the multiplicity of the input and/or output. We present the
examples of how prior information can be exploited in attention models and
discuss ongoing research efforts and open challenges in the area, providing the
first extensive categorization of the vast body of literature in this exciting
domain.
