SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive
  Connection
  While the self-attention mechanism has been widely used in a wide variety of
tasks, it has the unfortunate property of a quadratic cost with respect to the
input length, which makes it difficult to deal with long inputs. In this paper,
we present a method for accelerating and structuring self-attentions: Sparse
Adaptive Connection (SAC). In SAC, we regard the input sequence as a graph and
attention operations are performed between linked nodes. In contrast with
previous self-attention models with pre-defined structures (edges), the model
learns to construct attention edges to improve task-specific performances. In
this way, the model is able to select the most salient nodes and reduce the
quadratic complexity regardless of the sequence length. Based on SAC, we show
that previous variants of self-attention models are its special cases. Through
extensive experiments on neural machine translation, language modeling, graph
representation learning and image classification, we demonstrate SAC is
competitive with state-of-the-art models while significantly reducing memory
cost.
