Repulsive Attention: Rethinking Multi-head Attention as Bayesian
  Inference
  The neural attention mechanism plays an important role in many natural
language processing applications. In particular, the use of multi-head
attention extends single-head attention by allowing a model to jointly attend
information from different perspectives. Without explicit constraining,
however, multi-head attention may suffer from attention collapse, an issue that
makes different heads extract similar attentive features, thus limiting the
model's representation power. In this paper, for the first time, we provide a
novel understanding of multi-head attention from a Bayesian perspective. Based
on the recently developed particle-optimization sampling techniques, we propose
a non-parametric approach that explicitly improves the repulsiveness in
multi-head attention and consequently strengthens model's expressiveness.
Remarkably, our Bayesian interpretation provides theoretical inspirations on
the not-well-understood questions: why and how one uses multi-head attention.
Extensive experiments on various attention models and applications demonstrate
that the proposed repulsive attention can improve the learned feature
diversity, leading to more informative representations with consistent
performance improvement on various tasks.
