Bayesian Attention Modules
  Attention modules, as simple and effective tools, have not only enabled deep
neural networks to achieve state-of-the-art results in many domains, but also
enhanced their interpretability. Most current models use deterministic
attention modules due to their simplicity and ease of optimization. Stochastic
counterparts, on the other hand, are less popular despite their potential
benefits. The main reason is that stochastic attention often introduces
optimization issues or requires significant model changes. In this paper, we
propose a scalable stochastic version of attention that is easy to implement
and optimize. We construct simplex-constrained attention distributions by
normalizing reparameterizable distributions, making the training process
differentiable. We learn their parameters in a Bayesian framework where a
data-dependent prior is introduced for regularization. We apply the proposed
stochastic attention modules to various attention-based models, with
applications to graph node classification, visual question answering, image
captioning, machine translation, and language understanding. Our experiments
show the proposed method brings consistent improvements over the corresponding
baselines.
