Untangling tradeoffs between recurrence and self-attention in neural
  networks
  Attention and self-attention mechanisms, are now central to state-of-the-art
deep learning on sequential tasks. However, most recent progress hinges on
heuristic approaches with limited understanding of attention's role in model
optimization and computation, and rely on considerable memory and computational
resources that scale poorly. In this work, we present a formal analysis of how
self-attention affects gradient propagation in recurrent networks, and prove
that it mitigates the problem of vanishing gradients when trying to capture
long-term dependencies by establishing concrete bounds for gradient norms.
Building on these results, we propose a relevancy screening mechanism, inspired
by the cognitive process of memory consolidation, that allows for a scalable
use of sparse self-attention with recurrence. While providing guarantees to
avoid vanishing gradients, we use simple numerical experiments to demonstrate
the tradeoffs in performance and computational resources by efficiently
balancing attention and recurrence. Based on our results, we propose a concrete
direction of research to improve scalability of attentive networks.
