Is Attention Better Than Matrix Decomposition?
  As an essential ingredient of modern deep learning, attention mechanism,
especially self-attention, plays a vital role in the global correlation
discovery. However, is hand-crafted attention irreplaceable when modeling the
global context? Our intriguing finding is that self-attention is not better
than the matrix decomposition (MD) model developed 20 years ago regarding the
performance and computational cost for encoding the long-distance dependencies.
We model the global context issue as a low-rank recovery problem and show that
its optimization algorithms can help design global information blocks. This
paper then proposes a series of Hamburgers, in which we employ the optimization
algorithms for solving MDs to factorize the input representations into
sub-matrices and reconstruct a low-rank embedding. Hamburgers with different
MDs can perform favorably against the popular global context module
self-attention when carefully coping with gradients back-propagated through
MDs. Comprehensive experiments are conducted in the vision tasks where it is
crucial to learn the global context, including semantic segmentation and image
generation, demonstrating significant improvements over self-attention and its
variants.
