Visual Attention Emerges from Recurrent Sparse Reconstruction
  Visual attention helps achieve robust perception under noise, corruption, and
distribution shifts in human vision, which are areas where modern neural
networks still fall short. We present VARS, Visual Attention from Recurrent
Sparse reconstruction, a new attention formulation built on two prominent
features of the human visual attention mechanism: recurrency and sparsity.
Related features are grouped together via recurrent connections between
neurons, with salient objects emerging via sparse regularization. VARS adopts
an attractor network with recurrent connections that converges toward a stable
pattern over time. Network layers are represented as ordinary differential
equations (ODEs), formulating attention as a recurrent attractor network that
equivalently optimizes the sparse reconstruction of input using a dictionary of
"templates" encoding underlying patterns of data. We show that self-attention
is a special case of VARS with a single-step optimization and no sparsity
constraint. VARS can be readily used as a replacement for self-attention in
popular vision transformers, consistently improving their robustness across
various benchmarks. Code is released on GitHub (https://github.com/bfshi/VARS).
