O-ViT: Orthogonal Vision Transformer
  Inspired by the tremendous success of the self-attention mechanism in natural
language processing, the Vision Transformer (ViT) creatively applies it to
image patch sequences and achieves incredible performance. However, the scaled
dot-product self-attention of ViT brings about scale ambiguity to the structure
of the original feature space. To address this problem, we propose a novel
method named Orthogonal Vision Transformer (O-ViT), to optimize ViT from the
geometric perspective. O-ViT limits parameters of self-attention blocks to be
on the norm-keeping orthogonal manifold, which can keep the geometry of the
feature space. Moreover, O-ViT achieves both orthogonal constraints and cheap
optimization overhead by adopting a surjective mapping between the orthogonal
group and its Lie algebra.We have conducted comparative experiments on image
recognition tasks to demonstrate O-ViT's validity and experiments show that
O-ViT can boost the performance of ViT by up to 3.6%.
