ConViT: Improving Vision Transformers with Soft Convolutional Inductive
  Biases
  Convolutional architectures have proven extremely successful for vision
tasks. Their hard inductive biases enable sample-efficient learning, but come
at the cost of a potentially lower performance ceiling. Vision Transformers
(ViTs) rely on more flexible self-attention layers, and have recently
outperformed CNNs for image classification. However, they require costly
pre-training on large external datasets or distillation from pre-trained
convolutional networks. In this paper, we ask the following question: is it
possible to combine the strengths of these two architectures while avoiding
their respective limitations? To this end, we introduce gated positional
self-attention (GPSA), a form of positional self-attention which can be
equipped with a ``soft" convolutional inductive bias. We initialise the GPSA
layers to mimic the locality of convolutional layers, then give each attention
head the freedom to escape locality by adjusting a gating parameter regulating
the attention paid to position versus content information. The resulting
convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet,
while offering a much improved sample efficiency. We further investigate the
role of locality in learning by first quantifying how it is encouraged in
vanilla self-attention layers, then analysing how it is escaped in GPSA layers.
We conclude by presenting various ablations to better understand the success of
the ConViT. Our code and models are released publicly at
https://github.com/facebookresearch/convit.
