Toward Transformer-Based Object Detection
  Transformers have become the dominant model in natural language processing,
owing to their ability to pretrain on massive amounts of data, then transfer to
smaller, more specific tasks via fine-tuning. The Vision Transformer was the
first major attempt to apply a pure transformer model directly to images as
input, demonstrating that as compared to convolutional networks,
transformer-based architectures can achieve competitive results on benchmark
classification tasks. However, the computational complexity of the attention
operator means that we are limited to low-resolution inputs. For more complex
tasks such as detection or segmentation, maintaining a high input resolution is
crucial to ensure that models can properly identify and reflect fine details in
their output. This naturally raises the question of whether or not
transformer-based architectures such as the Vision Transformer are capable of
performing tasks other than classification. In this paper, we determine that
Vision Transformers can be used as a backbone by a common detection task head
to produce competitive COCO results. The model that we propose, ViT-FRCNN,
demonstrates several known properties associated with transformers, including
large pretraining capacity and fast fine-tuning performance. We also
investigate improvements over a standard detection backbone, including superior
performance on out-of-domain images, better performance on large objects, and a
lessened reliance on non-maximum suppression. We view ViT-FRCNN as an important
stepping stone toward a pure-transformer solution of complex vision tasks such
as object detection.
