An Empirical Study Of Self-supervised Learning Approaches For Object
  Detection With Transformers
  Self-supervised learning (SSL) methods such as masked language modeling have
shown massive performance gains by pretraining transformer models for a variety
of natural language processing tasks. The follow-up research adapted similar
methods like masked image modeling in vision transformer and demonstrated
improvements in the image classification task. Such simple self-supervised
methods are not exhaustively studied for object detection transformers (DETR,
Deformable DETR) as their transformer encoder modules take input in the
convolutional neural network (CNN) extracted feature space rather than the
image space as in general vision transformers. However, the CNN feature maps
still maintain the spatial relationship and we utilize this property to design
self-supervised learning approaches to train the encoder of object detection
transformers in pretraining and multi-task learning settings. We explore common
self-supervised methods based on image reconstruction, masked image modeling
and jigsaw. Preliminary experiments in the iSAID dataset demonstrate faster
convergence of DETR in the initial epochs in both pretraining and multi-task
learning settings; nonetheless, similar improvement is not observed in the case
of multi-task learning with Deformable DETR. The code for our experiments with
DETR and Deformable DETR are available at https://github.com/gokulkarthik/detr
and https://github.com/gokulkarthik/Deformable-DETR respectively.
