Rethinking Supervised Pre-training for Better Downstream Transferring
  The pretrain-finetune paradigm has shown outstanding performance on many
applications of deep learning, where a model is pre-trained on a upstream large
dataset (e.g. ImageNet), and is then fine-tuned to different downstream tasks.
Though for most cases, the pre-training stage is conducted based on supervised
methods, recent works on self-supervised pre-training have shown powerful
transferability and even outperform supervised pre-training on multiple
downstream tasks. It thus remains an open question how to better generalize
supervised pre-training model to downstream tasks. In this paper, we argue that
the worse transferability of existing supervised pre-training methods arise
from the negligence of valuable intra-class semantic difference. This is
because these methods tend to push images from the same class close to each
other despite of the large diversity in their visual contents, a problem to
which referred as "overfit of upstream tasks". To alleviate this problem, we
propose a new supervised pre-training method based on Leave-One-Out
K-Nearest-Neighbor, or LOOK for short. It relieves the problem of overfitting
upstream tasks by only requiring each image to share its class label with most
of its k nearest neighbors, thus allowing each class to exhibit a multi-mode
distribution and consequentially preserving part of intra-class difference for
better transferring to downstream tasks. We developed efficient implementation
of the proposed method that scales well to large datasets. Experimental studies
on multiple downstream tasks show that LOOK outperforms other state-of-the-art
methods for supervised and self-supervised pre-training.
