Solving ImageNet: a Unified Scheme for Training any Backbone to Top
  Results
  ImageNet serves as the primary dataset for evaluating the quality of
computer-vision models. The common practice today is training each architecture
with a tailor-made scheme, designed and tuned by an expert. In this paper, we
present a unified scheme for training any backbone on ImageNet. The scheme,
named USI (Unified Scheme for ImageNet), is based on knowledge distillation and
modern tricks. It requires no adjustments or hyper-parameters tuning between
different models, and is efficient in terms of training times. We test USI on a
wide variety of architectures, including CNNs, Transformers, Mobile-oriented
and MLP-only. On all models tested, USI outperforms previous state-of-the-art
results. Hence, we are able to transform training on ImageNet from an
expert-oriented task to an automatic seamless routine. Since USI accepts any
backbone and trains it to top results, it also enables to perform methodical
comparisons, and identify the most efficient backbones along the speed-accuracy
Pareto curve. Implementation is available
at:https://github.com/Alibaba-MIIL/Solving_ImageNet
