Vision Transformer Slimming: Multi-Dimension Searching in Continuous
  Optimization Space
  This paper explores the feasibility of finding an optimal sub-model from a
vision transformer and introduces a pure vision transformer slimming (ViT-Slim)
framework. It can search a sub-structure from the original model end-to-end
across multiple dimensions, including the input tokens, MHSA and MLP modules
with state-of-the-art performance. Our method is based on a learnable and
unified $\ell_1$ sparsity constraint with pre-defined factors to reflect the
global importance in the continuous searching space of different dimensions.
The searching process is highly efficient through a single-shot training
scheme. For instance, on DeiT-S, ViT-Slim only takes ~43 GPU hours for the
searching process, and the searched structure is flexible with diverse
dimensionalities in different modules. Then, a budget threshold is employed
according to the requirements of accuracy-FLOPs trade-off on running devices,
and a re-training process is performed to obtain the final model. The extensive
experiments show that our ViT-Slim can compress up to 40% of parameters and 40%
FLOPs on various vision transformers while increasing the accuracy by ~0.6% on
ImageNet. We also demonstrate the advantage of our searched models on several
downstream datasets. Our code is available at
https://github.com/Arnav0400/ViT-Slim.
