Automated Progressive Learning for Efficient Training of Vision
  Transformers
  Recent advances in vision Transformers (ViTs) have come with a voracious
appetite for computing power, high-lighting the urgent need to develop
efficient training methods for ViTs. Progressive learning, a training scheme
where the model capacity grows progressively during training, has started
showing its ability in efficient training. In this paper, we take a practical
step towards efficient training of ViTs by customizing and automating
progressive learning. First, we develop a strong manual baseline for
progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge
the gap brought by model growth. Then, we propose automated progressive
learning (AutoProg), an efficient training scheme that aims to achieve lossless
acceleration by automatically increasing the training overload on-the-fly; this
is achieved by adaptively deciding whether, where and how much should the model
grow during progressive learning. Specifically, we first relax the optimization
of the growth schedule to sub-network architecture optimization problem, then
propose one-shot estimation of the sub-network performance via an elastic
supernet. The searching overhead is reduced to minimal by recycling the
parameters of the supernet. Extensive experiments of efficient training on
ImageNet with two representative ViT models, DeiT and VOLO, demonstrate that
AutoProg can accelerate ViTs training by up to 85.1% with no performance drop.
Code: https://github.com/changlin31/AutoProg
