DynaBERT: Dynamic BERT with Adaptive Width and Depth
  The pre-trained language models like BERT, though powerful in many natural
language processing tasks, are both computation and memory expensive. To
alleviate this problem, one approach is to compress them for specific tasks
before deployment. However, recent works on BERT compression usually compress
the large BERT model to a fixed smaller size. They can not fully satisfy the
requirements of different edge devices with various hardware performances. In
this paper, we propose a novel dynamic BERT model (abbreviated as DynaBERT),
which can flexibly adjust the size and latency by selecting adaptive width and
depth. The training process of DynaBERT includes first training a
width-adaptive BERT and then allowing both adaptive width and depth, by
distilling knowledge from the full-sized model to small sub-networks. Network
rewiring is also used to keep the more important attention heads and neurons
shared by more sub-networks. Comprehensive experiments under various efficiency
constraints demonstrate that our proposed dynamic BERT (or RoBERTa) at its
largest size has comparable performance as BERT-base (or RoBERTa-base), while
at smaller widths and depths consistently outperforms existing BERT compression
methods. Code is available at
https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/DynaBERT.
