Efficient Transformer-based Large Scale Language Representations using
  Hardware-friendly Block Structured Pruning
  Pre-trained large-scale language models have increasingly demonstrated high
accuracy on many natural language processing (NLP) tasks. However, the limited
weight storage and computational speed on hardware platforms have impeded the
popularity of pre-trained models, especially in the era of edge computing. In
this work, we propose an efficient transformer-based large-scale language
representation using hardware-friendly block structure pruning. We incorporate
the reweighted group Lasso into block-structured pruning for optimization.
Besides the significantly reduced weight storage and computation, the proposed
approach achieves high compression rates. Experimental results on different
models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding
Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0x with zero or
minor accuracy degradation on certain task(s). Our proposed method is also
orthogonal to existing compact pre-trained language models such as DistilBERT
using knowledge distillation, since a further 1.79x average compression rate
can be achieved on top of DistilBERT with zero or minor accuracy degradation.
It is suitable to deploy the final compressed model on resource-constrained
edge devices.
