DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language
  Models
  Gigantic pre-trained models have become central to natural language
processing (NLP), serving as the starting point for fine-tuning towards a range
of downstream tasks. However, two pain points persist for this paradigm: (a) as
the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the
fine-tuning process can be time-consuming and computationally expensive; (b)
the fine-tuned model has the same size as its starting point by default, which
is neither sensible due to its more specialized functionality, nor practical
since many fine-tuned models will be deployed in resource-constrained
environments. To address these pain points, we propose a framework for
resource- and parameter-efficient fine-tuning by leveraging the sparsity prior
in both weight updates and the final model weights. Our proposed framework,
dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two
key objectives: (i) parameter efficient fine-tuning - by enforcing
sparsity-aware weight updates on top of the pre-trained weights; and (ii)
resource-efficient inference - by encouraging a sparse weight structure towards
the final fine-tuned model. We leverage sparsity in these two directions by
exploiting both unstructured and structured sparse patterns in pre-trained
language models via magnitude-based pruning and $\ell_1$ sparse regularization.
Extensive experiments and in-depth investigations, with diverse network
backbones (i.e., BERT, GPT-2, and DeBERTa) on dozens of datasets, consistently
demonstrate highly impressive parameter-/training-/inference-efficiency, while
maintaining competitive downstream transfer performance. For instance, our
DSEE-BERT obtains about $35\%$ inference FLOPs savings with <1% trainable
parameters and comparable performance to conventional fine-tuning. Codes are
available in https://github.com/VITA-Group/DSEE.
