{'fraction_non_alphanumeric': 0.03111540827986288, 'fraction_numerical': 0.0129208051331634, 'mean_word_length': 5.207310420076378, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fast DistilBERT on CPUs

Transformer-based language models have become the standard approach to solving natural language processing tasks. However, industry adoption usually requires the maximum throughput to comply with certain latency constraints that prevents Transformer models from being used in production. To address this gap, model compression techniques such as quantization and pruning may be used to improve inference efficiency. However, these compression techniques require specialized software to apply and deploy at scale. In this work, we propose a new pipeline for creating and running Fast Transformer models on CPUs, utilizing hardware-aware pruning, knowledge distillation, quantization, and our own Transformer inference runtime engine with optimized kernels for sparse and quantized operators. We demonstrate the efficiency of our pipeline by creating a Fast DistilBERT model showing minimal accuracy loss on the question-answering SQuADv1.1 benchmark, and throughput results under typical production constraints and environments. Our results outperform existing state-of-the-art Neural Magic's DeepSparse runtime performance by up to 50% and up to 4.1x performance speedup over ONNX Runtime. Source code is publicly available at https://github.com/intel/intel-extension-for-transformers.


Introduction and related work
Large Transformer-based Language Models (LMs) are evolving rapidly from millions of parameters, e.g., BERT-Large [3], to billions of parameters, e.g., Turing-Megatron [14], and GPT3 [2]. Those large models have demonstrated promising state-of-the-art (SoTA) accuracy on a wide range of NLP tasks. However, those models are inefficient and therefore unsuited to the limited computational sources and strict latency constraints in production.
To enable and increase the efficiency and scale of deployed Transformer models, additional model compression and optimization are usually required along with a dedicated inference engine. For example, DistilBERT [12], using knowledge distillation [4], shows minimal impact on downstream NLP tasks at a reduced size by 40%. Additionally, pruning [13] and quantization [17] are wellknown techniques to further compress a Transformer model. Recent works have proposed pruning Transformer models at pre-training to create sparse pre-trained LMs. It has been established that fine-tuning these sparse pre-trained LMs to downstream tasks can produce results that are competitive with other pruning methods, obviating the need for pruning. [18,7]. For example, a DistilBERT with 90% unstructured sparsity was produced with minimal impact on the accuracy of downstream tasks [18]; however, performance gains have not been demonstrated for this model. Similarly, Neural Magic published the throughput performance of a structured sparse DistilBERT with 80% sparsity, batch size 32 and sequence length 128, although latency performance was not provided [9]. Nvidia recently proposed novel 2:4 structured sparsity that is only available on Ampere architecture and above [10]. Quantization [6] [15] has been maturing in industry with two well-known approaches: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Typically, PTQ requires an offline calibration process on a representative calibration dataset, while QAT requires an additional fine-tuning stage simulating quantization inference while training. Although the approaches have proved successful in some typical models [16,17], it remains challenging to produce a quantized model based on a highly sparse model without sacrificing accuracy.
In this paper, we propose a new pipeline for creating and running Fast Transformer models on CPUs. We extend the model compression approach PruneOFA [18] by enabling CPU-friendly block-wise structured sparsity. We then apply an accuracy-aware post-training quantization approach to generate an 8-bit (INT8) sparse model. Furthermore, we demonstrate our models using our Transformer inference engine dedicated for running sparse & quantized Transformers on CPUs. We apply this pipeline on DistilBERT [12] to create Fast DistilBERT. We show that Fast DistilBERT can meet a requirement of under 1% accuracy loss vs. the DistilBERT baseline on the question-answering benchmark SQuADv1.1 [11]. We demonstrate up to 1.5x performance gain over Neural Magic's DeepSparse engine [8] and up to 4.1x performance gain over ONNX Runtime on common CPUs from Amazon Web Services (AWS) under typical production constraints.
Our main contributions are threefold: 1) Propose a hardware-aware extreme compression technique for fast Transformer models on CPUs. 2) Create an efficient Transformer inference runtime for sparse & quantized Transformer models. 3) Demonstrate new SOTA performance under typical constraints in common production environments.

Method description
In this section, we describe how to solve the challenges of applying model compression techniques on Transformer-based LMs.
Block-wise structured sparsity In this work, we accelerate sparse Transformer-based models with specialized sparse GEMM operators which require a structured sparsity of constant size blocks in the output dimension, see Section 3. To that end, we extend the model compression infrastructure proposed by Zafrir et al. [18] to create sparse pre-trained LMs with block-wise structured sparsity. Further details on block-wise pruning in Appendix A.1.
Fine-tuning with knowledge distillation Knowledge distillation is a widely used method for model compression [4,12]. Knowledge distillation while fine-tuning Transformer-based models may improve the accuracy of the model on the downstream task and bridge the accuracy gap caused by compression methods applied to the model [13,18,7]. Following these works we apply knowledge distillation while fine-tuning the block-wise sparse pre-trained LM to downstream tasks to mitigate the accuracy loss.
Post-training quantization PTQ is an effective approach to quantizing a model without additional training steps. It requires a calibration process using a representative dataset to determine the quantization parameters of the model. Converting FP32 operations to INT8 can increase their efficiency by up to 4x [1]. We apply PTQ with automatic accuracy-aware tuning on the sparse Transformer model to produce an optimized model using Intel ® Neural Compressor (INC) opensource tool for quantization [5].

Software acceleration
We develop a Transformer inference engine on CPU with advanced runtime, graph optimization, and sparse GEMM operators.
Advanced runtime We develop an advanced, cache friendly, memory allocator to enable buffer reuse and to reduce memory allocation overhead, in particular for a continuous demand to allocate/free small memory chunks. Moreover, we implement weight sharing that allows a single copy of weight to be shared across multiple instances running in parallel during inference. More details in  Sparse GEMM operators We implement INT8 sparse GEMM kernel to accelerate operations between dense input and sparse weights leveraging AVX512-VNNI instructions. Figure 1 illustrates how the kernel operates. Given a block-wise sparse weight, we broadcast the non-zero weight block to form a VNNI-format block A. Based on the mask in the sparse weight, we re-organize the corresponding input as another VNNI-format block B. Then, the kernel uses VNNI to produce the intermediate output given A and B, and add bias C as the final output. Algorithm 1 in the appendix contains the implementation of the kernel.

Experimental setup
To demonstrate the performance of the runtime optimizations described in Section 3, we produce a block-wise sparse pre-trained DistilBERT, and then fine-tune and quantize it for the questionanswering SQuADv1.1 benchmark [11] using the methods described in Section 2. Appendix A.2 details our model creation process. Table 1 lists the F1 scores of the DistilBERT baseline and our compressed model. To evaluate the performance of our optimizations, we evaluate our compressed model performance on a AWS c6i.12xlarge CPU instance for 3 scenarios: maximum ThroughPut (TP), minimum latency, and production. The production scenario requires maximum TP under 10 milliseconds (ms) latency per batch. Moreover, we compare our performance to other public inference runtimes for quantized and sparse models, ONNX Runtime and Neural Magic. We include the F1 results for their corresponding models in Table 1.

Results
All the results we present are an average over several measurements after several warm-up iterations. Figure 2 presents the relative performance of DistilBERT compressed models for the production scenario. Table 4 in Appendix A.4 shows the absolute performance accordingly. Our solution shows a consistent performance gain of 3.6x-4.1x over ONNX Runtime and outperforms Neural Magic by up to 50% for a range of sequence lengths.   6 Summary and future work In this paper, we presented a combined hardware-aware model compression technique (block-wise structured sparsity, knowledge distillation, and quantization) and demonstrated it with DistilBERT.
We created a new dedicated inference engine which unlocks the performance of extremely compressed Transformer-based LMs on CPUs. Our results outperform Neural Magic's inference solution by up to 50% and demonstrate up to 4.1x better performance than ONNX Runtime under production constraints. We plan to apply the new model compression technique to other popular Transformerbased models and demonstrate the inference efficiency using our inference solution. Code and models for reproducing the published results are available at https://github.com/intel/intel-extension-fortransformers. The default memory allocator usually creates a new buffer each time when receiving a memory allocation request, and therefore the data is less likely to be reused. To maximize the buffer reuse and make the data more cache friendly, we develop an advanced memory allocator as shown in Figure 3.

A.3.2 Weight sharing
Weight sharing is another useful runtime optimization used for efficient weight reuse in inference. Figure 4 shows how weight sharing works. The left one shows each inference instance (blue box) with the separate weight (grey box). If users want to run eight instances in parallel, eight copies of weight are needed, therefore lowering the overall inference efficiency due to lack of memory reuse. The right one shows one shared weight across multiple instances after applying weight sharing mechanism.

A.3.4 Sparse GEMM operators
We develop the optimized kernels for sparse GEMM operators based on the pre-defined block-wise structured sparsity pattern with constant block size 4. As described in Section 1, non-zero weight block is broadcasted to form a VNNI-format block; the sparse weight is compressed and the mask is  Note that for the weight with 4 non-dividable sparsity dimension, the additional padding is needed while this is not common in NLP models. For simplicity, we omit the special handling of padding in the sparse GEMM kernel implementation.

A.4.1 Software
We use the latest stable release ONNX Runtime v1.11.1 and Neural Magic v1.1.0 community edition (a436ca67) to measure the performance. Table 4 shows the absolute maximum throughput (under 10 ms latency constraint) which is used to derive the relative performance in Figure 2.