{'fraction_non_alphanumeric': 0.0333179278181589, 'fraction_numerical': 0.0093273391874291, 'mean_word_length': 4.756227327690447, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Latency Adjustable Transformer Encoder for Language Understanding

Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of efficient architecture development. This paper proposes an efficient transformer architecture that adjusts the inference computational cost adaptively with desired inference latency speedup. The proposed encoder model can work with fewer Floating Point Operations (FLOPs) than the original Transformer architecture. In fine-tuning phase, the proposed method detects more important hidden sequence elements (word-vectors) in each encoder layer by a proposed Attention Context Contribution (ACC) metric. It eliminates the less important word-vectors based on a new strategy. A mathematical inference speedup analysis is proposed to estimate the speedup accurately to adjust the latency and computational cost of fine-tuning and inference phases. After the fine-tuning phase, by the method offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections. The proposed method is applied to the BERTbase model for evaluation. Extensive experiments show that most of the word-vectors in higher BERT encoder layers have less contribution to the subsequent layers; hence, they can be eliminated to improve the inference latency. Experimental results on extensive sentiment analysis, classification, and regression benchmarks like GLUE showed that the method is effective in various datasets. The proposed method improves the inference latency of BERTbase by up to 4.8 times with less than 0.75% accuracy drop on average.


I. INTRODUCTION
Recently, Transformer [3] based architectures achieved remarkable success in various Natural Language Processing (NLP) tasks. However, the training difficulty, inference latency, and small size of available datasets are the main concerns with using these models in real applications [7]. Fine-tuning a pretrained language model on downstream tasks mitigates some of these concerns, including training effort and dataset size. GPT [4], BERT [5], and XLNet [6] are examples of recently state-of-the-art pre-trained language models. The number of parameters in these models is more than hundreds of millions. Hence the computational complexity bears billions of Floating Point Operations (FLOPs) for processing a single sample. In some cases, the inference phase computational effort is beyond the ability of resource-limited systems such as edge devices. Therefore baseline Transformer models cause unacceptable latency and energy usage limitations in such devices.
One of the most effective pre-trained language models is Bidirectional Encoder Representations from Transformers (BERT) [5], which has multi-layer transformer-based architecture. BERT base consists of an embedding layer and 12 stacked encoder layers. Each encoder contains an attention layer with 12 heads and multiple fully connected layers. There are about 110M parameters in BERT base model, the pre-training of which has taken 79 hours on 64 Tesla V100 GPUs [7].
In the literature, some methods are proposed to reduce the model computational effort by removing or sharing some layers or heads [8]- [11]. In addition, distillation-based methods produce a student network that is much faster with a smaller number of parameters [13], [17], [19]. The main limitations of these methods include pre-training from scratch, excessive training computational efforts, the need for supplement datasets, and significant accuracy degradation. This paper proposes a new encoder layer that speeds up the inference phases of BERT base model. The proposed hidden sequence elements (word-vectors) sort and elimination approach decreases the effective word-vectors in each layer and reduces the number of FLOPs. The main contributions in this work are as follows: • The novel structure and tuning policy are proposed to speed up the inference times with negligible accuracy degradation. • A new Attention Context Contribution (ACC) metric is proposed to observe the context contribution in each selfattention layer. It is shown that the context contribution is decreased at the last encoder layers significantly. • An accurate analytical inference speedup estimation is presented for the proposed method that helps a system designer determine the hyper-parameter selection at finetuning phase. • Offline-tuning property that adjusts the inference latency after the fine-tuning phase. This property controls the trade-off between latency and accuracy in a wide range of inference speedup selections. • The proposed method can easily be applied to transformer-based architectures alongside various compression and acceleration methods. Several methods are proposed in the literature to reduce the  model size and inference latency by controlling the number of  parameters and inference phase computational effort, respectively. ALBERT is a lighter version of BERT model [9]. It uses cross-layer weight sharing and a modified embedding layer structure, making the embedding size independent of the encoder's hidden size. Moreover, the new Sentence Order Prediction (SOP) task is utilized in the pre-training phase instead of the Next Sentence Prediction task. ALBERT reports stateof-the-art results with its parameter reduction and modified pre-training policy. ALBERT is nine times smaller and 1.2 times faster than BERT base , with less than 2.5% on average accuracy drop. A progressive layer dropping at the pre-training phase of BERT architecture offers up to 2.5 times speedup at the inference phase with more knowledge transferability and better generalization on downstream tasks [8]. DistilBERT [13], TinyBERT [19] and MobileBERT [17] use the knowledge distillation technique during the pre-training and offer up to 5.5 times faster inference on downstream tasks than BERT base . The main concerns with these methods are the pre-training effort of the model from scratch, extra training steps, and significant accuracy degradation.

II. RELATED WORK
FastBERT is proposed with adaptive inference time in which the model uses a sufficient number of encoder layers based on the inference certainty of a sample [14]. It dynamically adjusts the number of active layers to reduce the overall computational effort. ELBERT proposed the Confidence-Window Based Early Exit mechanism, improving the ALBERT inference speed [15]. In [16], the model weights are encouraged to be zero with the L0 regularization term. In [18], it is experimentally shown that using all layers of a pre-trained model in downstream tasks is unnecessary. Studying various layer dropping strategies shows that many model weights are not necessary.
A parameter reduction method was proposed by identifying the most critical heads in each encoder layer using layer-wise relevance propagation and pruning the redundant heads [10]. It showed that the earlier encoder layer's heads are much more critical than the last. Moreover, in [11], it is shown that around 17% of attention heads can be removed at the test time without significantly impacting inference performance. In [12], the magnitude weight pruning method is applied to BERT and shows that the model can be pruned once during pretraining rather than separately for each task without affecting performance.
PoWER-BERT proposes a word-vector elimination method that eliminates redundant word-vectors [20]. A supplement learn-able layer is added between each encoder self-attention and feed-forward layer, and an additional loss function term is used to train these supplement parameters. PoWER-BERT training consists of 3 steps "fine-tuning, elimination configuration search, and re-training" [20]. Funnel-Transformer [31] improves the transformer architecture by inserting a pooling layer in each self-attention layer. Funnel-Transformer compresses the sequence of hidden states to a shorter one and reduces the computation cost. This model is inspired by U-net [32] from computer vision.
The previous works are trying to reduce the model size and inference latency. These works show that significant part computations in the models appear unnecessary, and models with fewer computational complexity can be obtained from the original ones. The main drawbacks of these methods are pre-training from scratch, multi-phase fine-tuning, and significant inference accuracy drop. In these methods, the inference speedup is set in fine-tuning phase, and the achieved speedup is fixed in the inference phase. The proposed method tries to mitigate the previous drawbacks. It presents a powerful offline-tuning mechanism that controls the inference speedup of the model after the fine-tuning phase.

III. PROPOSED METHOD
This Section presents the analysis and reasoning behind the proposed method and its architecture. Analyzing the number of FLOPs in BERT base model shows that most FLOPs and latency occur in the encoder layers. The proposed word-vector elimination strategy at encoder layers is based on an insight into the amount of the contribution of word-vectors at different layers.
A. Background Fig. 1 shows BERT base model architecture and output classifier. The input sentence goes through the tokenizer layer and embedding layer. Then the context is fed into 12 cascaded encoder layers to be processed. The pooler and classifier layers are added in the last layer to perform proper output for the desired task. In the figure, T and H are the numbers of input word-vectors and the hidden state vector size, respectively. Each encoder consists of a Multi-Head Self-Attention layer and a Feed-Forward layer with several tensor multiplications. The residual branches and normalization layers increase the training stability. Table I shows the estimated FLOPs of BERT base at different layers. The amount of FLOPs is estimated based on the number of input tokens (T ), the number of encoder layers (L), the hidden state size (H), and the number of output labels (N ).

B. BERT FLOPs Analysis
As shown in the table, for a typical case 1 , almost all of the BERT base FLOPs occur in encoder layers. About 25% of FLOPs are in self-attention layers, and the rest are in feed-forward layers. The experimental results confirm the estimations and show that about 99.8% of the computational complexity is at the encoder layers.
The inference latency and computational effort are directly related to the number of Floating Point Operations (FLOPs). As a result, reducing the number of FLOPs at the encoder layers reduces the total number of FLOPs and inference latency.

C. Attention Context Contribution (ACC) metric
The context contribution of word-vectors at each selfattention layer is studied in this Section to find more important word-vectors at each encoder layer. For a given encoder layer, the Attention Context Contribution (ACC) metric measures the contribution of word-vectors in the layer's output. This metric is derived from a score vector (SV) to find more important word-vectors in each layer. For a given layer, the SV indicates the contribution of each input word-vector to the output of the attention layer. Fig. 2 illustrates the calculation of the Score Vector based on the attention probability matrix. Fig. 2 (left) shows the attention probability matrix of BERT base model. The rows of this matrix are normalized to one. By averaging the attention probability matrix over the heads, the aggregated probability matrix is resulted, Fig. 2 (center). The SV results from the sum of this matrix along the unnormalized dimension, Fig. 2 (right). The SV elements with higher score values correspond to the word-vectors that contribute more to the layer output. Fig. 3 illustrates BERT base SV results on the IMDB sentiment analysis task [21] for an input sample. As shown in earlier encoder layers, the SV values approximately have uniform distributions, and in the last encoder layers, the distribution of SV values changes to approximately δ distribution. The distribution of SV values in the last layers indicates that the concentration of scores is high on a few word-vectors and others have less contribution to the self-attention layer output.
The SV elements are positive and with an average of one. So, the median of SV is bounded. Suppose the median of the SV is lower than its average. In that case, more than half of the elements are lower than the average value, and only a few are higher than the average value. It suggests that most of the word-vectors do not bear enough contribution to the layer output, and only a few word-vectors influence the final result of the layer. The median of the Score Vector is interpreted as the proposed Attention Context Contribution metric.
The ACC metric calculation is explained in Algorithm. 1. The algorithm inputs are BERT encoder word-vectors and the model parameters. The algorithm output is the ACC metric vector that suggests the contribution of the word-vectors in each encoder self-attention layer. For encoder layer l, the first step is to prepare the pre-computed self-attention probability matrix, M l Att . Then the average of this matrix over its heads is computed, M l H . In the next step, the resulting matrix's summation over its unnormalized dimension is computed, M l S . This matrix is the Score Vector. Attention Context Contribution of the layer (E l ACC ) is the median of the Score Vector. Fig. 4 shows the ACC metric for BERT base encoder layers on IMDB [21] dataset and several General Language Understanding Evaluation (GLUE) [22] [5] Score Vector (SV) results on the IMDB sentiment analysis task [21] metric is reduced at later layers gradually. It indicates that the fraction of the word-vectors that contribute more in the last layers is less in those layers; hence more word-vectors can be eliminated. By eliminating less important word-vectors in each layer, the processed word-vectors are decreased gradually, and the inference time of the model is decreased.

D. Proposed Architecture
According to the discussion in Section III-B, almost all latency due to the FLOPs occur in the encoder layers. In addition, according to the observation in Section III-C, the word-vector contribution in the encoder decreases at later layers. Based on these observations, this Section proposes details of the method.
The proposed method is applied to the BERT base model for explanation and evaluation. The proposed architecture is the same as the original BERT with a modified multi-head self- Append E l ACC to E ACC 8: end for 9: return E ACC attention layer. Fig. 5 illustrates the proposed multi-head selfattention layer. The proposed architecture adds the Sort and Eliminate layers in the middle of the original BERT multihead self-attention layer. The Sort layer sorts the input context according to the computed layer Score Vector. The Eliminate layer eliminates a fraction of the sorted context with lower scores according to the Elimination-Rate (α ER ).
α ER hyper-parameter is a vector that sets the fraction of word-vectors that is eliminated in each encoder layer. For layer l, α l ER is the word-vector Elimination Rate obtained from the pre-computed Elimination-Profile (α EP ) vector and a pre-specified Speedup-Coefficient hyper-parameter (α SC ). α SC adjusts the trade-off between accuracy and speed in finetuning and offline-tuning phases.
α l EP is from the fitted curve to the ACC metric (red lines  [5] Encoder Attention Context Contribution (ACC) metric results (blue bars) and second degree fitted curves (red lines) on SST-2 [23], MRPC [25], QQP [26], QNLI [27], RTE [28], MNLI [29], STS-B [30] and IMDB [21] tasks.  Fig. 4) at layer l. In some tasks, the ACC values of layers are not smooth. To increase stability and prevent applying stress to the model in fine-tuning phase, in the proposed formulation, instead of ACC metric values (E ACC ), the fitted curve to the ACC metric (P ACC ) is used that presents a more smooth shape. The α EP lower bound is set to one to certify a monotonic descending shape of this hyper-parameter curve.
α l EP = min 1, The number of remaining word-vectors in each encoder layer is calculated according to the number of the layer input word-vectors and the α ER hyper-parameter. Equation (3) presents the number of remaining word-vectors (T l ) at the output of encoder layer l. The minimum number of remaining word-vectors at each encoder output is limited to one.
The effective number of processed word-vectors can be estimated using the number of word-vectors before and after the eliminate layer. Based on Table I, in each encoder layer, around 75% of FLOPs occur after the self-attention layer. Equation (4) presents the effective number of Processed wordvectors (P W l ) in layer l. Hence, Processed word-vectors in the original BERT base is: In these equations, L is the number of BERT encoder layers, and T is the number of encoder input word-vectors. Since almost all inference latency occurs in the encoder, the overall model computation effort is almost equal to the encoder stage. Equation (7) presents the proposed method inference time speedup to BERT base .
The above equation assumes that the number of remaining word-vectors in each encoder layer is more than a single wordvector. In other words, (3) returns the second argument, which is valid for most cases. For example, if L is 12 and α l EP is constant and equal to 0.8 for all layers, the method inference speed is around 3.09 times faster than BERT base .
In order to improve the speed of the proposed self-attention layer, the Sort and Eliminate layers are placed after the selfattention concatenation layer. With this change, the per-head calculations are reduced, and for a given layer, the eliminated word-vectors participate in forming the layer's output.

IV. EXPERIMENTS
This Section explores the performance of the method under the GLUE benchmark and IMDB sentiment analysis task. At the end of this Section, the offline-tuning property is investigated.

B. Model Settings
For the method's evaluation, the BERT base encoder selfattention layer is replaced with the proposed encoder selfattention layer. The hyper-parameters are the same as those in BERT base with extra α EP , and α ER hyper-parameters are added to set the word-vector elimination rate in each encoder layer. The hyper-parameters are: learning rate -[1.5 × 10 −5 , 4.5 × 10 and α EP vector that is obtained from (2). The cross-entropy loss function is used for classification tasks, and the Mean Square Error (MSE) loss function for regression tasks. The linear learning rate scheduler is used for all tasks. Initial linear learning warm-up is used for some tasks. In all tasks, AdamW optimizer with = 1 × 10 −8 , β1 = 0.9 and β2 = 0.999 is used. The model input sequence length is delicately fixed so that less than 1% of samples are truncated. The selected input sentence length for each dataset is specified in Table II.

C. Implementation Details
The model is implemented in the PyTorch 2 library on an Nvidia Tesla P100 GPU. The pre-trained BERT base-uncased [5] Hugging Face 3 implementation is used as the starting model. The datasets are from the Hugging Face Datasets library, and the GLUE results are verified on the official GLUE service 4 .

D. Experimental Results on GLUE and IMDB
The experimental results for extensive tasks are presented in Table II. The model compares to BERT base is 2.9 times faster with 0.7% drop on the test set. Also, the model inference time is up to 4.8 times faster than BERT base , with only 0.9% accuracy drop. These results indicate that the proposed method effectively improves transformer-based models' performance and resource usage with minimal accuracy degradation. Table II shows the experimental speedup, the expected speedup, and the number of FLOPs for each task. The number of FLOPs reduction is consistent with inference speedup and expected speedup predicted using (7). The slight differences are due to the computational overheads, including the computational effort of the tokenizer and output classifier layers that are not considered in (7). The consistency between the experimental speedup, the expected speedup, and the FLOPs reduction approves the hypothesis discussed in Section III-D.  average. In DistilBERT, a pre-trained general-purpose language presentation model must be trained from scratch with significant effort. The TinyBERT model speedup is 2.0 times with 0.4% accuracy drops on average. TinyBERT uses a twostage learning framework that performs transformer distillation at pre-training and fine-tuning phases with considerable excessive computational effort. The MobileBERT model speedup is 5.5 times with around 1.19% accuracy drop, but the starting point of this model is BERT large . The PoWER-BERT model speedup is 2.9 times with around 0.92% accuracy drop. However, the training of PoWER-BERT consists of 3 phases: "fine-tuning, elimination configuration search, and re-training" [20] that impose extra training effort.
The table shows that the proposed method results are competitive and more effective than previous baseline methods on extensive experimented tasks. Moreover, compared to recently reported methods, our method is only applied in fine-tuning phase.

E. Offline-Tuning Property
After the fine-tuning phase, the trade-off between inference accuracy and speed of the model can be adjusted with the offline-tuning property. This property controls each encoder layer's word-vector elimination rate by adjusting the Speedup-Coefficient hyper-parameter (α SC ) value. Fig. 6 presents the offline-tuning property. In the following experiments, the model first fine-tuned with α SC equal to one for a given task, and then in the offline-tuning phase, the α SC value changes from 0.85 to 1.2 to observe its effect on inference speedup and accuracy. Fig. 6 (a) shows the model inference speedup for a range of α SC hyper-parameter values on multiple tasks. The model inference speedup increased more than three times compared to BERT base , with the α SC value decreasing to 0.85. Fig. 6 (b) presents the accuracy stability on multiple tasks. It shows that the inference results are kept stable over a wide range of α SC values selections. Fig. 6 (c) shows that the inference results are almost unchanged even if the complexity of the model reduces by 70% according to the proposed method. In these experiments, the model's accuracy on validation sets is reported because of the GLUE test set verification limitations.
Two approaches can be offered to select the proper value of the α SC hyper-parameter in the fine-tuning phase. The first approach is using equation (7) and calculating the approximate value of the α SC based on the desired inference speedup. The second approach is selecting the α SC value with the help of suggested Fig. 6(c) curves. After the fine-tuning phase, the speedup can be accurately adjusted with the offline-tuning property.

CONCLUSIONS
This paper proposed a latency adjustable transformer encoder layer that significantly decreased Transformer based architectures inference time with a minor drop in accuracy. A new Attention Context Contribution (ACC) metric is proposed to observe the context contribution in each Transformer encoder layer and experimentally showed that the word-  [25], SST-2 [23], QNLI [27], IMDB [21] and MNLI-M/MM [29] tasks.
vector contribution is decreased in the last encoder layers. An accurate analytical speedup estimation is presented that helps a system designer determine the hyper-parameter selection. The offline-tuning property adjusts the model speedup after the fine-tuning phase. The proposed method is applied to the BERT base model, and its improvements are verified on the GLUE benchmark and IMDB dataset.