{'fraction_non_alphanumeric': 0.03679102696682841, 'fraction_numerical': 0.015392570201256861, 'mean_word_length': 4.62987012987013, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Can depth-adaptive BERT perform better on binary classification tasks

In light of the success of transferring language models into NLP tasks, we ask whether the full BERT model is always the best and does it exist a simple but effective method to find the winning ticket in state-of-the-art deep neural networks without complex calculations. We construct a series of BERT-based models with different size and compare their predictions on 8 binary classification tasks. The results show there truly exist smaller sub-networks performing better than the full model. Then we present a further study and propose a simple method to shrink BERT appropriately before fine-tuning. Some extended experiments indicate that our method could save time and storage overhead extraordinarily with little even no accuracy loss.


Introduction
Information retrieval is always of great concern, which Recently, pretrain-then-finetune paradigm has led to significant improvements in a wide range of the NLP tasks. Some pre-trained language models, like BERT (Devlin et al., 2019), achieve state-ofthe-art performance on numerous tasks, such as sentiment analysis, acceptability analysis, semantic similarity analysis and machine translation (Dolan and Brockett, 2005;Warstadt et al., 2019;Wang et al., 2018;Yang et al., 2019a;Hoang et al., 2019;Yang et al., 2020a). For learning general knowledge as much as possible in pre-training stage, the language models are constructed with plenties of units (i.e., the total parameters in BERT BASE are 110M, and in BERT LARGE are 340M) (Devlin et al., 2019) . They are usually too huge for the downstream tasks. Taking BERT BASE as an example, fine-tuning it on some complex downstream tasks requires several hours to finish one epoch (Strubell et al., 2019). That makes them impractical for resource-limited deployment scenarios and cost concerns incur, then hinders the applications. As we know, some compressed models (i.e., distilBERT, 66M (Sanh et al., 2019)) are capable of achieving comparable results. These lead us to think the magnitude of those large language models when transfer to target tasks, force us to compress models and accelerate their calculations (McCarley et al., 2019). However, existing model compression methods cost too much resources during fine-tuning, so we are eager to find a new way to solve the problem.
Existing model compression methods orienting to particular tasks fall into two categories, namely knowledge distillation (Buciluǎ et al., 2006) and pruning (Frankle and Carbin, 2018). For using knowledge distillation method, there needs to construct a complex teacher model and a much simple student. After the teacher fine-tuned thoroughly, the student imitates the teacher to predict and then makes decisions independently in test period. By doing so, resources like time and memory are saved in test period (Chia et al., 2019;Mirzadeh et al., 2020;Yang et al., 2020b;Wu et al., 2021). However, loading the teacher and student simultaneously and transferring knowledge from the teacher to the student during fine-tuning consume too much memory and time. Differently, pruning method only requires constructing one model, but mask parameters should be preserved to implement weight dropping. Matching the mask parameters with the original model and pruning the less important units during fine-tuning also need huge storage and time. (Joulin et al., 2016;Liu et al., 2018;Voita et al., 2019;Gordon et al., 2019;. Some common issues could be concluded that (1) they all target at accelerating calculations in test stage at the cost of more resource-consuming during fine-tuning. And (2) all the downstream tasks apply the standard language models indiscriminately and treat them as the superior basis of pruning without taking the complexities of different tasks into consideration. Naturally, the matching relations between pre-trained language models and target tasks are neglected. Compressing models based on them is unreasonable.
According to the lottery ticket hypothesis (Frankle and Carbin, 2018;Chen et al., 2020b), there is some redundancy in over-parameterization BERT model. While some experiments about binary text classification tasks (like CoLA (Warstadt et al., 2019) and ChnSentiCorpHou (Tan and Zhang, 2008)) indicate that the standard BERT model is not always the best choice, and sometimes we could find sub-networks (winning tickets) surpassing the full BERT on some tasks via layer truncation (Details are provided in section 2). Motivated by this phenomenon, we propose to truncate BERT adaptively before fine-tuning and save resources both in fine-tuning and test stages with little cost. Compared with the conventional model compressing methods, our truncation method is valuable. Because fine-tuning time is much longer (hundreds times or more) than test time, while conventional methods fail to reduce it but need more in practice. A simple way to estimate the truncating position based on the characteristics of tasks is also provided. In addition, we conduct some experiments on 8 binary text classification tasks (Binary text classification tasks are simple enough to highlight the issues and are the basis of most complex NLP tasks). The results show that our method is simple but effective.
In summary, our contributions include: • We observe that full BERT is not always the best when fine-tuning on some particular downstream tasks. Sometimes shrinking BERT by simply truncating layers leads to capable even better performances with less overhead.
• We put forward a simple but effective truncation method to find out the winning ticket in BERT before fine-tuning, and provide a measure to estimate truncating position.
• Experiments in this paper show that our method could surpass the traditional compressing methods on 7 tasks and even predict more precisely than full BERT on 3 tasks. In addition, our method saves about 75% finetuning time and 25% storage than knowledge distillation and pruning methods on average. Also, 20% storage and time are saved compared with original BERT-based models both in fine-tuning and testing while remaining near to 99% accuracy on average.

Full BERT is NOT always the best
Knowledge distillation and pruning methods accelerate testing of models by devoting additional resources during fine-tuning. That is impractical for some edge devices. In order to save cost both in fine-tuning and testing, we propose to truncate several consecutive transformer layers from top of the BERT (12-layer) and preserve the bottom part as a new encoder. In this paper, we reduce BERT ranging from 0 to 11 layers and obtain 12 different encoders. Each encoder is appended a task-specific fully-connected layer and a sigmoid layer. All these 12 shrunk models (namely trans12 to trans1) are fine-tuned on eight text classification tasks, including CoLA (Warstadt et al., 2019) and ChnSentiCorpHou (Tan and Zhang, 2008). More details of datasets, experimental settings and results are provided in Appendix A, B and C.
Here we take CoLA and ChnSentiCorpHou as examples and plot the test accuracy, training and test time of these 12 truncated models in Figure  1. There is no doubt that the training and test time are increasing dramatically as the number of transformer layer grows. However, we do not get far satisfying predictions with so many resources cost. With the increasement of the transformer layer, we find the accuracy is gradually fluctuating within a certain range. For CoLA (Figure 1 (left)), we find two winning tickets surpassing the full BERT when number of transformer layers are 9 and 10. While the models with 8 and 11 layers are almost capable of predicting as precisely as full BERT-based model. As for ChnSentiCorpHou (Figure 1 (right)), although we do not find a winning ticket better than full BERT, models with 2 to 12 layers predict similarly. It seems that smaller models could perform well enough with less time and memory compared with full BERT.

Truncating BERT before fine-tuning
The results in section 2 show there exist truncated BERT models performing better than full BERT on some binary classification tasks. This inspires us that shrinking massive BERT for simple tasks is valuable and necessary. In order to clearly understand this phenomenon and find a way to solve Figure 1: Accuracy, training time and test time of trans1-12. The reservation of accuracy arrives decimally hind two, and training and test time are rounded to the nearest whole number. In left subplot, trans5 reaches 90% accuracy compared with trans12, and the accuracy of models with more than 8 layers becomes changing slightly. In the right, almost all these 12 models achieve more than 95% accuracy of trans12, and the accuracy of models with more than 2 layers becomes fluctuating.
problems well with less time and memory, we further take a closer look at these results and make some analyses about BERT truncation. In this section, we will provide some discussions about when, where and how to truncate BERT model.

Where to truncate
As discussed in section 2, the test accuracy becomes fluctuating when the number of transformer layers larger than some certain values. We then compare the predictions of each model and capture something interesting. When the test accuracies of two models are similar, the predictions on the same instance are also close. In addition, all these 12 models provide consistent predictions on most of the instances, and divisions only arise over a few indistinguishable ones. While these divisions would become less and less with the test accuracies being closer.
Specifically, given a dataset S containing n examples x 1 , x 2 , · · · , x n . We construct 12 truncated BERT-based classification models, namely M 1 to M 12 (here, the subscript i of M denotes the number of transformer layers), to make predictions on the S. The test accuracy of a model M i is denoted as acc i (i = 1, · · · , 12). And the calculation of M i on x p (p = 1, · · · , n) is formulated as: Where σ is the sigmoid function, g i (·) is the function of a fully-connected layer, h i (·) is the func-tion of the encoder, and f i (·) is the function of the i-th transformer encoder layer. As mentioned above, when acc i is close to acc j , we find M i (x p ) is also close to M j (x p )(p = 1, · · · , n; i, j = 1, · · · , 12, and i < j). For classification task, we know σ(g i (·)) is a generalized linear transformation targeting at finding a hyperplane separating examples best without transforming the representations. Naturally, the prediction M i (·) mainly depends on the transformation of the encoder h i (·).
The higher separability of the hidden states produced by the encoder is, the higher prediction probability M i (·) and the higher test accuracy acc i of the model are. As a result, we could use the hidden states to help us decide where to truncate. Inspired by the distribution of points in high dimensional space, we treat the output representations of transformer layers in encoder as coordinates of point, and apply the separability theory of point set to expressing the distribution of examples in our datasets. We truncate BERT targeting at producing representations of dataset with maximum seperability.
Class Scatter Matrices, CSM : is a fraction of trace of between-cluster scatter and within-cluster scatter. It evaluates the overall distribution of an entire dataset.
Where tr(·) denotes the trace operation: calculating the sum of one's diagonal elements. c and n i are the number of classes in a dataset and number of examples in class i respectively. While m and m i are the mean vectors of whole dataset and class i separately. x i,j is the j-th instance in class i. The within-cluster scatter and between-cluster scatter are denoted as S W and S B . Separability Index, SI: calculates the average number of instances that share the same category label as their nearest neighbors. It is a measure evaluating the class overlap.
is the nearest neighbor of x i , and n is the number of examples in a dataset. Hypothesis Margin, HM: measures the sum of the distance between the hypothesis and the closest hypothesis that assigns the alternative label to the given instance.
Where nearhit(x i ) and nearmiss(x i ) denote the nearest instance to x i with the same and different label, respectively. And n is the number of instances in a dataset. · denotes the Euclidean norm. According to the definitions above mentioned, the larger these three separability values are, the more easily the examples are classified. We could apply our truncation method with the assistance of the separability measures.

When to truncate
As we all known, BERT is pre-trained on two general tasks, namely masked language model (MLM) and next sentence prediction (NSP). It means there is little disturbance of wrong prediction corresponding to specific task brought in. Making decisions on the basis of original pre-trained BERT is more objective.
What's more, once the fine-tuning processing starts, the predicting error would radially propagate back to each layer of the entire network. Breaking the network during this stage leads to reconstruct new matching relations among whole net, which is far more time-consuming.
On account of these factors, we decide to truncate BERT before fine-tuning. If we can make a reliable decision about where to truncate before fine-tuning, resources like time and storage would be saved extraordinarily.

How to truncate
In this sub-section, we provide a general algorithm about truncating BERT (12-layer) properly for downstream tasks by means of separability measures (Algorithm 1).
Given a target binary classification task and the corresponding dataset, we can estimate where to truncate before fine-tuning without building up the network. Our three-step method is simple enough and time-saving. In particular, all the instances in the dataset should be vectorized with BERT embeddings in the first step. Secondly, the vectorized instances are sent into the pre-trained BERT model and propagate forward once without backward propagation. In this step, we will obtain 12 hidden representations from different transformer layers of the BERT for every instance. The hidden representations produced by the same transformer layer are packed into a common set. Then in the third step, we calculate separability values for these 12 representation sets. The number of transformer layers related to the maximum separability is the suggested number of layers in the truncated BERT. We truncate BERT according to that and append the task-specific layer on to form the final model.
In this way, we can truncate BERT adaptively before fine-tuning according to target tasks. Putting these truncated BERT into force could save as much as time and memory with little expense.

Experiments
Observations in section 2 inspires us that the full BERT is not always the best choice for some binary classification tasks. And in section 3 we provide a simple but clear analysis about this phenomenon, Algorithm 1: Truncating algorithm Data: S = {x 1 , x 2 , · · · , x n } 1 for x i (i = 1, · · · , n) in S do 2 Vectorize x i as x i ; 3 Send x i in BERT; 4 for each transformer layer l j (j = 1, · · · , 12) in BERT do 5 Calculate hidden state h i,j ;

6
Collect h i,j into v j (v j = {h 1,j , · · · , h n,j }); 7 end 8 end 9 for each v j (j = 1, · · · , 12) do 10 Calculate separability value (csm j , si j or hm j ); 11 end 12 Truncate BERT according to the maximum separability value and introduce an effective method to shrink BERT for downstream tasks. In this section we will apply our method to 8 binary classification tasks and give a further exploration.

Datasets and tasks
In this sub-section, we will give a brief introduction about 8 corpora used in this paper, including ChnSentiCorpHou, WaiMai, SST-2, CoLA, Nikon-JD, AFQMC, MRPC and RTE. The ChnSentiCor-pHou is a sub-set of a series of Chinese sentiment analysis corpora organized by Tan et al. (Tan and Zhang, 2008 (Wang et al., 2018), and AFQMC 2 is selected from the Chinese language understanding evaluation (CLUE) benchmark (Xu et al., 2020). The Nikon-JD corpus is organized by our team. More details of these datasets are provided in Appendix A. In Table 1, we list some critical information about these corpora. We select two Chinese and two English single sentence classification tasks. In addition, two Chinese and two English sentence pair classification tasks are also chosen. The assignments in this section vary from sentiment classification and acceptability analysis to semantic similarity and natural language inference. Different assignment is related to different size of dataset and difficulty degree.

Implementations
We select the 12-layer BERT model (BERT BASE ) as our basic language model and apply the truncation method on these 8 datasets. For each single sentence task, we put a [CLS] and [SEP] at the beginning and the end of the given text separately. And for sentence pair task, we also concatenate two texts into a sequence with [SEP] token being the separator, besides the special pre-processing at beginning and end. Then send processed sequences into the BERT for following procedures. Before constructing models and fine-tuning, we utilize whole dev sets of ChnSentiCorpHou, Nikon-JD, MRPC and RTE, and randomly sample about 50%, 50%, 10% and 10% examples from dev sets of SST-2, CoLA, WaiMai and AFQMC separately. Then we calculate CSM, SI and HM with these prepared instances for each task, and according to that truncate BERT model adaptively. Next, we construct classification models based on the truncated BERT models, then fine-tune and test them on above mentioned corpora. More experimental details are in Appendix B.
For comparison, we also set up some baselines: BERT base: We construct binary classification models based on BERT BASE (Devlin et al., 2019) as the standard models.
Distil-Pretrain: We build up classification models based on the distilled language models (Distil-BERT (Sanh et al., 2019) and albert-chinese-tiny 3 ). In this way, the knowledge distillation theory is used in pre-training period.
Distil-task-biLSTM: We fine-tune a BERTbased classification model as the teacher, and construct a student using the biLSTM as the encoder with the assistance of the attention mechanism. Then let the student imitates prediction of the teacher.
Distil-task-kimCNN: We construct a student according to the kimCNN (Kim, 2014). While the teacher models are same as those of Distil-task-biLSTM.
Pruning-AGP: We use the Automated Gradual Pruner (AGP) (?) to implement the task-specific  Table 1: Corpora information. All these 8 are binary text classification tasks with a single sentence or sentence pair as the input. In the last column, the more the stars are, the more difficult the task is.
magnitude pruning on our tasks. Here we refer to the implementation provided in the Distiller library (?). Random truncation: Random truncation means selecting a random number ranging from 0 to 11 as the number of layers that should be truncated. Here, we use the mean test accuracy of 12 various truncated BERT-based models to denote the random accuracy.
Note that all these baselines use the BERT embeddings, the batch size and max sequence length are same as those of our models (Table 6). Differently, during implementing task-specific distillation, we set the learning rate as 1e-3.

Results
In order to evaluate whether our separability measures are appropriate or not and then choose a better one for truncating BERT, we firstly plot three measures with the test accuracies in Figure 2 (Here we only provide the results of CoLA and ChnSen-tiCorpHou, while others are shown in Appendix D). Obviously, we could easily find that the tendencies of these three measures are similar to the test accuracies to some extent. As revealed in the Equation 3-7, distance-based measures (CSM and HM) are more exact than the one based on number of instances (SI), while HM is easily influenced by some specific examples and loses effectiveness. Among these three measures, the CSM is more stable. To make a quantitative analysis, we use the Pearson's correlation coefficient (PCC) to evaluate the relations. The average of PCC in Table 2 also indicates that the CSM is more relevant to test accuracy. As a result, we choose the CSM measure to help us truncate BERT.
As shown in Figure 2 and Appendix D, when the number of transformer layers are 7, 11, 12, 10, 11, 12, 5 and 9, the CSM values reach the maximum on ChnSentiCorpHou, WaiMai, SST-2, CoLA, Nikon-   Table 3, Table 4 and Table  5, respectively. Obviously, our method, Truncated-CSM, is capable of the 99% language understanding capabilities of the full model in average, and almost surpasses the other compressing methods on all these tasks except the MRPC. Also, about 20% of time and memory are saved both in finetuning and test with little additional cost, which is impossible for knowledge distillation and pruning. What's more, our method is simple to implement and transfer to other tasks. On the contrary, we can draw conclusions from Table 4 and Table 5 that distillation methods are too complex to apply and consume far more time and storage from building up the teacher model to obtaining the student. The pruning methods are of the same circumstances in fine-tuning. The overhead of knowledge distillation and pruning methods in fine-tuning is awful and unbearable. However, we do not get satisfying predictions with these two kinds of ways.
According to these results, we confirm our method is effective and provide the following suggestions: • Only 1 layer of transformer encoder is enough to perform well on most simple tasks.
• For some complex tasks, a few more layers (3-5 layers) could produce satisfying results.
• More exact estimation could be made by randomly sampling some instances to calculate CSM value.

Related Work
In this section, we briefly review some work about pre-trained language models, knowledge distillation and model pruning.
Pre-trained language models: Since about 2017, there gradually appear some complex language models. The widely used BERT proposed by Devlin et al. (Devlin et al., 2019) achieves pretty good results on GLUE (Wang et al., 2018), SQuAD (Rajpurkar et al., 2016) and SWAG (Zellers et al., 2018). Before long, XLNet (Yang et al., 2019b), RoBERTa  and some others (Zafrir et al., 2019;Shen et al., 2020;Wang et al., 2021) emerge and push the evaluation scores of NLP tasks much higher. With the development of pre-trained language models, a wide range of NLP tasks obtain new state-of-the-art results again and again, including question answering (McCarley et al., 2019), language inference (Levesque et al., 2012;Williams et al., 2018), text classification (Joulin et al., 2016) and text summarization (Barzilay and Elhadad, 1999). Consequently, the language models has been an indispensable part in solving NLP assignments.
Knowledge distillation: Knowledge distillation is a method that can transfer knowledge from an ensemble or a large highly regularized model into a much smaller, distilled model. This concept is proposed by Caruana et al. (Buciluǎ et al., 2006) and further extended to Neural Networks by Hinton's team (Hinton et al., 2015). Some like SANH et al. (Sanh et al., 2019) and Jiao et al. (Jiao et al., 2020) only use this theory to compress models and accelerate processing. More are inspired not only to reduce the models but also transfer extra knowledge by means of changing input style or other factors during distillation Chen et al., 2020a;.

Conclusion
In this paper, we conduct a series of experiments to exploring whether it exist winning tickets when transferring the language models in downstream binary classification tasks and how to find them conveniently and effectively. Different from conventional model compressing methods, we insist to truncate BERT by a brute force mean that is dropping out several consecutive transformer layers once. In addition, a simple way to estimate the size of BERT before fine-tuning is provided by us. In this way, the resources are saved tremendously without losing too much accuracy. More effectively, some simple task could be solved only use one transformer layer sometimes. Currently, there is a little work studying the overparameterization problem of state-of-the-art networks based on language models. And researchers still fail to comprehend and explain these complex and massive deep neural networks. Also, there is little work solving the resource-consuming problem well. However, these problems should be attached importance to and solved well before the development of neural models reaching a higher level. Now we throw away a brick in order to attract more researchers to pay attention to then solve these problems.
In future, we will extend these experiments to other more complex NLP tasks and other language models, and explore more effective ways to find winning tickets in language models. Also, we are eager to find out more effective ways to solve these problems better.