{'fraction_non_alphanumeric': 0.042060854001534136, 'fraction_numerical': 0.018345691638967015, 'mean_word_length': 4.619612068965517, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
When does Further Pre-training MLM Help? An Empirical Study on Task-Oriented Dialog Pre-training

Further pre-training language models on in-domain data (domain-adaptive pre-training, DAPT) or task-relevant data (task-adaptive pre-training, TAPT) before fine-tuning has been shown to improve downstream tasks’ performances. However, in task-oriented dialog modeling, we observe that further pre-training MLM does not always boost the performance on a downstream task. We find that DAPT is beneficial in the low-resource setting, but as the fine-tuning data size grows, DAPT becomes less beneficial or even useless, and scaling the size of DAPT data does not help. Through Representational Similarity Analysis, we conclude that more data for fine-tuning yields greater change of the model’s representations and thus reduces the influence of initialization.


Introduction
Pre-trained models such as BERT (Devlin et al., 2019) and GPT2 (Radford et al., 2019) have been used in a wide range of NLP tasks and achieved superior performance. These models usually follow the pre-train and fine-tune paradigm, which adopts unsupervised pre-training on large-scale corpora and supervised fine-tuning for downstream task adaption. However, the pre-training corpora are in the general domain, while the data of downstream tasks fall in more task-specific domains.
To bridge the data distribution gap, further pretraining has been applied and shows consistent improvements (Sun et al., 2019). According to the training data used in this process, Gururangan et al. (2020) termed domain-adaptive pre-training (DAPT), which uses the data in the same domain of the target task and task-adaptive pre-training (TAPT), which uses much less unlabeled training data from the target task than DAPT. They found that DAPT masked LM leads to performance gains under both high-and low-resource settings and TAPT is beneficial with or without DAPT. DAPT has shown effectiveness for task-oriented dialog modeling. Wu et al. (2020) further pretrained BERT on 9 task-oriented dialog corpora and outperformed BERT on four downstream tasks, especially in the few-shot setting. Gu et al. (2020) further pre-trained GPT-2 on 13 dialog corpora ranging from chitchats to task-oriented dialogs, leading to better results on three task-oriented datasets.
However, does further pre-training always help? Mehri et al. (2020) performed DAPT on 700M opendomain dialogs and TAPT, but the resulting model only outperforms BERT in 4 out of 7 task-oriented dialog datasets. We also observe that replacing BERT with TOD- BERT-mlm (Wu et al., 2020) that is further pre-trained MLM on 101K task-oriented dialogs does not always bring a significant difference on downstream tasks. So far, however, there has been little discussion about when and why further pre-training on in-domain data can boost the performance on a downstream task and how the DAPT data size can affect this.
In this paper, we conduct an empirical study on the effect of further pre-training BERT BASE on task-oriented dialogs. Our experiments are organized around the following research questions: • RQ1 When can DAPT improve the performance on a downstream task? • RQ2 How does the amount of data for DAPT affect the performance on a downstream task? We evaluate further pre-trained models on five downstream tasks involving seven task-oriented dialog datasets. Our main findings are summarized as follows: (1) DAPT and TAPT do not always improve fine-tuning performance: the effect varies for different tasks, models, and fine-tuning data sizes. (2) DAPT is more beneficial in the low-resource setting. As the fine-tuning data size grows, the model's representations change more, implying that the in-fluence of pre-training decays, thus the benefit of DAPT decreases or even vanishes. (3) Increasing the amount of data for DAPT mostly improves the performance in the relative low-resource setting.

Further Pre-training
We further pre-train BERT BASE uncased model using masked language modeling loss with 15% tokens masked. Our DAPT dataset consists of several multi-turn task-oriented dialog datasets, including Schema (Rastogi et al., 2020), Taskmaster-1&2 (Byrne et al., 2019, MetaLWOZ (Li et al., 2020), MSR-E2E (Li et al., 2018), SMD (Eric et al., 2017), Frames (El Asri et al., 2017, WOZ (Mrkšić et al., 2017), andCamrest (Wen et al., 2017), which has 103K dialogs (13M words) in total. To investigate RQ2, we also use 25%, 5% and 1% dialogs to perform DAPT. For TAPT, we use the training set of each downstream task. We use 95% dialogs for training and select the best checkpoint with the lowest MLM loss on the other 5% dialogs. To obtain a training sample D 1:t = {U 1 , S 1 , ..., U t } where U i , S i are user's utterance and system's utterance respectively, we randomly pick a dialog D and sample a turn t ∈ [1, T ] uniformly, where T is the length of D. Then all the utterances are concatenated into a sequence as the model input: and [SYS] are two special tokens prepended to user's and system's utterances respectively. See Appendix A for the hyper-parameter setting.

Evaluation
We conduct comprehensive evaluations on 5 downstream tasks. Models on these tasks are adapted from TOD-BERT (Wu et al., 2020), DialoGLUE (Mehri et al., 2020), or ConvLab-2 (Zhu et al., 2020. See Appendix B for fine-tuning details. Intent Classification (IC) is a sequence classification problem, where models take an utterance as input and predict its intent. We use three datasets: HWU (Liu et al., 2019) that has 64 intents and 26K utterances, BANKING (Casanueva et al., 2020) that has 77 intents and 13K utterances, and OOS (Larson et al., 2019) that has 151 intents and 24K utterances. We pass the representation of [CLS] token to a linear layer for prediction.
Slot Filling (SF) requires models to extract slots' values in an utterance, which is often formulated as a sequence tagging problem. We use REST8K dataset (Coope et al., 2020) that has 5 slots and 8K utterances. We add a linear layer on the top of the tokens' representations to predict BIO format tags.
Semantic Parsing (SP) aims at identifying both intents and slots' values in an utterance. We use TOP dataset (Gupta et al., 2018) that has 45K utterances spanning 25 intents and 36 slots and MultiWOZ 2.3 dataset (Han et al., 2020) that has 10K dialogs and 143K utterances spanning 7 domains, 13 intents, and 25 slots. We use two linear layers to predict intent and tokens' tags respectively.
Dialog State Tracking (DST) is the task of recognizing user constraints throughout the conversation. We use MultiWOZ dataset version 2.1 (Eric et al., 2020) that has 30 domain-slot pairs to track. We adopt two BERT-based models: TripPy (Heck et al., 2020) and TOD-DST (Wu et al., 2020). Both models use BERT to encode dialog history.
Dialog Act Prediction (DAP) is a multi-label sequence classification problem, where models predict the intents of the system response given the dialog history. We use two datasets: MultiWOZ and GSIM (Shah et al., 2018) that contains 6 intents and 3K dialogs. For each intent, we feed the representation of [CLS] token to a linear layer and predict whether the intent is in the response.
As for evaluation metrics, we use accuracy for intent prediction, macro-F1 for slot filling and dialog act prediction, exact-match for semantic parsing, and joint goal accuracy for dialog state tracking.

Representational Similarity Analysis
Representational similarity analysis (RSA) is a technique to measure the similarity between models' representations (Laakso and Cottrell, 2000). Following Merchant et al. (2020), we encode samples from a test dataset and randomly select the same n = 5000 tokens as stimuli, whose contextual representations at each layer are used to compute an n × n pairwise cosine similarity matrix. The final similarity score between two models' representations at a certain layer is computed as the Pearson correlation between the flattened upper triangular of the two similarity matrices.

Full Data Experiments
We fine-tune BERT, TOD-BERT-mlm (Wu et al., 2020) that is further pre-trained on 9 task-oriented  Table 1: Performance on downstream tasks. We report the means and standard deviations across three random seeds for BERT. Note that TOD-BERT-mlm has pre-trained on MultiWOZ dataset. A task is in red if further pretraining all outperform BERT by at least one standard deviation. The best task performances are boldfaced.
datasets including MultiWOZ, and our further pretrained models on downstream tasks using the same hyper-parameters. The results are shown in Table 1.
To measure the performance variance, we fine-tune BERT three times using different random seeds and report means and standard deviations. We evaluate the effect of DAPT, denoted by ∆ DAPT , through averaging the improvements of the models with different DAPT data sizes (100%, 25%, 5%, 1%) compared with BERT. Similarly, ∆ TAPT is the benefit of TAPT. We find that ∆ DAPT and ∆ TAPT sometimes are small or even negative, indicating that DAPT and TAPT does not always improve performances on downstream tasks. Consistently, TOD-BERTmlm does not always outperform BERT significantly. Only on tasks in red in Table 1, the benefits of further pre-training are larger than the standard deviations of BERT. In some cases, interestingly, further pre-training can lead to inferior performance. Even on the same MultiWOZ dataset, further pre-training effects vary according to the model architectures used (TripPy, TOD-DST) and the downstream tasks (SP, DST, DAP). Compared with DAPT and DAPT+TAPT, TAPT obtains similar results but requires much lower training cost, which is worth trying before DAPT.
3.2 RQ1: When can DAPT improve the performance on a downstream task?
Since further pre-training does not help in some cases, we want to explore when DAPT can improve the performance on a downstream task. We first show that DAPT does improve the model's LM ability on downstream datasets (Figure 1), and TAPT can more efficiently improve this ability. This means that further pre-training does reduce the data distribution gap (for LM) between pre-training and fine-tuning but does not guarantee task performance improvement. A possible hypothesis is that further pretraining encodes shallow domain knowledge that has obvious influence only when there are insufficient labeled data providing taskspecific knowledge for fine-tuning. By further pre-training, a model learns the co-occurrence of words and their context, which can be viewed as a kind of statistics feature of the target domain. When the fine-tuning data are deficient, this general domain knowledge can alleviate the lack of taskspecific knowledge. However, models can learn to encode task-specific knowledge directly through fine-tuning when there are sufficient labeled data and thus rely less on further pre-training.
To verify the hypothesis, we use RSA to assess the representation similarity between fine-tuned models and their initializations for different finetuning data sizes. As illustrated in Figure 2, for both BERT and the full data DAPT model, the RSA similarity decreases as the fine-tuning data size grows, especially on the top layers. We ob-  Table 2: Comparison of full and 10% data fine-tuning. We use the same 10% data and average the performance of 3 runs using different random seeds. The RSA similarity averaged across layers is between full data DAPT model and its fine-tuned counterpart. We also report which DAPT data size performs best as "best DAPT".
serve similar trends for all tasks, supporting that less fine-tuning data highlights the importance of pre-training. We also fine-tune the models with and without DAPT in the low-resource setting. Table 2 compares the average performance gain of DAPT (∆ DAPT ) and RSA similarity of full data DAPT model and its fine-tuned counterpart (averaged across layers) with full data and low-resource fine-tuning. In the low-resource setting, RSA similarity increases, and DAPT is more beneficial in most cases, which means the knowledge learned through DAPT is more useful when the fine-tuning data are deficient.
3.3 RQ2: How does the amount of data for DAPT affect the performance on a downstream task?
We have shown that models can encode general domain knowledge after DAPT and improve the per- Figure 3: RSA on OOS test set between BERT and further pre-training: 1%, 5%, 25%, and 100% data DAPT, TAPT, and TAPT after full data DAPT. formance on a downstream task in the low-resource setting. However, how much gain can we obtain by enlarging the DAPT data? To investigate this question, we perform DAPT with 1%, 5%, 25%, and 100% data, ranging from 1020 dialogs to 102K dialogs. From Figure 1, we can see that the more data used in DAPT, the stronger language model on downstream datasets we can get. We also show the change of the model's representation caused by further pre-training in Figure 3. Like fine-tuning, using more data for DAPT brings greater change, and the trend is similar for all tasks. It is also worth noting that compared with DAPT, TAPT changes the model more efficiently in the target dataset. However, change brought by enlarging DAPT data does not guarantee performance improvement. We compare how much data the best DAPT model used in both full data and low-resource finetuning. As shown in Table 2, including more data for DAPT may not always improve downstream task performance. Nevertheless, when there are less fine-tuning data, the best model needs more data for DAPT.

Conclusion
In this work, we conduct an empirical study to investigate the effect of further pre-training MLM on task-oriented dialogs. Different from earlier findings (Sun et al., 2019; Gururangan et al., 2020), neither DAPT nor TAPT always improves performances on downstream tasks in our experiments. In the low-resource setting, however, DAPT is more helpful, and the size of DAPT data needed to perform best increases. Through RSA, we find that as the fine-tuning data grows, the impact of model initialization fades away, which could be the explanation. We also show that although further pre-training can improve the model's LM ability on downstream datasets, this may not contribute much to downstream tasks under pre-train and fine-tune paradigm, calling for novel pre-training objectives and effective ways to use pre-trained models. ward a realistic and diverse dialog dataset.

A.1 Dataset Description
In this section, we present a detailed description of the data we use for further pre-training.
Domain Adaptive Pre-Training (DAPT): We use the pure text of several multi-turn task-oriented dialog datasets for DAPT, including Schema 2 (Rastogi et al., 2020), Taskmaster Table 3 shows the corpus statistics of each dataset. Note that for Schema, we only use its train set as the corpus, and for other datasets, we combine their train, dev, and test sets. For validation, to evaluate the pre-training performance on each corpus separately, we split 5% of the dialogs from each corpus and compute masked language modeling losses on them respectively. For DAPT, we merge the other 95% of each corpus. To reduce the gap between pre-training and fine-tuning, we remove system side utterances at the beginning and the end in each dialog to ensure that the first sentence and the last sentence of each dialog are both from the user side.
Task Adaptive Pre-Training (TAPT): We use the pure text of each downstream task dataset for TAPT and delete the system side utterances at the beginning and the end of each dialog. Similar to DAPT, we use 95% dialogs for training and 5% for validation.