{'fraction_non_alphanumeric': 0.042530203319909636, 'fraction_numerical': 0.022853026880136202, 'mean_word_length': 4.665739194954554, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues

The use of pre-trained language models has emerged as a promising direction for improving dialogue systems. However, the underlying difference of linguistic patterns between conversational data and general text makes the existing pre-trained language models not as effective as they have been shown to be. Recently, there are some pre-training approaches based on open-domain dialogues, leveraging large-scale social media data such as Twitter or Reddit. Pre-training for task-oriented dialogues, on the other hand, is rarely discussed because of the long-standing and crucial data scarcity problem. In this work, we combine nine English-based, human-human, multi-turn and publicly available task-oriented dialogue datasets to conduct language model pre-training. The experimental results show that our pre-trained task-oriented dialogue BERT (ToD-BERT) surpasses BERT and other strong baselines in four downstream task-oriented dialogue applications, including intention detection, dialogue state tracking, dialogue act prediction, and response selection. Moreover, in the simulated limited data experiments, we show that ToD-BERT has stronger few-shot capacity that can mitigate the data scarcity problem in task-oriented dialogues.


Introduction
Recent advances in pre-training using self-attention encoder architectures (Devlin et al., 2018;Lan et al., 2019) have been commonly used in many NLP applications. Such models are usually constructed based on a massive scale of general text corpora, such as English Wikipedia or books (Zhu et al., 2015). The distributed representations are learned self-supervised from the raw text. By further fine-tuning these representations, breakthroughs have been continuously reported for various downstream tasks, especially those of natural language understanding.
However, previous work (Rashkin et al., 2018;Wolf et al., 2019) shows that there are some deficiencies in the performance to directly apply finetuning on conversational corpora. One possible reason could be the intrinsic difference of linguistic patterns between human conversations and writing text, resulting in a large gap of data distributions (Bao et al., 2019). Therefore, pre-training dialogue language models using chit-chat conversational corpora from social media, such as Twitter or Reddit, has been recently investigated, especially for dialogue response generation (Zhang et al., 2019) and retrieval (Henderson et al., 2019b) tasks. Although these open-domain dialogues are diverse and easyto-get, they are usually short, noisy and without specific chatting goals.
Task-oriented dialogues, on the other hand, have explicit goals (e.g. restaurant reservation or ticket booking) and many conversational interactions. But each of these datasets is usually small and scattered since obtaining and labeling such data is difficult and expensive. Moreover, task-oriented dialogues have clear user and system behaviors where the user has his/her goal and the system has its belief and database information, which makes the language understanding component and dialogue policy learning more essential than those chit-chat scenarios.
In this paper, we aim to prove this hypothesis: self-supervised language model pre-training using task-oriented corpora can learn better representations than existing pre-trained models for those task-oriented downstream tasks. We emphasize that what we care the most is not whether our pretrained model can achieve state-of-the-art results on each downstream task, since most of the current best models are built on top of pre-trained models, which can be easily replaced by ours. In our ex-periments, we avoid adding too many additional components on top of pre-training architectures when fine-tuning on each downstream task, and simply rely on the learned representations to show the full strength of a pre-trained model.
We collect and combine nine English-based, human-human, multi-turn, and publicly available task-oriented dialogue corpora to train a taskoriented dialogue BERT (ToD-BERT). In total, there are around 100k dialogues with 1.4M utterances across 60 different domains. Like BERT (Devlin et al., 2018), ToD-BERT is formulated as a masked language model, and uses the deep bidirectional Transformer (Vaswani et al., 2017) encoder as its model architecture. Unlike BERT, ToD-BERT incorporates response selection task into pretraining and adds two special tokens for user and system to model the corresponding behavior. We select BERT architecture simply because it is the most widely used model in NLP research recently. Note that the unified datasets we combine can be easily applied to pre-train any existing language models.
We test ToD-BERT on four common downstream tasks of task-oriented dialogue systems, including intention detection, dialogue state tracking, dialogue act prediction, and response selection. This is what we observe: ToD-BERT outperforms BERT and other strong baselines, including GPT-2 (Radford et al., b) and DialoGPT (Zhang et al., 2019), on all the selected downstream tasks, which further confirms its effectiveness for improving dialogue language understanding. More importantly, ToD-BERT has stronger few-shot capacity than BERT on each task, implying that it can reduce the need for expensive human-annotated labels in the future. ToD-BERT can be easily leveraged and adapted to new task-oriented dialogue datasets, especially those with few training examples. Our source code will be released soon to facilitate future research. 1 .

Related Work
General Pre-trained Language Models, which are trained on massive general text such as Wikipedia and BookCorpus, can be roughly divided into two categories: uni-directional or bidirectional attention mechanisms. GPT (Radford et al., a) and b) are representatives of uni-directional language models using a Transformer decoder, where the objective is to maximize left-to-right generation likelihood. These models are commonly applied in natural language generation tasks. On the other hand, BERT (Devlin et al., 2018), RoBERTa  and their variances are pre-trained using a Transformer encoder with bi-directional token prediction. These models are usually evaluated on classification tasks such as GLUE benchmark (Wang et al., 2018) or span-based question answering tasks (Rajpurkar et al., 2016).
Some language models can support both unidirectional and bi-directional attention, such as UniLM (Dong et al., 2019). Conditional language model pre-training is also proposed, for example, CTRL (Keskar et al., 2019) is a conditional Transformer model, trained to condition on control codes that govern style, content, and task-specific behavior. Recently, multi-task language model pretraining with unified sequence-to-sequence generation is proposed. Text-to-text Transformer (T5) (Raffel et al., 2019) unifies multiple text modeling tasks, and achieves the promising results in various NLP benchmarks.
Dialogue Pre-trained Language Models are mostly trained on open-domain conversational data from Reddit or Twitter for dialogue response generation. Transfertransfo (Wolf et al., 2019) achieves good performance on ConvAI-2 dialogue competition using GPT-2. DialoGPT (Zhang et al., 2019) is an extension of GPT-2 that is pre-trained on Reddit data for open-domain response generation. Con-veRT (Henderson et al., 2019a) pre-trained a dual transformer encoder for response selection task on large-scale Reddit (input, response) pairs. PLATO (Bao et al., 2019) uses both Twitter and Reddit data to pre-trained a dialogue generation model with discrete latent variables. All of them are designed to cope with the response generation task for opendomain chatbots.
Pretraining for task-oriented dialogues, on the other hand, has few related works. Budzianowski and Vulić (2019) first apply the GPT-2 model to train on response generation task, which takes system belief, database result, and last dialogue turn as input to predict next system responses. It only use one dataset to train its model because few public datasets have database information available. Henderson et al. (2019b) pre-trained a response selection model for task-oriented dialogues. They first pre-train on Reddit corpora and then fine-tune Name # Dialogue # Utterance Avg. Turn # Domain MetaLWOZ  37,884 432,036 11.4 47 Schema (Rastogi et al., 2019) 22,825 463,284 20.3 17 Taskmaster (Byrne et al., 2019) 13,215 303,066 22.9 6 MWOZ (Budzianowski et al., 2018) 10,420 71,410 6.9 7 MSR-E2E  10,087 74,686 7.4 3 SMD (Eric and Manning, 2017) 3,031 15,928 5.3 3 Frames (Asri et al., 2017) 1,369 19,986 14.6 3 WOZ (Mrkšić et al., 2016) 1 on target dialogue domains, but their training and fine-tuning code is not released. Peng et al. (2020) focus on the natural language generation (NLG) task, which assumes dialogue acts and slot-tagging results are given to generate a natural language response. By pre-training on a set of annotated NLG corpora, it can improve conditional generation quality using a GPT-2 model.

Method
In this section, we first discuss each dataset used for our task-oriented pre-training and how we process the data. Then we introduce the selected pretraining base model and its objective functions.

Datasets
We collect nine different task-oriented datasets which are English-based, human-human, multiturn and publicly available. In total, there are 100,707 dialogues, which contain 1,388,152 utterances over 60 domains. Dataset statistics is shown in Table 1. • Taskmaster (Byrne et al., 2019): This dataset includes 13,215 dialogues comprising six domains, including 5,507 spoken and 7,708 written dialogs created with two distinct procedures.
One is a two-person Wizard of Oz approach that one person acts like a robot and the other is a self-dialogue approach in which crowdsourced workers wrote the entire dialog themselves. It has 22.9 average conversational turns in a single dialogue, which is the longest among all taskoriented datasets listed.
• MWOZ (Budzianowski et al., 2018): Multi-Domain Wizard-of-Oz dataset contains 10,420 dialogues over seven domains, and it has multiple domains in a single dialogue. It has a detailed description of the data collection procedure, and user goal, system act, and dialogue state labels. Different from most of the existing corpora, it also provides full database information.
• MSR-E2E : Microsoft end-toend dialogue challenge has 10,087 dialogues in three domains, movie-ticket booking, restaurant reservation, and taxi booking. It also includes an experiment platform with built-in simulators in each domain.
• SMD (Eric and Manning, 2017): Stanford multidomain dialogue is an in-car personal assistant dataset, comprising 3,301 dialogues and three domains: calendar scheduling, weather information retrieval, and point-of-interest navigation.
It is designed to smoothly interface with knowledge bases, where a knowledge snippet is attached with each dialogue as a piece of simplified database information.
• Frames (Asri et al., 2017): This dataset is composed of 1,369 human-human dialogues with an average of 14.6 turns per dialogue, where users are given some constraints to book a trip and assistants who search a database to find appropriate trips. Different from other datasets, it has labels to keep track of different semantic frames, which is the decision-making behavior of users, throughout each dialogue.
• WOZ (Mrkšić et al., 2016) and Cam-Rest676 (Wen et al., 2016): These two corpora use the same data collection procedure and same ontology from DSTC2 (Henderson et al., 2014). They are one of the first task-oriented dialogue datasets that use Wizard of Oz style with text input instead of speech input, which improves the models capacity for the semantic understanding instead of its robustness to automatic speech recognition errors.

ToD-BERT Model
We train our ToD-BERT based on BERT (Devlin et al., 2018) architecture using two loss functions: masked language modeling loss (MLM) and response selection loss (RSL). Note that the dataset we combine can be used to pre-train any existing language model architecture, and here we select BERT simply because it is the most widely used model in NLP research recently. We use the BERT-Base uncased model, which is a transformer selfattention encoder (Vaswani et al., 2017) with 12 layers and 12 attention heads with its hidden size d B = 768.
To capture speaker information and the underlying interaction behavior in dialogues, we add two special tokens, [USR] and [SYS], to the bytepair embeddings (Mrkšić et al., 2016). We prefix the special token to each user utterance and system response, and concatenate all the utterances in the same dialogue into one flat sequence, as shown in Figure 1. For example, for a dialogue D = {S 1 , U 1 , S 2 , U 2 , . . . , S n , U n }, where n is the number of dialogue turns and each S i or U i contains a sequence of words, the input of the pretraining model is processed as "[SYS] S 1 [USR] U 1 . . . " with standard positional embeddings and segmentation embeddings. Masked language modeling is a common pretraining strategy for BERT-like architectures, in which a random sample of the tokens in the input sequence is selected and replaced with the special token [MASK]. The MLM loss function is the cross-entropy loss on predicting the masked tokens. In the original implementation, random masking and replacement is performed once in the beginning and saved for the duration of the training, here we conduct token masking dynamically during batch training. ToD-BERT is initialized from BERT, a good starting parameter set, then is further pretrained on those task-oriented corpus mentioned above. The MLM loss function is where M is the total number of masked tokens and P (x m ) is the predicted probability of the token x m .
Response selection loss can also be used for dialogue language model pre-training since it does not require any additional human annotation. Pretraining with RSL can bring us several advantages: 1) we can learn a better representation for the [CLS] token, since it is essential for all the downstream tasks, and 2) we encourage the model to capture underlying dialogue sequential order and structure information. Different from the original next sentence prediction (NSP) objective, which concatenates two segments A and B together to predict whether A and B are consecutive text (binary classification), we apply the concept of dual encoder response selection (Henderson et al., 2019a) and simulate multiple negative samples. We first draw a batch of dialogues {D 1 , . . . , D b } and cut each dialogue at a randomly selected turn t. For example, D 1 will be separated into two segments, one is the context {S 1 1 , U 1 1 , . . . , S 1 t , U 1 t } and the other is the response {S 1 t+1 }. We use the ToD-BERT to separately encode the context and its corresponding response.
Afterwards, we have a context matrix C ∈ R b×d B and a response matrix R ∈ R b×d B by taking the embedding of their [CLS] tokens from the b dialogues. We simply treat other responses in the same batch as negative samples during pre-training. The RSL objective function is The overall pre-training loss function is the weighted-sum of L mlm and L rsl , and in our experiments we simply add them. We gradually reduce the learning rate without a warm-up period. We optimize ToD-BERT with AdamW (Loshchilov and Hutter, 2017) and train with a dropout of 0.1 on all layers and attention weights. GELU activation functions (Hendrycks and Gimpel, 2016) is used. Models are trained early-stop using perplexity of a held-out development set, with mini-batches containing 32 sequences of maximum length 512 tokens.

Downstream Tasks
We emphasize that what we care the most in this paper is: whether our ToD-BERT, a pre-trained language model using multiple task-oriented corpora, can show any advantage over BERT. Therefore, we try to avoid adding too many additional components on top of their architecture when fine-tuning on each downstream task and simply rely on their learned representations. Also, we always use the same architecture with the same number of parameters for a fair comparison. We select four common task-oriented downstream tasks to evaluate our pre-trained ToD-BERT: intent classification, dialogue state tracking, dialogue act prediction, and response selection. All of them are core components in modularized taskoriented systems. We briefly introduce them below: Intent classification task is a multi-class classification problem, where we input a sentence U and models predict one single intent class over M possible intents.
where F is a pre-trained language model that takes a sequence of tokens as input, and we use its [CLS] embeddings as the output representation. W 1 ∈ R d B ×M is a trainable linear mapping. The model is trained with cross-entropy loss between the predicted distributions P int and the true intent labels.
Dialogue state tracking task can be treated as a multi-class classification problem using a predefined ontology. Unlike intent classification, we input dialogue history X (a sequence of utterances, e.g., 6.9 average turns in MWOZ) and a model predicts values for each (domain, slot) pair at each dialogue turn. Each corresponding value v j i , the i-th value for the j-th (domain, slot) pair, is passed into a pre-trained model and fixed the representation during training. The number of slot projection layers |G j | is equal to the number of (domain, slot) pairs: where Sim is the cosine similarity function, and S j ∈ R |v j | is the probability distribution of the j-th (domain, slot) pair over its possible values. The model is trained with cross-entropy loss summed over all the (domain, slot) pairs.
Dialogue act prediction task is a multi-label classification problem because a system response may contain multiple dialogue acts, e.g., request and inform users at the same time. Model take dialogue history as input and predict a binary result for each possible dialogue act: where W 2 ∈ R d B ×N is a trainable linear mapping, N is the number of possible dialogue acts, and each value in A is between [0, 1] after a Sigmoid layer. The model is trained with binary cross-entropy loss and the i-th dialogue act is considered as a triggered dialogue act if A i > 0.5.
Response selection task is a ranking problem , aiming to retrieve the most relative system response from a candidate pool. We use dual encoder strategy (Henderson et al., 2019b) and compute similarity scores between source X and target Y , where Y i is the i-th response candidate and r i is its cosine similarity score. Source X can be truncated and we limit the context lengths to the most recent 256 tokens in our experiments. We randomly sample several system responses from the corpus as negative samples. Although it may not be a true negative sample, it is a common way to train a ranker and evaluate its results (Henderson et al., 2019a).
We pick up several datasets, OOS, DSTC2, GSIM, and MWOZ, for downstream tasks evaluation. The first three corpora are not included in the pretrained task-oriented datasets. For MWOZ, to be fair, we do not include its test set dialogues during the pre-training stage. Details of each evaluation dataset are discussed in the following: • OOS (Larson et al., 2019): The out-of-scope intent dataset is one of the largest annotated intent datasets, including 15,100/3,100/5,500 samples for the train, validation, and test sets, respectively. It covers 151 intent classes over 10 domains, including 150 in-scope intent and 1 outof-scope intent. The out-of-scope intent means that a user utterance that does not fall into any of the predefined intents. Each of the intents has 100 training samples. We use this dataset to evaluate the performance of the intent classification task.
• DSTC2 (Henderson et al., 2014): DSTC2 is a human-machine task-oriented dataset, which has 1,612/506/1117 dialogues for train, validation, and test sets, respectively. We follow  to map the original dialogue act labels to universal dialogue acts, which results in 19 different system dialogue acts. We use this dataset to evaluate the performance of the dialogue act prediction and response selection tasks.
• GSIM (Shah et al., 2018a): GSIM is a humanrewrote machine-machine task-oriented corpus, including 1500/469/1039 dialogues for the train, validation, and test sets, respectively. We combine its two domains, movie and restaurant domains, into one single corpus. It is collected by Machines Talking To Machines (M2M) (Shah et al., 2018b) approach, a functionality-driven process combining a dialogue self-play step and a crowd-sourcing step. We map its dialogue act labels to universal dialogue acts , resulting in 13 different system dialogue acts. We use this dataset to evaluate the performance of the dialogue act prediction and response selection tasks.
• MWOZ (Budzianowski et al., 2018): MWOZ is the most common benchmark for task-oriented dialogues, especially for dialogue state tracking. It has 8420/1000/1000 dialogues for train, validation, and test sets, respectively. Across seven different domains, in total it has 30 (domain, slot) pairs that need to be tracked in the test set. We use its revised version MWOZ 2.1 from Eric et al. (2019), which has the same dialogue transcripts but with cleaner state label annotation. We use this dataset to evaluate the performance of dialogue state tracking, dialogue act prediction, and response selection tasks.

Results
For each downstream task, we first conduct the experiments using the whole dataset, then we simulate the few-shot setting to show the strength of our ToD-BERT. We run at least three times with different random seeds for each few-shot experiment to reduce the variance of data sampling, and we report its mean and standard deviation for these limited data scenarios. We investigate two versions of ToD-BERT, one is ToD-BERT-mlm that only uses MLM loss during pretraining, and the other is ToD-BERTjnt which is jointly trained with the MLM and RSL objectives. We compare ToD-BERT with BERT and other baselines, including two other strong pretraining models GPT-2 (Radford et al., b) and DialoGPT (Zhang et al., 2019).

Intent Classification
ToD-BERT outperforms BERT and other strong baselines in one of the largest intent classification datasets, as shown in Table 2. We evaluate accuracy on all the data, only the in-domain intents, and only the out-of-scope intent. Note that there are two ways to predict out-of-scope intent, one is to treat it as an additional class, and the other is to set a threshold for prediction confidence. Here we report the results of the first setting. ToD-BERT-jnt achieves 86.6% accuracy over the 151 intent classes, 96.2% accuracy over the defined 150 intent classes, and has 89.9% accuracy of the out-of-scope intent. Besides, we conduct 1-shot and 10-shot experiments by randomly sampling one and ten utterances from each intent class in the training set. To reduce the variance data sampling, the numbers reported are averaged over three runs. ToD-BERT-jnt has 6.2% all-domain accuracy improvement and 7.6% in-domain accuracy improvement compared with BERT for the 1-shot setting. We found that the results of ToD-BERTjnt are not consistently better than ToD-BERT-mlm. One possible reason could be that the input of intent classification task is only a single utterance without Model Acc (all) Acc (in) Acc (out) Recall (out)

Full Data
dialogue context, and two utterances in different domains could have similar system responses, which makes the RSL objective sub-optimal.

Dialogue State Tracking
Two evaluation metrics are commonly used in dialogue state tracking task, joint goal accuracy and slot accuracy. The joint goal accuracy compares the predicted dialogue states to the ground truth at each dialogue turn, where the ground truth includes slot values for all the possible (domain, slot) pairs.
The output is considered as a correct prediction if and only if all the predicted values exactly match its ground truth values. The slot accuracy, on the other hand, individually compares each (domain, slot, value) triplet to its ground truth label.
In Table 3, we first compare BERT with ToD-BERT on the MWOZ dataset (the 2.1 version) and find the latter has 2.4% joint goal accuracy improvement. Since the original ontology provided by Budzianowski et al. (2018) is not complete (some labeled values are not included in the ontology), we create a new ontology of all the possible annotated values. We also list several wellknown dialogue state trackers as reference, including DSTReader , HyST , TRADE , and ZS-DST (Rastogi et al., 2019). ToD-BERT outperforms DSTReader, HyST, and TRADE by 11.6%, 9.9% and 2.4% joint goal accuracy, respectively.
We also report the few-shot experiments using 1%, 5%, 10% and 25% data for dialogue state tracking. Note that 1% of data has around 84 dialogues. Each result shown is averaged over three different runs. ToD-BERT outperforms BERT in all the setting, which further show the strength of task-oriented dialogue pre-training. ToD-BERT surpasses BERT by 2.7%, 2.2%, 5.9%, 3.5% in 1%, 5%, 10%, and 25% data setting, respectively.

Dialogue Act Prediction
We conduct experiments on three different datasets and report micro-F1 and macro-F1 scores for the dialogue act prediction task, a multi-label classifica-   tion problem. For the MWOZ dataset, we remove the domain information from the original system dialogue act labels, for example, the "taxi-inform" will be simplified to "inform". This process reduces the number of possible dialogue acts from 31 to 13. For DSTC2 and GSIM corpora, we follow  to apply universal dialogue act mapping that maps the original dialogue act labels to a general dialogue act format, resulting in 19 and 13 system dialogue acts in DSTC2 and GSIM, respectively.
We run two other baselines, MLP and RNN, to further show the strengths of BERT-based models. The MLP model simply takes bag-of-word embeddings to make dialogue act prediction, and the RNN model is a bi-directional GRU network. In Table 4, one can observe that in full data setting, ToD-BERT consistently works better than BERT and other baselines, no matter which datasets or which evaluation metrics.
In the few-shot experiments, we run three times and report the results. ToD-BERT-mlm outper-forms BERT by 3.5% micro-F1 and 6.6% macro-F1 on MWOZ corpus in the 1% data scenario. We also found that when 10% of training data can achieve good performance that is close to the full data training.

Response Selection
To evaluate response selection in task-oriented dialogues, we follow the k-to-100 accuracy, which is becoming a research community standard (Yang et al., 2018;Henderson et al., 2019a). The k-of-100 ranking accuracy is a Recall@k metric, which indicates whether the relevant response occurs in the top k ranked candidate responses. The k-of-100 metric is computed using random batches of 100 examples so that the responses from other examples in the batch are used as random negative candidates. This allows efficient computing the metric across many examples in batches. While it is not guaranteed that the random negatives will indeed be "true" negatives, the 1-of-100 metric still provides a useful evaluation signal that correlates with downstream tasks. We run five different ran-dom seeds to sample random batches and report the average results.
In Table 5, we conduct response selection experiments on three datasets, MWOZ, DSTC2, and GSIM. ToD-BERT-jnt achieves 65.8% 1-to-100 accuracy and 87.0% 3-to-100 accuracy on MWOZ, which surpasses BERT by 18.3% and 11.5%, respectively. The advantage of the ToD-BERT-jnt is more obvious under the few-shot scenario. In the 1% data setting, ToD-BERT-jnt has around 20% 1-to-100 accuracy improvement and 22% 3-to-100 accuracy improvement on DSTC2. We do not report ToD-BERT-jnt for MWOZ few-shot setting because it is not fair to compare them with others as the full MWOZ training set is used during pretraining. 2 We found that the response selection results are sensitive to the training batch size since the larger the batch size the harder the prediction. In our experiments, we set batch size equals to 25.

Visualization
In Figure 2, we visualize the embeddings of BERT and ToD-BERT given the same input, the utterances in the test set of MWOZ. Each sample point is an utterance representation, which is passed through a pretrained model and reduced its highdimension features to a two-dimension point using the t-distributed stochastic neighbor embedding (tSNE) method. Since we know the true domain label for each utterance, we use different colors to represent different domains. As one can observe, ToD-BERT has more clear group boundaries than BERT.
To analyze the results quantitatively, we run Kmeans, a common unsupervised clustering algorithms, on top of the output embeddings of BERT and ToD-BERT. We set K for K-means equal to 10 and 20. After the clustering, we can assign each utterance in the MWOZ test set to a predicted class. We then compute the normalized mutual information (NMI) between the clustering result and the true domain label for each utterance. Here is what we observe: ToD-BERT consistently achieves higher NMI scores than BERT. For K=10, ToD-BERT has a 0.143 NMI score and BERT only has 0.094. For K=20, ToD-BERT achieves a 0.213 NMI score while BERT has 0.109. 2 The pre-trained ConveRT model (Henderson et al., 2019a) achieves 5.2% ± 0.1% 1-to-100 accuracy and 10.4% ± 0.2% 3-to-100 accuracy on MWOZ. Since they only released code for model inference, we report their results without finetuning.

Conclusion
We propose task-oriented dialogue BERT (ToD-BERT) that is trained on nine English-based, human-human, multi-turn and publicly available task-oriented datasets across over 60 domains. ToD-BERT outperforms BERT on four dialogue downstream tasks, including intention classification, dialogue state tracking, dialogue act prediction, and response selection. It also has clear advantage in the few-shot experiments than limited labeled data is available. ToD-BERT is easy-todeploy and will be open-sourced, allowing the NLP research community to apply or fine-tune on any task-oriented conversational problem. Lastly, the nine task-oriented datasets we combined can be leveraged to train and test any other latest pretrained architectures in the future.