{'fraction_non_alphanumeric': 0.0359930467977777, 'fraction_numerical': 0.01363372984764307, 'mean_word_length': 4.672853828306264, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enhancing Dialogue Generation via Multi-Level Contrastive Learning

Most of the existing works for dialogue generation are data-driven models trained directly on corpora crawled from websites. They mainly focus on improving the model architecture to produce better responses but pay little attention to considering the quality of the training data contrastively. In this paper, we propose a multi-level contrastive learning paradigm to model the fine-grained quality of the responses with respect to the query. A Rank-aware Calibration (RC) network is designed to construct the multi-level contrastive optimization objectives. Since these objectives are calculated based on the sentence level, which may erroneously encourage/suppress the generation of uninformative/informative words. To tackle this incidental issue, on one hand, we design an exquisite token-level strategy for estimating the instance loss more accurately. On the other hand, we build a Knowledge Inference (KI) component to capture the keyword knowledge from the reference during training and exploit such information to encourage the generation of informative words. We evaluate the proposed model on a carefully annotated dialogue dataset and the results suggest that our model can generate more relevant and diverse responses compared to the baseline models.


Introduction
Response generation for dialogue systems has stimulated great interests for researchers recently [Ritter et al., 2011;Shang et al., 2015;Vinyals and Le, 2015]. The core idea of dialogue generation is to formulate the task as a sequence translation problem and translate the query to a response. One common neural model is the sequence-to-sequence (S2S) encoder-decoder framework [Cho et al., 2014;Sutskever et al., 2014;Bahdanau et al., 2015]. Many approaches have been proposed to improve the basic S2S model for better human-computer conversation performance Xing et al., 2017;Shao et al., 2017;Xu et al., 2017;Pei and Li, 2018;Wang et al., 2018; * This work was done when Xin Li was an intern as Tencent AI Lab.   Gao et al., 2019a;He and Glass, 2020;Tian et al., 2020].
Despite their popularity, these approaches assume that each training sample, namely, query-response pair, contributes equally to the model and ignore the consideration of different response quality contrastively. Table 1 depicts some example responses for a particular query in a dialogue dataset. Both of the first and the second response are relevant to the query but the first one is obviously better when considering informativeness and interestingness. The third response is acceptable for the conversation but quite universal, meaning that, it can also be used to answer other queries. Thus, its quality is not as good as that of the first two responses. The fourth response is poor since it directly copies part of the query. Although the fourth response is not acceptable, it is still better when compared with the fifth response, which is completely irrelevant to the query. Some initial attempts [Shang et al., 2018;Liu et al., 2018] have been conducted to consider the quality of the training data. Following the idea of instance weighting [Jiang and Zhai, 2007], [Shang et al., 2018] pre-train a calibration network to calculate the response quality score for each training sample (i.e., query-response pair) and update the model with the weighted combination of the sample loss. Similarly, [Liu et al., 2018] estimate the instance score based on the corpus-level n-gram co-occurrence and the length of the response. Both of them are simple to implement but they still have some limitations: (1) The calibration network is only trained on relevant responses and irrelevant responses from other queries and therefore cannot capture the fine-grained response quality, as exemplified in Table 1; (2) The instance weighting strategy treats all tokens in the response as equal importance to the query by assigning them with the same quality score, which may erroneously encourage the generation of some uninformative words in the relevant responses (e.g., "since" and "afraid" in the first response in Table 1).
To tackle the issues mentioned above, we introduce the Contrastive Learning paradigm [Hadsell et al., 2006;He et al., 2019;Iter et al., 2020] to model the multi-level fine-grained quality of the responses with respect to the query. Specifically, we develop a Rank-aware Calibration (RC) network aiming for modeling the fine-grained quality and characterizing the response properties (e.g., relevance and informativeness) that will affect the conversation experience with a multi-scale response quality score. The rank-aware calibrator adopts the strategies of pointwise regression and pairwise ranking for gauging the quality of the query-response pair. Besides, to address the second limitation aforementioned, we design a more exquisite strategy to consider the different importance of tokens instead of simply scaling the training sample loss with the response-level quality score. Concretely, we propose to conditionally sample a response via Monte-Carlo Rollout [Yu et al., 2017; for each gold standard response token and deem the quality scores of the sampled responses as the importance of the tokens in the sample loss estimation.
It is also observed that some meaningful words such as "university" and "martyrs cemetery" in the fifth response in Table 1 are very likely to receive low quality scores due to the irrelevance to the query. Thus, we propose Knowledge Inference (KI) component to explicitly encourage the generation of the informative tokens in the gold standard responses. This component firstly associates the query and the decoder hidden representation with the memories of the informative tokens and then incorporates the summarized memories into each decoding step.
In summary, our contributions are as follows: • To enhance the performance of dialogue generation, we propose a multi-level contrastive learning paradigm to model the fine-grained quality of the responses with respect to the query.
• We propose a Rank-aware Calibration (RC) network to construct the multi-level contrastive objectives. We further design a strategy to calibrate the model training with tokenlevel quality information.
• We propose to reconsider the generation of some informative words erroneously punished by the calibrator via a tailor-made Knowledge Inference (KI) component.
• We build a dataset with fine-grained response annotations and conduct extensive evaluations. The experimental results validate the effectiveness of the proposed framework.
Code and the labelled dataset will be public to facilitate the research.

Overview
Given a query-response pair (X, Y), where the query word sequence X = {x 1 , x 2 , · · · , x n } and the response word se-quence Y = {y 1 , y 2 , · · · , y m }. The aim of the dialogue generation task is to maximize the conditional probability p(Y|X) and generate the relevant and meaningful responses with respect to the query.
As shown in Figure 1, the model proposed in this paper consist of three major components: the backbone S2S model, a rank-aware calibration network (RC) for response quality estimation, and a knowledge inference component (KI) for encouraging the generation of informative words. For RCbased response quality estimation, we apply Monte-Carlo Rollout with the policy network G β to perform response sampling conditioned on the current token y t and feed the sampled responseŶ t ∈ R m to the RC, where three contrastive quality estimation components D ns , D pt and D pair , namely, negative sampling component, pointwise ranking component, and pairwise ranking component are included. The produced quality scores of theŶ t are q ns , q pt , q pair respectively. For knowledge inference (KI), a knowledge memory K ∈ R n k ×dim k and a knowledge attention layer are designed. n k refers to the number of the knowledge memories for the informative tokens. The knowledge summary c k t will also serve as the input for the t-th decoding step.

Response Generation
The backbone generation model is an attention-based sequence-to-sequence framework. Specifically, in encoding, we employ a bi-directional LSTM [Hochreiter and Schmidhuber, 1997] to map the query input x i to the distributed hidden representation h i ∈ R 2dim h (i ∈ [1, n]) as follows: (1) where the operation of an LSTM unit on x t is denoted as LSTM(x i , h i−1 ). In decoding, the computational formula of the decoding hidden representation s t at the t-th time step is below: (2) The context vector c t is computed as the weighted combination of the encoder hidden memories H = {h 1 , · · · , h n } along with the attention signal a t ∈ R n . Then s t is fed to a softmax layer to calculate the probability distribution of the candidate words:

Rank-Aware Quality Estimation
As shown in Figure 1, the Rank-aware Calibrator (RC) model is composed of three components: D ns , D pt and D pair . Each of the component contains a scoring network for calculating the quality score, either performing semantic matching or learning a joint semantic representation for a query and a response. The matching-based scoring network g M maps a query and a response to vector representations and employs a bi-linear layer to measure the semantic relatedness between these two representations. The joint-representationbased scoring network g J learns the joint semantic representation for the query and the response and utilize a non-linear transformation layer to obtain the quality score. To exploit the fine-grained response quality, we propose three ranking strategies, namely, Negative Sampling, Pointwise Ranking and Pairwise Ranking, to pre-train the components D ns , D pt and D pair respectively. Negative Sampling Negative sampling is the strategy for optimizing the model D ns . Specifically, for each query response pair (X, Y), we sample a response Y − from other queries in the training dataset to construct a negative sample (X, Y − ). Then, we augment the original training dataset with the sampled negative data and train the scoring network by maximizing the following objective: where g ns J denotes the score obtained based on the joint representation of X and Y * . The binary variable b is set as 1 if Y * is from the original training dataset. Otherwise, b is 0. Optimizing this classification-based objective can rank the relevant responses higher than the irrelevant ones. Pointwise Ranking As mentioned in the Introduction section, only differentiating relevant responses from the irrelevant responses is not enough. Since the multi-level quality annotations are available in the training dataset, one direct way for modeling the fine-grained response quality is to minimize the gap between the predicted quality scores and the gold standard multi-level scores. Thus, we propose a regressionbased criterion for optimizing the model D pt . The objective function is given by: (5) where q g is the gold-standard quality score for the queryresponse pair (X, Y) and g pt M is the matching-based scoring function. With this objective, the D pt is enforced to pay attention to different levels of quality scores rather than just 0-1 relevance. In other words, D pt can exploit how good the response Y is apart from its relevance/irrelevance to the query X. Pairwise Ranking The pointwise ranking strategy adopted in D pt is to directly model the absolute quality scores. Compared to the pointwise ranking, the pairwise ranking strategy implicitly exploits the multi-level quality scores by learning to assign preference for the pre-built paired responses Y ++ and Y + from the same query X. Here, Y ++ and Y + denote the responses belonging to the adjacent quality levels and Y ++ is the better one. To optimize the scoring component g pair M in D pair , the pairwise ranking strategy rewards the better response in the ranking pair and penalizes the other one so that the margin of their quality estimations is as close as possible to the pre-specified value ∆. The loss function is as follows: where the score function g pair M calculates the quality scores of the response via matching the semantic representations of the response and the query. Then, we traverse all of the possible response pairs under the query X to make the model D pair aware of different quality levels.
Intuitively, we can utilize the RC model to estimate the quality of each training sample and leverage the quality scores to adjust the weight of the training sample loss. However, as mentioned above, such response-level strategy ignores the consideration of the importance of different tokens in the same response. In this paper, we give a solution that performing token-level calibration with the fine-grained quality estimation.

Token-Level Calibration
The aim of the vanilla sequence-to-sequence (S2S) model is to minimize the following negative log-likelihood (NLL) objective function: where D denotes the training dataset or a training batch and p(Y|X) is the conditional generation probability given by Eq 3.
Recall that response-level quality estimation treats all of the tokens in the response as equal importance and cannot accurately reflect the importance of the token in the response, which may erroneously encourage the generation of some uninformative words. In order to integrate the quality information more accurately, we propose to calibrate the S2S training with token-level quality information. Specifically, we propose to measure the quality of the sampled response conditioned on each token and deem the quality score estimated by rank-aware calibrator as the importance of the corresponding token in the sample loss estimation.
Given the training sample X, Y and the gold standard response token y t at the t-th decoding step, we firstly apply Monte-Carlo search with a rollout policy G β (see the "Rollout" component in Figure 1), which is set identical to the current generator (i.e, S2S model), to conduct the sampling conditioned on y t (that is, perform generation with the starting token as y t ). LetŶ t be the sampled response, then, we employ the pre-trained rank-aware calibrator to calculate the fine-grained quality score forŶ t . Specifically,Ŷ t is fed to the components D r , D pt and D pair separately and the final quality score is the mean value of the outputs from the scoring models: To reduce the variance of the sampling, we perform the Monte-Carlo search N times, yielding a set of simulated responses {Ŷ 1 t , · · · ,Ŷ N t } for the response token y t . The final token-level quality estimations for y t is: where q i t is the fine-grained quality score of the sampled re-sponseŶ i t . Here, q t characterizes the contribution of the t-th generation step to computing the training sample loss. Compared to [Shang et al., 2018;Liu et al., 2018], which treat each generation step equally, performing the weighted sum of the losses at each generation step can alleviate the incorrect bonus on some uninformative words. The calculations of the sample loss L RC (X, Y) and the total loss L RC are given below:

Knowledge Inference
Recall the issue that some meaningful words in the irrelevant responses may be erroneously penalized due to the topical irrelevance. For example, in the fifth response in Table 1, the tokens "university" and "martyrs" will be assigned low quality scores and thus the generation of these words will be suppressed after the training. To reduce such kind of side effect, we propose Knowledge Inference (KI) component. The key idea of this component is to guide the generation with the useful knowledge. In this paper, we regard keywords (i.e., informative words) as the knowledge. Firstly, we utilize TextRank model [Mihalcea and Tarau, 2004] to extract the keywords from the responses in the training dataset. Then, we construct a knowledge memory K ∈ R n k ×dim k , where n k memory vectors correspond to n k informative words obtained from the training corpus. After building the knowledge memory, we leverage the context-aware decoding hidden representation s t to pay attention to the memory vectors k j (j ∈ (1, . . . , n k )) and summarize the knowledge via weighted combination: where W K is a parameter matrix in the KI component and tanh denotes the hyperbolic tangent function. c k t is the summarized knowledge at the t-th generation timestep. To incorporate the knowledge into the generation, we feed the concatenated c k t and decoder hidden representation s t to the ultimate softmax layer for producing the token y t .
Learning the alignment between the memory K and the decoder hidden representation s t from scratch is difficult because the model has no prior knowledge of the informative words. Thus, we regard the extracted informative words from the current ground-truth response Y as additional supervision signal to help the training of K. Given the informative words Y K extracted from Y, the objective of the KI component is defined as follows: where the binary variable b K j is the indicator of the existence of the j-th informative word and δ denotes the sigmoid activation function. p(k j |s t ) is the alignment score between k j and s t . By introducing the objective L KI , the model can learn better knowledge memory K and its alignment with s t based on more feedback from both the S2S component and the KI component. Consequently, there will be a higher potential generating informative words.

Joint Training
The parameters of the pre-trained rank-aware calibrator model are kept fixed all the time. Although the loss with rankaware calibration, i.e., L RC , reduce the possibility of generating unimportant words, it may weaken the language model constraint on the generated responses. Thus, we combine the NLL loss L NLL with L RC during the training. The loss L KI is also included to encourage the generation of the informative words. The final training objective J (θ) of the proposed framework is as follows: 3 Experimental Setup

Dataset
To investigate the effectiveness of the proposed framework, we build a dialogue dataset with multi-level fine-grained  To process such quality labels, we convert L1 to L5 to the normalized quality scores, namely, 0.0, 0.25, 0.5, 0.75 and 1.0 respectively. We crawl 534,381 query-response pairs from Douban Group, a famous forum in China 1 , and recruit human annotators to annotate the multi-level quality labels for each sample. We split the annotated dataset into three parts, where 522,881 query-response pairs are for training, 10000 queryresponse pairs are for validation and the remaining 1500 query-response pairs are for testing. There is no overlap among the queries in training, validation and testing set.

Settings
We compare our model with the following baselines and comparison models: • S2S: It is the standard attention-based sequence-tosequence model [Bahdanau et al., 2015].
• S2S-RW [Liu et al., 2018]: S2S with Re-Weighting. It is an extended S2S model with response quality measurement based on the corpus-level n-gram co-occurrence statistics and the length of the response.
• S2S-CN [Shang et al., 2018]: S2S with Calibration Networks. It is an enhanced S2S model where a pre-trained calibration networks is introduced to adjust the weight of the sample loss.
• S2S-GT: S2S with Ground Truth. It is a variant of the models [Liu et al., 2018] and [Shang et al., 2018], where we replace the instance quality measurement with the ground truth quality scores.
• S2S-DF: S2S with Data Filtering. It is a S2S model trained on the high quality data where we adopt the data filtering strategy proposed in [Xu et al., 2018] to obtain the topically related query-response pairs. 2 • S2S-DF + : A variant of S2S-DF. We perform data filtering according to the gold standard multi-level quality score.  Specifically, we only preserve query-response pairs of L4 or L5 quality for training the model. 3 We divide the comparisons with the baseline models into two groups according to using the Maximum Mutual Information (MMI) decoding  or not. The first group of comparison is under the setting w/o MMI, where the decoding is identical to the normal beam search. The second group is under the setting w/ MMI, where an inverse S2S (response-to-query) model is introduced to rerank the Nbest (N is set as 50) hypothesis generated from the standard S2S (query-to-response) model. We employ a two-layer bidirectional LSTM as the encoder and a two-layer unidirectional LSTM as the decoder. The dimension dim h of both the encoder and the decoder hidden representations is 500. The word embeddings are randomly initialized and the size of each word vector is 300. For the model training, we employ Adam [Kingma and Ba, 2014] as optimizer, with the initial learning rate being 0.0001 and the decay rate being 0.9. The dimension of the keyword memory vector dim k is 30. In the rank-aware calibrator, we employ multi-channel CNN [Kim, 2014] to extract sentence-level features. The sizes of convolutional filter are 1, 2, 3 and the number of filters is 100 for each size.

Evaluation Metrics
We evaluate our model and the comparison models using the following evaluation metrics: Word Overlap: we employ BLEU-1, BLEU-2 and BLEU-3 to measure the word overlap between the generated and the gold standard responses. Embedding Similarity: Actually, BLEU metric does not  correlate strongly with human judgements. Thus, we introduce the embedding-based metrics [Liu et al., 2016] to measure the similarity between the generated results and the ground truth. Diversity: Following , we calculate the ratios of the distinct unigrams (Dist-1) and bigrams (Dist-2) in the generated responses, and use the metrics to measure how diverse and informative the responses are.
Human Evaluation: Apart from the automatic evaluations, we randomly sample 200 queries in the testing dataset and recruit three helpers to judge the quality of each generated response from the best models on the automatic metrics. The rating criteria, which is same as that in [Shang et al., 2018], is as follows: +2: The response is not only relevant and natural, but also informative and interesting. +1: The response can be used as a reply to the message, but is too universal like "Yes, I see" , "Me too" and "I don't know". 0: The response cannot be used as a reply to the message. It is either semantically irrelevant or disfluent. For each model, we report the ratio of each score (+2, +1 or 0) and the average score as the human evaluation results.  Table 2) and w/ MMI (the second group in Table 2), our model consistently outperforms the baseline and the comparison models on all of the metrics, demonstrating the effectiveness of the proposed components. Specifically, our model is better than all of the non MMI-based baselines on Dist-1 and Dist-2, suggesting that the proposed model can generate more diverse and informative responses. The probable reason is that we introduce a knowledge inference component to encourage the generation of some informative words erroneously punished by the models adopting instanceweighting strategies (i.e., S2S-RW, S2S-CN and S2S-GT). We also observe that OURS achieves better performance on BLEU scores and Embedding Similarity compared with S2S-GT. This finding shows that token-level training loss estimation together with rank-aware calibration is a better strategy than just scaling the sample loss with an instance weight. Another interesting finding is that training model with only the high-quality data, as done in S2S-DF and S2S-DF + , is still worse than the instance weighting strategy adopted in S2S-GT, indicating that soft data filtering via quality estimation is a more suitable way for integrating quality information into the model training.

Main Results
From the human evaluation results in Table 3,  0). Specifically, OURS w/ MMI increases 3.5% +2 responses and reduces 2.0% 0 responses compared with S2S w/ MMI. Without introducing the additional knowledge and diversity modeling, the pure S2S model tend to produce more generic responses than the MMI-based models, i.e., S2S w/ MMI and OURS w/ MMI. We also find that taking the instance quality into consideration can lead to less generation of the 0 responses, for example, OURS w/ MMI / S2S-GT w/ MMI versus S2S w/ MMI. The above observations are somewhat consistent with those in the Table 2.

Ablation Study
To further investigate the effectiveness of the proposed component, we compare our model with the ablated models. The results of the ablation study are listed in Table 4. OURS significantly outperforms the ablated models on the relevance metrics (i.e., Embedding Similarity) but the performance gap on the diversity metrics is small. It is reasonable because the aim of the proposed components is not to model the diversity. Removing either the KI component or the RC component will result in performance degradation, especially for the case removing the RC component, our model degenerates to the S2S model together with the knowledge inference (KI) component. Comparing with OURS, the relevance scores of the OURS w/o RC drastically decrease (-6%, -6%, -7% on Average, Extrema and Greedy respectively), indicating that our token-level rank-aware calibration is helpful for improving the relevance of the generated responses. Without RC & KI components, our model will become equivalent to the standard S2S model and the metric scores will drop for further step.

Case Visualization and Analysis
In Figure 2, we visualize the token-level quality scores calculated from Eq 8. Each cell corresponds to a Chinese word and it uses depth of color to represent the importance of this word. We can observe that the importance score for the token "Sichuan", a province in China, is significantly larger than those for other tokens when the query is talking about "hometown" (the upper query). Similarly, the token "cloudy" and "rainy" are highlighted by the rank-aware calibrator when the topic of the query is related to the weather. These examples demonstrate that our token-level strategy armed with Monte-Carlo Rollout can accurately characterize the quality of token with respect to the query, leading to better calibration for the training. Table 5 shows some example outputs. The informative words appearing in the training dataset are wrapped in the  brackets and coupled with underlines. According to these cases, we find that our model (i.e., OURS w/ MMI) can produce some interesting responses with the meaningful words. For example, the response for the third query mentions "Fantasy Westward Journey", the name of a very popular online game in China. Similarly, the response answers "Baidu", a Chinese IT company, for the fifth query. We attribute these phenomena to the KI component, which explicitly encourages the generation of the important words. Besides, our model can give replies closely relevant to some queries without explicit topic, for example, the first and the the fourth query, it is probably because our model introduces finegrained quality information via token-level rank-aware calibration and the probability of generating generic response such as "oh, yes" and "I also want to know" is thus reduced.

Related Work
Neural Dialogue Generation is usually formulated as a sequence translation problem [Ritter et al., 2011;Shang et al., 2015;Wang et al., 2018] and the sequence-to-sequence (S2S) encoder-decoder framework [Cho et al., 2014;Sutskever et al., 2014;Bahdanau et al., 2015] is applied. Various approaches have been proposed to improve the S2S model for better human-computer conversation, for example, via introducing topic or keyword information into the generation process [Mou et al., 2016;Xing et al., 2017;Gao et al., 2019a], diversifying the generated responses with additional memories or objectives Zhou et al., 2018], and arming the S2S model with GAN or other advanced techniques [Xu et al., 2017;Du et al., 2018;Tao et al., 2018;Cai et al., 2019;Gao et al., 2019b]. Different from the above works, [Shang et al., 2018;Liu et al., 2018;Csáky et al., 2019;Cai et al., 2020] attempt to incorporate the quality modeling of training data into S2S training. The basic idea of these approaches is either instance weighting [Jiang and Zhai, 2007] or filtering data of low quality [Wojciechowski and Zakrzewicz, 2002]. The potential issue of these strategies is that the tokens within each training instance are treated equally and the informative words in a low-quality response may be erroneously punished. Similarly, the model guided by these strategies tends to generate some boring but frequently-used words.

Conclusion
We propose a multi-level contrastive learning paradigm to exploit the fine-grained response quality to calibrate the training of the response generation models. We design a Rank-aware Calibration (RC) network to construct the contrastive optimization objectives. We further build a Knowledge Inference (KI) component to capture the keyword knowledge from the reference during training and exploit such information to encourage the generation of informative words. We evaluate the proposed model on a carefully annotated short-text conversation dataset and the results suggest that our model can generate more relevant and diverse responses compared to the baseline models.