{'fraction_non_alphanumeric': 0.02474917978222624, 'fraction_numerical': 0.013551424088250677, 'mean_word_length': 4.90938465861197, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics

Though generative dialogue modeling is widely seen as a language modeling task, the task demands an agent to have a complex natural language understanding of its input text to carry a meaningful interaction with an user. The automatic metrics used evaluate the quality of the generated text as a proxy to the holistic interaction of the agent. Such metrics were earlier shown to not correlate with the human judgement. In this work, we observe that human evaluation of dialogue agents can be inconclusive due to the lack of sufficient information for appropriate evaluation. The automatic metrics are deterministic yet shallow and human evaluation can be relevant yet inconclusive. To bridge this gap in evaluation, we propose designing a set of probing tasks to evaluate dialogue models. The hand-crafted tasks are aimed at quantitatively evaluating a generative dialogue model's understanding beyond the token-level evaluation on the generated text. The probing tasks are deterministic like automatic metrics and requires human judgement in their designing; benefiting from the best of both worlds. With experiments on probe tasks we observe that, unlike RNN based architectures, transformer model may not be learning to comprehend the input text despite its generated text having higher overlap with the target text.


Introduction
Kahneman (2014) explains decision making with a two system model; an action is decided by either of the two systems -System 1 and System 2. System 1's decisions are fast and often impulsive -like pulling one's hand when a vessel is hot, attempting to dodge an object thrown at, etc. In these examples, the system had sufficient information to act quickly and it did so. But, he argues about other scenarios that require higher cognition, where the decision making requires meticulous analysis of facts through a chain of reasoning mechanisms. Such decisions are carried out by System 2. Example scenarios include engaging in debate with a person on say climate change, inflation, politics etc., or a dialogue with a hotel representative to go over a list of holiday options and deciding on the best choice based on price, timing, location and negotiating for deals. Such scenarios require better understanding before action through asking a series of questions. The two systems work differently and require information at different granularity. One of the primary reasons behind it is that, we, humans respond swiftly when the effect of actions is observed immediately and require additional information when the decision is not imminent.
The decision problems in artificial intelligence have variety and can be perceived as a behavior of either of the two systems. From the description and examples cited in (Kahneman, 2014), Natural Language Processing tasks like text summarization (Luhn, 1958;Kupiec et al., 1995;Gambhir and Gupta, 2017), question answering (Rajpurkar et al., 2016;Reddy et al., 2019), sentiment analysis (Dave et al., 2003), dialogue modeling (Vinyals and Le, 2015;Serban et al., 2015;Bordes et al., 2016;Li et al., 2017;Parthasarathi and Pineau, 2018;Neelakantan et al., 2019) require careful understanding of the input, as in System 2, to make a decision and act. Other challenging tasks like caption generation (Vinyals et al., 2015), PoS Tagging (Klein and Manning, 2003), machine translation (Wang and Waibel, 1997;Kalchbrenner and Blunsom, 2013), language modeling (Bengio et al., 2003;Lang et al., 1990) require mostly the System 1 dynamics. The task of dialogue modeling requires learning through interaction, often, from humans. The model is expected to understand the input text for it to interact, and the interaction can be meaningful only when the language understanding is better. Approaches for solving dialogue task include information retrieval based approached like selecting a response from a set of canned responses (Lowe et al., 2015a) or keeping track of very specific information which are a priori marked as informative slot-value pairs (Guo et al., 2018;Asri et al., 2017); using the current information state to select response.
Generative dialogue modeling (Vinyals and Le, 2015;Lowe et al., 2015a;Serban et al., 2015;Li et al., 2016Li et al., , 2017Parthasarathi and Pineau, 2018) -a sub-research in dialogue modeling aims to generate a response as a sequence of tokens with one token at a time conditioned on an input text. The task formulation encapsulates the mechanism for decision making with a free style text generation (Vinyals and Le, 2015). Neural machine translation (NMT) (Bahdanau et al., 2014) also followed a similar task formulation and was seeing a dramatic increase in its popularity when generative dialogue models research was getting traction. The successful concepts in NMT got borrowed to other research in NLP like dialogue, text summarization, PoS Tagging which are also sequence mapping problems.
Despite mapping a sequence of text to another text being common between generative dialogue models and NMT, a dialogue generation is required to stay coherent, understand the information in the text, anticipate a response and others in addition to generating text. These underlying mechanisms do not get evaluated with uninformative automatic evaluation metrics like BLEU (Papineni et al., 2002), embedding based metrics (Wieting et al., 2015;Rus and Lintean, 2012;Landauer and Dumais, 1997), ROUGE (Lin, 2004), Perplexity Score, METEOR (Lavie and Agarwal, 2007), F-1 and by training on uninformative dialogue datasets like Ubuntu corpus (Lowe et al., 2015b), Reddit, Twitter (Ritter et al., 2011) which do not foster a decision making component with information in addition to sequence generation. To mitigate the requirement of additional information for dialogue generation external knowledge sources like wikipedia (Scheepers, 2017) and NELL (Carlson et al., 2010) was tried with reasonable success in dialogue generation (Parthasarathi and Pineau, 2018;Dinan et al., 2018).
But, the issues in evaluation -automatic evaluation metrics uncorrelated with human judgementshowcased by Liu et al. (2016) is still an open problem. Attempts to mimic human scores for better evaluation metric (Lowe et al., 2017) and other met-rics that aim to correlate to the human judgement (Sinha et al., 2020;Tao et al., 2018) still have a long way to go. Dependency on human evaluation is not any better as it is subjective, requires more time, cost, effort and has issues of disagreement among scorers in evaluation (Li et al., 2019). But, many dialogue research promote the costly yet inconclusive human evaluation as a way for thorough evaluation of dialogue models. The evaluation of surface level token generation -automatic or with a human -does not evaluate the underlying understanding of a dialogue model, which is required of a model to have meaningful conversation. The language understanding component of an agent more often than not goes unnoticed with only token-level evaluation metrics.
Simpler probing tasks (Belinkov and Glass, 2019;Jawahar et al., 2019;Anand et al., 2019) test the performance of classifiers that train on the representation learnt by a model. A probe task is a backward reasoning task, where a model reasons out its understanding of the input through simple questions as classification task. It could be seen that probe tasks can evaluate whether the model does careful reasoning on the facts presented as like in System 2. For example, in representation learning for reinforcement learning, learning to predict the position of an enemy from the encoding of input state shows whether the representation is discriminative of the game features. The probe tasks allow a way to quantify the understanding of a model and articulate a meaningful discussion around the success or failure of a model, prevent over-fitting to spurious patterns, identify unwanted biases in a model among others. Drawing inspirations and analysing the existing literature to the best of our knowledge we propose a set of probing tasks 1 for evaluating the language understanding of generative dialogue models on chit-chat and goaloriented dialogues - (Table 1). Our probe tasks also help us to understand the difference in learning behaviour of recurrent neural network (RNN) models and Transformer models which was previously not evident from the token-level evaluation methods.
Our contributions in the paper are: • Showcasing the significantly high variance in human evaluation of dialogues.
• Proposing a list of probe tasks -2 semantic, 1 The code repository can be found at https://github.com/ppartha03/Dialogue-Probe-Tasks-Public 13 information specific and 3 downstream as an alternate evaluation of dialogue systems.
• Finding that the representation learnt by recurrent neural network based models is better at solving the probe tasks than the one by transformer model.

Language Generation
Setting up probe-tasks to understand the underlying workings of the neural models is not unique. There has been quite a lot of work (Conneau et al., 2018;Belinkov and Glass, 2019;Elazar et al., 2020) to understand the embedded information in vector embedding of sentences. The objective of designing probe tasks is to evaluate the inductive bias of a model to learn a task-specific hidden representation that can be used to solve a series of simple tasks.
As it is easier to control the biases in probing tasks than in the downstream tasks, research in language generation has analysed models on probing tasks like using encoder representation to identify words in input (WordCont) to measuring encoder sensitivity to shifts in bigrams (Conneau et al., 2018;Belinkov and Glass, 2019). Dialogue generation task requires understanding and reasoning on the information from a user before generating text and language generation is only a part of the task. The probing tasks on language generation are useful in probing the generation aspect of dialogue models but are not complete to measure the dialogue modeling capabilities of models, which involves the agent having a better understanding of the input context.

Reinforcement Learning
Reinforcement learning (RL) is another area of research where large models make it hard to deduce whether a model is biased in the right ways to act in an environment. Dialogue generation shares similarity with the sequential action prediction in RL by their large combined space of possible states and actions. To avoid learning from spurious correlation in the data, it is important to verify if the agents are getting biased by relevant features in input. Anand et al. (2019) learn state representation for an RL agent in an unsupervised setting and introduce a set of probing tasks to evaluate the representation learnt by agents. This includes using an annotated dataset with markers for position of the agent, current score, items in inventory, target's location among others. The authors train a shallow linear classifier and measure its performance; which serves as a metric for the representational soundness of the learning algorithm. Due to the similarity between RL and dialogue, we draw inspirations from Anand et al. (2019)'s probing tasks on game playing agent.

Probing tasks in Vision
Applications of computer vision like caption generation for images (Vinyals et al., 2015) or videos (Donahue et al., 2015) use attention based models to parse over the hidden states of a convolutional neural network (ConvNet) (LeCun et al., 1998). The attention over the ConvNet features are visualized to observe the words corresponding to different parts of the image. Visualizing the attention has been one of the qualitative probe task for text generation conditioned on images (Xu et al., 2015).

Software Unit Testing
It is also interesting to draw parallels to Unit Testing in software engineering (Koomen and Pol, 1999), where the smallest software components of a system are tested for their design and logical accuracy. The only difference between a deterministic application software and a stochastic decision making ML module is that the behavior of the ML system is data-driven while for a software system it is driven by logic. Despite the difference, the unit testing and probing tasks share a common ground. Analogous to an application software, the decision making modules also have smaller decision components. In most models, these smaller components are latent but still they have to be evaluated as they indirectly contribute to the performance of the modules. These latent components can be explicitly validated to qualify a model's understanding and to keep a check on irrelervant and/or irresponsible biases the agent may have picked up from the dataset. Such evaluations on ML systems will have positive effect on users' trust when they are deployed.

Dialogue Probing Tasks
Dialogue is a complex decision making problem, which is sequential and requires the agent to have sufficient understanding of the context before generating a response. The quality of the generated text is not a sufficient metric to evaluate the model's understanding of the input. An evaluation metric for dialogue requires a profound understanding of grammar, semantics, as well as domain specific and general knowledge; this makes the problem of coming up with the evaluation metric for dialogue AI-Complete (Yampolskiy, 2013). In a way, if we have a perfect evaluation metric, the metric itself is an AI system which explains the rationale behind the need for human evaluation. But, human evaluation is difficult to scale up, and agreement among annotators is hard to come by, and formulating the right questions takes effort (Li et al., 2019); so, instead of spending human effort in appropriately evaluating the dialogue, we propose to design probe tasks -semantic, syntactic, information specific and downstream tasks -for each dialogue dataset. The tasks proposed and discussed in this paper are shown in Table 1.

Semantic and Syntactic Probe Tasks
Similar to probing a language generation model, some probe tasks for dialogue generation models include measuring sensitivity of models to context by shuffling the input (Sankar et al., 2019), testing if the model can predict a mid frequency token in the context (WordCont)(Belinkov and Glass, 2019), and testing if the model understands how far is it in the conversation by using its context encoding (UtteranceLoc) (Sinha et al., 2020) . To formally compare different dialogue generation models, we use UtteranceLoc and WordCont probe tasks to evaluate the semantic understanding.

Information Specific Probe Tasks
Apart from language generation, a dialogue agent is expected to understand specific information from the context to help it in generating the appropriate response. The surface level observation of "nice human like text" or other syntactic or semantic features do not shed light on the underlying mechanisms. Hence, it is imperative that we systematically probe also the information processing with simple questions on agent's encoding of the input context. To this end, we propose 12 information specific probe tasks that track the understanding of a dialogue model on the input text. The tasks are listed in Table 1.

Downstream Probing Tasks
A Dialogue agent is not only expected to predict the next utterance, and track the information but also perform an appropriate downstream task. An example could be of an agent carrying Task Task Name Description Semantic UtteranceLoc * How long has the conversation been happening ? WordCont + Which mid-frequency word is encoded in the context ?

Information Specific
IsMultiTopic Does the conversation have more than one topic ? NumAllTopics How many topics does this conversation have ? RepeatInfo Which information provided by the user is repeated ? NumRepeatInfo What many number of recent information are repeats ?

AllTopics
What are all the topics discussed so far ?

RecentSlots
What is the recent information given by the user ? NumRecentInfo How many information did the user provide recently ?

RecentValues
What are the details of the recent information ? AllSlots What all information are given by the user so far? AllValues What are the details of in all the information ?

RecentTopic
What is the current topic of the dialogue ? NumAllInfo How many information did the user provide so far ? PersonalInfo + What characteristics of your persona can you infer from the conversation history ?

Downstream task
ActionSelect Which downstream task (database query) follows the current conversation ? EntitySlots What information should is required to construct the query ?

EntityValues
What values should be passed to the query ? Table 1: List of probe tasks on the two different datasets. The color of the row identifies its difficulty -green, yellow and red for easy, medium and hard respectively. This categorization is further discussed in Section 5. + -Task on PersonaChat, * -Task on both the datasets, 'none' -Task on MultiWoZ.
out a follow-up action Hotel-recommend with guesthouse-price and cheap as slot and value to a user context -"I am looking for a lowprice hotel to stay." The dialogue state tracking measures the performance of a model on such tasks (Henderson et al., 2014) but such often goes unnoticed in the evaluation of generative models. Works like that of Neelakantan et al. (2019) use entity, values and action information to train on the dialogue generation task but the performance of a generative dialogue model without explicitly training on the downstream tasks are not compared. Towards that, we propose ActionSelect, EntitySlots, EntityValues probe tasks. The details of the task are shown in Table 1.

Datasets
We experiment the proposed set of probing tasks on  Table 2. The data sets represent the two major styles in dialogue and we use probe tasks accordingly. For experiments in this paper, we used BLEU score (Papineni et al., 2002) on the validation set as the metric for model selection, which is commonly used in dialogue research. The results of the experiments did not differ with using other model selection metrics like ROUGE-F1, METEOR or Vector-Based (Average BERT embedding) which are tabled and explained in Appendix D.

Models
We train 5 commonly used generative dialogue models for 25 epochs on the two datasets.
LSTM ENCODER-DECODER The architecture (Vinyals and Le, 2015) has an LSTM cell to encode the input context only in the forward direction. For a sequence of words in the input context The decoder outputs one token every step. We used two layer LSTM cell; the first layer applies recurrent operation on the input to the model while the layer above recurs on the outputs of the layer below. The encoder final hidden state (from the 2nd layer) is passed as an input to the decoder. We train the model with cross entropy loss as shown in where y t is the t th ground truth token distribution in the output sequence,ŷ t is model generated token and p is the model learned distribution over the tokens. We train the model with Adam (Kingma and Ba, 2014) optimizer with teacher forcing (Williams and Zipser, 1989).

LSTM ENCODER-ATTENTION DECODER
The architecture is similar to the LSTM Encoder-Decoder with an exception of an attention module to the decoder. The attention module (Bahdanau et al., 2014) linearly combines the encoder hidden states h t T 1 as an input to the decoder LSTM at every step of decoding, unlike only having the last encoder hidden state.
HIERARCHICAL RECURRENT ENCODER DE-CODER The model has encoding done by two encoder modules acting at different levels (Sordoni   , 2015); sentence encoder to encode the sentences that feeds in as input to the context encoder.
Both the encoders are LSTMs. The decoder is an attention decoder.

BI-LSTM ENCODER-ATTENTION DECODER
The encoder is a concatenation of two LSTMs that can read the input from forward and backward direction (Schuster and Paliwal, 1997). The hidden state is computed as the summation of the hidden states of the two encoders. The decoding is done with an attention decoder.
TRANSFORMER ARCHITECTURE This stateof-the-art architecture (Vaswani et al., 2017;Rush, 2018) is a transductive model that has multiple layers of attention to predict the output. We used the architecture in an encoder-decoder style by splitting half the layers for encoding and the remainder for decoding. We perform the probe tasks on the encoder hidden state. The size of the models used in the experiments are detailed in Table 3. For the probing tasks, we select the untrained model, model with the best BLEU score on validation, and model from the last training epoch. We use packages pytorch (Paszke et al., 2017) and scikit-learn (Pedregosa et al., 2011) for our experiments.

Motivation for Dialogue Probe Tasks
Although criticism on automatic metrics for dialogue evaluation (Liu et al., 2016;Sankar et al., 2019) is widely accepted, the human evaluation, though straightforward, does not validate the holistic understanding of the models being compared against. Successful evaluation requires careful crafting of the appropriate question and is contingent on the understanding of the same by the human participants. Here, the evaluation expects a human participant to understand not only a model's ability to generate meaningful text but also verify if it understood the conversation thus far; which requires more details for the human participants to have an agreement. We hypothesize that the generated text presented to the participants are greatly dependent on the choice of seed values and a slight variation could result in a model generating a very different response. We verify that the human participants cannot identify the difference between two models by posing an alternate hypothesis where we expect the participants to fail in rating alike the two responses selected from two different runs of the same model with different seed values.
To verify our hypothesis, we train the models on the two datasets and performed human evaluation experiment with 500 volunteers through amazon's Mechannical Turk (Buhrmester et al., 2016;Miller et al., 2017). For an even more challenging scenario to validate our hypothesis, we chose the goal-oriented dataset as the context-response variance is relatively lower than in PersonaChat. For  the study, we sample 2000 context-response pairs from Bi-LSTM Attention model from two different seeds. We chose this model as this had the lowest variance in BLEU scores (Table 4). We ask the participants to select the response that they think is more relevant to the given context, similar to Li et al. (2015). The annotators can select either of the responses or a Tie. We show the participants the responses generated by the model with same model parameters, but different seeds. For every contextresponse pair, we collected 3 feedback from different participants (Distribution corresponding to the 3 different human responses are shown with legend HumanExp1, HumanExp2 and HumanExp3 in Figure 1). Usually human evaluation is done on 100-500 responses. To understand the variance in this set up and the lack of information at the token generation level, we sample 50000 sets of 200 human responses from the collected 2000 responses and compute the fraction of times there was a tie. We observed that distribution over the fraction of times the human participants selected a Tie was centered around 35% ( Figure 1) with all of the probability mass within 50%. This shows that (a) text generated by the same model can have variance with the seeds, and the variance is significant (b) attributing the choice of seed value to the performance of a model creates confusion in the evaluation. The results show that the scores based only on the text generated by a model cannot be extrapolated to be the performance of the model architecture in the dialogue task. Optimizing on the seed value does not guarantee reproducible results, which is necessary to progress the field further. Further, the dependence of the model generated text on seed value raises a valid concern; whether a model parameter chosen by the seed value can mimic the surface level token generation of a model that actually understands the context. The lack of clarity leads to inconclusiveness of studies with human evaluation only on the generated text.
Although variance due to the seeds can be reduced with averaging results over multiple seed values, human evaluations are expensive and requires the same effort for setting them up every time. Whereas probe-tasks allow cheaper extension for evaluating over multiple seed values that can effectively reduce the variance on an appropriate set of probe tasks.

Probing Tasks
We train the models on the two different datasets without any auxiliary tasks. To understand the evolution on the probe task from beginning of the training till end, we compare with 3 different parameter configurations of every model -Untrained, Last epoch, and BestBLEU. We save the model parameters while training on the end-to-end dialogue gen-eration task and evaluate on the probing tasks as a post analysis. We use Logistic Regression classifier 2 implementation from scikit-learn (Pedregosa et al., 2011) with default parameters except the max_iter set to 250 for all the probing tasks, invariably. We train the classifier using the encoder representations on the probe tasks with the training set and evaluate with the validation set. The evaluation metric is F1-score with micro averaging in multi-class prediction tasks. The data preparation for the probing tasks are discussed in detail in Appendix B.
PROBE TASKS ON PERSONACHAT The models are evaluated on the three probing tasks relevant to chit-chat dialogue generation (Table 5) -two semantic and one information specific. Utterance-Loc and WordCont measures if the encoded context suggests semantic awareness of the model while PersonalInfo measures the amount of knowledge the model has about its persona from encoding of conversation history. In other words, it evaluates the extent to which persona can be identified from the context encoding with a linear classifier. A better performance in these tasks indicate that the model has an understanding that conversations involve assuming different persona, utterances follow a temporal sequence and hence the encoding has to be different. Would a human have such an understanding ? Yes.
The PersonalInfo task here is not very specific to identifying personal information but acts as an indicator to the information embedded in dialogues that goes unnoticed in the encoding. It was surprising to see that none of the models scored a reasonable F1. Transformer model scored higher on BLEU score (Table 4) but performance of transformer on PersonalInfo task was decreasing with increased training ( Table 5).
The semantic tasks UtteranceLoc and WordCont evaluate whether the model understands how far in the conversation is it in and if it can identify mid-frequency words in the target response. Bi-LSTM model performed the best in UtteranceLoc while the Transformer model was not in the top 3. Transformer performed the best in WordCont.
We hypothesize that the Transformer model can learn extensive specific information in the input because of the size of its attention but finds it diffi- 2 We also trained a nonlinear model -multi-layer perceptron for probe tasks (Appendix D). The results had a similar trend holding good the remainder of the discussion in the paper.  cult to learn general information like in Utterance-Loc. Also, we observed that the inductive biases of the SEQ2SEQ models enable random projections that are informative even without training. This correlates with independent observations on the results in (Tallec et al., 2019) which suggests random projections on temporal information can hold information. Similarly, Transformer architecture's random representation is also informative; but visualization of the encoder hidden states in low dimensional space does not show prominent cluster formations unlike in SEQ2SEQ - Figure  4 in Appendix D. The SEQ2SEQ models have a smaller manifold due to recurrent multiplication that regularizes its representation to observe structures, whereas Transformer network's attention operations project the context on to a larger manifold that prevents loss in encoding 3 . This explains the SEQ2SEQ models performing well on Utterance-Loc while Transformer model performing well on WordCont. The difference between the two classes of models is much more evident on the probing tasks in MultiWoZ dataset.
PROBE TASKS ON MULTIWOZ Unlike chitchat dialogues, goal-oriented datasets naturally provide probe-tasks that can validate the understanding of a model on the task. The probe tasks as shown in Table 1 enable the hidden representation of an end-to-end goal-oriented dialogue agent to be interpretable. We tested the models on 16 different probe tasks -1 Semantic, 12 Information specific and 3 downstream tasks. These probe tasks are de-signed to reveal the models' ability to understand information in conversation history only through indirect signal -language generation.
In majority of information specific tasks and in the downstream tasks (Tables 6,7), we observed that SEQ2SEQ models performed significantly better than the Transformer model. Interestingly, we observed a pattern in Transformer in the two datasets, that the model's performance on the probe tasks decreased from the beginning of training till the end on all of the tasks, while for the rest of the models there was learning involved.
To understand this phenomenon better, we downsampled the encoder representation of the contexts with PCA to 2 components (Figure 2). Although the visualization is not an accurate indicator of what happens in the high-dimensional space, this helps in getting a reasonable understanding of the models' internals. First, the models definitely learn to cluster in the encoder hidden state that help the decoder in generating appropriate responses. Second, the range of the two axes are different for SEQ2SEQ and Transformer models. We observed that the SEQ2SEQ models, mostly, has spread out to a larger manifold from the beginning of the training to end. But, the spreading out has been constrained by the non-linear operations like tanh and sigmoid. Whereas, in the case of Transformer the manifold in an untrained model is much larger (∼ 100×) and eventually shrinks it during training, but still way larger (∼ 40×) than the embedding manifolds of SEQ2SEQ. This could be explained by the absence of tanh or sigmoid non-linearity and the stacking attention operations that only linearly combines the previous layer.
This observation reasons the need for deeper layers in Transformer when training on large data for improved performance; the attention layers help in spreading the data in a large manifold thereby the model can retain almost all of the information it was trained on. But, the reverse of generalizing from a small data is hard to come by because the model does not have sufficient direct information to cluster except the surface level signal of predicting the right tokens. This helps the Transformer model to perform well on the token prediction task or language modelling, while abstracting information and generalizing appears to be a difficult task as is observed from its performance on probing tasks.
The SEQ2SEQ models have inductive biases to squish the input through tanh or sigmoid op-erations. From the visualizations and from other results, we hypothesize that this aids the model in learning a regularized representation in a low-data set up. But, this can potentially be unhelpful when the input is a large set of samples and has rich structure as that requires a model to aggressively spread out. Transformer architecture can thrive in such a set up and that can be validated by the performance of large Transformer models like GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), BERT (Devlin et al., 2018, 2019, RoBERTa (Liu et al., 2019) etc., whereas the SEQ2SEQ models are adept at learning unsupervised structures for better understanding of the input as evaluated through the probe tasks. Also we note that the performance in probe tasks can be a pseudo metric to measure the capacity of the model in generalizing to unobserved structures in inputs in a low data scenario.

Discussion
We articulated so far that mere token-level evaluation of complex language understanding tasks have high bias in automatic metrics and high variance in human evaluation. The inductive biases of models allow for different representations of the input as observed through the visualization of the embedding in Figure 2. Although the notion of a wellstructured representation is subjective, the representations can be quantitatively evaluated with the probing tasks. Such an evaluation also allows an interpretable way for evaluating the understanding of language generation models in dialogue tasks.
But, while deciding on the probe tasks to evaluate the models, we observed that most of the data collected for end-to-end dialogue generation tasks do not provide tasks for sanity check. Absence of probe tasks lead to draw imperfect correlations like the one between token-level accuracy and language understanding. The probe tasks show that the two are very different. To build a holistic model that can understand and perform token-level generation one may have to chose an appropriate inductive bias to train the model. At this point one may wonder, why not train the model with all the probe-tasks as auxiliary tasks for an improved performance ? Although it is a possibility, such a set up does not evaluate a model's ability to generalize to unseen dialogue tasks. Such systematic generalizations depend on the ability of a model in having a profound understanding of its input. One could potentially   train a model with a fraction of the probe-tasks as auxiliary and evaluate on the rest, we leave that for future work.
DIALOGUE MODELS In open domain dialogues, without sufficient information to track and probe, the dialogue models cease to be a dialogue modeling tool without ways to interpret the models' language understanding. As an alternate to tokenlevel evaluation, comparison of different model architectures can be meaningfully made with an aggregate metric on the probe tasks in three groups of difficulty -easy ((Ave. SEQ2SEQ) Untrained F1 > .50), medium(0.25 < Untrained F1 ≥ 0.50), and hard (Untrained F1 < 0.25). Such an analysis, as shown in Table 8, allows better inspection of the model results and a fairer comparison between the models. We can see from Table 8 that the models have difficulty in solving harder probe tasks. The results can be used as motivation for coming up with novel inductive biases for neural architectures that address one or a group of aspects in the language understanding of generative dialogue models.
DIALOGUE DATASETS The challenges in dialogue modeling has been evolving majorly because of the complex datasets. But, datasets on chit-chat dialogues often have little to no auxiliary tasks to evaluate the dialogue management abilities of a model. This limits the practitioners to validate the models only on the text generation abilities which, in this paper, is shown to have no correlation with the model's ability to manage internal states in a dialogue.
Goal oriented datasets provide rich set of information that allows evaluating the internal workings of generative dialogue models. Further, probe tasks can also serve as a way to compare datasets in similar domain to rank them on the difficulty in   comprehending the input context. Such a set up invites datasets that challenge models on their understanding. We observed that the chit-chat dialogue datasets need to have a richer set of probing tasks; and dialogue state tracking for non-goal oriented dialogues could be a way forward.

Conclusion
We propose a set of probing tasks to compare endto-end generative dialogue models. We observed that mimicking surface level token prediction is easier for models than learning representations that understand the context. The results of the experiments on probe tasks showed that SEQ2SEQ models perform better than transformer model in encoding information in the context that often goes unsupervised. We also found some probe tasks that all of the models find difficult to solve; this invites novel architectures that can handle the language understanding aspects in dialogue generation. Although language modeling is required for a dialogue model, the performance in token prediction alone cannot be a proxy for the model's ability to understand a conversation. Hence, systematically identifying issues with probe tasks can help in building better models for the task and collecting datasets that allow holistic evaluation of a model's performance.

A Model Parameters
• For SEQ2SEQ models, we used a 256 unit hidden size LSTM with 2 layers and a 128 unit input embedding dimension. The learning rate we used for all the models is 4E-3.
• For Transformer, we used a 512 unit hidden size, 512 unit input embedding dimension, 2 attention header and 4 layers.
• We used Adam as the optimizer to optimize on the cross-entropy loss.
• We averaged the results over 3 different seeds.
• We used a truncated history of last 100 tokens as context to keep the training uniform across the models.

B.1 MultiWoZ
We use the minimal information in the annotated json dataset to add additional details to the dialogue state. The annotation has texts of user and agent utterances with slot and value pairs parsed from the user utterance. We parse and process the slot-value pairs data into information like task topics, recent slot, among others used in the probe tasks. For UtteranceLoc, we bucket the utterances based on the position they occur in a conversation.

B.2 PersonaChat
PersonaChat dataset has persona details along with text utterances from two different users. We extract only the non-stop words from the persona information for PersonalInfo task. For WordCont, we order the vocab descending frequency and select 500 words that occur for 1000-3000 times in the dataset.

C Distribution of Information
Before creating the probe tasks to test the model, we wanted to ensure that the outputs of the tasks have a spread out distribution over the outputs such that the model can be evaluated on its understanding with the probe tasks. The distributional analysis on MultiWoZ data is shown in Figure 3 suggests that in addition to diversity in token level there is diversity in the underlying information which charcterizes the response.
To that end, we create probe tasks with the annotated information extracted from the dataset to evaluate the understanding of the dialogue architectures.

D Additional Experiments
Similar to the encoder representation observed on the MultiWoZ dataset, we observed the encoder representation between the first and the last epoch with training on PersonaChat dataset. PersonaChat dataset being an open-domain dialogue dataset, we did not observe strong tendency to cluster in the encoder representation. Non-existence of clusters shows that the model struggle to summarize the information available, explaining the lower performance on BLEU score and probe tasks on the dataset.

D.1 Comparison of Model Selection on probe tasks
We experimented on selecting models based with METEOR, ROUGE-F1(F1), Average vector with BERT encoding as alternate selection metric to BLEU on the two datasets  We compared the performance of these models on the probe tasks to understand if there is any correlation between the metric and higher perfor- mance on the probe tasks. But, we did not find any meaningful correlation between the scores and selection metric.

D.2 Analysis of Non-linear Probing
We experimented with multi-layer perceptron with ReLU non-linearity to analyze the trend in the model performance on the probing-tasks on the two different datasets in Tables 13,14,12. Although there was some difference in the performance of in-dividual probe tasks, the global trend of SEQ2SEQ models' performance improving with epochs while Transformer's performance decreasing with epochs remained constant.

D.3 Performance Evolution on Probe Tasks
Although we observed in Table 6, 7 and 5 that Transformer model architecture's performance decreasing while its BLEU score is higher in the task, we experimented to observe the evolution of the The encoder representation manifold shows that Transformer model has high capacity to encode a large dataset but not the SEQ2SEQ models. model performance on these probe tasks in Figures  5,6 and 7. We observed the trend in transformer in medium and hard probe tasks that its performance decreased and almost always stayed below the SEQ2SEQ models. This shows that the BLEU score and performance on probe tasks do not correlate, as inductive biases in a model can force the model to overfit to the patterns in text without actually understanding it.

E Human Evaluation Details
We collected human annotation on comparing the relevance between two responses from the same model architecture but with different seeds. We used ParlAI's mturk framework to set up our human evaluation (Figure 8).
We provide the users a detailed set of instructions on relevance of the generated text with four different examples. On top of that, we start the data collection with a few sanity check questions (the participant is not told that it is a test question) where the correct answers were obvious. When the participant failed thrice, we soft block them and do not allow them to participate in our data collection.
With this set up, we collected data only from 508 participants out of 1004 who took the sanity check task. This ensured the quality of the data collected.