{'fraction_non_alphanumeric': 0.03187820674573135, 'fraction_numerical': 0.011383071186250596, 'mean_word_length': 5.009772535804549, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems

The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues remain largely unaddressed. This is despite emotions and dialogue success having equally important roles in a natural system. Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks. To lay a foundation for studies on emotions in task-oriented dialogues, we introduce EmoWOZ, a large-scale manually emotion-annotated corpus of task-oriented dialogues. EmoWOZ is based on MultiWOZ, a multi-domain task-oriented dialogue dataset. It contains more than 11K dialogues with more than 83K emotion annotations of user utterances. In addition to Wizard-of-Oz dialogues from MultiWOZ, we collect human-machine dialogues within the same set of domains to sufficiently cover the space of various emotions that can happen during the lifetime of a data-driven dialogue system. To the best of our knowledge, this is the first large-scale open-source corpus of its kind. We propose a novel emotion labelling scheme, which is tailored to task-oriented dialogues. We report a set of experimental results to show the usability of this corpus for emotion recognition and state tracking in task-oriented dialogues.


Introduction
Incorporating human intelligence into conversational artificial intelligence (AI) has been a challenging and long-term goal (Picard, 1997). Emotional intelligence, defined as the ability to regulate, perceive, assimilate, and express emotions, is a key component of general intelligence (Mayer et al., 1999). Such emotion awareness can help the conversational AI generate more emotionally and semantically appropriate responses . Dialogue systems generally fall into two classes. Taskoriented systems converse with users to help complete tasks determined by user goals. Chit-chat systems are set up to mimic the unstructured conversations or 'chats' characteristic of human-human interaction (Jurafsky and Martin, 2009). Chat-oriented systems are typically modelled in a supervised fashion with large available corpora (Vinyals and Le, 2015). In contrast, task-oriented systems track the user goal throughout the dialogue and a policy is typically trained via some form of reinforcement learning (RL) to conduct dialogue towards successful goal completion (Young, 2002). Moreover, the scope of the dialogue can also be extended during this process, e.g. by adding new domains to the dialogue system (Madotto et al., 2021). Consequently, the distribution of data from which a task-oriented system learns can change. Emotions appear in both chit-chat and task-oriented dialogues. However, the cause of emotion may differ as well as their role. Chit-chat dialogues are a means to express emotion. Speakers may discuss emotional ex-periences (Li et al., 2017), or topics that induce emotions such as news broadcasts (Lubis et al., 2017). In task-oriented dialogues, the user is primarily interested in achieving their goal. While an emotional situation may be a reason to interact with the system, e.g. the user just missed a flight and needs to rebook one, the emotion the user exhibits is more often a reaction to potential goal completion or failure. Since the emotion is centred around the user goal, it is more contextual and subtle. Therefore, besides inferring emotional states from dialogue utterances, an agent also needs to reason about emotion-generating situations (Poria et al., 2021). Substantial research efforts in emotion recognition in conversations (ERC) have been invested in chit-chat dialogues Ghosal et al., 2020). There are several public ERC corpora containing chitchat dialogues (Li et al., 2017;Zahiri and Choi, 2018) and conversational data from social media (Zhou and Wang, 2018). These corpora can tremendously accelerate the building of emotional chatbots using data-driven approaches . In task-oriented dialogues, recognising emotions is equally important but remains largely unaddressed. Using RL to optimise a dialogue policy necessitates a feedback signal. While it is accepted that the feedback signal needs to correlate with user satisfaction (Ultes et al., 2017), this feedback signal is often based on hand-coded rules. Could an emotional model instead be directly used to provide such a feedback signal? Could it also be used to support emotion-aware natural language generation (Mairesse and Walker, 2007), or even improve dialogue state tracking through multi-task learning (Heck et al., 2020a)? Existing corpora are small in size, and labels are limited to sentiment polarity, creating a bottleneck, so these questions remain largely unexplored. In this work, we present EmoWOZ, a largescale manually labelled corpus for emotion in taskoriented dialogues. EmoWOZ is derived from Multi-WOZ (Budzianowski et al., 2018), one of the largest multi-domain corpora and the benchmark dataset for various dialogue modelling tasks, from dialogue state tracking (Heck et al., 2020b;Lin et al., 2021) to policy optimisation (He et al., 2022). We also collected and annotated human-machine dialogues as a complement. Our contributions are as follows: • We construct a corpus containing task-oriented dialogues with emotion labels, comprising more than 11K dialogues and 83K annotated user utterances. To the best of our knowledge, this is the first large-scale open-source corpus & code 1 for emotion recognition in task-oriented dialogues.
• We propose a novel labelling scheme, containing 7 emotion classes, adapted from the Ortony, Clore and Collins (OCC) model (Ortony et al., 1988), specifically tailored to capture an array of emotions in relation to user goals in task-oriented dialogue.
• We report a series of emotion recognition baseline results to show the usability of this corpus. We also empirically show that the emotion labels can be used to improve the performance of other task-oriented dialogue system modules, in this case, a dialogue state tracker (DST).

Emotion Models
Within the area of affective computing, emotion models are commonly grouped into two types: dimensional models and categorical models. Dimensional models describe emotions as a combination of values across a set of dimensions. The longest established dimensions are valence and arousal, as proposed by Russell (1980) in the circumplex model of emotion. Valence measures the positivity, while arousal measures the activation. Happiness, for example, is an emotion with positive valence and high activation. Additional dimensions, namely dominance and expectancy (Fontaine et al., 2007), have also been proposed to further describe and distinguish complex emotions.
Categorical models group emotions into distinct categories. The "Big six" theory is one of the most wellknown theories on universal emotions. Based on studies of facial expressions, Ekman (1992) proposed six 1 https://doi.org/10.5281/zenodo.5865437 basic human emotions which are influenced neither by culture nor other social influences: happiness, anger, sadness, disgust, fear, surprise. Parrott (2001) conceptualised over a hundred emotions into a tree-structured list and identified six primary emotions from it. Ortony et al. (1988) proposed the Ortony, Clore and Collins (OCC) emotion model, which is explicitly developed for implementation in computers. In the OCC model, 22 emotion types are described as a valenced reaction to one of three cognitive elicitors: consequences of events, actions of agents, or aspects of objects. For example, dissatisfied is specified as disapproving of someone else's blameworthy action. These cognitive aspects are in line with the cognitive process of a computational agent, making the OCC model suitable for building emotional artificial agents. However, the use of this model for dialogue agents is not yet widespread. In a similar spirit, Gross and Thompson (2007) formulated the process of emotion regulation as the attention, appraisal, and response originated from various situations.
Although there are corpora with real-valued annotation of multiple emotion dimensions (Preoţiuc-Pietro et al., 2016;Buechel and Hahn, 2017), researchers often focus on the valence dimension and annotate with discrete classes (Socher et al., 2013), often called sentiment polarity. Emotion datasets also consider emotions from various categorical models in the annotation scheme (Li et al., 2017;, but some datasets have domain-specific labels. For instance, Zhou and Wang (2018) leverage common emojis in social media posts. The Topical-Chat dataset (Gopalakrishnan et al., 2019) introduces curious to dive deeper in addition to other basic emotions.
In this work, we propose a novel set of 7 emotions and motivate it using OCC model as the basis. We aim for this scheme to capture the cognitive context of emotions while retaining the simplicity of labels that facilitates large-scale crowd-sourcing of emotion annotations.

Emotion Dialogue Datasets
Early works on ERC focus on speech signals (Cowie et al., 2001;Riccardi and Hakkani-Tür, 2005;Carrión and López-Cózar, 2008). More recently, there are increasing number of text-based ERC datasets focusing on chit-chat dialogue. Chit-chat dialogue lends itself well to affective computing research due to its open-domain set-up, where conversation topics are diverse and not restricted to a particular task. One of the largest such corpora is DailyDialog (Li et al., 2017), which contains conversations between English learners on various topics ranging from relationships to money. Other similar datasets include EmoryNLP (Zahiri and Choi, 2018) and MELD . They contain multiparty dialogues from the TV show Friends. TV recordings in talk show format have also been utilised to collect emotion-rich and topic-specific dialogues (Lubis  Table 1: Comparison of our corpus to similar corpora. Values in bold indicate the best value for each metric. For label type, "Emo" stands for emotion categories and "Sent" stands for sentiment polarities. For corpora providing both emotion and sentiment labels, agreement metrics are measured for emotion labels. DSTC1, SentiVA, and TML refer to works by Shi and Yu (2018), Saha et al. (2020), and Wang et al. (2020Wang et al. ( ), respectively. et al., 2015. Unfortunately, existing data suitable for task-oriented corpora, such as customer service chat logs, are typically not within the public domain.
There also exist a few corpora concerning the affective aspect of task-oriented dialogues. Wang et al. (2020) proposed a large-scale sentiment classification corpus containing customer service dialogues in Chinese. However, this dataset is not publicly available. Saha et al. (2020) annotated dialogues from bAbI (Bordes et al., 2017) with sentiment for policy optimisation. Since dialogues are machine-generated, it is unclear how well these emotions match real human emotions and whether sentiment on its own sufficiently captures emotional nuances in task-oriented dialogue. In a similar spirit, Shi and Yu (2018) annotated the DSTC1 dataset with user sentiment. Unfortunately, containing only 50 dialogues, the dataset is very limited in terms of coverage and application in machine learning. To summarise, existing corpora are either limited in size or not publicly available, limiting further works on emotions in task-oriented dialogue systems. Furthermore, their annotation schemes focus on sentiment polarities, overlooking the effect of goals on users' emotional states.

Task-oriented Dialogues
MultiWOZ: Our dataset covers the entirety of Multi-WOZ, which was constructed using the Wizard-of-Oz framework (Kelley, 1984). It contains over 10k dialogues. Each dialogue was completed by two workers, each acting as the user or the operator, to achieve specified goals such as information retrieval or making reservations. There are 7 domains in total. A single dialogue or even a single turn can span multiple domains.
Complementary Dialogues: We envisage emotions as learning signal for dialogue system optimisation. Since emotions in task-oriented dialogue systems can be a direct effect of the user perception of the ability of the system to fulfill their goal, the policy performance can largely influence emotion distribution. During the life span of a data-driven task-oriented dialogue system, the distributions of dialogues and emotions may change as the policy learns and improves over time. An immediate impact of such a distributional shift is the increase in the number of negative emotions due to failed dialogues during the early stages of learning. Therefore, in addition to the human wizard policy in MultiWOZ, it is important that EmoWOZ covers a variety of dialogues which represent the emotions throughout such a dialogue system life span. We complement MultiWOZ with human-machine dialogues from a machine-generated policy (DialMAGE). To elicit more genuine reactions, we let subjects directly interact with a machinegenerated policy instead of human wizards trying to make machine-like mistakes. We launched a dialogue interactive task on Amazon Mechanical Turk, where workers are asked to retrieve information by interacting with the learning policy. We start with a policy trained in a supervised fashion on MultiWOZ that achieved a task success rate of 55% when evaluated with the ConvLab-2 (Zhu et al., 2020) rule-based user simulator. Throughout the task, the policy learned and improved as user feedback on task success is used for further training using RL. The policy reached a final human-rated success rate of 73%. Similar to , the policy uses a recurrent neural network (RNN) based model to produce multiple actions in a single turn, followed by the ConvLab-2 template-based NLG module for response generation.

Emotion Annotation Scheme
EmoWOZ focuses on user emotions rather than system ones. We believe recognising user emotions is the starting point for building emotion-aware task-oriented dialogue systems. We use the OCC model to arrive at  Table 2: Comparison between the OCC model and our labelling scheme. Emotions that do not occur in our dataset are marked as "not applicable to our dataset". {User, negative, impolite} has too few instances and {neutral, impolite} is not strong enough to be considered as abusive and therefore are not modelled for now. For simplicity, emotion words in blue are used to represent each emotion category. The OCC model is illustrated in Appendix A.
specific emotion categories. For that, we consider the following aspects: 1. Elicitor or cause: The OCC model defines three main elicitors of emotion: events, agents, and objects.
In task-oriented dialogues, events describe the situation which brings the user to interact with the system. For example, a user may be looking for a hotel for an upcoming trip or asking for the police information after a robbery. Agents are participants of the dialogue: the user and the system. Objects are equal to entities being talked about in the dialogue, such as the recommended hotel or the nearest police station. In our dataset, an object is always associated with either the operator, who proposes it, or an event, which drives the need for it. For this reason, we do not consider the object as an elicitor alone. On the other hand, within the agent category, it is important to distinguish between the user and the system. Therefore, we arrive at three elicitors for our annotation scheme: 1) the system, 2) the user, and 3) events (or facts).

Valence:
In essence, the OCC model describes emotion as a valenced reaction towards an elicitor. Valence is a dimension which expresses the positivity or negativity of emotion. For example, successfully achieving a goal is likely to bring positive valence, while a misunderstanding with an agent is likely to cause negative valence. As EmoWOZ will demonstrate in a later section, valence is highly related to task success or failure, making it an important signal for a task-oriented system. We distinguish neutral and emotional utterances, and further separate emotional utterances into those with negative and positive valence. 3. Conduct: Conduct is not a part of the OCC model, but given the rising concern of how humans behave when interacting with virtual assistants (Cercas Curry and Rieser, 2018), we decided to include it. Conduct describes the politeness of users and is usually associated with emotional acts. Politeness can indicate the degree of valence. For example, the user can express very strong dissatisfaction through rudeness. It also helps distinguish emotions such as those associ-ated with apology or abuse, which are both intrinsically negative.
Considering all combinations of these three aspects for annotation leads to a large number of classes. When choosing the final set of classes we were guided by whether or not a particular emotion category occurs in the database and the potential impact of that emotion category on the dialogue policy. We also carried out several trials and considered the ease of communicating to the annotator how to label such instances. We finally arrive at a set of 6 non-neutral emotion categories: An emotion elicited by the operator is defined as satisfied if it is positive, and dissatisfied if it is negative. Positive emotion caused by an event gives us excited, and negative fearful. In terms of negative emotions expressed towards the system, we consider user conduct to distinguish between dissatisfied and abusive, since they require very different responses from the system (Curry and Rieser, 2019). In terms of the negative emotions that users may direct toward themselves, we single out apologetic behaviours since it features in human-human information-seeking dialogues. Emotion categories and their attributes in the abovementioned aspects and their relation to the original OCC model are shown in Table 2.

Emotion Annotation Setup
We crowd-source the emotion annotation on Amazon Mechanical Turk in a controlled manner. As suggested by Carrión and López-Cózar (2008) to improve the annotation quality, workers are shown the dialogue history up to the utterance they are required to label. Each emotion category is followed by a list of emotion words that best fit into the category and an explanation. Due to the high subjectivity in the emotion annotation (Devillers et al., 2005), each dialogue is annotated by three different workers. We also implement several measures to ensure the quality of the emotion labels: Qualification tests: The test contains fifteen questions, seven are straight-forward and eight are more complex. The test also serves as a tutorial. For diffi-cult questions, hints are provided to guide the workers to identify implicit emotions and use contextual information (see Appendix B).
Hidden tests: We pre-label more than 1000 utterances containing obvious emotions and use them as sanity checks. The hidden tests serve as an indicator of worker reliability. If a worker scores above 80% on the hidden tests, we assume that the worker is reliable. Otherwise, the workers' submission is subject to manual review.
Review for outliers: We use a simple lexicon-based recogniser and manually annotate a small batch to have an estimate of the overall emotion distribution. If the label distribution in a worker's submissions deviates substantially from our prior belief, we mark them for manual review.
Annotation limit: We limit each worker to annotate at most 500 dialogues to ensure a diversity of workers and to avoid that workers adapt to our approval policy. Overall, we had 215 workers, each annotating 160 dialogues on average.

Linguistic Style
Dialogues from MultiWOZ and DialMAGE differ linguistically. As seen in Table 3, DialMAGE has longer dialogues than MultiWOZ as it takes longer for the machine-generated policy to accomplish user goals. Meanwhile, users use simpler and shorter sentences when talking to a machine. Especially when the system under-performs, users are discouraged to converse with it (see sample dialogues with annotations in Appendix C). We will analyse the impact of these differences on emotion recognition in Section 5.

Emotion Distribution
According to Table 4, the most common non-neutral emotion in EmoWOZ is satisfied, followed by dissatisfied. This is expected in task-oriented dialogues as users mainly express emotion in relation to their goals. While MultiWOZ contains more neutral utterances, it has a more diverse emotion distribution than DialMAGE. MultiWOZ contributes most satisfied utterances whereas DialMAGE contributes most dissatisfied utterances. This is in line with their respective dialogue-generating setup.
Sometimes users also express emotion to engage or provoke the operator. MultiWOZ contains more apologetic and less abusive utterances than DialMAGE, suggesting that users tend to be more polite when talking to human operators. Dialogues from MultiWOZ also contain more event-elicited emotions (fearful and excited) than DialMAGE. Users are more talkative when conversing with human operators. Users may describe a miserable situation they were experiencing, hoping to be helped and comforted. A human operator would naturally show empathy. In MultiWOZ, the operator sometimes asks if the user is alright when the user is looking for help from a robbery. When talking to machines, users tend not to express such chit-chat-style emotions due to the expected incapability of the machine to reciprocate. This indicates that an emotionally intelligent agent will allow dialogues that are emotionally richer and more nuanced, even in a task-oriented setting.

Inter-annotator Agreement
We measure the inter-annotator agreement by computing Fleiss' Kappa (Fleiss, 1971). Fleiss' Kappa for EmoWOZ is 0.602, suggesting a substantial agreement. Fleiss' Kappa for MultiWOZ is 0.611, higher than 0.465 for DialMAGE. Emotions in DialMAGE are more challenging to annotate because users express emotion less explicitly when they know that they are talking to a machine that does not react to emotions. Annotators often have to infer the user's implicit emotions from dialogue history, for example, based on repetitions or misunderstanding. Among all utterances, 72.1% see a full agreement among three annotators, 26.4% see a partial agreement, and 1.5% see no agreement. The count of each case in each subset can be found in Appendix D. Utterances for which no agreement is reached are resolved manually. Figure 1 illustrates the confusion matrix between annotators' labels and the golden labels. Most disagreements occur between non-neutral emotions and neutral, as well as abusive and dissatisfied. A reasonable explanation is that workers adopt different valence or impoliteness thresholds when they make decisions. Note that dissatisfied is rarely confused with abusive, but rather with neutral, suggesting that the ambiguity lies in when an expression of dissatisfaction is considered  Table 5: Comparison of baseline models. We report the F1 for each emotion label (Neutral, Fearful, Dissatisfied, Apologetic, Abusive, Excited, Satisfied) on EmoWOZ as well as Macro and Weighted F1 (excluding neutral) on EmoWOZ and its subsets. "Ctx." stands for "context". * indicates statistically significant difference with p < 0.05 between the best and the second best values in each column. Please refer to Appendix F.1 for more detailed results. to be rude or abusive, and not due to the similarity between abuse and dissatisfaction. On the other hand, confusions between fearful and dissatisfied suggest workers may also interpret elicitors differently. For example, a user may express negative emotions after the agent informed that there is no attraction meeting the user's criteria. While the emotion is caused by the fact that there is no match, one can also argue that the operator failed to suggest alternative options. We believe differences on interpretations are natural to a certain extent, as emotion appraisal may differ across individuals (Kuppens et al., 2007).

Emotion Recognition in Dialogue
Emotion recognition aims to recognise emotion within an utterance. Unlike utterances in isolation, emotion recognition in dialogues is highly contextual with respect to the dialogue history. As baselines, we compare two models originally developed for chit-chat emotion recognition as well as various BERT-based models. We believe emotion recognition is the first step towards an emotion-aware task-oriented agent, as a means for a deployed agent to obtain emotion information during an interaction.

Baselines
BERT (Devlin et al., 2019): BERT is used as the utterance encoder. Each user turn is encoded in isolation without any dialogue context. The [CLS] token from a bert-base-cased model is used as the feature representation, which is then fed into a linear output layer for classification.
ContextBERT: The set-up is identical to that of BERT, except that the entire dialogue history and the current user utterance are concatenated in the reversed order to form one long sequence. We add "User:" and "System:" to mark the speaker of each turn. DialogueRNN : The model combines gated recurrent units (GRUs) with an attention mechanism to capture the long-term trajectory of the dialogue. We experiment with using GloVe embeddings (Pennington et al., 2014) or the [CLS] representation from BERT as input features. When GloVe is used, a convolutional neural network (CNN) layer is used as a feature extractor to generate utterance representations. This CNN layer is dropped when using BERT features. COSMIC (Ghosal et al., 2020): This model also combines GRUs with the attention mechanism. In addition to utterance representations from a pretrained language model (LM), it supplements input features with common-sense knowledge extracted from a pre-trained commonsense transformer model called COMET (Bosselut et al., 2019). Although the original paper uses RoBERTa as input features, we found that BERT results in a better sequence representation for emotion recognition on our data. Therefore we use BERT as the utterance encoder in our experiments.

Experimental Setup
We perform a recognition task on the 7 emotions proposed in our annotation scheme 2 . All models are implemented in PyTorch (Paszke et al., 2019). For COS-MIC and DialogueRNN, we use the code provided by the respective papers. We include more details on the hyperparameters of each model in Appendix E. To split EmoWOZ into training, validation, and testing sets, we  Table 6: Performance of ContextBERT in cross-dataset experiments. We report the F1 for each emotion label (Neutral, Fearful, Dissatisfied, Apologetic, Abusive, Excited, Satisfied), as well as Macro and Weighted F1 (excluding neutral). * indicates statistically significant difference with p < 0.05 between the best and the second best values in each column. For detailed results, please refer to Appendix F.1.
keep the original split of MultiWOZ and further divide DialMAGE with a ratio of 8:1:1, leading to 9,234, 1,100, and 1,100 dialogues in each set. We run each task on 5 different seeds and report the average performance.

Results and Discussion
Recognition on emotion classes. Table 5 summarises the performance of baseline models. Since almost 70% of the annotations are neutral, we exclude it when calculating average F1 scores. In general, models that take into account context information perform better on the full EmoWOZ. This shows the importance of context or dialogue-level features in emotion recognition in task-oriented dialogues. An exception is DialogueRNN with GloVe feature, which underperforms in EmoWOZ macro F1, likely due to the non-contextual embedding used. On the other hand, BERT scores very well on MultiWOZ dialogues but performs poorly on DialMAGE for both setups. This suggests that emotions in MultiWOZ are less contextdependent. BERT, the only non-contextual model among our baselines, performs well for apologetic, excited, and satisfied, potentially due to the existence of distinguishable keywords associated with these emotions such as "thank you" for satisfied and "sorry" for apologetic. These emotion labels do not benefit much from context. In contrast, BERT produces a significantly worse F1 on dissatisfied, probably because users tend to express dissatisfaction more implicitly, for instance via repetition or correction, making dialogue-level features necessary. Figure 2 shows two dialogues with implicit emotions and predictions made by respective baseline models. In example 1, the system gives the wrong time of arrival, eliciting mild annoyance from the user. BERT predicts neutral because in isolation, the utterance has no words suggesting dissatisfaction. All other models correctly recognise dissatisfied, as they capture the misunderstanding occurs in previous dialogue turns. Example 2 presents a similar but more implicit case, where all models fail. This shows that EmoWOZ contains contextualised emotions that are more implicit and subtle, requiring more sophisticated features and models. Complementarity between MultiWOZ and Dial-MAGE. Due to different linguistic features and emotion distributions in MultiWOZ and DialMAGE, one concern is that the models learn to predict emotion based on these statistical artifacts. According to Table 3, the most obvious difference is the average utterance length (5.8 in DialMAGE and 11.8 in MultiWOZ). A naive model may simply recognise the data source from word count and predict the most likely emotion from that source. Table 7 presents how ContextBERT trained on EmoWOZ predicts emotion in long Dial-MAGE and short MultiWOZ utterances. The emotion distribution in model prediction is vastly different from that in the complementing subset. Clearly, the model does not simply count words to decide on the underlying emotion.  Table 7: Emotion distribution in labels and Con-textBERT prediction. See Appendix F.5 for full results. Recall and precision on satisfied and dissatisfied for task-oriented dialogue. We further investigate the change in F1 of each emotion on MultiWOZ by looking at the change in recall and precision after complementing MultiWOZ with DialMAGE. We believe it is necessary to distinguish recall and precision, as for some emotions, one may be more important than the other. The relative importance of recall and precision for each emotion class depends on its implication to a task-oriented dialogue system and the consequence of false recognition. Most importantly for task-oriented dialogue system, a high recall of dissatisfied is desirable because the system should not miss any failure in dialogues. Failing to recognise dissatisfaction can trigger more anger from the user and therefore impair task completion (see Figure C.2). On the other hand, a high precision may be more desirable for all other emotions to ensure proper affective response from the system. When the relative importance of recall and precision of the emotion is taken into account, complementing Mul-tiWOZ with DialMAGE is beneficial to {dissatisfied} for higher recall and {fearful, excited, satisfied} for higher precision, see Table 8. Detailed results can be found in Appendix F.4.

Emotions for Dialogue State Tracking
In task-oriented dialogues, dialogue state tracking (DST) aims to continuously track the user's goal and intent as the dialogue progresses (Young et al., 2010). We hypothesise that the user emotion can help inform the system about their goal. To investigate this, we train a dialogue state tracker that incorporates an additional task to predict one of 7 emotional classes on MultiWOZ 2.1 (Eric et al., 2020). We utilise the outof-task training approach and the available code presented in (Heck et al., 2020a). We follow the multitask learning (MTL) algorithm, where on each training step, the same model is trained on two different batches, one from the main task (DST) and one from the auxiliary task (emotion recognition). Since neutral emotion provides limited information on the user goal, we remove a half of the neutral utterances when performing MTL. We show that additional emotion labels can lead to a significant improvement (p < 0.02) in the joint goal accuracy (JGA) of DST (see Table 9).

Conclusion
In this work, we examined emotions and their expression in the context of task-oriented dialogues, where emotions are centred around a user goal. We used the OCC model as a starting point to derive a comprehensive annotation scheme beyond sentiment polarity for emotions in relation to user goals. We designed a set of 7 emotions that differ in terms of valence, conduct and elicitor to capture the cognitive context of emotions, while maintaining labeling simplicity. With EmoWOZ, we present a publicly available, large-scale human-annotated emotion corpus consisting of Wizard-of-Oz style as well as dialogues with a machine-generated policy.
Our intention with EmoWOZ is to overcome the lack of large emotionlabelled corpora to support research towards emotionaware task-oriented dialogue systems, for dialogues closer to human-human interactions. We apply various emotion recognition models to EmoWOZ and examined the effect of context for different emotions. In cross-dataset experiments we analysed the complementarity of WOZ-style data and machine-generated policy data. Our results show that recognising context-dependent and implicit emotions from task-oriented dialogues is a challenging task that will benefit from further research. EmoWOZ provides an ideal test bed for that. Lastly, we leveraged emotion recognition in the dialogue state tracking task to exemplify the utility of emotion labels in dialogue modeling. We hope this dataset can offer insights beyond the scope of emotion recognition and push the performance of downstream tasks in task-oriented dialogue modelling. In future work, we plan to investigate tailored models for emotion recognition in task-oriented dialogues that take advantage of high-level features such as dialogue acts or belief states. We are also interested in using emotion as a feedback signal within reinforcement learning policy optimisation.        Table F1: Performance of baseline models on emotion classification including cross-dataset experiments. For cross-dataset experiments, the "X → Y"s in the 'Set-up' column represents the training and evaluation set-up, where X is the training set and Y is the test set. E stands for EmoWOZ, M stands for MultiWOZ, and D stands for DialMAGE. M → D, for example, means to train on MultiWOZ and test on DialMAGE. Extreme values for "Apologetic" and "Abusive" in DialMAGE ("* → D"s) are caused by their rarity in the test set (1 and 5 occurrences respectively).   Table F4: Precision, recall and F1 score of ContextBERT for all emotions when trained on MultiWOZ and EmoWOZ respectively, and tested on MultiWOZ. A → B represents how the value change after complementing MultiWOZ with DialMAGE in training. A is the value when trained on MultiWOZ and B is the value when trained on EmoWOZ. Values with statistical significance (p < 0.1) are bolded and colored where red indicates a drop and green indicates an improvement. For recognising user emotions in task-oriented dialogues, a high precision is more desirable for neutral, fearful, apologetic, abusive, excited, and satisfied where as a high recall is more desirable for dissatisfied.  Table F5: Emotion distribution in model predictions (trained on EmoWOZ).