{'fraction_non_alphanumeric': 0.04771241830065359, 'fraction_numerical': 0.013435003631082063, 'mean_word_length': 4.490829346092504, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa

We present EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa, a simple yet expressive scheme of solving the ERC (emotion recognition in conversation) task. By simply prepending speaker names to utterances and inserting separation tokens between the utterances in a dialogue, EmoBERTa can learn intra- and inter- speaker states and context to predict the emotion of a current speaker, in an end-to-end manner. Our experiments show that we reach a new state of the art on the two popular ERC datasets using a basic and straight-forward approach. We've open sourced our code and models at https://github.com/tae898/erc.


Introduction
The scope of emotion recognition is very wide, ranging from stills of face images, audio data to the actual utterances or text as in tweets. In this paper, we focus on emotion recognition in conversation (ERC), which is a subfield of emotion recognition. More specifically, the task is to predict the emotion of a current speaker who's engaging in a conversation with one person or more. Recognizing emotion is important to areas such as affective computing and human-robot communication, in which it can be an important feedback mechanism.
As humans use multiple sensory inputs to have a conversation (e.g., vision, voice, etc.) the ERC task can also include multiple modalities (e.g., visual, audio, text, etc.). Here, we report on our first experiments on the text modality, leaving using multiple modalities within our framework to future work.
Since the introduction of the Transformer (Vaswani et al., 2017), transformer-based deep neural network models have become the dominating neural network model in sequence modeling. Especially, pretrained encoder-only models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown that they can be success-fully fine-tuned for downstream tasks, such as sentence classification and question answering. ERC can be seen as a special case of sequence modeling, since emotions are expected to be triggered by a preceding event in any modality.
Our approach to ERC enriches such transformer models by including the speaker identity in the sequence information spanning multiple utterances. By adapting the RoBERTa sequence representation, we improve the SOTA on two popular benchmark datasets.

Related Work
Most of the existing works on ERC combine different kinds of neural network architectures (e.g., CNNs, RNNs, Transformers, GNNs, etc.) (Li et al., 2020b), (Li et al., 2020c), (Ishiwatari et al., 2020), (Wang et al., 2020), (Hazarika et al., 2021), (Sheng et al., 2020), (Ghosal et al., 2020), (Ghosal et al., 2019). The biggest downside of such approaches is that each part of the model is responsible for extracting their own features. These extracted features might not be ideal for the other parts of the model. Also, since these models are combinations of sub-models, it is hard to understand what each sub-model is contributing and how to improve the overall model. Some of the approaches try to take advantage of external knowledge bases (Ghosal et al., 2020), (Zhong et al., 2019a, which adds even more complexity to the model. A substantial number of approaches are heavily based on RNNs (e.g., GRU) to model the sequence (Jiao et al., 2019), (Lu et al., 2020. The biggest problem with this is that it inherently decouples the word embedding extraction and the sequence modeling, whereas BERT-like models tackle them at once, often leading to a better performance. Also, they have to rely on external decontextualized word embedding extractors (e.g., GloVe (Pennington et al., 2014) or word2vec (Mikolov et al., 2013)). Furthermore, training an RNN is very inefficient since "backpropagation through time" has to wait until the last input of a sequence has been processed.
The approaches that are most closely related to us are HiTrans (Li et al., 2020a) and DialogXL (Shen et al., 2020). HiTrans packs multiple utterances with [CLS] tokens prepended into one input sequence. This sequence is first fed into a BERT and then to another transformer. DialogXL is based on XLNet (Yang et al., 2019). Our approach differs in that we just use RoBERTa and encode the speaker information with multiple utterances.

Problem Definition
Let's say a dialogue of M utterances is given and I interlocutors are engaging in a conversation. Then the dialogue can be expressed as a list of vectors: where an utterance x t contains several words (tokens). Each utterance x t is spoken by a unique speaker ∈ I. Since this is a supervised setup, every utterance x t has one corresponding label y t that is annotated by a human.
The simplest way to solve this problem is to come up with a function f that takes x t as an input and outputs the correct label y t . However, this doesn't take context into account. It's easily conceivable that the function f should also consider the past utterances [x 1 , x 2 , ..., x t−1 ] or even the future utterances [x t+1 , x t+2 , ..., x M ] to model the context.

EmoBERTa
EmoBERTa starts from the pretrained roberta-large model (Liu et al., 2019). Since the task is basically a sequence classification task, we simply add a randomly initialized linear layer with the softmax nonlinearity to the first hidden state (this state corresponds to the [CLS] token) of the last layer of the pretrained model.
We chose RoBERTa, among the many BERTlike models, because its structure is not only relatively simple, but also it can deal with more than two segments. The original authors of RoBERTa simply used two </s> tokens consecutively as [SEP] token, which separates the first and the second segments. Although the pretrained model has not been trained on more than two segments, we show that EmoBERTa can be generalized to three segments per input sequence.
The first, second, and third segments contain the past utterances, the current utterance, and the future utterances, respectively, in a dialogue. Each utterance is prepended with the name of a speaker so that the model is aware which utterance is spoken by whom. The task is to predict the emotion of the current utterance.
RoBERTa uses <s> and </s> as [CLS] and [EOS] tokens, respectively. Building an input sequence for EmoBERTa is outlined in Algorithm 1. Example sequences can be found in Figure 1.

Training
The loss is calculated as the sum of cross entropy loss and L 2 weight decay (Krogh and Hertz, 1991). We use adaptive gradient descent (Kingma and Ba, 2015), (Loshchilov and Hutter, 2019) with gradual linear warmup learning rate scheduling (Goyal et al., 2017). The peak learning rate was determined using Optuna (Akiba et al., 2019). Mixed floating point precision was used to reduce the training time and increase the batch size (Micikevicius et al., 2017). See Appendix A.1 for the details (e.g., hyperparameters, training time, hardware, etc.). We mostly used the huggingface transformer pytorch library for training (Wolf et al., 2020).
Then the training loss function is (1) where y (i) , is a one-hot label vector,ŷ (i) is the softmax output vector given the input x (i) , N is the number of training data samples, C is the number of classes, λ is a L 2 regularization rate, and w are the weights of the model. In practice, the data samples are batched, and stochastic gradient descent is used.
Although the weights w were tuned to minimize the loss value, the final model is chosen where the weighted f 1 score on the validation split is the highest, since that's the metric that we report. We test EmoBERTa on the two popular ERC datasets. MELD (Poria et al., 2019) is a multimodal (visual, audio, and text) and multi-party (more than two interlocutors in a dialogue) conversational dataset. It was collected from the TV series Friends. The seven emotions are neutral, joy, surprise, anger, sadness, disgust, and fear. Weighted f 1 score is used to evaluate the performance, as the class distribution is highly imbalanced.

Datasets and Evaluation Metrics
IEMOCAP (Busso et al., 2008) is a multimodal (visual, audio, and text) and dyadic (only two interlocutors in a dialogue) conversational dataset. Ten actors participated in the data collection. Although the original dataset contains 11 different emotions, only six of them are used for evaluation. They are neutral, frustration, sadness, anger, excited, and happiness. As MELD, the class distribution is highly imbalanced and thus a weighted f 1 score will be used for evaluation. Unlike MELD, IEMOCAP does not officially have the names of the speakers. Therefore, we gave each actor a random name. See Appendix A.2 for the details.
Some statistics on train, val, and test splits of both datasets can be found at Table 1.

Baselines
We compare our model with the models we mentioned in Section 2.  EmoBERTa shows very good results: max weighted f 1 scores (%) of 65.61 (MELD) and 68.57 (IEMOCAP) respectively and above the best reported SOTA, especially considering that no modifications were made to the original RoBERTa model architecture. We also trained a model without the speaker names prepended, which drops the performance: weighted f 1 scores (%) of 65.07 (MELD) and 64.02 (IEMOCAP) respectively, pro-viding evidence that encoding the speaker information helps.

Quantitative Analysis
Especially on IEMOCAP dataset, although we had to come up with random names for the actors, EmoBERTa was able to learn what's important. As for IEMOCAP, the results were better using only past utterances, rather than using both past and future utterances. We believe that this is due to the fact that IEMOCAP has many more utterances than MELD, and thus we couldn't fit all the past and future utterances in one sequence, meaning that only using past utterances can fit more past utterances than aiming for both. Past utterances were apparently more useful than future utterances to predict the emotion of a current utterance.
There was a bigger performance gain by incorporating past and/or future utterances in IEMOCAP than MELD (12.48 vs. 2.15). We believe that this is due to the fact that the nature of IEMOCAP is more contextual than that of MELD.

Qualitative Analysis
To get more insight in the value of encoding the speaker, we did a qualitative analysis on 10 correctly and 10 incorrectly classified random samples from each test split.
Our manual inspection shows that the model tries to learn the dynamics of the interlocutors in the beginning layer, as the current speaker tokens attend to the interlocutor tokens (We observed this behavior from all the 20 MELD and 20 IEMOCAP random test samples). As for MELD, in all 100% of the correctly classified samples, based on the top 10 attended tokens, the <s> token of the last layer attended to the speaker token of the target (current) utterance, whereas this ratio was only 60% for the incorrectly classified samples. This verifies that the current speaker token increasingly contains important information, as the tokens move on to the higher layers. Figure 1 shows a visualization of one correctly and one incorrectly classified random samples.
As the <s> token in the last layer focuses on the current speaker and his/her utterance, we believe that this information is what the model finds most useful to make the final prediction.
Note that in the incorrectly classified example, the <s> token in the last layer does not focus on the current speaker but some random punctuation marks throughout the conversation, thus leading to an incorrect prediction.

See Appendix A.3 for the random IEMOCAP test samples.
<s>PHOEBE: I think she took it pretty well. You know Paolo's over there right now, so... ROSS: Ah...ooh! Well, looks like, uh, we kicked your butts. JOEY: No-no, she kicked our butts. You could be on the Olympic standing-there team. ROSS: Come on, two on one. CHANDLER: What are you still doing here? She just broke up with the guy, it's time for you to swoop in! ROSS: What, now? JOEY: Yes, now is when you swoop!</s></s>JOEY: You gotta make sure that when Paolo walks out of there, the first guy Rachel sees is you, She's gotta know that you're everything he's not!</s></s>JOEY: You're like, like the anti-Paolo! CHANDLER: My Catholic friend is right. CHANDLER: She's distraught. CHANDLER: You're there for her. CHANDLER: You pick up the pieces, and then you usher in the age of Ross!</s> <s>PHOEBE: I think she took it pretty well. You know Paolo's over there right now, so... ROSS: Ah...ooh! Well, looks like, uh, we kicked your butts. JOEY: No-no, she kicked our butts. You could be on the Olympic standing-there team. ROSS: Come on, two on one. CHANDLER: What are you still doing here? She just broke up with the guy, it's time for you to swoop in! ROSS: What, now? JOEY: Yes, now is when you swoop!</s></s>JOEY: You gotta make sure that when Paolo walks out of there, the first guy Rachel sees is you, She's gotta know that you're everything he's not!</s></s>JOEY: You're like, like the anti-Paolo! CHANDLER: My Catholic friend is right. CHANDLER: She's distraught. CHANDLER: You're there for her. CHANDLER: You pick up the pieces, and then you usher in the age of Ross!</s> (a) A correctly classified example. Both the prediction and the truth are joy.

Conclusion
In this paper, we showed that our new model, EmoBERTa, outperforms other models in the ERC task. Since EmoBERTa can directly attend to the input tokens and interlocutor names, we can easily observe the attention coefficients to see which part of the dialogue the model finds most important to make a final classification.  We see a similar behavior as in MELD. Again, in the incorrectly classified example, the <s> in the last layer does not attend to the current speaker.