{'fraction_non_alphanumeric': 0.03246895429947762, 'fraction_numerical': 0.011292189308392505, 'mean_word_length': 4.860678276810265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues

Indirect speech such as sarcasm achieves a constellation of discourse goals in human communication. While the indirectness of figurative language warrants speakers to achieve certain pragmatic goals, it is challenging for AI agents to comprehend such idiosyncrasies of human communication. Though sarcasm identification has been a well-explored topic in dialogue analysis, for conversational systems to truly grasp a conversation’s innate meaning and generate appropriate responses, simply detecting sarcasm is not enough; it is vital to explain its underlying sarcastic connotation to capture its true essence. In this work, we study the discourse structure of sarcastic conversations and propose a novel task – Sarcasm Explanation in Dialogue (SED). Set in a multimodal and code-mixed setting, the task aims to generate natural language explanations of satirical conversations. To this end, we curate WITS, a new dataset to support our task. We propose MAF (Modality Aware Fusion), a multimodal context-aware attention and global information fusion module to capture multimodality and use it to benchmark WITS. The proposed attention module surpasses the traditional multimodal fusion baselines and reports the best performance on almost all metrics. Lastly, we carry out detailed analysis both quantitatively and qualitatively.


Introduction
The use of figurative language serves many communicative purposes and is a regular feature of both oral and written communication (Roberts and Kreuz, 1994). Predominantly used to induce humour, criticism, or mockery (Colston, 1997), paradoxical language is also used in concurrence with hyperbole to show surprise (Colston and Keller, 1998) as well as highlight the disparity between expectations and reality (Ivanko and Pexman, 2003). While the use and comprehension of sarcasm is a * Equal contribution Figure 1: Sarcasm Explanation in Dialogues (SED). Given a sarcastic dialogue, the aim is to generate a natural language explanation for the sarcasm in it. Blue text represents the English translation for the text.
cognitively taxing process (Olkoniemi et al., 2016), psychological evidence advocate that it positively correlates with the receiver's theory of mind (ToM) (Wellman, 2014), i.e., the capability to interpret and understand another person's state of mind. Thus, for NLP systems to emulate such anthropomorphic intelligent behavior, they must not only be potent enough to identify sarcasm but also possess the ability to comprehend it in its entirety. To this end, moving forward from sarcasm identification, we propose the novel task of Sarcasm Explanation in Dialogue (SED).
For dialogue agents, understanding sarcasm is even more crucial as there is a need to normalize its sarcastic undertone and deliver appropriate responses. Conversations interspersed with sarcastic statements often use contrastive language to convey the opposite of what is being said. In a real-world setting, understanding sarcasm goes beyond negat-ing a dialogue's language and involves the acute comprehension of audio-visual cues. Additionally, due to the presence of essential temporal, contextual, and speaker-dependent information, sarcasm understanding in conversation manifests as a challenging problem. Consequently, many studies in the domain of dialogue systems have investigated sarcasm from textual, multimodal, and conversational standpoints (Ghosh et al., 2018;Castro et al., 2019;Oraby et al., 2017;Bedi et al., 2021). However, baring some exceptions (Mishra et al., 2019;Dubey et al., 2019;Chakrabarty et al., 2020), research on figurative language has focused predominantly on its identification rather than its comprehension and normalization. This paper addresses this gap by attempting to generate natural language explanations of satirical dialogues.
To illustrate the proposed problem statement, we show an example in Figure 1. It contains a dyadic conversation of four utterances u 1 , u 2 , u 3 , u 4 , where the last utterance (u 4 ) is a sarcastic remark. Note that in this example, although the opposite of what is being said is, "I don't have to think about it," it is not what the speaker means; thus, it enforces our hypothesis that sarcasm explanation goes beyond simply negating the dialogue's language. The discourse is also accompanied by ancillary audio-visual markers of satire such as an ironical intonation of the pitch, a blank face, or roll of the eyes. Thus, conglomerating the conversation history, multimodal signals, and speaker information, SED aims to generate a coherent and cohesive natural language explanation associated with sarcastic dialogues.
For the task at hand, we extend MASAC (Bedi et al., 2021) -a sarcasm detection dataset for codemixed conversations -by augmenting it with natural language explanations for each sarcastic dialogue. We name the dataset WITS 1 . The dataset is a compilation of sarcastic dialogues from a popular Indian TV show. Along with the textual transcripts of the conversations, the dataset also contains multimodal signals of audio and video.
We experiment with unimodal as well as multimodal models to benchmark WITS. Text, being the driving force of the explanations, is given the primary importance, and thus, we compare a number of established text-based sequence-to-sequence systems on WITS. To incorporate multimodal information, we propose a unique fusion scheme of 1 WITS: "Why Is This Sarcastic" Multimodal Context-Aware Attention (MCA2). Inspired by , this attention variant facilitates deep semantic interaction between the multimodal signals and textual representations by conditioning the key and value vectors with audio-visual information and then performing dot product attention with these modified vectors. The generated audio and video information-informed textual representations are then combined using the Global Information Fusion Mechanism (GIF). The gating mechanism of GIF allows for the selective inclusion of information relevant to the satirical language and also prohibits any multimodal noise from seeping into the model. We further propose MAF (Modality Aware Fusion) module where the aforementioned mechanisms are introduced in the Generative Pretrained Models (GPLMs) as adapter modules. Our fusion strategy outperforms the textbased baselines and the traditional multimodal fusion schemes in terms of multiple text-generation metrics. Finally, we conduct a comprehensive quantitative and qualitative analysis of the generated explanations.
In a nutshell, our contributions are four fold: • We propose Sarcasm Explanation in Dialogue (SED), a novel task aimed at generating a natural language explanation for a given sarcastic dialogue, elucidating the intended irony. • We extend an existing sarcastic dialogue dataset, to curate WITS, a novel dataset containing human annotated gold standard explanations. • We benchmark our dataset using MAF-TAV B and MAF-TAV M variants of BART and mBART, respectively, that incorporate the audio-visual cues using a unique context-aware attention mechanism. • We carry out extensive quantitative and qualitative analysis along with human evaluation to assess the quality of the generated explanations.

Reproducibility:
The source codes and the dataset can be found here: https://github.com/LCS2-IIITD/MAF.git. Sarcasm and Text: Joshi et al. (2017) presented a well-compiled survey on computational sarcasm where the authors expanded on the relevant datasets, trends, and issues for automatic sarcasm identification. Early work in sarcasm detection dealt with standalone text inputs like tweets and reviews (Kreuz and Caucci, 2007;Tsur et al., 2010;Joshi et al., 2015;Peled and Reichart, 2017). These initial works mostly focused on the use of linguistic and lexical features to spot the markers of sarcasm (Kreuz and Caucci, 2007;Tsur et al., 2010). More recently, attention-based architectures are proposed to harness the inter-and intra-sentence relationships in texts for efficient sarcasm identification (Tay et al., 2018;Xiong et al., 2019;Srivastava et al., 2020). Analysis of figurative language has also been extensively explored in conversational AI setting. Ghosh et al. (2017) utilised attentionbased RNNs to identify sarcasm in the presence of context. Two separate LSTMs-with-attention were trained for the two inputs (sentence and context) and their hidden representations were combined during the prediction.

Related Work
The study of sarcasm identification has also expanded beyond the English language. Bharti et al.  (2021) proposed a code-mixed multi-party dialogue dataset, MASAC, for sarcasm and humor detection. In the bimodal setting, sarcasm identification with tweets containing images has also been well explored (Cai et al., 2019;Xu et al., 2020;Pan et al., 2020) . In summary, much work has been done in sarcasm detection, but little, if any, effort has been placed into explaining the irony behind sarcasm. This paper attempts to fill this gap by proposing a new problem definition and a supporting dataset.

Dataset
Situational comedies, or 'Sitcoms', vividly depict human behaviour and mannerism in everyday reallife settings. Consequently, the NLP research community has successfully used such data for sarcasm identification (Castro et al., 2019;Bedi et al., 2021). However, as there is no current dataset tailored for the proposed task, we curate a new dataset named WITS, where we augment the already existing MASAC dataset (Bedi et al., 2021) with explanations for our task. MASAC is a multimodal, multi-party, Hindi-English code-mixed dialogue dataset compiled from the popular Indian TV show, 'Sarabhai v/s Sarabhai' 2 . We manually analyze the data and clean it for our task. While the original dataset contained 45 episodes of the TV series, we add 10 more episodes along with their transcription and audio-visual boundaries. Subsequently, we select the sarcastic utterances from this augmented dataset and manually define the utterances to be included in the dialogue context for each of them. Finally, we are left with 2240 sarcastic dialogues with the number of contextual utterances ranging from 2 to 27. Each of these instances is manually   annotated with a corresponding natural language explanation interpreting its sarcasm. Each explanation contains four primary attributes -source and target of sarcasm, action word for sarcasm, and an optional description for the satire as illustrated in Figure 1. In the explanation "Indu implies that Maya is not looking good.", 'Indu' is the sarcasm source, 'Maya' is the target, 'implies' is the action word, while 'is not looking good' forms the description part of the explanation. We collect explanations in code-mixed format to keep consistency with the dialogue language. We split the data into train/val/test sets in an 80:10:10 ratio for our experiments, resulting in 1792 dialogues in the train set and 224 dialogues each in the validation and test sets. The next section illustrates the annotation process in more detail. Table 1 and Figure 2 show detailed statistics of WITS.

Annotation Guidelines
Each of the instance in WITS is associated with a corresponding video, audio, and textual transcript such that the last utterance is sarcastic in nature. We first manually define the number of contextual utterances required to understand the sarcasm present in the last utterance of each dialogue. Further, we provide each of these sarcastic statements, along with their context, to the annotators who are asked to generate an explanation for these instances based on the audio, video, and text cues. Two annotators were asked to annotate the entire dataset. The target explanation is selected by calculating the cosine similarity between the two explanations. If the cosine similarity is greater than 90% then the shorter length explanation is selected as the target explanation. Otherwise, a third annotator goes through the dialogue along with the explanations and resolves the conflict. The average cosine similarity after the first pass is 87.67%. All the final selected explanations contain the following attributes: • Sarcasm source: The speaker in the dialog who is being sarcastic. • Sarcasm target: The person/ thing towards whom the sarcasm is directed. • Action word: Verb/ action used to describe how the sarcasm is taking place. For e.g. mocks, insults, taunts, etc. • Description: A description about the scene which helps in understanding the sarcasm. Figure 1 represents an example annotation from WITS with its attributes.

Proposed Methodology
In this section, we present our model and its nuances. The primary goal is to smoothly integrate multimodal knowledge into the BART architecture. To this end, we introduce Multimodal Aware Fusion (MAF), an adapter-based module that comprises of Multimodal Context-Aware Attention (MCA2) and Global Information Fusion (GIF) mechanisms.  Given the textual input sarcastic dialogue along with the audio-video cues, the former aptly introduces multimodal information in the textual representations, while the latter conglomerates the audiovisual information infused textual representations. This adapter module can be readily incorporated at multiple layers of BART/mBART to facilitate various levels of multimodal interaction. Figure 3 illustrates our model architecture.

Multimodal Context Aware Attention
The traditional dot-product-based cross-modal attention scheme leads to the direct interaction of textual representations with other modalities. Here the text representations act as the query against the multimodal representations, which serve as the key and value. As each modality comes from a different embedding subspace, a direct fusion of multimodal information might not retain maximum contextual information and can also leak substantial noise in the final representations. Thus, based on the findings of , we propose multimodal fusion through Context Aware Attention. We first generate multimodal information conditioned key and value vectors and then perform the traditional scaled dot-product attention. We elaborate on the process below. Given the intermediate representation H generated by the GPLMs at a specific layer, we calculate the query, key, and value vectors Q, K, and V ∈ R n×d , respectively, as given in Equation 1, where W Q , W K , and W V ∈ R d×d are learnable parameters. Here, n denotes the maximum sequence length of the text, and d denotes the dimensionality of the GPLM generated vector.
Let C ∈ R n×dc denote the vector obtained from audio or visual representation. We generate multimodal information informed key and value vec-torsK andV , respectively, as given by . To decide how much information to integrate from the multimodal source and how much information to retain from the textual modality, we learn matrix λ ∈ R n×1 (Equation 3). Note that U k and U v ∈ R dc×d are learnable matrices.
Instead of making λ k and λ v as hyperparameters, we let the model decide their values using a gating mechanism as computed in Equation 3. The matrices of W k 1 , W k 2 , W v 1 , and W v 2 ∈ R d×1 are trained along with the model.
Finally, the multimodal information infused vec-torsK andV are used to compute the traditional scaled dot-product attention. For our case, we have two modalities -audio and video. Using the context-aware attention mechanism, we obtain the acoustic-information-infused and visualinformation infused vectors H A and H V , respectively (c.f. Equations 4 and 5).

Global Information Fusion
In order to combine the information from both the acoustic and visual modalities, we design the GIF block. We propose two gates, namely the acoustic gate (g a ) and the visual gate (g v ) to control the amount of information transmitted by each modality. They are as follows: Here, W a , W v ∈ R 2d×d and b a , b v ∈ R d×1 are trainable parameters, and ⊕ denotes concatenation. The final multimodal information fused representa-tionĤ is given by Equation 8.
This vectorĤ is inserted back into GPLM for further processing.

Experiments, Results and Analysis
In this section, we illustrate our feature extraction strategy, the comparative systems, followed by the results and its analysis. For a quantitative analysis of the generated explanations, we use the standard metrics for generative tasks -ROUGE-1/2/L (Lin, 2004), BLEU-1/2/3/4 (Papineni et al., 2002), and METEOR (Denkowski and Lavie, 2014). To capture the semantic similarity, we use the multilingual version of the BERTScore .

Feature Extraction
Audio: Acoustic representations for each instance are obtained using the openSMILE python library 3 . We use a window size of 25 ms and a window shift of 10 ms to get the non-overlapping frames. Further, we employ the eGeMAPS model (Eyben et al., 2016) and extract 154 dimensional functional features such as Mel Frequency Cepstral Coefficients (MFCCs) and loudness for each frame of the instance. These features are then fed to a Transformer encoder (Vaswani et al., 2017) for further processing.
Video: We use a pre-trained action recognition model, ResNext-101 (Hara et al., 2018), trained on the Kinetics dataset (Kay et al., 2017) which can recognise 101 different actions. We use a frame rate of 1.5, a resolution of 720 pixels, and a window length of 16 to extract the 2048 dimensional visual features. Similar to audio feature extraction, we employ a Transformer encoder (Vaswani et al., 2017) to capture the sequential dialogue context in the representations.

Comparative Systems
To get the best textual representations for the dialogues, we experiment with various sequence-tosequence (seq2seq) architectures. RNN: We use the openNMT 4 implementation of the RNN seqto-seq architecture. Transformer (Vaswani et al., 2017): The standard Transformer encoder and decoder are used to generate explanations in this case. Pointer Generator Network (See et al., 2017): A seq-to-seq architecture that allows the generation of new words as well as copying words from the input text for generating accurate summaries. BART (Lewis et al., 2020): It is a denoising autoencoder model with standard machine translation architecture with a bidirectional encoder and an auto-regressive left-to-right decoder. We use its base version. mBART (Liu et al., 2020): Following the same architecture and objective as BART, mBART is trained on large-scale monolingual corpora in different languages 5 .

Results
Text Based: As evident from Table 2, BART performs the best across all the metrics for the textual modality, showing an improvement of almost 2-3% on the METEOR and ROUGE scores when compared with the next best baseline. PGN, RNN, and Transformers demonstrate admissible performance considering that they have been trained from scratch. However, it is surprising to see mBART not performing better than BART as it is trained on multilingual data. We elaborate more on this in Appendix A.1.  Multimodality: Psychological and linguistic literature suggests that there exist distinct paralinguistic cues that aid in comprehending sarcasm and humour (Attardo et al., 2003;Tabacaru and Lemmens, 2014). Thus, we gradually merge auditory and visual modalities using MAF module and obtain MAF-TAV B and MAF-TAV M for BART and mBART, respectively. We observe that the inclusion of acoustic signals leads to noticeable gains of 2-3% across the ROUGE, BLEU, and METEOR scores. The rise in BERTScore also suggests that the multimodal variant generates a tad more coherent explanations. As ironical intonations such as mimicry, monotone, flat contour, extremes of pitch, long pauses, and exaggerated pitch (Rockwell, 2007) form a significant component in sarcasm understanding, we surmise that our model, to some extent, is able to spot such markers and identify the intended sarcasm behind them.
We notice that visual information also contributes to our cause. Significant performance gains are observed for MAF-TV B and MAF-TV M , as all the metrics show a rise of about 3-4%. While MAF-TA B gives marginally better performance over MAF-TV B in terms of R1, RL, and B1, we see that MAF-TV B performs better in terms of the rest of the metrics. Often, sarcasm is depicted through gestural cues such as raised eyebrows, a straight face, or an eye roll (Attardo et al., 2003). Moreover, when satire is conveyed by mocking someone's looks or physical appearances, it becomes essential to incorporate information expressed through visual media. Thus, we can say that, to some extent, our model is able to capture these nuances of non-verbal cues and use them well to normalize the sarcasm in a dialogue. In summary, we conjecture that whether independent or together, audio-visual signals bring essential information to the table for understanding sarcasm.   Table 3 reports the ablation study. CONCAT1 represents the case where we perform bimodal concatenation ((T ⊕ A), (T ⊕ V )) instead of the MCA2 mechanism, followed by the GIF module, whereas, CONCAT2 represents the simple trimodal concatenation (T ⊕ A ⊕ V ) of acoustic, visual, and textual representations followed by a linear layer for dimensionality reduction. In comparison with MCA2, CONCAT2 reports a below-average performance with a significant drop of more than 14% for MAF-TAV B and MAF-TAV M . This highlights the need to have deftly crafted multimodal fusion mechanisms. CONCAT1, on the other hand, gives good performance and is competitive with DPA and MAF-TAV B . We speculate that treating the audio and video modalities separately and then merging them to retain the complimentary and differential features lead to this performance gain. Our proposed MAF outperforms DPA with gains of 1-3%. This underlines that our unique multimodal fusion strategy is aptly able to capture the contextual information provided by the audio and video signals.

Ablation Study
Replacing the GIF module with simple addition, we observe a noticeable decline in the performance across almost all metrics by about 2-3%. This attests to the inclusion of GIF module over simple addition. We also experiment with fusing multimodal information using MAF before different layers of the BART encoder. The best performance was obtained when the fusion was done before the sixth layer of the architecture (c.f. Appendix A.2).

Result Analysis
We evaluate the generated explanations based on their ability to correctly identify the source and target of a sarcastic comment in a conversation. We report such results for mBART, BART, MAF-TA B , MAF-TV B , and MAF-TAV B . BART performs better than mBART for the source as well as target identification. We observe that the inclusion of audio (↑ 10%) and video (↑ 8%) information dras- BART Monisha sahil ko walnut brownie ki matlab wo khane wali. Walnut Brownie to Monisha Sahil means she eats
(b) Explanation related to dialogue MONISHA: Ladki ka naam Ajanta Kyon Rakha? Why did they named the girl Ajanta? INDRAVARDHAN: Kyunki uski maa ajanta caves dekh rahi thi Jab vo Paida Hui haha. Because her mother must be watching the Ajanta caves when she was born haha.

Gold
Indravadan Ajanta ke naam ka mazak udata hai Indravardhan makes fun of Ajanta's name BART Indravardhan Monisha ko taunt maarta hai ki uski maa ajanta caves dekh rahi thi Jab vo Paida Hui Indravardhan taunts Monisha as her mother was watching Ajanta Caves when she was born.

MAF-TAVB
Indravadan ajanta ke naam ka mazak udata hai Indravardhan makes fun of Ajanta's name (c) Explanation related to sarcasm  tically improves the source identification capability of the model. The combination of both these nonverbal cues leads to a whopping improvement of more than 13% for the same. As a result, we infer that multimodal fusion enables the model to incorporate audio-visual peculiarities unique to each speaker, resulting in improved source identification. The performance for target identification, however, drops slightly on the inclusion of multimodality. We encourage future work in this direction.
Qualitative Analysis. We analyze the best performing model, MAF-TAV B , and its corresponding unimodal model, BART, and present some examples in Table 4. In Table 4a, we show one instance where the explanations generated by the BART as well as MAF-TAV B are neither coherent nor comply with the dialogue context and contain much scope of improvement. On the other hand, Table 4b illustrates an instance where the explanation generated by MAF-TAV B adheres to the topic of the dialogue, unlike the one generated by its unimodal counterpart. Table 4c depicts a dialogue where MAF-TAV B 's explanation better captures the satire than BART. We further dissect the models based on different modalities in Appendix A.3.
Human Evaluation. Since the proposed SED task is a generative task, it is imperative to man-ually inspect the generated results. Consequently, we perform a human evaluation for a sample of 30 instances from our test set with the help of 25 evaluators 6 . We ask the evaluators to judge the generated explanation, given the transcripts of the sarcastic dialogues along with a small video clip with audio as well. Each evaluator has to see the video clips and then rate the generated explanations on a scale of 0 to 5 based on the following factors 7 : • Coherence: Measures how well the explanations are organized and structured. • Related to dialogue: Measures whether the generated explanation adheres to the topic of the dialogue. • Related to sarcasm: Measures whether the explanation is talking about something related to the sarcasm present in the dialogue. Table 6 presents the human evaluation analysis with average scores for each of the aforementioned categories. Our scrutiny suggests that MAF-TAV B generates more syntactically coherent explanations when compared with its textual and bimodal counterparts. Also, MAF-TAV B and MAF-TV B generate explanations that are more focused on the conversation's topic, as we see an increase of 0.55 points in the related to the dialogue category. Thus, we reestablish that these models are able to incorporate information that is explicitly absent from the dialogue, such as scene description, facial fea-  tures, and looks of the characters. Furthermore, we establish that MAF-TAV B is better able to grasp sarcasm and its normalization, as it shows about 0.6 points improvement over BART in the related to sarcasm category. Lastly, as none of the metrics in Table 6 exhibit high scores (3.5+), we feel there is still much scope for improvement in terms of the generation performance and human evaluation. The research community can further explore the task with our proposed dataset, WITS.

Conclusion
In this work, we proposed the new task of Sarcasm Explanation in Dialogue (SED), which aims to generate a natural language explanation for sarcastic conversations. We curated WITS, a novel multimodal, multiparty, code-mixed, dialogue dataset to support the SED task. We experimented with multiple text and multimodal baselines, which give promising results on the task at hand. Furthermore, we designed a unique multimodal fusion scheme to merge the textual, acoustic, and visual features via Multimodal Context-Aware Attention (MCA2) and Global Information Fusion (GIF) mechanisms. As hypothesized, the results show that acoustic and visual features support our task and thus, generate better explanations. We show extensive qualitative analysis of the explanations obtained from different models and highlight their advantages as well as pitfalls. We also perform a thorough human evaluation to compare the performance of the models with that of human understanding. Though the models augmented with the proposed fusion strategy perform better than the rest, the human evaluation suggested there is still room for improvement which can be further explored in future studies.

A.1 Embedding Space for BART and mBART
We compared various text based unimodal methods for our task. Although BART is performing the best for SED, it is important to note that BART is pre-trained on English datasets (GLUE (Wang et al., 2018) and SQUAD (Rajpurkar et al., 2016)).
In order to explore how the representation learning is being transferred to a code-mixed setting, we analyse the embedding space learnt by the model before and after fine-tuning it for our task. We considered three random utterances from WITS and created three copies of them-one in English, one in Hindi (romanised), and one without modification i.e. code-mixed. Figure 4 illustrates the PCA plot for the embeddings obtained for these nine utterance representations obtained by BART before and after fine-tuning on our task. It is interesting to note that even before any fine-tuning the Hindi, English, and code-mixed representations lie closer to each other and they shift further closer when we fine-tune our model. This phenomenon can be justified as out input is of romanised codemixed format and thus we can assume that representations are already being captured by the pretrained model. Fine-tuning helps us understand the Hindi part of the input. Table 7 shows the cosine distance between the representations obtained for English-Hindi, English-Code mixed, and Code mixed-Hindi utterances for the sample utterances. It can be clearly seen that the distance is decreasing after fine-tuning.

A.2 Fusion at Different Layers
We fuse the multimodal information of audio and video in the BART encoder using the proposed fusion mechanism before different layers of the BART encoder. Table 8 shows the results we obtain when the fusion happens at different layers. We obtain the best results when the fusion happens before layer 6 i.e. the last layer of the encoder. This can be attributed to the fact that since there is only one layer of encoder after the fusion, the multimodal information is being retained efficiently and thus being decoded more accurately.  A.3 More Qualitative Analysis Table 9 highlights one of many cases where BART is able to capture the essence of sarcasm in a better way when compared to mBART. While mBART gives us an incorrect and incoherent explanation, BART generates an explanation which essentially means the same as the ground truth explanation. The inclusion of audio modality in the unimodal system often helps in generating preferable explanations, as shown in Table 10. AVII-TA is able to capture the essense of sarcasm in the dialogue while the unimodal systems were not able to do so. Furthermore, video modality facilitates even better understanding of sarcasm as illustrated in Table 11. AVII-TV is able to generate the best results while audio may act as noise in this particular example.

MAF-TV B Maya mocks monisha kyunki wo rhe theek hai
Maya mocks Monisha because she is okay.

MAF-TAV B
Maya kehti hai ki uske speech bure hai Maya says that she didn't like the speech. Table 9: BART v/s mBART: An example where explanation generated by BART is better than mBART. SAHIL: Ek minute, kya hai maa ji, humaare naatak mein ek bhi stree patra nahi hai, sare ladke hai. One minute, what is it ma'am, we don't have any female parts in our play, all are male PRABHAVATI: To uss mein bhi kaunsi badi baat hai, mai ladka ban jaungi. Mere paas pant shirt to hai, moonche aapki de dena! So what is the big deal in it, I'll play a male. I have pant shirt, you give me your mustache. INDRAVARDHAN: Cancel! Naatak cancel! Maa ji huaa aisa ki humaari jo bahu hai, uska ek chota sa accident ho gaya, to iss liye natak cancel! Monisha le jaao inhe. Cancel! Play cancel! Ma'am, what happened is, that our daughter in law had a small accident, that is why the play is cancelled. Monisha take her. SAHIL: Aur aate aate apna ek chota sa accident bhi kara ke aao! And when you come, have a small accident too! Gold Sahil Monisha pe gussa hai as usne Prabhavati as an actress le aya. Sahil is angry on Monisha that she hired Prabhavati as an actress.