FiLMing Multimodal Sarcasm Detection with Attention
  Sarcasm detection identifies natural language expressions whose intended
meaning is different from what is implied by its surface meaning. It finds
applications in many NLP tasks such as opinion mining, sentiment analysis, etc.
Today, social media has given rise to an abundant amount of multimodal data
where users express their opinions through text and images. Our paper aims to
leverage multimodal data to improve the performance of the existing systems for
sarcasm detection. So far, various approaches have been proposed that uses text
and image modality and a fusion of both. We propose a novel architecture that
uses the RoBERTa model with a co-attention layer on top to incorporate context
incongruity between input text and image attributes. Further, we integrate
feature-wise affine transformation by conditioning the input image through
FiLMed ResNet blocks with the textual features using the GRU network to capture
the multimodal information. The output from both the models and the CLS token
from RoBERTa is concatenated and used for the final prediction. Our results
demonstrate that our proposed model outperforms the existing state-of-the-art
method by 6.14% F1 score on the public Twitter multimodal sarcasm detection
dataset.
