{'fraction_non_alphanumeric': 0.027831030607522973, 'fraction_numerical': 0.012494215640906987, 'mean_word_length': 5.207632334837916, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Analysis of Multimodal Document Intent in Instagram Posts

Present-day, a majority of representation style on social media (i.e., Instagram) tends to combine visual and textual content in the same message as a consequence of building up a modern way of communication. Message in multimodality is essential in almost any type of social interaction especially in the context of social multimedia content online. Hence, effective computational approaches for understanding documents with multiple modalities are needed to identify the relationship between them. This study extends recent advances in authors intent classification by putting forward an approach using Image-caption Pairs The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24–26, 2020. The Association for Computational Linguistics and Chinese Language Processing


Introduction
Up to date, a majority of representation style on social media tends to combine visual and textual content in the same message, the growing of multimodal documents is thus at a staggering rate. The developing multimodal document builds up a modern way of communication. The fact is that social multimedia, such as Instagram, inevitably hosting the way for information conveying. When reading a post on a multimodal social media, a simple question is raised: how do people analyze the relationship between image and text?
Regarding the meaning of a word, we found out that it would help to know the difference between denotation and connotation, which can be something suggested or implied by a word or constructions of words. As in semiotics, the terms denotation and connotation may be seen as different ways comprising a particular semantic domain, making up of the two obligatory relata of the sign function -(1) the expression and content, and (2) of a portion of the world in correspondence of the content, that is, the referent (Wason & Jones, 1963).
The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24-26, 2020. The Association for Computational Linguistics and Chinese Language Processing To access multimodal data in a computational way, an essential inclination of current studies utilized an amount of data with annotation for solving different scales of computer science problems in data mining and multimedia (Jin et al., 2010). Besides, a substantial body of related research documents the tendency of assuming images accompanied by basic text labels or captions such as interpreting author intent (Kruk et al., 2019). With the growing volume of multimodal social media, the detection of the relationships between text and image has become more critical. Research engaging fully with the multimodal aspects of the texts and images on social media is still in its infancy. Some researchers (i.e., Castro  As a consequence, two major sets of research questions are addressed in this study. The first purpose is to describe whether the combined modality "stronger" -means which has better performance in the multimodal classification task -than individual modalities or not

Taxonomies
Since this study focuses on exploring the ways that images and text interact, we adapt three Promotive: promote events, products, organizations, etc.
Exhibitionist: create a self-image for the user using selfies, pictures of belongings, etc.
Expressive: express emotion, attachment, or admiration at an external entity or group.
Informative: relay information regarding a subject or event using factual language.
Provocative/Discrimination: directly attach an individual or group.

Multimodal Document Understanding
The literature is full of discussions surrounding the definitions of modality, and scholars have debated its nature for decades. "A modality is a communication channel, for instance, related As discussed in the last paragraph, such a combination of diverse modalities sometimes results in Meaning Multiplication, a metaphor first promoted by the socio-functional semiotician Jay Lemke (Lemke, 1998). Prior work (Bateman, 2014) states that "under the right condition, the value of a combination of different modes of meaning can be worth more than the information (whatever that might be) that we get from the modes when used alone.
In other words, text 'multiplied by' images is more than text simply occurring with or alongside images. (...) Somehow the meanings of one and the meanings of the other resonate so as to produce more than the sum of the parts". As for NLP perspective, Morency and The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24-26, 2020. The Association for Computational Linguistics and Chinese Language Processing Baltrušaitis (2017) provided the view that, with the goals of recognizing language and vision projects such as image and video captioning, Multimodal Machine Learning is, inevitably, a vibrant multi-disciplinary research field for Artificial Intelligence, for example, integrating and modeling multiple communicative modalities, like linguistics, acoustic, and visual messages. In this study, we focus primarily on two modalities: the natural language that can be written, and visual signal which is represented with images. So far, seminal work on defining modality was carried out.
In order to interpret and reason about multimodal messages, it is necessary to develop a computational model that can not only deal with the heterogeneity of the data and the contingency often found between modalities but understand the dependencies across modalities. Additionally, in this study, it also requires the knowledge of the multimodal language, and thus, a multimodal framework was created and briefly introduced below. Most researchers working on exploring the relationship between text and image and extracting meaning often assigning a subordinate role to either text or images, i.e., image captioning, visual question answering, which claimed that text is a subordinate modality to image. Thus, our work builds on the framework of Marsh and White (2003)  CNN showed to be one of the best CNN models for image recognition. However, existing approaches to this task primarily rely on the textual content, but ignoring the other increasingly vibrant multimodal data sources, like images. This kind of ignoring will somehow enhance the robustness of these text-based models. Inspired by the recently proposed BERT architecture, a multimodal BERT architecture is applied, firstly by (Yu & Jiang, 2019), to obtain target-sensitive textual representations in order to model intramodality dynamics.

Fusion:
The fourth face-off remarks on how to combine information from two or more different modalities to perform a prediction, discrete or continuous. Take an image for example, the visual description of an image is fused with a caption to predict authors intent.
The varying predictive power and noise typology may be the consequence of information coming from different modalities. Baltrušaitis et al. (2018) claim that two types of multimodal representations -joint and coordinated. Joint representation often projects The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24-26, 2020. The Association for Computational Linguistics and Chinese Language Processing multimodal input into a common space while coordinated representation project each modality into a separate but coordinated space where only one modality is present at test time.

Co-learning:
The last challenge suggests transferring knowledge between different modalities, their representation, and their predictive models. It will be unexpectedly dominant when one of the modalities has limited resources such as a lack of annotated data, noisy data, and unreliable labels.

Datasets
The study comprises data labeling and text analysis of a corpus published posts from the discourse community Instagram. The primary criterion for selecting objects was that they are Random Forest (RF) (Breiman, 2001). In this study, both Machine Learning models were trained on both multimodal features. Our model takes input image (Img), text (Txt), or both (Img+Txt), plus modality-specific encoders, a fusion layer, and a class prediction layer.
Consistent with our purpose that caption is seen as an integration, BERT sentence embedding fits the most because it considers caption as a whole sentence. For the combined encoding model, we take advantage of a simple fusion strategy that linearly projects encoded vectors from both modalities in the same embedding space and then adds two vectors (Kruk et

Result and Discussion
We use a 906-sample dataset and only use a corresponding image and text information which is aligned manually for each post. Due to the small dataset, 10-fold cross-validation is conducted in our implementation. In order to report the result, the classification accuracy (ACC) and area under the ROC curve (AUC) are reported, using micro-average across all The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24-26, 2020. The Association for Computational Linguistics and Chinese Language Processing classes (Jeni et al., 2013;Stager et al., 2006). Additionally, for image, we use 2048 dimensional vectors trained from scratch. For character-based embeddings, we use a pretrained model with layers resulting in a 512-dimensional vector. The results will be shown after training in DTC and RF models. Due to the small dataset, we conducted a 10-fold crossvalidation. To report the result, I'll present the classification accuracy (ACC) and area under the ROC curve (AUC) with micro-average across all classes. There will be two tables summarizing the main results as follows. First, the result with Decision Tree Classifier is shown in Table 1. Overall, the result shows a striking effect of text embedding on performance. For all the taxonomies, text embedding was significantly superior to the other models. It outperforms consistently than just using images embedding and the combined model. Following Kruk's work, we hypothetically assume that the combined model would help across the board. However, it has been disproved from several 'aspects of the result. For the contextual and semiotic relationship, text embedding and combined model both performs much better than image model for more than 20%. Toward this result, this might because Sentence-BERT is originally designed for text classification, so it outperforms in the text classification task. As for author's intent, the performance of text embedding (82.20%) is just slightly higher than image embedding (81.02%), and reaches almost 15% difference from the combined model (67.23%). The reason might be that the data of author's intent is more homogeneous than the others. Digging into the annotation details, when annotating the author's intent, annotators tend to label the target with high identical consistency, hence making the high performance in the author's intent.

Conclusion
This study utilizes a computational model to capture the complex relationship between text and image modality, and how they cue authors intent in Instagram posts. In response to the first research question that if the combined modality stronger than individual modalities, text did assign a subordinate role to image. Toward this result, this might because Sentence-BERT is originally designed for text classification, and it performs very strong in the text classification task. Up to this point, these results are consistent with the prior studies which indicate that either text or image should be assigned a subordinate role. Although these two modalities encode different information on the use of classifying the relationship, the result suggests that the incorporating of text and image needs more effort to complement each other. The second main finding is that we present the results of the relationships of author's intent, the contextual relationship and the semiotic relationship between the ICPs.
Furthermore, among the all studied modalities, the captions are no doubt the strongest feature for classifying relationships, in that the caption encoder (Sentence-BERT) shows its powerful advantages in text classification. In addition, we make two comparisons: the results between the two Machine Learning models and the result between this study and that of Kruk et al (2019).
Even though there are a variety of tasks being carried in the study, the design of the present study is not without limitations. The first limitation concerns the data size used in this current study. 906 posts might be too small to make a classification with high accuracy. The deep learning model used in these tasks needs more data to get better training. This may probably be one of the reasons that we cannot reach better performance in classifying the relationship with the image embedding model. The second limitation is rooted in the labor for annotation.
The Kappa score in this study is to the average of 0.5953 (moderate agreement). If it may reach a substantial agreement (0.61-0.80) or even almost perfect agreement (0.81-1.00), the result should appear to be better. In order to raise agreement between annotators, we should have completed more annotating norms with an abundance of details besides the two norms applied in this study. Third, a more stable architecture to build the image describer is needed.
To characterize the images by representing an n-dimensional feature vector, it is necessary to explore the image describer to better generalize the results. Last but not least, the synergistic model in our task did not outperform the model of single modality. This issue might because the linearly project vectors did not highlight the significance of image embedding. Having The 32nd Conference on Computational Linguistics and Speech Processing (ROCLING 2020) Taipei, Taiwan, September 24-26, 2020. The Association for Computational Linguistics and Chinese Language Processing acknowledged the limitations above, future studies should be alerted to the disadvantages of this study.
For future studies, new possibilities can be listed. Since sentence-BERT is good at processing the text input, the literal meaning of the image automatically generated from the computer vision technique should be involved as a part of the image encoder to reach a possibly better performance. Additionally, it is suggested that future research should explore more linguistic features of multimodal documents to improve the working. More solid visual features and other meta-data features are needed. Further, considering a large amount of literature in predicting sentiments of multimodal documents, the key challenge is to collect a sufficient amount of training labels to train a discriminative model for multimodal prediction. Although preliminary research in the area is already being undertaken by researchers, more extensive research would be necessary to make any definite claims along these lines.