{'fraction_non_alphanumeric': 0.030628480509148768, 'fraction_numerical': 0.007292495359321135, 'mean_word_length': 4.974782149458674, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Affective Feedback Synthesis Towards Multimodal Text and Image Data

In this paper, we have defined a novel task of affective feedback synthesis that deals with generating feedback for input text&corresponding image in a similar way as humans respond towards the multimodal data. A feedback synthesis system has been proposed and trained using ground-truth human comments along with image-text input. We have also constructed a large-scale dataset consisting of image, text, Twitter user comments, and the number of likes for the comments by crawling the news articles through Twitter feeds. The proposed system extracts textual features using a transformer-based textual encoder while the visual features have been extracted using a Faster region-based convolutional neural networks model. The textual and visual features have been concatenated to construct the multimodal features using which the decoder synthesizes the feedback. We have compared the results of the proposed system with the baseline models using quantitative and qualitative measures. The generated feedbacks have been analyzed using automatic and human evaluation. They have been found to be semantically similar to the ground-truth comments and relevant to the given text-image input.


INTRODUCTION
Multimodal data processing has emerged as an important sub-domain in Artificial Intelligence (AI) research due to the fast growth of multimedia data in the last few years [26]. One of the goals of AI is to enable machines to respond to multimodal data just like humans do. A response could be feedback, answer, caption, vocal signal, facial reaction, bodily gesture, etc. [35]. Humans portray different emotions through various modalities, among which text and image are known to contain the human emotions and intentions most effectively [29]. Machines and systems capable of generating affective feedback towards multimodal text and image data could be very useful [11]. Here the term 'affective' or 'human-like' is used in the sense that the feedback synthesis system should be able to synthesize feedback towards a multimodal input in a way humans do.
The ability to synthesize feedback towards multimodal data could be helpful in various applications such as determining user response towards products, social behavior analysis, evaluating multimodal educational content (for example, slides, blogs & books), and predicting the user engagement in advertisements [11]. Multimodal feedback synthesis systems can also be used to predict emotions that multimedia contents would induce in the users and hence predict the success of the contents [32]. Likewise, the success of advertisements and educational content can also be predicted upfront by gauging the kind of feedback users are likely to have towards them [4]. Moreover, the analysis of multimodal data generated by patients can help predict their mental-states [30].
Among the existing research tasks, multimodal summarization and dialog generation are related to the affective feedback synthesis task introduced in our work [12,15,19]. The text and image summarization approaches use the information from a single modality. Whereas multimodal summarization leverages the information from multiple modalities to fine-tune the summary [55]. However, most multimodal summarization approaches do not use actual human responses to train their models. Likewise, textual and visual dialog generation models are also not trained on human responses. They also suffer from the problem of short and uninformative dialogs' generation [48]. Moreover, affective feedback generation can be abstract and more difficult compared to the aforementioned tasks. For instance, identifying is a common challenge in affective feedback generation that can alter the conveyed sentiment. To this end, the proposed multimodal feedback synthesis system has been trained using actual human comments along with text and image inputs. The relevance of the comments is decided based on their number of likes (upvotes). We have incorporated textual and visual features that enable the proposed system to generate informative feedback considering the textual and visual context of the inputs.
In this paper, we have proposed a novel task to synthesize affective feedback towards text-image inputs, which is illustrated in Fig. 1. The idea is to build a model that could generate contextually relevant feedback as a new modality from two given modalities, i.e., news image and text. Affective feedback can be considered a special type of summary produced by humans based on their state of mind induced on processing the emotional context of the inputs [31]. Affective feedback aligns with the sentiment being expressed and is potentially useful in analyzing human behavior. We have also constructed a large-scale dataset, i.e., IIT Roorkee Multimodal Feedback (IIT-R MMFeed) dataset, for multimodal feedback synthesis. For simplicity, the 'IIT-R MMFeed' dataset has been referred to as the MMFeed dataset in this paper. The MMFeed dataset has been constructed by crawling the news articles through Twitter feeds, and it contains images, text, human comments, and the number of likes for the comments.
The proposed system contains textual and visual encoders (as shown in Fig. 3) to which input text and image are given as the inputs. The textual encoder uses a multi-headed self-attention-based transformer [46] and extracts textual features in the form of textual context vectors. The visual encoder uses a Faster Region-Based Convolutional Neural Networks (R-CNN) model [41] and extracts visual features in the form of a visual context vector. Each text encoder output vector is concatenated with a visual context vector and passed through a feedforward network that outputs multimodal contextual vectors with the same dimension as the text encoder vectors. Further, the decoder block takes textual context vector, multimodal context vectors, and ground-truth comments as inputs and generates the feedback as output. The generated feedbacks are evaluated using qualitative and quantitative methods for their relevance with the ground-truth comments and inputs.
The important contributions made in this paper are listed as follows: • The novel task has been defined to generate feedback towards multimodal input containing image and text similar to how humans do. The task aims to synthesize a new modality, i.e., feedback that is contextually relevant to the two given modalities, i.e., news image and text. • A large-scale dataset, 'IIT-R MMFeed dataset,' has been constructed for multimodal feedback synthesis by crawling news articles through Twitter feeds. It contains images, text, and Twitter user comments, along with the number of likes (upvotes) for each comment. • A multimodal feedback synthesis system has been proposed. The textual and visual encoders extract textual and visual features using a text transformer and a Faster R-CNN model. The decoder synthesizes the feedback according to the multimodal context that combines textual and visual features.
The organization of the remainder of the paper is as follows: the research tasks related to affective feedback synthesis are surveyed in Section 2. Section 3 describes the dataset construction process and the proposed system. The experiments and results have been discussed in Section 4 and Section 5 respectively. The major conclusions are drawn in Section 6 along with the future research directions.

RELATED WORK
This Section surveys the research advances related to the task of affective feedback synthesis introduced in this paper.

Text Summarization
There are two major types of text summarization approaches, viz. abstractive and extractive summarization [10,12]. The extractive summarization is focused on extracting useful words from the given text. In this context, Narayan et al. [33] worked on neural extractive summarization and utilized side information such as the title and captions of the news images. The abstractive summarization expresses the main content of a given text using different words instead of creating the summary by selecting the words or sentences from the text. The abstractive summarization of the sentences has been explored through neural attention-based models [43]. The extractive summarization is limited by the vocabulary set of the input text, whereas abstractive summarization approaches are not able to generate factually consistent and human-like summaries [20]. Moreover, text summarization does not leverage the information from multiple modalities to fine-tune the summary.

Image Summarization
Summarization of a scene from an image collection is an existing research problem [45]. In this context, Pan et al. [36] proposed an approach to generate a visual summarization for a set of images. They considered the social attractiveness features such as image quality and aesthetics while summarizing the images. On the other hand, Samani et al. [44] considered semantic features along with social attractiveness features to summarize the images in a context-sensitive manner. The image summarization approaches consider only visual information to produce the summary of the contents, whereas the multimedia contents often contain a combination of visual and textual information [23]. Moreover, the output of these approaches is in visual form only, which is more challenging to infer the emotion and context-related information as compared to textual and multimodal outputs [21]. The aforementioned limitation of text summarization and image summarization of not leveraging the information from multiple modalities are addressed by multimodal summarization approaches [55].

Multimodal Summarization
In the context of multimodal summarization, Chen and Zhuge [6] used recurrent networks to generate a summary from text and images. They performed simultaneous summarization of images and text documents and aligned the sentences and images to generate the summaries. On the other hand, Zhu et al. [55] used a pointer generator network for multimodal summarization instead of performing manual alignment of the text and images. They picked the most relevant image among the input images for a data sample and selected the essential keywords from the text inputs by performing extractive summarization. Summarization of multimedia news has also been explored using extractive summarization approaches [7]. In an attempt to utilize videos instead of images for multimedia news articles, Li et al. [51] implemented self-attention to automatically choose a suitable video frame based on the semantic meaning of the article. They modeled the article's semantic meaning along with the input video jointly.
Multimodal processing faces the challenges of either missing out on specific modalities or biasing the results on a particular modality [35]. Zhu et al. used Multimodal reference [56] to handle the modality-bias problem in multimodal summarization. They used the multimodal reference for guidance, designed an objective function, and proposed a novel evaluation metric based on the joint multimodal representation that considered the loss of image selection and summary generation. Context-aware summarization techniques have also been explored for multimodal summarization. In this direction, Li et al. [25] incorporated aspect coverage and corresponding use-cases for various product categories. Synthesis of affect during multimodal summarization has not been explored to its full potential. Moreover, the abovementioned research did not consider human comments towards the multimodal input data while training their summarization models. On the other hand, the proposed system is trained on human-generated comments along with multimodal inputs containing text and images. It generates the feedback as a new modality from two input modalities, i.e., image and text.

Textual Dialog
Textual dialog systems consider text input questions and generate text output responses in the form of a dialog. In this context, Zhou et al. [53] developed chatting agents that maintain and use a memory of emotional keywords. In another work, Xu et al. [50] worked on conversation modelings for dialog generation, whereas [52] used conditional variational autoencoders to understand the diverse use-cases for dialog generation models. Researchers have also worked on increasing the relevance, diversity, and originality of generation results [49]. In this context, Gu et al. [15] trained a conditional generative network for dialogue modeling. They modeled the data distribution by training a generative adversarial network considering multiple possible probability distributions of the topics and sentiments in the latent variable space. In an attempt to incorporate human-ness in the generated dialogues, Zhao et al. [54] used conditional autoencoders and included emojis in the synthesized responses. Most of the real-life multimedia context is expressed through multiple modalities where combining the complementary information from various modalities helps understanding the underlying emotional context effectively [17]. However, textual dialog systems consider only the textual context while generating the response towards input questions. There is a need to consider the corresponding visual context as well, which has given rise to the development of visual dialog systems as explored in the following Section.

Visual Dialog
Visual dialog (VisDial) is another closely related task to Multimodal Affective Feedback Synthesis, which aims to generate dialogs about visual input contents. In this context, Kang et al. [19] proposed a Dual Attention Network (DAN) to resolve visual reference between given image and dialogue history. They implemented multi-head attention to learn the relationships between the given question and the dialog history and bottom-up attention to model the image features and output dialogs' representations. In a similar work, Chen et al. [5] implemented dual-channel reasoning to learn the context from the image and dialog history together. The dual-channel reasoning proposed by them enabled them to learn rich semantic representations of the questions compared to the single-channel reasoning approach followed by DAN. In another work, Niu et al. [34] used recursive attention for finding the image component, referred to by a particular text entity. Their work was extended by Park et al. [38] for multi-view settings of recursive attention. Multimodal VisDial synthesis has also been explored for audio-visual data [1,16]. To effectively model the visual features, Jiang et al. [18] implemented a region-based graph attention network to learn question-aware relationships of the input image and dialog history-aware question features. This inspired us to incorporate attention-based visual feature extraction in our proposed work.
The existing VisDial methods are not able to solve our problem of affective feedback synthesis. We have empirically found that they could not synthesize meaningful feedback, and the responses are limited to a few words. A little work has been carried out in this context. For instance, Wu et al. [48] attempted to handle the problem of short and uninformative dialogs generation by editing the generated dialog with informative words based on the context of the input. We have incorporated textual and visual features that enable the proposed system to generate feedback considering the textual and visual context of the inputs. The appropriate textual and visual feature extraction networks have been determined through extensive ablation studies in Section 4.3. GRU, Bidirectional GRU, and Transformer have been explored for textual feature extraction. The VGG and ResNet models have been explored to extract global visual features, whereas R-CNN variants have been explored for local feature extraction. Moreover, the attention mechanism has been incorporated in textual and visual encoders.

Evaluation of Machine Generated Sentences
The quality of machine-generated sentences such as translations, text summaries, image captions, dialogues, etc., is subjective. Various quantitative metrics have been used in the literature for their evaluation. The importance of their extensive evaluation has been pointed out [3]. Various quantitative metrics based on recall, precision, and sensitivity have also been used. For example, Zhu et al. [56] used a recall-oriented metric, ROUGE, to evaluate the machine-generated visual dialog against the textual reference sentence. They also evaluated their output against the reference description of input images using precision value. The ranking-based evaluation metrics such as 'Recall@k' [42] and 'Mean Reciprocal Rank' [8] have also been utilized to rank the machine-generated sentences against a set of references sentences.
Further, multimodal summarization and VisDial systems are prone to the modality-bias problem where they tend to consider one of the input modalities more than the others [56]. Subjective human evaluation is helpful to evaluate whether the output is relevant to both the input modalities, i.e., text and image [22]. For the exhaustive evaluation of the feedback generated by the proposed system, we have implemented five metrics based on recall, precision, and precision (namely BLEU Score, ROUGE, Meteor, CIDEr, and SPICE) and two ranking based metrics (Recall@k and Mean Reciprocal Rank). Human evaluation has also been carried out to subjectively evaluate the relevance of the generated feedback with the input text, image, and ground-truth comments.
As discovered from the literature surveyed above, synthesis of affect during multimodal summarization has not been explored to its full potential. Most of the above methods generated a textual response towards multimodal data; however, they did not use actual human responses to train their models. An adequate dataset for multimodal feedback synthesis is also not available. With that as an inspiration, a method to synthesize affective feedback towards multimodal data has been proposed in this work. A large-scale dataset has also been constructed for multimodal feedback synthesis.

PROPOSED SYSTEM
This paper proposes a multimodal system to synthesize feedback for given text and image data as humans do. The formulation of the proposed task has been described as follows, along with the construction of the MMFeed dataset and proposed affective feedback synthesis system's architecture.

Problem Formulation
Given a multimodal input = { , }, where = { 1 , 2 , . . . } is the news text and is an image ( denotes the length of the news text sequence), the proposed system generates affective, i.e., human-like feedback = { 1 , 2 , . . . } where denotes the length of the feedback sequence. The problem is to generate a new modality, i.e., feedback from two given modalities, i.e., news image and text. The generated feedback is considered 'affective' or 'human-like' in the sense that the proposed system synthesizes them in a way humans do.

Dataset Construction
A large-scale dataset, IIT Roorkee Multimodal Feedback (IIT-R MMFeed) dataset, has been constructed for multimodal feedback synthesis. The procedure to compile and pre-process the dataset has been described in Appendix A. It has been collected by crawling news articles from corresponding Twitter handles (such as 'TIME,' 'CNN,' 'NYTimes,' 'BBCBreaking,' etc.) using NLTK 1 & newspaper3k 2 libraries and Tweepy API 3 . The MMFeed dataset consists of 77,790 samples collected through 9,479 Tweets containing image, text, and user comments, along with the number of likes (upvotes) for each comment. Table 1 describes various parameters of the dataset while example data instances and the dataset's distribution as per the number of comments each image has been depicted in Table 2 and Fig. 2 respectively.
The MMFeed dataset stands out from the existing datasets such as 'New York Times Articles & Comments (2020)' dataset [9] as it contains the images as well as the no. of likes (upvotes) that are helpful to evaluate the relevance of the comments in the context of the input image & text. The MMFeed dataset includes data from various genres such as sports, politics, current affairs, etc., which can be utilized for the proposed feedback synthesis system's robust training. As the human users have generated comments in response to input text and image, they can be considered the combined representation of the textual and visual contexts. Moreover, the number of likes associated with the comments can be considered to denote the comments' relevance to the input image and text. The proposed system has been trained on the input text and image and evaluated against the comments.   3. The schematic architecture of the proposed system. Here, three textual encoder blocks and three decoder blocks are used along with a visual encoder network. The dotted arrows represent the residual connections. The similarity module generates the similarity score, which denotes the significance of the generated feedback.

Proposed System
The proposed system's overall architecture is described in Fig. 3 and elaborated in the following Sections. The first two blocks are textual and visual encoders that work in parallel. Input text and image are given as the input to these blocks, respectively. The output of the textual encoder is a time series of vectors (textual context vectors, z*), and the output of the visual encoder is a single vector (visual context vector, g*). Each text encoder output vector is concatenated with g* and passed through a feedforward network that outputs another time series of vectors (multimodal contextual vectors, y*) with the same dimension as the text encoder vectors. Next is the decoder block for which z* and y* are the two inputs. During training, the ground-truth comments are provided as the third input against which the model is trained. During testing, feedback is generated as an output. The similarity module evaluates the similarity between the input comments and generated feedback.

Text Encoder.
Encoding of textual data is achieved by multi-headed self-attention-based transformer [46]. The encoder block includes a self-attention layer and a feedforward layer. Its sub-layers (i.e., feedforward and self-attention layers) have residual connections around them, and each layer is followed by a normalization layer.
Where , , and are the matrices packed with all the queries, keys, and values, respectively, whereas denotes the Transpose of the matrix and is the scaling factor.
We concatenated the attention heads and multiplied the weight matrix 0 to generate an intermediate vector ′ , which is passed through the feed-forward layer to generate the textual context vector, * . The multiple attention layers are stacked to learn the information simultaneously from different representation positions. The keys, values and queries are linearly projected ℎ times to , and dimensions. As shown in Eq. 2, the attention function is operated on the projected keys, values, and queries simultaneously to get -dimensional output.
Where ℎ denotes the number of attention heads whereas , , and are the parameter matrices with the projections of queries, keys, values, and output, respectively.
The encoder stack has a fully connected feed-forward network (FFN) for each layer. The input layer of this network is of 256 (Need to verify) dimensions, which is the same as the dimension of the output layer, while the hidden layers are of 512 dimensions. The network's output for input is computed as per Eq. 3.
Where is the input; 1 & 2 are bias terms, and 1 & 2 are the weight matrices.

Image Encoder.
We have used a pre-trained Faster R-CNN [41] model to extract the visual features. The input image is fed to a series of convolutional layers to obtain the feature maps. Then the RPN runs a sliding window of size * through them to generate the anchor boxes. The total number of anchor boxes for a feature map is * * , where and denote the height and width of a feature map, and is the number of anchors for each position. An objectiveness score defined as the Intersection Over Union (IoU) is given to each anchor box. This score can be either positive or negative (Eq. 4). The anchors which do not have a score associated with them do not contribute to the training. These anchors are passed into the classification (object classification) and regression (object localization) layers which output the classified boxes in the image.
Eq. 5 shows the loss function used for training the image encoder model. Here, 1601 classes were used for the anchor box. The model is trained to classify the anchor box, and its output is passed to the Visual Attention Network.
Where & * are the predicted probability and ground-truth value of anchors denoting whether they contain an object or not; denotes the coordinates of the predicted anchors; * denotes the ground-truth coordinates associated with the bounding boxes; & are the classifier loss and regression loss terms; & denote the normalization parameters for mini-batch (of size 256) and regression and is a hyper-parameter with a value of 10.
The rationale behind choosing a Faster R-CNN for object detection during visual feature extraction is governed by its speed and applicability in our use case. The Faster R-CNN model is computationally Faster in extracting the visual features than the other models of its family (namely, R-CNN [14], and Fast R-CNN [13] that incorporated ROI (Region of Interest) pooling layer to speed up the conventional R-CNN model). The Faster R-CNN replaces the conventional selective search-based region proposal with a Region Proposal Network (RPN). The reason why Faster R-CNN was used over the much Faster You only look once (YOLO) model [39] is due to the inefficiency of YOLO while identifying small objects. We could not afford to lose any such information in our image data. The YOLO model also struggles in identifying objects with skewed aspect ratios, while this is not the case with the Faster R-CNN model. Moreover, the proposed task of feedback synthesis does not need to be real-time. Finally, the Faster R-CNN model was chosen for the implementation.
Visual Attention: Visual context vector * is computed using Eq. 6 where is the global feature vector while , , denote compatibility score, feature vector and attention vector for state and anchor box .
Where denotes the compatibility score for state and box ; is the global feature vector; * is the visual context vector (final output of attention for state ) whereas & denote the feature vector & attention vector for state and box .

Multimodal Fusion Block.
Generation of multimodal context is achieved by combining textual context and visual context in Multimodal fusion block. The output of the text encoder (textual context vectors, * ) and the output of the visual encoder (visual context vector, * ) are given as inputs to the Multimodal fusion block. The Multimodal Fusion block computes the multimodal context vectors as follows -The output of the text encoder is a time series of vectors. Each of these vectors is concatenated with g* and passed through a feedforward network that outputs a time series of vectors with the same dimension as the text encoder vectors, which is termed as Multimodal context vectors ( * ). The attention module incorporates textual, visual, and multimodal attention. The textual and visual attention have been described above, whereas multimodal attention is as follows.
Multimodal Attention: Multimodal context vector * is calculated in Eq. 7 by concatenating * & * and passing through a feed-forward layer. Further, * and * are fed as input to each decoder block. * = ( * , * ) .
Where * , * and * denote multimodal, textual, and visual context vectors; is the weight matrix, and denotes the transpose operation.

Decoder.
The decoder block takes multimodal context vectors and * , textual context vectors, * as input. During training, it takes the ground-truth comments as additional input against which the model is trained. The comments' embeddings are produced, and positional encoding is added for them in a similar way to how the encoder encodes the text. During testing, the decoder synthesizes the feedback using * and * . Various layers of the decoder block are listed as follows.
Each attention layer has a residual connection around it, and a normalization layer follows it.
(a) Self-attention layer: where query, keys, and values are the decoder representation.
(b) Encoder-decoder attention layer: where a query is decoder representation and * is assigned to keys and values.
(c) Multimodal-attention layer: implemented to achieve multimodal attention with architecture similar to self-attention layer except the queries are decoder representations and * is assigned as keys and values.
(d) Feed-forward layer: which finally generates the feedback as decoder's output.

Similarity Module. The similarity module uses a pre-trained Sentence Bidirectional Encoder
Representations from Transformer (SBERT) [40] based teacher-student model. The teacher and student models convert comments and feedback to language-agnostic vectors and produce their embeddings, _ and _ . The language-agnostic vectors produce similar embeddings for the sentences using different words but portraying similar meaning. The teacher model maps the comments to _ . The student model^maps the comments and feedbacks to _ and _ , respectively. We have trained a student model^such that^( _ ) ≈ ( _ ) and ( _ ) ≈ ( _ ). The similarity score ( ) between _ and _ is minimized as per Eq. 8. Taking MSE between them helps compute the closeness of their embeddings and hence the similarity between them. We have fine-tuned the similarity module using parallel commentfeedback pairs (( 1 , 1 ), . . . ,( , )) where the feedbacks with more than 80% similarity during human evaluations have been considered.
Where is the number of feedback-comment pairs, whereas _ & _ denote comment embedding and feedback embedding.
English SBERT model [40] has been used as the teacher model whereas RoBERTa [28] has been used as the student model^. The semantic similarities of the embeddings of the comments & feedbacks have been found using cosine similarity as per Eq. 9.
Where _ and _ are the embeddings for comment and feedback, respectively, and denotes the transpose operation.
To evaluate the feedbacks against ground-truth comments, Mean Reciprocal Rank (MRR) & Recall@k (also known as 'Recall Rate@k') (defined in Section 4.2). The MRR & Recall@k are computed to denote whether the feedback generated is similar to the input comments, which are sorted based on the no. of likes. The 'Recall@k' is computed for the generated feedback denoting whether it is similar to top ranked comments. The comments for an image are ranked based on their no. of likes. Then the semantic similarity between the comments' embeddings and the embeddings of generated feedback is found using cosine similarity. Finally, 'Recall@k' is computed for the generated feedback denoting whether it is similar to top ranked comments where is a user-definable integer.

Training Strategy & Parameter Tuning
The baselines and proposed system described in the following sections have been trained for 30 epochs on GTX 1080Ti GPU with 3584 NVIDIA CUDA Cores and 11 GB GDDR5X Memory. The experiments have been performed using 5-fold cross-validation and 80%-20% training-testing split. The hyper-parameter values are described as follows.

Evaluation Metrics
The generated feedbacks are evaluated using qualitative and quantitative metrics for their relevance with the ground-truth comments and inputs. We have incorporated two phases of automatic evaluation using five quantitative metrics (BLEU, CIDEr, ROUGE, SPICE, and METEOR) and two qualitative metrics (Recall@k and MRR) along with the human evaluation to evaluate the generated feedbacks holistically. For the quantitative evaluation of the feedbacks, the following metrics have been used in the first phase of automatic evaluation. These metrics can evaluate machine-generated sentences such as summaries, image descriptions, and translations against benchmark results and human references based on recall, precision, and sensitivity.
• BLEU Score [37]: BLEU (bilingual evaluation understudy) is a precision based metric that compares candidate sentence with reference sentences to judge the quality of the candidate translation. In this work, 4-gram BLEU score has been used. The second phase of automatic evaluation uses 'Recall@k' [42] and 'Mean Reciprocal Rank' [8] to judge the relevance of the generated feedbacks against the ground-truth comments. These metrics have been defined as follows.
• Mean Reciprocal Rank (MRR): If the ℎ feedback is most similar with the ℎ most liked comment, then the rank and reciprocal ranks of the ℎ feedback, i.e., and are calculated as per Eq. 10.
Where denotes the ℎ comment sorted by the number of likes whereas & are the Rank & Reciprocal rank of the ℎ feedback. Mean Reciprocal Rank (MRR) denotes the average of the reciprocal ranks of all the feedback samples considered in the study, and it is given by Eq. 11.
Where is the number of feedback samples and denotes the rank of the ℎ feedback.
• Recall@k: In general, Recall@k denotes whether a data sample matches with any of the top relevant samples. It has been adapted to evaluate whether the generated feedback matches with top relevant ground-truth comments. We have found the similarity score of the feedback and find the ranks for all the comments where rank denotes the rank of the corresponding comment among all the comments when sorted by no. of likes. As shown in Eq. 12, if the comment with which feedback shows maximum similarity score is in top comments, it will get a score of 1 for Recall@k, else 0.
Where is the ℎ feedback; is the ℎ comment sorted by number of likes and denotes the rank of the ℎ feedback.
Moreover, to ensure that the generated feedbacks represent and are relevant to the given inputs, they have been evaluated by the human readers alongside the objective measures. Manual evaluation has also been carried out by having 50 human evaluators read the generated feedbacks and evaluate them against input image, text, and comments for their similarity.

Ablation Studies
Ablation studies have been performed to analyze the effect of using visual information, attention mechanism, and region extraction. Table 3 summarises these studies.

Effect of using Visual
Modality. The feedbacks are first generated considering only the text input, and then the complementary visual information has also been considered. The qualitative scores improved on including visual modality along with the textual modality.

Effect of using Attention.
The model is first trained on textual attention only; then, the visual attention is also fed to further fine-tune the output. It has been observed that the output with both textual and visual attention was more human-like than the one with textual attention only. It should be noted that the scores reduced a bit on including visual features along with textual, though they improved significantly on including the attention mechanism.

Effect of using Region Proposal.
Better performance has been observed on incorporating region extraction along with using textual & visual information with attention. In Section 4.3.1 global visual features using VGG and ResNet were included, whereas this Section incorporates local features using Faster R-CNN based region proposals. The aforementioned ablation studies on using visual modality, attention, and region proposal have been summarized in Table 3. The experiments are performed using complete data, data with low (up to 5), mid (between 13 and 50), and high (more than 30) no. of comments per image. The split was done using the following thresholds -low-comments: up to 5 comments per image or tweet, mid-comments: between 13 and 50 comments per image, and high-comments: more than 30 comments per image. The performance for various combinations is shown in Table 4. It is observed that experimenting with the complete dataset produced similar results as samples with 13 to 50 comments; however, it was computationally expensive.  Table 3). On the other hand, using 13-50 comments per image (mid-comments) resulted in similar quantitative performance scores as training with complete data (Observed from Table 4). The final implementation has been carried out using region proposal along with visual & textual attention and the data samples with 13-50 comments per image.

Models
The architectures of the baselines and proposed system have been determined based on the ablation studies performed in Section 4.3. As observed earlier, the feedbacks generated considering textual, and corresponding visual features had better scores; hence, all the baseline models have included textual and visual encoders.
• Baseline 1: The first baseline uses Gated Recurrent Units (GRU) for the textual encoder, whereas residual network (ResNet) is used for the visual encoder network. The choice of GRU over Long Short Term Memory (LSTM) architecture is guided through experimental observations. LSTM corresponded to 20% more memory consumption as compared to GRU, whereas their performance was comparable.
• Baseline 2 -The second baseline retains ResNet for visual encoding; however, it replaces the textual encoder network with Bidirectional GRU (BiGRU). The intuition behind using BiGRU was to enable a particular word's embedding to embody its contextual meeting, which changes according to the words appearing before and after it.
• Baseline 3 -Textual Encoder: This model further replaces the textual encoder with a text transformer model, whereas the attention mechanism is incorporated along with the ResNetbased visual encoder. The choice of using text transformer is governed by its off-the-shelf performance in various language analysis problems such as summarization, translation, caption generation, etc. [46].
• Proposed System -Textual Encoder: The proposed system uses text transformer as textual encoder, whereas a Faster R-CNN based region proposal mechanism is incorporated along with ResNet extracted features for visual encoding. The incorporation of local features extracted by R-CNN resulted in a better performance as compared to using only the global features extracted by ResNet.
The proposed system's implementation code and MMFeed dataset constructed in this paper can be accessed at github.com/MIntelligence-Group/MMFeed.

RESULTS AND DISCUSSION
The feedbacks generated by the proposed system have been evaluated using the quantitative and qualitative measures described in Section 4.2, and they have been compared with the Baseline methods.

Quantitative Results
In the first phase of automatic evaluation, the generated feedbacks are quantitatively evaluated using BLEU, CIDEr, ROUGE, SPICE, and METEOR metrics. The results have been shown in Table 6, which indicate the increased informative-ness for the feedbacks generated by the proposed method as compared to the baseline models.

Qualitative Results
The second automatic evaluation phase evaluates the generated feedbacks using MRR and Recall@k metrics. As discussed in Table 7, 98.67% of the feedbacks are relevant to one of the top 7 groundtruth comments, whereas they show the MRR of 0.3042, denoting that a majority of the feedbacks portrayed the most similarity with the top 3 (corresponding to MRR of 0.33) or top 4 (corresponding to MRR of 0.25) comments itself. The ground-truth comments and generated feedback portray similar sentiment and context using different words. To ensure that the generated feedbacks represent and are relevant to the given inputs, they have also been evaluated by the human readers alongside the objective measures. Table 8 summarises the results of the same. It should be noted that the 'Score' described in Fig. 4 is for the automatic evaluations of the generated feedbacks while ' ' described in Table 8 corresponds to their human evaluation. As per the human evaluations, the ground-truth comments show 72.86% and 74.90% similarity with the input text and image, respectively. These numbers are 65.34% and 67.96% respectively for the generated feedbacks. At the same time, the similarity between the feedbacks and comments has been found to be 80.17%.

Discussion
The proposed system has been trained using human comments along with input text & images to generate feedback towards multimodal content just as humans do. The evaluation results advocate that the generated feedbacks are relevant to the corresponding text & image input. We hope that the novel problem of multimodal human-like feedback synthesis and the MMFeed dataset proposed in this paper will inspire the researchers for further advancements in this context.
The proposed system's architecture has been evolved through progressive experiments. The model is first trained on textual attention only; then, the visual attention is also fed to fine-tune the output further. The output with both textual & visual attention has been observed to be more human-like than the one with textual attention only. Further ablation studies have been performed to analyze the effect of the feature extraction techniques. Various architecture choices for textual and visual encoders have also been evaluated. Finally, a transformer-based textual encoder and Faster R-CNN-based visual encoder have been implemented. The incorporation of the region proposal increased the computational cost to 63 times because of the large size of the bounding boxes. On the other hand, using the data samples with 13-50 comments per image for training resulted in similar performance as training with complete data, although it took 5.5X less computational time. Finally, a trade-off was chosen between training the proposed system with region proposals and using the data samples with 13-50 comments per image. Further, the baselines and proposed system converged in terms of validation loss in 18-23 epochs. The models have been trained for 30 epochs as a safe upper bound. The generated feedbacks have been evaluated for relevance with the input image & text and ground-truth comments using two automatic and one manual evaluation phases. The improvements in the evaluation scores on incorporating attention & region proposal advocate that the generated feedbacks represent the corresponding text & image input. The automatic & human evaluation results affirm that the generated feedbacks are relevant to the input text & image. The generated feedbacks have been observed to learn the in-context information from the training data. For example, many training data samples contained information about politics and corona-virus. The context about the same was reflected in some of the generated feedback.
One challenge with the evaluation of affective feedback generation is that multiple feedback can be contextually similar and may convey the same information. Thus given the ground truth, the evaluation is difficult as our goal is to produce contextually similar feedback. The meaning of the term 'Human-like' is to be taken more in the sense that the model can generate contexts similar to how a human would do. Though the syntax & semantics of the feedbacks are not entirely correct, the proposed system can learn and generate the words with respect to the text and image inputs. The minor errors in the model's understanding of the human language can be attributed to the noise (special characters, sentence phrases, and multilingual symbols) present in the comments of the training data.

CONCLUSIONS AND FUTURE WORK
In this work, we have introduced a novel task to generate human-like feedback for text & image data and proposed an affective feedback synthesis system. We have also constructed a large-scale multimodal feedback synthesis dataset. Automatic and human evaluations have been carried out to evaluate the generated feedbacks' relevance with the input text & image and similarity with the comments.
In the future, we aim to improve the syntactic and semantic correctness while generating long feedback sentences and extend the dataset to include more than one image per news article. It is also planned to work on the news articles' sentiment classification and genre classification. For the automatic evaluation of the generated feedbacks, existing evaluation metrics such as SPICE, CIDEr, ROUGE, etc., have been used, which are broadly used for various types of machine-generated sentences such as automatic translations, summaries, and image captions. There is a need to design an automatic evaluation metric specifically for evaluating multimodal feedbacks that would aid or replace the human evaluation process. We will focus on that as well in our future work.

A APPENDIX: DATA CRAWLING & PRE-PROCESSING PROCEDURE
Algorithm 1 shows the procedure to crawl and pre-process the data instances during the construction of the MMFeed dataset. The MMFeed dataset along with the proposed system's implementation code can be accessed through this link: github.com/MIntelligence-Group/MMFeed.