{'fraction_non_alphanumeric': 0.030364282541092847, 'fraction_numerical': 0.01046201688138605, 'mean_word_length': 5.0179788798289, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Survey on visual sentiment analysis

: Visual Sentiment Analysis aims to understand how images affect people, in terms of evoked emotions. Although this field is rather new, a broad range of techniques have been developed for various data sources and problems, resulting in a large body of research. This paper reviews pertinent publications and tries to present an exhaustive overview of the field. After a description of the task and the related applications, the subject is tackled under different main headings. The paper also describes principles of design of general Visual Sentiment Analysis systems from three main points of view: emotional models, dataset definition, feature design. A formalization of the problem is discussed, considering different levels of granularity, as well as the components that can affect the sentiment toward an image in different ways. To this aim, this paper considers a structured formalization of the problem which is usually used for the analysis of text, and discusses it's suitability in the context of Visual Sentiment Analysis. The paper also includes a description of new challenges, the evaluation from the viewpoint of progress toward more sophisticated systems and related practical applications, as well as a summary of the insights resulting from this study.


Introduction and Motivations
Nowadays, the amount of public available information encourages the study and development of Sentiment Analysis algorithms that analyse huge amount of users' data with the aim to infer reactions about topics, opinions, trends to understand the mood of the users who produce and share information through the web. The aim of Sentiment Analysis is to extract the attitude of people toward a topic or the intended emotional affect the author wishes to have on the readers. The tasks of this research field are challenging as well as very useful in practice. Sentiment analysis finds several practical applications, since opinions influence many human decisions either in business and social activities. As instance, companies are interested in monitoring people opinions toward their products or services, as well as customers rely on feedbacks of other users to evaluate a product before they purchase it. With the growth of social media (i.e., reviews, forums, blogs and social networks), individuals and organizations are increasingly using public opinions for their decision making [1].  of the field by describing five emerging tasks related to Visual Sentiment Analysis, with proper references, in Section 5. The paper introduces some additional challenges and techniques that could be investigated, proposing suggestions for new methods, features and datasets. In this sense, the proposed paper aims to serve as a guide for research scientist that are interested in Visual Sentiment Analysis. Indeed, this paper is intended to be a reference source of contents in the field, by providing a structured description and analysis of the past works, a detailed presentation of the different existing datasets with pros and cons, as well as a perspective on current challenges and for future directions. Section 6 reports the insights as outcome of the proposed study. Then, Section 7 concludes the paper by providing a summary of the previous sections and by suggesting directions for future works.

State of the Art
Visual Sentiment Analysis is a recent research area. Most of the works in this new research field rely on previous studies on emotional semantic image retrieval [7,8,9,10], which make connections between low-level image features and emotions with the aim to perform automatic image retrieval and categorization. These works have been also influenced by empirical studies from psychology and art theory [11,12,13,14,15,16]. Other research fields close to Visual Sentiment Analysis are those considering the analysis of the image aesthetic [17,18,19,20,21], interestingness [22], affect [23] and popularity [24,25,26,27].
The first paper on Visual Sentiment Analysis aims to classify images as "positive" or "negative" and dates back on 2010 [28]. In this work the authors studied the correlations between the sentiment of images and their visual content. They assigned numerical sentiment scores to each picture based on their accompanying text (i.e., meta-data). To this aim, the authors used the SentiWordNet [29] lexicon to extract sentiment score values from the text associated to images. This work revealed that there are strong correlations between sentiment scores extracted from Flickr meta-data (e.g., image title, description and tags provided by the user) and visual features (i.e., SIFT based bag-of-visual words, and local/global RGB histograms).
In [30] a study on the features useful to the task of affective classification of images is presented. The insights from the experimental observation of emotional responses with respect to colors and art have been exploited to empirically select the image features. To perform the emotional image classification, the authors considered the 8 emotional output categories as defined in [31] (i.e., Awe, Anger, Amusement, Contentment, Excitement, Disgust, Sad, and Fear).
In [32] the authors built a large scale Visual Sentiment Ontology (VSO) of semantic concepts based on psychological theories and web mining (SentiBank). A concept is expressed as an adjective-noun combination called Adjective Noun Pair (ANP) such as "beautiful flowers" or "sad eyes". After building the ontology consisting of 1.200 ANP, they trained a set of 1.200 visual concept detectors which responses can be exploited as a sentiment representation for a given image. Indeed, the 1.200 dimension ANP outputs (i.e., the outputs of the ANP detectors) can be exploited as features to train a sentiment classifier. To perform this work the authors extracted adjectives and nouns from videos and images tags retrieved from YouTube and Flickr respectively. These images and videos have been searched using the words corresponding to the 24 emotions defined in the Plutchik Wheel of Emotion [33], a well known psychological model of human emotions. The authors released a large labelled image dataset composed by half million Flickr images regarding to 1.200 ANPs. Results show that the approach based on SentiBank concepts outperforms text based method in tweet sentiment prediction experiments. Furthermore, the authors compared the SentiBank representation with shallow features (colour histogram, GIST, LBP, BoW) to predict the sentiment reflected in images. To this end, they used two different classification models (LinearSVM and Logistic Regression) achieving significant performance improvements when using the SentiBank representation. The proposed mid-level representation has been further evaluated in the emotion classification task considered in [30], obtaining better results.
In 2013, Yuan et al. [34] employed scene-based attributes to define mid-level features, and built a binary sentiment classifier on top of them. Furthermore, their experiments demonstrated that adding a facial expression recognition step helps the sentiment prediction task when applied to images with faces.
In 2014, Yang et al. [35] proposed a Sentiment Analysis approach based on a graphical model which is used to represent the connections between visual features and friends interactions (i.e., comments) related to the shared images. The exploited visual features include saturation, saturation contrast, bright contrast, cool color ratio, figure-ground color difference, figure-ground area difference, background texture complexity, and foreground texture complexity. In this work the authors considered the Ekman's emotion model [36].

May 19, 2020
Chen et al. [37] introduced a CNN (Convolutional Neural Network) based approach, also known as "SentiBank 2.0" or "DeepSentiBank". They performed a fine-tuning training on a CNN model previously trained for the task of object classification to classify images in one of a 2.096 ANP category (obtained by extending the previous SentiBank ontology [32]). This approach significantly improved the ANP detection with respect to [32]. Similarly to [32], this approach provides a sentiment feature (i.e., a representation) of an image that can be exploited by further systems.
In contrast to the common task of infer the affective concepts intended by the media content publisher (i.e., by analysing the text associated to the image by the publisher), the method proposed in [38] tries to predict what concepts will be evoked to the image viewers.
In [39] a pre-trained CNN is used as a provider of high-level attribute descriptors in order to train two sentiment classifiers based on Logistic Regression. Two types of activations are used as visual features, namely the fc7 and fc8 features (i.e., the activations of the seventh and eighth fully connected layers of the CNN respectively). The authors propose a fine-grained sentiment categorization, classifying the polarity of a given image through a 5-scale labelling scheme: "strong negative", "weak negative", "neutral", "weak positive", and "strong positive". The evaluation of this approach considers two baseline methods taken from the state of the art, namely low-level visual features and SentiBank, both introduced in [32], in comparison with their approaches (the fc7 and fc8 based classifiers). The experimental setting evaluates all the considered methods on two real-world dataset related to Twitter and Tumblr, whose images have been manually labelled considering the above described 5-scale score scheme. The results suggest that the methods proposed in [39] outperform the baseline methods in visual sentiment prediction.
The authors of [40] proposed to use a progressive approach for training a CNN (called Progressive CNN or PCNN) in order to perform visual Sentiment Analysis in terms of "positive" or "negative" polarity. They first trained a CNN architecture with a dataset of half million Flickr images introduced in [32]. At training time, the method selects a subset of training images which achieve high prediction scores. Then, this subset is used to further fine-tune the obtained CNN. In the architecture design they considered a last fully connected layer with 24 neurons. This design decision has been taken with the aim to let the CNN learn the response of the 24 Plutchik's emotions [33]. An implementation of the architecture proposed in [40] is publicly available. The results of experiments performed on a set of manually labelled Twitter images show that the progressive CNN approach obtain better results with respect to other previous algorithms, such as [32] and [34].
Considering that the emotional response of a person viewing an image may include multiple emotions, the authors of [41] aimed to predict a distribution representation of the emotions rather than a single dominant emotion from (see Figure 1). The authors compared three methods to predict such emotion distributions: a Support Vector Regressor (based on hand crafted features related to edge, color, texture, shape and saliency), a CNN for both classification and regression. They also proposed a method to change the evoked emotion distribution of an image by editing its texture and colors. Given a source image and a target one, the proposed method transforms the color tone and textures of the source image to those of the target one. The result is that the edited image evokes emotions closer to the target image than the original one. This approach has been quantitatively evaluated by using four similarity measures between distributions. For the experiments, the authors consider a set of 7 emotion categories, corresponding to the 6 basic emotions defined by Ekman in [36] and the neutral emotion. Furthermore, the authors proposed a sentiment database called Emotion6. The experiments on evoked emotion transfer suggest that holistic features such as the color tone can influence the evoked emotion, albeit the emotion related to images with high level semantics are difficult to be shaped according to an arbitrary target image.
In [42] the textual data, such as comments and captions, related to the images are considered as contextual information. Differently from the previous approaches, which exploit low-level features [23], mid-level features [32,34] and Deep Learning architectures [40,41], the framework in [42] implements an unsupervised approach (USEA -Unsupervised SEntiment Analysis). In [43] a CNN pre-trained for the task of Object Classification is fine-tuned to accomplish the task of visual sentiment prediction. Then, with the aim to understand the contribution of each CNN layer for the task, the authors performed an exhaustive layer-perlayer analysis of the fine-tuned model. Indeed, the traditional approach consists in initializing the weights obtained by training a CNN for a specific task, and replacing the last layer with a new one containing a number of units corresponding to the number of classes of the new target dataset. The experiments performed in this paper explored the possibility to use each layer as a feature extractor and training individual classifiers. This layer by layer study allows measuring the performance of the different layers which is useful to understand how the layers affect the whole CNN performances. Based on the layer by layer analysis, the authors proposed several CNN architectures obtained by either removing or adding layers from the original CNN.  Even though the conceptual meaning of an image is the same for all cultures, each culture may have a different sentimental expression of a given concept. Motivated by this observation, Jou et al. [44] extended the ANP ontology defined in [32] for a multi-lingual context. Specifically, the method provides a multilingual sentiment driven visual concept detector in 12 languages. The resulting Multilingual Visual Sentiment Ontology (MVSO) provides a rich information source for the analysis of cultural connections and the study of the visual sentiment across languages.
Starting by the fact that either global features and dominant objects bring massive sentiment cues, Sun et al. [45] proposed an algorithm that extracts and combines features from either the whole image and "salient" regions. These regions have been selected by considering proper objectness and sentiment scores aimed to discover affective local regions. The proposed method obtained good results compared with [32] and [40] on three widely used datasets presented in [32] and [40].
In [46] Katsurai and Satoh exploited visual, textual and sentiment features to build a latent embedding space where the correlation between the projected features from different views is maximized. This work implements the CCA (Canonical Correlation Analysis) technique to build a 3-view embedding which provides a tool to encode inputs from different sources (i.e., a text and an image with similar meaning/sentiment are projected nearby in the embedding space) and a method to obtain a sentiment representation of images (by simply projecting an input feature to the latent embedding space). This representation is exploited to train a linear SVM classifier to infer positive or negative polarity. The authors used a composition of RGB histograms, GIST, SIFT based Bag of Words and two mid-level features defined in [47] and [32] as visual features. The textual feature is obtained using a Bag of Words approach form text associated to the image, crawled from Flickr and Instagram. The sentiment features are obtained starting from the input text and exploiting an external knowledge base, called SentiWordNet [29], a well-known lexical resource used in opinion mining to assign sentiment scores to words.
The works in [48] and in [49] perform emotional image classification of images considering multiple emotional labels. As previously proposed in 2015 by Peng [41], instead of training a model to predict only one sentiment label, the authors considered a distribution over a set of pre-defined emotional labels. To this aim, they proposed a multi-task system which optimizes the classification and the distribution prediction simultaneously. In [48] the authors proposed two Conditional Probability Neural Networks (CPNN), called Binary CPNN (BCPNN) and Augmented CPNN (ACPNN). A CPNN is a neural network with one hidden layer which takes either features and labels as input and outputs the label distribution. Indeed, the aim of a CPNN is to predict the probability distribution over a set of considered labels. The authors of [49] changed the dimension of the last layer of a CNN pre-trained for Object Classification in order to extract a probability distribution with respect to the considered emotional labels, and replaced the original loss layer with a function that integrates the classification loss and sentiment distribution loss through a weighted combination. Then the modified CNN has been fine-tuned to predict sentiment distributions. Since the majority of the existing datasets are built to assign a single emotion ground truth to each image, the authors of [49] proposed two approaches to convert the single labels to emotional distribution vectors, which elements represent the degree to which each emotion category is associated to the considered image. This is obtained considering the similarities between the pairwise emotion categories [33]. The experimental results show that the approach proposed in [49] outperforms eleven baseline Label Distribution Learning (LDL) methods, including BCPNN and ACPNN proposed in [48].
In [50] the authors extended their previous work [43] in which they first trained a CNN for sentiment analysis and then empirically studied the contribute of each layer. In particular, they used the activations in each layer to train different linear classifiers. In this work the authors also studied the effect of weight initialization for fine-tuning by changing the task (i.e., the output domain) for which the fine-tuned CNN has been originally trained. Then, the authors propose an improved CNN architecture based on the experimental results and observations. Then, the authors propose an improved CNN architecture based on the empirical insights.
The authors of [51] exploited a large tweets dataset named Twitter for Sentiment Analysis (T4SA) to finetune existing CNN previously trained for objects and places classification (VGG19 [52] and HybridNet [53]) for the task of Visual Sentiment Analysis, where the image labels have been defined based on the tweet texts.
The proposed system has been compared with the CNN and PCNN presented in [40], DeepSentiBank [37] and MVSO [44] obtaining better results on the built dataset.
The system proposed in [54] represents the sentiment of an image by extracting a set of ANPs describing the image. Then, the weighted sum of the extracted textual sentiment values is computed, by using the related ANP responses as weights. The approach proposed in this paper takes advantage of the sentiment of the text composing the ANPs extracted from images, instead of only considering the ANP responses defined in SentiBank [32] as mid-level representations. In particular, the sentiment value of an extracted ANP is defined by summing the sentiment scores defined in SentiWordNet [29] and SentiStrength [64] for the pair of adjective and the noun words of th ANP. A logistic regressor is used to infer the sentiment orientation by exploiting the scores extracted from the textual information, and a logistic classifier is trained for polarity prediction by exploiting the traditional ANP responses as representations. Then, the two schemes are combined by employing a late fusion approach. The authors compared their method with respect to three baselines: a logistic regression model based on the SentiBank mid-level representation, the CNN and PCNN methods proposed in [40]. Experiments show that the proposed late fusion method outperforms the method based only on the mid-level representation defined in SentiBank, demonstrating the contribute given by the sentiment coefficients associated to the text composing the extracted ANPs. However, the CNN and PCNN approaches proposed in [40] exhibit better performances than the late fusion method. The problem of the noise in the text associated to social images by users is very common in the state of the art approaches.
The work in [55] presents a method for Visual Sentiment Analysis that integrates saliency detection into visual attention learning aimed to let the sentiment classifier focus on salient objects in the image. In particular, a fully connected neural network is trained to extract salient regions in the input image using multi-scale feature maps. During prediction, the image weighted by the attention map is fed to a CNN trained for Visual Sentiment Analysis. The proposed approach has been evaluated on the tasks of polarity prediction and emotion classification using the Mikels' 8 emotional model [31].
The paper in [56] addresses the challenge of image sentiment polarity prediction by proposing a novel source of text for this task, dealing with the issue related to the use of text associated to images provided by users, which is commonly used in most of the previous works and is often noisy due its subjective nature. In particular, the authors built an image sentiment classifier that exploits a representation based on both visual and textual features extracted from an image, and evaluated the performances obtained by using the text provided by users (subjective) and the text extracted from the visual content by using four deep models trained on different tasks. Several experiments have been performed by combining different features based on subjective (i.e., user-provided) and objective (i.e., automatically extracted) text. The experiments revealed that employing a source of text automatically extracted from the images, in lieu of the text provided by users, improves the classifier performances.
In [57] the authors combined the visual and textual information of social media posts to predict the overall sentiment of daily news. The system has been evaluated on a dataset of images crawled from Instagram.
The authors of [58] proposes a joint visual-textual sentiment analysis system trying to exploit more than one modality. However, the authors further incorporated a cross-modality attention mechanism and semantic embedding learning based on bidirectional recurrent neural networks with the aim to design a model able to focus on the visual and textual features that mostly contribute to the sentiment classification. The intuition beyond this approach is that not both text and image contribute equally to the sentiment classification, as visual and textual information differ in their contribution to sentiment analysis. The attention mechanism allows to properly weight input that may deliver inconsistent sentiment, such as the noisy text provided by users [56].
Huang et al. [59] propose an image-text sentiment model named Deep Multimodal Attentive Fusion (DMAF). This approach defines two attention models aimed to learn effective sentiment classifiers for visual and textual inputs, named Visual Attention Model and Semantic Attention Model respectively. Furthermore, an additional Multimodal Attention Model is employed with the aim to exploit the correlation between visual and textual features. Then, a late fusion approach is used to combine the three attention models.
In [61] the authors trained a joint embedding for images and text for sentiment analysis. In particular, they formulated an Adjective Noun Pair (ANP) detection task in images computed on features extracted from an continuous embedding space that encodes inter-concept relationships between different ANPs. In [62] the authors presented a multitask framework to combine multimodal information (i.e., images and text) when it is available, while being able to handle the cases when a modality is missing. In particular, this approach adds two auxiliary image-based and text-based classifiers to the classic multimodal framework. Experiments empirically demonstrated that such a multi-task learning approach can also improve generalization by acting as a regularization mechanism. The work in [60] presents a method that combines visual and textual features by employing an ensemble learning approach. In particular, the authors combined five state of the art classifiers based on visual and textual inputs. The system classifies a givne image using the 8 emotions scheme proposed by Mikels [31]. The authors of [63] combined the use of CNNs and saliency detection to develop a system that first performs a prediction of sentiment of the whole image. Then, if salient regions are detected, performs the sentiment prediction on the sub-images that depict such sub-areas. Finally, the global and local predictions are fused by a weighted sum.
May 19, 2020 The paper in [65] presents an overview in the field of affective image content analysis with respect to two main tasks, namely affective gap and perception subjectivity.
Soleymani et al. [66] present a brief overview of the methods adopted to address three different fields: multi-modal spoken reviews and vlogs, sentiment analysis in humans-machines face to face interactions and sentiment of images and tags in social media.
The paper in [67] presents an overview on textual sentiment analysis of social networks contents. Then, a focus on multi-modal approaches that combines textual and visual features for visual sentiment analysis is presented. The works described in this section have led to significant improvements in the field of Visual Sentiment Analysis. However these works address the problem considering different emotion models, datasets and evaluation methods (see Table 1). So far, researchers formulated this task as a classification problem among a number of polarity levels or emotional categories, but the number and the type of the emotional outputs adopted for the classification are arbitrary. The difference in the adopted emotion categories makes result comparison difficult. Moreover, there is not a strong agreement in the research community about the use of an universal benchmark dataset. Indeed, several works evaluated their methods on their own datasets. Many of the mentioned works present at least one of the said issues. Previous surveys give a brief overview of the field followed by a focus on specific issues [66,65,67]. In contrast, the proposed study introduces the basic task in Visual Sentiment Analysis (i.e., classifying an image as positive or negative) and then extends the discussion by deeply analysing each aspect, covering a larger number of topics, derived tasks and new challenges related to Visual Sentiment Analysis with high level of detail.

How to represent the emotions?
Basically, the goal of a Visual Sentiment Analysis system is to determine the sentiment polarity of an input image (i.e., positive or negative). Several works aims to classify the sentiment conveyed by images into 2 (positive, negative) or 3 polarity levels (positive, neutral, negative) [32,28,40]. However, there are also systems that adopt more then 3 levels, such as the 5-level sentiment scheme used by Xu et al. [39] or the 35 "impression words" used by Hayashi et al. [69]. Beside the polarity estimation, there are systems that perform the sentiment classification by using a set of emotional categories, according to an established emotion model based on previous psychological studies. However, each emotional category usually corresponds to a positive or negative polarity [30]. Thus, these systems can be evaluated also for the task of polarity estimation. Generally, there are two main approaches for emotion modelling: • Dimensional approach: emotions are represented as points in a 2 or 3 dimensional space (see Figures 3 and 4). Indeed, as discussed in several studies [11,13,14,16], emotions have three basic underlying dimensions: valence, arousal and control (or dominance). However, as can be seen from Figure 3, the control dimension has a small effect. Therefore, a 2D emotion space is often used, obtained by considering only the arousal and the valence axis.
• Category approach: according to this model there are a number of basic emotions. Having just a few emotional categories is convenient for tasks such as indexing and classification. This set of descriptive words can be assigned to regions of the VAC (Valence-Arousal-Control) space. Thus it can be considered a quantized version of the dimensional approach. The choice of the emotional categories in not an easy task. Since emotion belongs to the psychology domain, the insights and achievements from cognitive science can be beneficial for this problem.
What are the basic emotions? There are several works that aim to ask this question. As we observed in Section 2, the most adopted model is the Plutchnik's Wheel of Emotions [33]. This model defines 8 basic emotions with 3 valences each. Thus it defines a total of 24 emotions (see Figure 2).
According to Ekman's theory [36] there are just five basic emotions ("anger", "fear", "disgust", "surprise" and "sadness"). Another emotions categorization is the one defined in a psychological study by Mikels et al. [31]. In this work the authors perform an intensive study on the International Affective Picture System (IAPS) in order to extract a categorical structure of such dataset [70]. As result, a subset of IAPS have been categorized in eight distinct emotions: "amusement", "awe", "anger", "contentment", "disgust", "excitement", "fear" and "sad". A deeper list of emotion is described in Shaver et al. [71], where emotion concepts are organized in a hierarchical structure. Furthermore, a cognitive perspective analysis on basic emotions and their relations is presented in [72].
As we discussed in this Section, there is a wide range of research on identification of basic emotions. By way of conclusion, the 24 emotions model defined in Plutchnik's theory [33] is a well established psychological model of emotions. This model is inspired by chromatics in which emotions are organized along a wheel scheme where bipolar elements are placed opposite one to each other. Moreover the three intensities provides a richer set of emotional valences. For these reasons it can be considered the reference model for the identification of the emotional categories.

Existing datasets
There are several sources that can be exploited to build Sentiment Analysis datasets. The general procedure to obtain a human labelled set of data is to perform surveys over a large number of people, but in the context of Sentiment Analysis the collection of huge opinion data can be alternatively obtained by exploiting the most common social platforms (Instagram, Flickr, Twitter, Facebook, etc.), as well as websites for collecting business and products reviews (Amazon, Tripadvisor, Ebay, etc.). Indeed, nowadays people are used to express their opinions and share their daily experiences through the Internet Social Platforms.
In the context of Visual Sentiment Analysis, one of the first published dataset is the International Affective Picture System (IAPS) [70]. This dataset has been developed with the aim to produce a set of evocative color images that includes contents from a wide range of semantic categories. This work provides a set of standardized stimuli for the study of human emotional process. The dataset is composed by hundreds of pictures related to several scenes including images of insects, children, portraits, poverty, puppies and diseases, which have been manually rated by humans by means of affective words. This dataset has been used in [30] in combination with other two datasets built by the authors, which are publicly available 1 .
In [73] the authors considered a subset of IAPS extended with subject annotations to obtain a training set categorized in distinct emotions according to the emotional model described in [31] (see Section 3.1). However, the number of images of this dataset is very low.
In [30], the authors presented the Affective Image Classification Dataset. It consists of two image sets: one containing 228 abstract painting and the other containing 807 artistic photos. These images have been labelled by using the 8 emotions defined in [31].
The authors of the dataset presented in [28] considered the top 1.000 positive and negative words in SentiWordNet [29] as keywords to search and crawl over 586.000 images from Flickr. The list of image URLs as well as the collected including title, image resolution, description and the list of the associated tags is available for comparisons 2 .
The Geneva Affective Picture Database (GAPED) [74] dataset includes 730 pictures labelled considering negative (e.g., images depicting human rights violation scenes), positive (e.g., human and puppies) as well as neutral pictures which show static objects. All dataset images have been rated considering the valence, arousal, and the coherence of the scene. The dataset is available for research purposes 3 .
In 2013 Borth et al. [32] proposed a very large dataset (∼ 0.5 million) of pictures gathered from social media and labelled with ANP (Adjective Noun Pair) concepts. Furthermore, they proposed Twitter benchmark dataset which includes 603 tweets with photos. It is intended for evaluating the performance of automatic sentiment prediction using features of different modalities (text only, image only, and text-image combined). This dataset has been used by most of the state-of-the-art works as evaluation benchmark for Visual Sentiment Analysis, especially when the designed approaches involve the use of Machine Learning methods such as in [40] and in [34] for instance, due to the large scale of this dataset.
Marchewka et al. [75] presented the Nencki Affective Picture System (NAPS), a dataset of high-quality photos depicting people, faces, animals, objects and landscapes. The photos have been manually rated by 204 participants using a 3D dimensional approach and a 9-point rating scale for each dimension.
The Emotion6 dataset, presented and used in [41], has been built considering the Elkman's 6 basic emotion categories [36]. The number of images is balanced over the considered categories and the emotions associated with each image is expressed as a probability distribution instead of as a single dominant emotion.
In [40] You et al. proposed a dataset with 1,269 Twitter images labelled into positive or negative by 5 different annotators. Given the subjective nature of sentiment, this dataset has the advantage to be manually labelled by human annotators, differently than other datasets that have been created collecting images by automatic systems based on textual tags or predefined concepts such as the VSO dataset used in [32].
The work in [76] introduced a large scale dataset named Image-Emotion-Social Net (IESN), including over 1 million images crawled from Flickr, labelled considering either the dimensional (i.e., valence, arousal and dominance) and the categorical (i.e., Mikels' categories [31]) schemes. An important contribute of this work is the inclusion of information related to about 8.000 users and their interactions in the social platform (e.g., 1 www.imageemotion.org 2 http://www.l3s.de/ minack/flickr-sentiment 3 http://www.affective-sciences.org/home/research/materials-and-online-research/research-material/ May 19, 2020 Vadicamo et al. [51] crawled ∼ 3M tweets from July to December 2016. The collected tweets have been filtered considering only the ones written in English and including at least an image. The sentiment of the text extracted from the tweets has been classified using a polarity classifier based on a paired LSTM-SVM architecture. The data with the most confident prediction have been used to determine the sentiment labels of the images in terms of positive, negative and neutral. The resulting Twitter for Sentiment Analysis dataset (T4SA) consists of ∼ 1M tweets and related ∼ 1.5M images.
Datasets such as GAPED, NAPS and IAPS rely on emotion induction. This kind of datasets are very difficult to be built in large scale and maintained over time. The Machine Learning techniques and the recent Deep Learning methods are able to obtain impressive results as long as these systems are trained with very large scale datasets (e.g., VSO [32]). Such datasets can be easily obtained by exploiting the social network platforms by which people share their pictures every day. These datasets allowed the extensive use of Machine Learning systems that requires large scale datasets. This furthered the building of very large datasets such as T4SA in the last few years. Table 2 summarizes the main dataset just reported with details about the number of images, the source (e.g., Social Platform, paintings, etc.) and the labelling options.

Features
One of the most difficult step for the design of a Visual Sentiment Analysis system, and in general for the design of a data analysis approach is the selection of the data features that better encode the information that the system is aimed to infer. Image features for Visual Sentiment Analysis can be categorized within three levels of semantics: theory. These studies suggest that some low-level features, such as colors and texture can be used to express the emotional effect of an image [30].
• Mid-level features -This group of features bring more semantic, thus they are more interpretable and have stronger associations with emotions [77]. One example is given by the scene-based 102dimensional feature defined in [34]. Furthermore, many of the aforementioned works on Visual Sentiment Analysis exploit the 1200-dimensional mid-level representation given by the 1200 Adjective-Noun Pairs (ANP) classifiers defined by Borth et al. [32].
• High-level features -These features describe the semantic concepts shown in the images. Such a feature representation can be obtained by using pre-trained classification methods or semantic embeddings [46].
In 2010 Machajdik and Hanbury [30] performed an intensive study on image emotion classification by properly combining the use of several low and high visual features. These features have been obtained by exploiting concepts from and art theory [12,15], or exploited in image retrieval [78] and image classification [17,9] tasks. They selected 17 visual features, categorized in 4 groups: • color: mean saturation and brightness, 3-dimensional emotion representation by Valdez et al. [15], hue statistics, colorfulness measure according to [17], number of pixels of each of the 11 basic colors [79], Itten contrast [12], color histogram designed by Wang Wei-ning et al. [9]; • texture: wavelet textures for each HSB channel, features by Tamura et al. [80], and features based on GLCM (i.e., correlation, contrast, homogeneity, and energy for the HSB channels); • composition: the number of resulting segments obtained after the application of a waterfall segmentation (denoted as "level of detail" in [30]), depth of field (DOF) [17], statistics on the line slopes by using the Hough transform (denoted as "dynamics"), rule of thirds; • content: number of detected front faces, number of the biggest face pixels, count of skin pixels, ratio of the skin pixels over the face size.
Most of the mentioned works in Visual Sentiment Analysis combine huge number of hand-crafted visual features. Although all the exploited features have been proven to have a direct influence on the perceived emotion by previous studies, there is not agreement about which of them give the most of the contribution on the aimed task. Besides the selection of proper hand-crafted features, designed with the aim to encode the sentiment content conveyed by images, there are other kind of approaches that lean on representation learning techniques based on Deep Learning [37,39,40]. By employing such representation methods image features are learned from the data. This avoid the designing of a proper feature for the task of Visual Sentiment Analysis, because the system automatically learns how to extract the needed information from the input data. These methods requires huge amounts of labelled training data, and an intensive learning phase, but obtain better performances in general.
Another approach, borrowed from the image retrieval methods, consists on combining textual and visual information through multimodal embedding systems [46,56,61,62]. In this case, features taken from different modalities (e.g., visual, textual, etc.) are combined to create a common vector space in which the correlations between projections of the different modalities are maximized (i.e., an embedding space).
In the diagram depicted in Figure 5 the related works described in Section 2 are arranged temporally. For each work, the diagram reports the input features considered by the proposed system. It is possible to notice that early works were mostly based on hand-crafted visual features until 2016. Then, most of the works implemented systems able to automatically learn meaningful features from the input raw image (see branch 2017 in Figure 5). Lastly, the mentioned works published in 2019 combine the contribution of both the raw image and the associated metadata, as they can exploit large dataset built considering images crawled from social media platforms, which offer several additional textual metadata associated to images.
So far, there is not an established strategy to select the visual features that allows to address the problem. Most of the previous exploited features demonstrated to be useful, but recent results on Visual Sentiment Analysis suggest that it's worth investigating the use of representation learning approaches such as Convolutional Neural Networks and multimodal embeddings that are able to embody the contribute of multiple sources of information.

Problem analysis
In this section we propose a formulation of the problem, which highlights the related issues and the key tasks of Visual Sentiment Analysis. This allows to better focus the related sub-issues which form the Visual Sentiment Analysis problem and support the designing of more robust approaches. Moreover, to address the overall structure of the problem is useful to suggest a common framework helping researchers to design more robust approaches. Starting from the definition of the Sentiment Analysis problem applied to the natural language text given by Liu [81], we propose to generalize the definition in the context of Visual Sentiment Analysis.
Text based Sentiment Analysis can be performed considering different levels of detail: • at the document level the task is to classify whether a whole document (i.e., the whole input) expresses a positive or negative sentiment. This model works on the underling assumption that the whole input discusses only one topic; • at the sentence level the task is to find each phrase within the input document and determine if each sentence express a positive or negative (or neutral) sentiment; • the entity and aspect level performs finer-grained analysis by considering all the opinions expressed in the input document and defining a sentiment score (positive or negative) for each detected target.
Similarly, if the subject of the analysis is an image, we can: • consider a Sentiment Analysis evaluation for the whole image. These systems work with global image features (e.g., color histograms, saturation, brightness, colorfulness, color harmony, etc.); • consider an image as a composition of several sub-images according to its specific content. A number of sub-images is extracted and the sentiment analysis is performed on each sub-image obtained by exploiting methods such as multi-object detection, image segmentation, objectness extraction [82]; May 19, 2020 • define a set of image aspects, in terms of low level features, each one associated to a sentiment polarity based on previous studies [30,73]. This is essentially the most fine-grained analysis to be considered.
When a system aims to perform Sentiment Analysis on some textual content, basically it is looking for the opinions in the content and extracting the associated sentiment. An opinion consists of two main components: a target (or topic), and a sentiment. The opinions can be taken from more than one person, this means that the system has to take into account also the opinion holder. Furthermore, opinions can change over time, thus also the time an opinion is expressed has to be taken into account. According to Liu [81], an opinion (or sentiment) is a quintuple where e i is the name of an entity, a ij is an aspect related to e i , s ijkh is the sentiment score with respect to the aspect a ij , h k is the opinion holder, and t l is the time when the opinion is expressed by the opinion holder h k . The sentiment score s ijkh can be expressed in terms of polarity, considering positive, negative or neutral polarity; or with different levels of intensity. The special aspect "GENERAL" is used when the sentiment is expressed for the whole entity. In this case, either the entity e i and the aspect a ij represent the opinion target.
This definition is given in the context of opinion analysis applied on textual contents which express positive or negative sentiments. In the case of Sentiment Analysis applied on visual contents there are some differences. Indeed, when the input is a text, Sentiment Analysis can easily lean on context and semantic information extracted directly from the text. Thus the problem is to be considered into the NLP domain. When the input is an image, because of the affective gap between visual content representations and semantic concepts such as human sentiments, the task to associate the visual features with sentiment labels or polarity scores results challenging. Such affective gap can be defined as: "the lack of coincidence between the measurable signal properties, commonly referred to as features, and the expected affective state in which the user is brought by perceiving the signal" [83].
In the following paragraphs each of the sentiment components previously defined (entity, aspect, holder and time) are discussed in the context of Visual Sentiment Analysis.

Entity and Aspects
The entity is the subject (or target) of the analysis. In the case of Visual Sentiment Analysis the entity is the input image. In general, an entity can be viewed as a set of "parts" and "attributes". The set of the entity's parts, its attributes, plus the special aspect "GENERAL" forms the set of the aspects. This structure can be transferred to the visual domain considering different levels of visual features. Indeed, as mentioned above, in the case of visual contents, Sentiment Analysis can be performed considering different level of visual detail. The most general approach performs Sentiment Analysis considering the whole image, this corresponds to apply a Visual Sentiment Analysis method on the "GENERAL" aspect. The parts of an image can be defined by considering a set of sub-images. This set can be obtained by exploiting several Computer Vision techniques, such as background/foreground extraction, image segmentation, multi object recognition or dense captioning [84,85]. The attributes of an image regards its aesthetic quality features, often obtained by extracting low-level features. Exploiting this structured image hierarchy, a sentiment score can be achieved for each aspect. Finally, the scores are combined to obtain the sentiment classification (e.g., data can be used as input features of a regression model).
Instead of representing the image parts as a set of sub-images, an alternative approach can rely on a textual description of the depicted scene. The description of a photo can be focused on a specific task of image understanding. By changing the task, we can obtain different descriptions of the same image from different points of view. Then, these complementary concepts can be combined to obtain the above described structure. Most of the existing works in sentiment analysis of social media exploit textual information manually associated to images by performing textual Sentiment Analysis. Although the text associated to social images is widely exploited in the state-of-the-art to improve the semantics inferred from images, it can be a very noisy source because it is provided by the users; the reliability of such input is often based on the capability and the intent of the users to provide textual data that are coherent with respect to the visual content of the image. There is no guarantee that the subjective text accompanying an image is useful. In addition, the tags associated to social images are often selected by users with the purpose to maximize the Textual information provided by the user Figure 6: The paper in [56] extracts the text describing a given image by exploiting four different deep learning architectures. The considered architectures are used to extract text related to objects, scene and image description. The figure shows also the text associated to the image by the user (i.e., title, description and tags) in the bottom. The subjective text presents very noisy words which are highlighted in red. The words that appears in both sources of text are highlighted in green.
retrieval and/or the visibility of such images by the platform search engine. In Flickr, for instance, a good selection of tags helps to augment the number of views of an image, hence its popularity in the social platform. These information are hence not always useful for sentiment analysis. Starting from this observation, the authors of [56] presented a work on Image Polarity Prediction exploiting Objective Text extracted directly from images, and experimentally compared such text with respect to the Subjective (i.e., user provided) text information commonly used in the state-of-the-art approaches. For a deeper analysis, a comprehensive treatise of image tag assignment is presented in [86].
As discussed in [87], the semantic of an image can be expressed by means of an object category (i.e., a class). However the tags provided by users usually include several additional terms, related to the object class, coming from a larger vocabulary. As an alternative, the semantic could be expressed by using multiple keywords corresponding to scenes, object categories, or attributes. Figure 6 shows an example taken from [56]. The textual information below the image is the text provided by the Flickr's user. Namely the photo title, the description and the tags are usually the text that can be exploited to make inferences on the image. This example shows how the text can be very noisy with respect to any task aimed to understand the sentiment that can be evoked by the picture. Indeed the title is used to describe the tension between the depicted dogs, whereas the photo description is used to ask a question to the community. Furthermore, most of the provided tags include misleading text such as geographical information (i.e., Washington State, Seattle), information related to the camera (i.e., Nikon, D200), objects that are not present in the picture (i.e., boy, red ball, stick) or personal considerations of the user (i.e., my new word view). Moreover, in the subjective text there are many redundant terms (e.g., dog). Another drawback of the text associated to social images is that two users can provide rather different information about the same picture, either in quality and in quantity. Finally, there is not guarantee that such text is present; this is an intrinsic limit of all Visual Sentiment Analysis approaches exploiting subjective text.
Starting from the aforementioned observations about the user provided text associated to social images, the paper in [56] exploits an objective aspect of the textual source that comes directly from the understanding of the visual content. This text is achieved by employing a set of deep learning models trained to accomplish different visual inference tasks on the input image. At the top right part of Figure 6 the text automatically extracted with different scene understanding methods is shown. In this case, the inferred text is very descriptive and each model provides distinctive information related to objects, scene, context, etc. The objective text extracted by the three different scene understanding methods has a pre-defined structure, therefore all the images have the same quantity of textual objective information. Such approach provides an alternative user-independent source of text which describes the semantic of images, useful to address the issues related to the inherent subjectivity of the text associated to images.
Several papers faced the issues related to the subjective text associated to images, such as tag refinement and completion [86,88,89,90], which aims at alleviating the number of noisy tags and enhancing the number of informative tags by modelling the relationship between visual content and tags.

Holder
Emotions are subjective, they are affected by several factors such as gender, individual background, age, environments, etc. However, emotions also have the property of stability [91]. This means that the average emotion response of a statistically large set of observers is stable and reproducible. The stability of emotion response enables researchers to generalize their results, when obtained on large datasets.
Almost all the works in Visual Sentiment Analysis ignore the sentiment holder, or implicitly consider only the sentiment of the image publisher. In this context at least two holders can be defined: the image owner and the image viewer. Considering the example of an advertising campaign, where the owner is the advertising company and the viewers are the potential customers, it's crucial the study and analysis the connection between the sentiment intended by the owner and the actual sentiment induced to the viewers.
These days, the social media platforms provide a very powerful mean to retrieve real-time and large scale information of people reactions toward topics, events and advertising campaigns. The work in [38] is the first that distinguishes publisher affect (i.e., intent) and viewer affect (i.e., reaction) related to the visual content. This branch of research can be useful to understand the relation between the affect concepts of the image owner and the evoked viewer ones, allowing new user centric applications.
Zhao et al. [76] proposed a method to predict the personalized emotion induced by an image for each individual viewer (i.e., user in the social platform). To this aim, the authors considered several contextual factors such as the social context, temporal evolution and the location. This work required the definition of an ad-hoc dataset which includes information about a large scale of involved users, beside the set of images. User profiling helps personalization, which is very important in the field of recommendation systems. The insights that could be obtained from such a research branch can be useful for several business fields, such as advertisement and User Interface design (UI). And the community of user interface designers started to take into account the emotional effect of the user interfaces toward users who are interacting with a website, product or brand. The work in [92] discusses about methods to measure user's emotion during an interface interaction experience, with the aim to assess the interface design emotional effect. Progresses in this field promote the definition of new design approaches such as Emotional UI [93], aimed to exploit the emotions conveyed by visual contents. Indeed, emotions have been traditionally considered to be something that the design evoked, now they represent something that drives the design process. While so far, designers focused on "user friendly" design (i.e., interfaces easy to use), now they need to focus on design that stimulates and connects the product with users deeply.
Although the interesting cues discussed in this paragraph, currently the development of Visual Sentiment Analysis algorithms that concern the sentiment holder find difficulties due the lack of specific datasets. In this context, the huge data shared on the social media platforms can be exploited to better understand the relationships between the sentiment of the two main holders (i.e., owner/publisher and viewer/user) through their interactions.

Time
Although almost all the aforementioned works ignore this aspect, the emotion evoked by an image can change depending on the time. This sentiment component can be ignored the most of times, but in specific cases is determinant. Moreover, is very difficult to collect a dataset related to the changes in the emotion evoked by images over time. For example, the sentiment evoked by an image depicting the World Trade Center (Figure 7) is presumably different if the image is shown before or after 9/11.
Although there are not works on Visual Sentiment Analysis that analyse the changes of image sentiments over time, due to the specificity of the task and the lack of image datasets, there are several works that exploits the analysis of images over time focused on specific cognitive and psychology applications. As an example, the work in [94] employed a statistical framework to detect depression by analysing the sequence of photos posted on Instagram. The findings of this paper suggest the idea that variations in individual psychology reflect in the use social media by the users, hence they can be computationally detected by the analysis of the user's posting history.
In [95] the authors studied which objects and regions of an image are positively or negatively correlated with memorability, allowing to create memorability maps for each image. This work provides a method May 19, 2020 to estimate the memorability of images from many different classes. To collect human memory scores, the adopted experimental procedure consists of showing several occurrences of the same images at variable time intervals. The employed image dataset has been created by sampling images from a number of existing dataset, including images evoking emotions [30].

Challenges
So far we discussed on the current state of the art in Visual Sentient Analysis, describing the related issues, as well as the different employed approaches and features. This section aims to introduce some additional challenges and techniques that can be investigated.

Popularity
One of the most common application field of Visual Sentiment Analysis is related to social marketing campaigns. In the context of social media communication, several companies are interested to analyse the level of people engagement with respect to social posts related to their products. This can be measured as the number of post's views, likes, shares or by the analysis of the comments. These information can be further combined with web search engine and companies website visits statistics, to find correlations between social advertising campaigns and their aimed outcomes (e.g., brand reputation, website/store visits, product dissemination and sale, etc.) [96,97,98].
The popularity of an image is a difficult quantity to define, hence to measure or infer. However human beings are able to predict what visual contents other people will like in specific contexts (e.g., marketing campaigns, professional photography). This suggest that there are some common appealing factors in images. So far, researches have been trying to gain insights into what features make an image popular.
As mentioned in the previous sections, the quality of the text associated to images is often pre-processed in order to avoid noisy text. On the other hand, the users who want increase the reach of their published contents are used to associate popular tags to their images, regardless their relevance with the image content. This is motivated by the fact that image associated tags are used by the image search engines in these platforms, so the use of popular tags highs the reach of the pictures. Therefore, in this context, the tags associated to images become a crucial factor to understand the popularity of images in social platforms. For instance, Yamasaki et al. [99] proposed an algorithm to estimate the social popularity of images uploaded on Flickr by using only text tags. Other features should be also taken into account such as: number of user followers and groups, which represent the reach capability of the user. These factors make the task of popularity prediction very different from the task of sentiment polarity classification in the selection of features, methods and measures of evaluation.
The work in [26] considers the effect of 16 features in the prediction of the image popularity. Specifically, they considered image context (i.e., day, time, season and acquisition settings), image content (i.e., image content provided by detectors of scenes, faces and dominant colors), user context and text features (i.e., image tags).

May 19, 2020
The authors cast the problem as a binary classification by splitting the dataset between images with high and low popularity measure. As popularity measures the authors considered the views and the comments counts. Their study highlights that comments are more predictable than views, hence comments are more correlated with the studied features. The experimental results show that the accuracy values achieved only considering textual features (i.e., tags) outperform the performances of the classification based on other features and the combinations of them, for both comments and views counts classifications.
In 2014 Khosla et al. [25] proposed a log-normalized popularity score that has been then commonly used in the community. Let c i be a measure of the engagement achieved by a social media item (e.g., number of likes, number of views, number of shares, etc.), also known as popularity measure. The popularity score of the i th item is defined as follows: where T i is the number of days since the uploading of the image on the Social Platform. Equation 2 normalizes the number of interactions reached by an image by dividing the engagement measure by the time. However, the measures c i related to social posts are cumulative values as they continuously collect the interactions between users and the social posts during their time on-line. Therefore, this normalization will penalize social media contents published in the past with respect to more recent contents, especially when the difference between the dates of posting is high. Indeed, the most of the engagement obtained by a social media item is achieved in the first period, then the engagement measures become more stable. There are very few works which takes into account the evolution of the image popularity over time. For example, the study presented in [100] shows that photos obtain most of their engagement within the first 7 days since the date of upload. However, this study is focused on Flickr, and each social platform has its own mechanisms to show contents to users.
In [25] the authors analysed the importance of several image and social cues that lead to high or low values of popularity. In particular they considered the relevance of user context features (e.g., mean views, number of photos, number of contacts, number of groups, average groups' members, etc.), image context features (e.g., title length, description length, number of tags), as well as image features (e.g., GIST, LBP, BoW color patches, CNN activations features, etc.). Is interesting to notice that, differently than several works on sentiment polarity prediction in which the text concerning the images (i.e., title, description and tags included in the post) is semantically analysed in order to achieve sentiment related insights on the image content, in this work only the length of the text associated to the images is considered.
Cappallo et al. [101] address the popularity prediction problem as a ranking task by exploiting a latent-SVM objective function defined such that the ranking of the popularity scores between pairs of images is maintained. They considered the number of views and comments for Flickr images and the number of re-tweets and favorites for Twitter.
The problem of image popularity prediction is also addressed in [24], whose experiments suggest that some sentiment ANPs defined in VSO [32] have a correlation with popularity.
In [102] the authors considered the number of likes achieved within the first hour after the image posting (early popularity) to predict the popularity after a day, a week or a month. This study has been performed on Instagram images and the dataset is publicly available 5 . The images are categorized as popular or not popular considering a popularity threshold obtained with the Pareto principle (80 % -20 %). Three features representing information that is retrieved within the first hour of image upload (i.e., early information) are evaluated: social context (based on the user's number of followers), image semantics (based on image caption and NLP), and early popularity in the first hour. The binary classification is performed by using a Gaussian Naive Bayes Model. The experimental results show that the early popularity feature significantly outperforms the other evaluated features. Furthermore, the authors compared the proposed semantic feature with the features proposed by [26] for the task of popularity binary classification considering the MIR-1M Flickr dataset [103], obtaining better accuracy rates.
Most of the works addressing the problem of popularity prediction follow a very similar pipeline. First, a set of interesting features that have been demonstrating correlation with the images sentiment or popularity is selected. Then a model for each distinctive feature is trained to understand the predictive capability of each feature. In the above discussed works, the popularity prediction task is cast as a ranking or a regression problem. Therefore, the exploited algorithms are ranking SVM and latent SVM in the case of ranking, and May 19, 2020 SVR for regression. Then, the features are combined with the aim to improve the performances of the method. The evaluation is usually quantified through the Spearman's correlation coefficient.
Although the task of image popularity prediction is rather new, there are interesting datasets available for the development of massive learning systems (i.e., deep neural networks). The Micro-Blog Images 1 Million (MBI-1M) dataset is a collection of 1M images from Twitter, along with accompanying tweets and metadata. The dataset was introduced by the work in [101]. A subset of the the Trec 2013 micro-blog track tweets collection [104] has been selected.
The MIR-1M dataset [103] is a collection of 1M photos from Flickr. These images have been selected considering the interestingness score used by Flickr to rank images.
The Social Media Prediction (SMP) dataset is a large-scale collection of social posts, recently collected for the ACM Multimedia 2017 SMP Challenge 6 . This dataset consists of over 850K posts and 80K users, including photos from VSO [32] as well as photos collected from personal users' albums [105,106,107]. In particular, the authors aimed to record the dynamic variance of social media data. Indeed, the social media posts in the dataset are obtained with temporal information (i.e., posts sequentiality) to preserve the continuity of post sequences. Two challenges have been proposed: • Popularity Prediction: the task is to predict a popularity measure defined for the specific social platform (e.g., number of photo's views on Flickr) of a given image posted by a specific user; • Tomorrow's Top Prediction: given a set of photos and the data related to the past photo sharing history, the task is to predict the top-n popular posts (i.e., ranking problem over a set of social posts) on the social media platform in the next day.
The SMP dataset includes features such as unique picture id (pid) and associated user id (uid). From these information one can extract almost all the user and photo related data available in Flickr. Some metadata of the picture and user-centered information are also included in the dataset. Moreover, the popularity scores (as defined in Equation 2) are provided. The SMP dataset furthered the development of time aware popularity prediction methods, which exploit time information to define new image representation spaces used to infer the image popularity score at a precise time or at pre-defined time scales. Li et al. [108] extracted multiple time-scale features from a set of timestamps related to the photo post. As instance, the timestamp "postdate" is used to define several features with different time scales: "season of year", "month of year", etc. The framework presented in [109] exploits an ensemble learning method to combine the outputs of an SVR and a CART (Classification And Regression Tree) models, previously trained to estimate the popularity score. The models have been trained by exploiting features extracted from user's information, image meta-data and visual aesthetic features extracted from the image. In particular, the authors take into account the post duration (i.e., the number of days the image was posted), the upload time, day and month.

Popularity Dynamics
Equation 2 normalizes the number of interactions reached by an image by dividing the engagement measure by the time. However, the measures c i related to social posts are cumulative values as they continuously collect the interactions between users and the social posts during their time on-line. Therefore, this normalization will penalize social media contents published in the past with respect to more recent contents, especially when the difference between the dates of posting is high. In other words, such a score does not take into account the evolution of the popularity over time. As consequence, two images with the same popularity dynamic could be ranked differently, depending on the download time. In particular, the most of the engagement obtained by a social media item is achieved in the first period, then the engagement measures become more stable. For example, the study presented in [100] shows that photos obtain most of their engagement within the first 7 days since the date of upload. However, this study is focused on Flickr, and each social platform has its own mechanisms to show contents to users. There are very few works which takes into account the evolution of the image popularity over time (i.e., the dynamics of the image popularity). The authors of [110] presented a work on image popularity dynamic prediction. An important contribution is the definition of a dataset of ∼ 20K Flickr images, crawled within the first 2 hours from posting, and their daily engagement scores for 30 days. The approach in [110] considers two difference properties of a sequence: the sequence shape describes the trend of the popularity, whereas the sequence scale gives a degree of popularity in the whole period (i.e., the maximum value of engagement achieved after 30 days). The authors performed a detailed analysis of the task, demonstrating that for a given sequence of popularity, the scale and shape properties are unrelated.

May 19, 2020
Such a study defined a set of 50 typical shapes for the 30-days sequences that can be used as prototypes for the popularity dynamics, as well as a method to predict the shape and the scale of an image, given a set of post features. At test time, the two outputs are combined to infer the whole popularity sequence. Since the employed features are available at the moment of the post, this method is able to predict the whole sequence of popularity over 30 days with a daily granularity even before the image is posted.

Image Virality
A recent emerging task, closely related to image popularity, is the prediction of the level of virality of an image. The image virality is defined as the quality of a visual content (i.e., images or videos) to be rapidly and widely spread on social networks [111]. Differently than popularity, the virality score takes into account also the number of resubmission of an image by different users. Therefore, images that became popular when they are posted, but not reposted, are not considered to be viral [112]. These often involve images which content itself is less relevant, but are related to current events that drawn attention to the image in a specific period such as a flash news, or a tragedy. The work in [111] focused on understanding the influence of image parts on its virality. In particular, the authors presented a method for the task of simultaneously detection and localicazion of virality in images. The detection consists on the prediction of the virality score on an image. The localization aims to detect which areas in an image are responsable for making the image viral, this allows to produce an heatmap which highlights the relevant areas of the input image.

Relative Attributes
As discussed in previous sections, several Visual Sentiment Analysis works aim to associate an image one sentiment label over a set of emotional categories or attributes. However, given a set of images that have been assigned to the same emotional category (e.g., joy), it would be interesting to determine their ranking with respect the specific attribute. Such a technique could suggest, for example, if a given image A conveys more "joy" than another image B. For this purpose, several works on relative attributes can be exploited [113,114,115,116]. Furthermore, a ground truth dataset can be built by exploiting human annotators. Given a pair of images, the annotator is requested to indicate which image is closer to the attribute. In this way it's possible to obtain a proper ranking for each sentiment attribute.

Common Sense
With the aim to reduce the affective and cognitive gap between images and sentiments conveyed by them, we further need to encode the "affective common-sense". An Halloween picture can be classified as a negative image by an automatic system which considers the image semantics, however the knowledge of the context (i.e., Halloween) should affect the semantic concepts conveyed by the picture, hence its interpretation. This corresponds to the "common-sense knowledge problem" in the field of knowledge representation, which is a sub-field of Artificial Intelligence. Clearly, besides inferential capabilities, such an intelligent program needs a representation of the knowledge. By observing that is very difficult to build a Sentiment Analysis system that may be used in any context with accurate classification prediction, Agrawal et al. [117] considered contextual information to determine the sentiment of text. Indeed, in this paper is proposed a model based on common-sense knowledge extracted from ConceptNet [118] ontology and context information. Although this work addresses the problem of Sentiment Analysis applied on textual data, as discussed above, the knowledge of the context related to what an image is depicting should affect its interpretation. Moreover, such results on textual analysis can be transferred to the visual counterpart. Furthermore, emerging approaches based on the Attention mechanism could be exploited to add such a context. The Attention mechanism is a recent trend in Deep Learning, it can be viewed as a method for making the Artificial Neural Network work better by letting the network know where to look as it is performing its task. For example, in the task of image captioning, the attention mechanism tells the network roughly which pixels to pay attention to when generating the text [119,120].

Emoticon/Emoji
In this section we discuss about the possibility to exploit text ideograms, such emoticons and emoji, in the task of Sentiment Analysis on both visual and textual contents. An emoticon is a textual shorthand that represents a facial expression. The emoticons have been introduced to allow the writer to express feelings and emotions with respect to a textual message. It helps to express the correct intent of a text sentence, improving the understanding of the message. The emoticons are used to emulate visual cues in textual communications Figure 8: Some examples of the Emoji Sentiment Ranking scores and statistics obtained by the study conducted in [125]. The sentiment bar (10th column) shows the proportion of negativity, neutrality and positivity of the associated emoji.
with the aim to express or explicitly clarify the writer's sentiment. Indeed, in real conversations the sentiment can be inferred from visual cues such as facial expressions, pose and gestures. However, in textual based conversations, the visual cues are not present.
The authors of [121] tried to understand if emoticons could be useful as well on the textual Sentiment Analysis task. In particular, they investigated the role that emoticons play in conveying sentiment and how they can be exploited in the field of Sentiment Analysis. The authors manually labelled 574 emoticons as positive or negative, and combined this emoticon-lexicon with the text based Sentiment Analysis to perform document polarity classification considering both sentence and paragraph levels. A step further the emoticon, is represented by the emoji. An emoji is an ideogram representing concepts such as weather, celebration, food, animals, emotions, feelings, and activities, besides a large set of facial expressions. They have been developed with the aim to allow more expressive messages. Emojis have become extremely popular in social media platforms and instant messaging systems. For example, in March 2015, Instagram reported that almost half of the texts on its platform contain emojis [122].
In [123], the authors exploited the expressiveness carried by emoji, to develop a system able to generate an image content description in terms of a set of emoji. The focus of this system is to use emoji as a means for image retrieval and exploration. Indeed, it allows to perform an image search by means of a emoji-based query. This approach exploits the expressiveness conveyed by emoji, by leaning on the textual description of these ideograms (see the 10th column in Figure 8). The work in [124] studied the ways in which emoji can be related to other common modalities such as text and images, in the context of multimedia research. This work also presents a new dataset that contains examples of both text-emoji and image-emoji relationships.
Most of them contains also strong sentiment properties.
In [125] the authors presented a sentiment emoji lexicon named Emoji Sentiment Ranking. In this paper, the sentiment properties of the emojis have been deeply analysed, and some interesting conclusions have been highlighted. For each emoji, the Emoji Sentiment Ranking provides its associated positive, negative and neutral scores, represented by values between -1 and +1. The authors also proposed a visual tool, named sentiment bar, to better visualize the sentiment properties associated to each emoji (see Figure 8). The data considered in this analysis consists of 1.6 million labelled tweets. This collection includes text written in 13 different languages. The authors found that the sentiment scores and ranking associated to emojis remain stable among different languages. This property is very useful to overcome the difficulties addressed in multilingual contexts. This lexicon represents a precious resource for many useful applications 7 .
The work in [126] presented a dataset of 4M Twitter images with their associated emojis. Based on such large scale dataset, the authors defined a deep neural model for image-emoji embedding. The proposed system has been evaluated on the tasks of emoji prediction, sentiment analysis and emotion prediction.
The results and the insights obtained in [123,124,126] and [125] could be combined to exploit the sentiment conveyed by emoji on the task of Visual Sentiment Analysis, rather than exploiting textual information as in the most of the previous works. For instance, an image could be represented by considering the distribution of the associate emojis as a sentiment feature, taking a cue from the approach presented in [41].
By a few years, Facebook has released a new "reactions" feature, which allows users to interact with a Facebook post by using one of six emotional reactions (Like, Love, Haha, Wow, Sad, and Angry), instead of just having the option of "liking" a post. These reactions corresponds to a meaningful subset of emoji.

Discussion
The papers presented in Section 2 represent the most relevant works in the field fo Visual Sentiment Analysis aimed to infer the emotions evoked by pictures. As previously mentioned, there is not agreement on these works about meaningful input features, benchmark datasets nor emotion categories. For this reason, we further investigated these aspects in details in Section 4.The basic polarity classification task considers two or three polarity levels (i.e., negative, neutral and positive). However, usually more sophisticated psychology based emotional models are used, such as the Plutchnik's wheel of emotions [33], the Ekman's 5 basic emotions [36], the 35 impression words [69], or the Mikels' 8 basic emotions [31]. Moreover, the works that aim to infer a distribution over a set of sentiment categories [41,48,49] employ variations of Mikels' or Ekman's models. Other works, employ specific categorizations such as the 5-scale sentiment categorization used in [39], or the ANP annotation introduced in [32].
The employed datasets can be organized into two main categories: psychological based or social media based.
Datasets based on psychological studies [70,73,30], such as IAPS and GAPED, have different pros and cons. Indeed, the photos have been designed to select stimuli eliciting specific ranges of emotions, and the data can lean on reliable labelling (i.e., human annotations supported by tools/scientists). However, this kind of dataset requires high efforts to be built and maintained over time. As consequence, is not possible to build large scale datasets. Moreover, considering the selection process of the photos, they results very different kind of images with respect to the ones shared through social media platforms.
More recent efforts have been spent in building large scale datasets of images shared through social media platforms (e.g., Flickr, Twitter, Facebook, Instagram, etc.). These media provide an easy way to retrieve images (i.e., automatic crawling from social media platforms) at very large scale. Moreover, these images are often associated with accompanying metadata that provides additional context and help for the automatic labelling of the images (e.g., by exploiting NLP on metadata). However, such automatic labelling is often unreliable (e.g., metadata includes sarcam, personal considerations, misleading text, etc.). Therefore, such large scale datasets need further filtering processes before their exploitation in Visual Sentiment Analysis systems.
Feature selection is crucial for any data analysis process. Data features are supposed to encode the information that the system will be able to infer. There are three general approaches in the field of Image Sentiment Analysis: find correlations between visual features (e.g, colors, textures, etc..) and sentiment, learn image features from data (e.g., training a CNN), exploiting a multi-modal representation learning approach by combining visual and textual information through multimodal embedding systems (i.e., defining a new vector space).

Summary and Future Directions
In this paper we have discussed the main issues and techniques related to Visual Sentiment Analysis. The current state of the art has been analysed in detail, highlighting pros and cons of each approach and dataset. Although this task has been studied for years, the field is still in its infancy. Visual Sentiment Analysis is a challenging task due to a number of factors that have been discussed in this paper.
The results discussed in this study, such as [30], agree that the semantic content has a great impact on the emotional influence of a picture. Images having similar colour histograms and textures could have completely different emotional impacts. As result, a representation of images which express both the appearance of the whole image and the intrinsic semantic of the viewed scene is needed. Early methods in the literature about Visual Sentiment Analysis tried to fill the so called affective gap by designing visual representations. Some approaches build systems trained with human labelled datasets and try to predict the polarity of the images. Other approaches compute the polarity of the text associated to the images (e.g., post message, tags and comments) by exploiting common Sentiment Analysis systems that works on textual contents [29,127], and try to learn Machine Learning systems able to infer that polarity from the associated visual content. These techniques have achieved interesting improvements in the tasks of image content recognition, automatic annotation and image retrieval [128,87,129,130,131,132,133]. However, it is impossible to know if such user provided text is related to the image content or to the sentiment it conveys. Moreover, the text associated to images is often noisy. Therefore, the exploitation of such text for the definition of either polarity ground truth or as an input source for a sentiment classifier have to address with not reliable text sources. A specific study on this issue is reported in [56]. Furthermore, some approaches exploit a combination of feature modalities (often called views) to build feature space embeddings in which the correlation of the May 19, 2020 multi-modal features associated to the images that have the same polarity is maximized [46,56,61,62]. The results achieved by several discussed works suggest that exploiting multiple modalities is mandatory, since the sentiment evoked by a picture is affected by a combination of factors, beside the visual information. This has been further confirmed by recent works that add the attention mechanism to the multi-modality approach [58,59]. Studies in psychology and art theory suggested some visual features associated to emotions evoked by images. However, the most promising choice is given by representations automatically learned through neural networks, autoencoder and feature embedding methods. These approaches are able to find new feature spaces which capture contributes from the different input factor which the sentiment is affected by. The recent results in representation learning confirm this statement.
To this end, one important contribution is given by the availability of large and robust datasets. Indeed, in this study, we highlighted some issues related to the existing datasets. Modern social media platforms allows the collection of huge amount of pictures with several correlated information. These can be exploited to define either input features and "ground truth". However, as highlighted before, these textual information need to be properly filtered and processed, in order to avoid the association of noisy information to the images.
Systems with broader ambitions could be developed to address the new challenges (e.g., relative attributes, popularity prediction, common-sense, etc.) or to focus on new emerging tasks (e.g., image popularity prediction, sentiment over time, sentiment by exploiting ideograms, etc.). For instance, ideograms help people to reduce the gap between real and virtual communications. Thanks to the diffusion of social media platforms, the use of emojis has been growing for years, and they are now integrated to the way people communicate in the digital world. They are commonly used to express user reactions with respect to messages, pictures, or news. Thus, the analysis of such new communication media could help to improve the current state of the art performances.
This paper aimed to give a comprehensive overview of the Visual Sentiment Analysis problem, the relative issues, and the algorithms proposed in the state of the art. Relevant points with practical applications in business fields which would benefit from studies in Sentiment Analysis on visual contents have been also discussed. This survey addresses all the aspects related to Visual Sentiment Analysis, and for each of these, relevant references, datasets and results are presented. It also proposes a critical perspective for each considered issue. The paper aims to serve as a starting point for researchers that want to tackle tasks in the field of Visual Sentiment Analysis and related challenges, as it is presented as a structured and critique review of previous works, description of the available datasets, features and techniques. It also aims to clear the way for new solutions, by suggesting new and interesting techniques and sources of information that can be explored to tackle the problem. Although the psychological basis (e.g., emotional models) and some techniques are often applied on both images and videos, the presented study is mostly devoted to the analysis of still images. Future extensions of this paper could be devoted to the review of the state of the art in Visual Sentiment Analysis applied on videos. In the case of videos, the temporal dimension and the presence of the audio or other type of temporal signals (e.g., EEG [134]) pave the way for a number of sentiment related tasks on visual contents such as multi-modal continuous emotional state recognition [134] or scene popularity [135,136], among others. Several datasets designed for these tasks are already available [134,135,136].