{'fraction_non_alphanumeric': 0.02629391764261428, 'fraction_numerical': 0.010691348696637702, 'mean_word_length': 4.83447211813974, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Apples to Oranges: Evaluating Image Annotations from Natural Language Processing Systems

We examine evaluation methods for systems that automatically annotate images using co-occurring text. We compare previous datasets for this task using a series of baseline measures inspired by those used in information retrieval, computer vision, and extractive summarization. Some of our baselines match or exceed the best published scores for those datasets. These results illuminate incorrect assumptions and improper practices regarding preprocessing, evaluation metrics, and the collection of gold image annotations. We conclude with a list of recommended practices for future research combining language and vision processing techniques.


Introduction
Automatic image annotation is an important area with many applications such as tagging, generating captions, and indexing and retrieval on the web. Given an input image, the goal is to generate relevant descriptive keywords that describe the visual content of the image. The Computer Vision (CV) literature contains countless approaches to this task, using a wide range of learning techniques and visual features to identify aspects such as objects, people, scenes, and events.
Text processing is computationally less expensive than image processing and easily provides information that is difficult to learn visually. For this reason, most commerical image search websites identify the semantic content of images using co-occurring text exclusively. But co-occurring text is also a noisy source for candidate annotations, since not all of the text is visually relevant. Techniques from Natural Language Processing help align descriptive words and images. Some examples of previous research use named-entity recognition to identify people in images (Deschacht and Moens, 2007); term association to estimate the "visualness" of candidate annotations (Boiy et al., 2008;Leong et al., 2010); and topic models to annotate images given both visual and textual features (Feng and Lapata, 2010b).
Image annotation using NLP is still an emerging area with many different tasks, datasets, and evaluation methods, making it impossible to compare many recent systems to each other. Although there is some effort being made towards establishing shared tasks 1 , it is not yet clear which kinds of tasks and datasets will provide interesting research questions and practical applications in the long term. Until then, establishing general "best practices" for NLP image annotation will help advance and legitimitize this work. In this paper, we propose some good practices and demonstrate why they are important.

Image Annotation Evaluation in CV and NLP
In this section, we first review related work in image annotation evaluation in computer vision, specific challenges, and proposed solutions. We then relate these challenges to the NLP image annotation task and some of the specific problems we propose to address.

Related Work in Computer Vision
The work of Müller et al. (2002) is one of the first to address the issue of evaluation for image annotation systems. While using the exact same annotation system, dataset, and evaluation metric, they dramatically improve the apparent performance of their system by using dataset pruning heuristics.
Others have criticized commonly-used CV datasets for being too "easy" -images with the same keywords are extremely similar in low-level features such as orientation, lighting, and color; while differences between images with different keywords are very clear (Westerveld and de Vries, 2003;Ponce et al., 2006;Hervé and Boujemaa, 2007;Tang and Lewis, 2007). These features are unwittingly exploited by certain algorithms and obscure the benefits of using more complex techniques (Ponce et al., 2006). The problem is further exacerbated by evaluation metrics which essentially prefer precision over recall and are biased towards certain keywords. Annotations in test data might not include all of the "correct" keywords, and evaluation metrics need to account for the fact that frequent keywords in the corpus are safer guesses than keywords that appear less frequently (Monay and Gatica-Perez, 2003).
New baseline techniques, evaluation metrics, and datasets for image annotation have been developed in response to these problems. Makadia et al. (2008; define a basic set of low-level features, and propose new baselines for more complex systems to evaluate against. Barnard et al. (2003) present a normalized loss function to address the preference towards precision in evaluation metrics. New datasets are larger and provide more diverse images, and it is now easy to obtain multiple human-annotations per image thanks to distributed services such as Amazon's Mechanical Turk, and the ESP game (von Ahn and Dabbish, 2004). Hanbury (2008) provides an overview of popular CV annotation datasets and methods used for building them.

Image Annotation using Natural Language Processing
Many of the problems from CV image annotation are also applicable to NLP image annotation, and bringing NLP to the task brings new challenges as well. One of these challenges is whether to allow infrequent words to be pruned. In CV annotation it is typical to remove infrequent terms from both the keyword vocabulary and the evaluation data because CV algorithms typically need a large number of examples to train on. However, using NLP systems and baselines one can correctly annotate using keywords that did not appear in the training set. Removing "unlearnable" keywords from evaluation data, as done in (Boiy et al., 2008;Feng and Lapata, 2010b), artificially inflates performance against simple baselines such as term frequency. Nearly all NLP annotation datasets use naturallyoccurring sources of images and text. A particularly popular source is news images alongside captions or articles, which are collected online from sources such as Yahoo! News (Berg et al., 2004;Deschacht and Moens, 2007). There are also domainspecific databases with images and descriptions such as the art, antiques, and flowers corpora used in Boiy et al. (2008). Wikipedia has also been used as a source of images and associated text (Tsikrika et al., 2011). These sources typically offer well-written and cleanly formatted text but introduce the problem of converting text into annotations, and the annotations may not meet the requirements of the new task (as shown in Section 3.1). Obtaining data via image search engines is a common practice in CV (Fei-Fei et al., 2004; and can also be used to provide more challenging and diverse instances of images and co-occurring text. The additional challenge for NLP is that text content on many websites is written to improve their rank in search engines, using techniques such as listing dozens of popular keywords. Co-occurring text for retrieved images on popular queries may not be representative of the task to be performed.

Datasets
In this paper, we examine two established image annotation datasets: the BBC News Dataset of Feng and Lapata (2008) (henceforth referred to as BBC), and the general web dataset of Leong et al. (2010) (henceforth referred to as UNT). These datasets were both built to evaluate image annotation systems that use longer co-occurring text such as a news article or a webpage, but they use data from differ- ent domains, different sources of gold image annotations, different preprocessing procedures, and different evaluation measures. Table 1 provides an overview of the datasets; while this section covers the source of the datsets and their gold annotations in more detail.

BBC
The BBC Dataset (Feng and Lapata, 2008) 2 contains news articles, image captions, and images taken from the BBC News website. Training instances consist of a news article, image, and image caption from the same news story. Test instances are just the image and the article, and hold-out the caption as a source of gold image annotations.
Using news image captions as annotations has 2 Downloaded from http://homepages.inuf.ed.
ac.uk/s0677528/data.html the disadvantage that captions often describe background information or relate the photo to the story, rather than listing important entities in the image. It also fails to capture variation in how humans describe images, since it is limited to one caption per image. 3 However, captions are a cheap source of data; BBC has ten times as many images as UNT.
To address the problem of converting natural language into annotations, a large amount of preprocessing is performed. The established preprocessing procedure for this dataset is to lemmatize and POStag using TreeTagger (Schmid, 1994) then remove all but the "descriptive" words (defined as nouns, adjectives, and certain classes of verbs). This leaves a total text vocabulary of about 32K words, which is further reduced by removing words that appear fewer than five times in the training set articles. Table 1 shows the number of word tokens and types after performing these steps. 4

UNT
The UNT Dataset (Leong et al., 2010) 5 consists of images and co-occurring text from webpages. The webpages are found by querying Google Image Search with frequent English words, and randomly selecting from the results.
Each image in UNT is annotated by five people via Mechanical Turk. In order to make human and system results comparable, human annotators are required to only select words and collocations that are directly extracted from the text, and the gold annotations are the count of how many times each keyword or collocation is selected. The human annotators write keywords into a text box; while the collocations are presented as a list of candidates and annotators mark which ones are relevant. Human annotators tend to select subsets of collocations in addition to the entire collocation. For example, the gold annotation for one image has "university of texas", "university of texas at dallas", "the university of texas", and "the university of texas at dallas", each selected by at least four of the five annotators. Additionally, annotators can select multiple forms of the same word (such as "tank" and "tanks"). Gold annotations are stemmed after they are collected, and keywords with the same stem have their counts merged. For this reason, many keywords have a higher count than the number of annotators. 4 We are unable to reproduce work from Feng & Lapata (2008;2010a;2010b) and Feng (2011). Specifically, our vocabulary counts after preprocessing (as in Table 1) are much higher than reported counts, although the number of tokens per article/caption they report is higher than ours. We have contacted the authors, who confirmed that they took additional steps to reduce the size of the vocabulary, but were unable to tell us exactly what those steps are. Therefore, all system and baseline scores presented on their dataset are of our own implementation, and do not match those reported in previous publications. 5

Baselines
We run several baselines on the datasets. Term frequency, tf*idf, and corpus frequency are features that are often used in annotation systems, so it is important to test them on their own. Document Title and tf*idf are both baselines that were used in the original papers where these datasets came from. Sentence extraction is a new baseline that we propose specifically for the BBC dataset, in order see if we can exploit certain properties of the gold annotations, which are also derived from sentences.

Term Frequency
Term frequency has been shown to be a powerful feature in summarization (Nenkova and Vanderwende, 2005). Words that appear frequently are considered more meaningful than infrequent words. Term frequency is the number of times a term (excluding function words) appears in a document, divided by the total number of terms in that document. On the UNT dataset we use the stopword list included with the MALLET 6 toolkit, while the BBC dataset doesn't matter because the function words have already been removed.

tf*idf
While term frequency baseline requires the use of an ad hoc function word list, tf*idf adjusts the weights of different words depending on how important they are in the corpus. It is a standard baseline used for information retrieval tasks, based on the intuition that a word that appears in a smaller number of documents is more likely to be meaningful than a word that appears in many documents. tf*idf is the product of term frequency and inverse document frequency -idf (t i ) = log N n i where N is the number of documents in the corpus, and n i is the number of documents that contain the term t i . For the BBC Dataset, we base the idf weights on the document frequency of the training articles. For UNT, we use the reported tf*idf score which uses the British National Corpus to calculate the idf scores. 7

Corpus Frequency
Image annotations in both NLP and CV tend to be distributed with a relatively small number of frequently occuring keywords, and a long tail of keywords that only appear a few times. For UNT, we use the total keyword frequency of all the gold annotations, except for the one document that we are currently scoring. For BBC, we only measure the frequency of keywords in the training set captions, since we are specifically interested in the frequency of terms in captions.

Document Title
For BBC, the news article headline, and for UNT, the title of the webpage.

Sentence Extraction
Our baseline extracts the most central sentence from the co-occurring text and uses descriptive words from that sentence as the image annotation. Unlike sentence extraction techniques from Feng and Lapata (2010a), we determine which sentence to extract using the term frequency distribution directly. We extract the sentence with the minimum KLdivergence to the entire document. 8

System Comparison
In addition to the baselines, we compare against the Mix LDA system from Feng and Lapata (2010b). In Mix LDA, each instance is represented as a bag of textual features (unigrams) and visual features (SIFT features quantized to discrete "image words" using k-means). A Latent Dirichlet Allocation topic model is trained on articles, images, and captions from the training set. Keywords are generated for an unseen image and article pair by estimating the distribution of topics that generates the test instance, then multiplying them with the word distributions in each topic to find the probability of textual keywords for the image. Text LDA is is the same model but only using words and not image features. 8 One could also think of this as a version of the KLSum summarization system (Haghighi and Vanderwende, 2009) that stops after one sentence.

Evaluation
The evaluation metric and the source of gold annotations is described in Table 1. For the baselines 4.1, 4.2, 4.3 and the Mix LDA system, the generated annotation for each test image is its ten most likely keywords. We also run all baselines and the Mix LDA system on an unpruned version of the dataset, where infrequent terms are not removed from training data, test data, or the gold annotations. The purpose of this evaluation is to see if candidate keywords deemd "unlearnable" by the Mix LDA system can be learned by the baselines.

Results
The evaluation results for the BBC Dataset are shown in Table 2. Clearly, term frequency is a stronger baseline than tf*idf by a large margin. The reason for this is simple: since nearly all of BBC's function words are removed during preprocessing, the only words downweighted by the idf score are common -but meaningful -words such as police or government. This is worth pointing out because, in many cases, the choice of using a term frequency or tf*idf baseline is made based on what was used in previous work. As we show here and in Section 6.3, the choice of frequency baseline should be based on the data and processing techniques being used.
We use the corpus frequency baseline to illustrate the difference between standard and includeinfrequent evaluations. Since including infrequent words doesn't change which are most frequent in the dataset, precision for corpus frequency doesn't change. But since infrequent words are now included in the evaluation data, we see a 0.5% drop in recall (since corpus frequency won't capture infrequent words). Compared to the other baselines, this is not a large difference. Other baselines see a larger drop in recall because they have both more gold keywords to estimate and more candidate keywords to consider. tf*idf is the most affected by this, because idf overly favors very infrequent keywords, despite their low term frequency. In comparison, the term frequency baseline is not as negatively affected and even improves in precision because there are some cases where a word is very important to an article in the test set but just didn't appear very often in the training set (see Table 3 for examples). But the base-   Table 3: Examples of gold annotations from the test section of the BBC Dataset. The bolded words are the ones that appear five or more times in the training set; the unbolded words appear fewer than five times and would be removed from both the candidate and gold keywords in the standard BBC evaluation. lines with the best precision are the Doc Title and Sentence Extraction baselines, which do not need to generate ten keywords for every image.
While sentence extraction has a lower recall than term frequency, it is the only baseline or system that has improved recall when including infrequent words. This is unexpected because our baseline selects a sentence based on the term frequency of the document, and the recall for term frequency fell. One possible explanation is that extraction implicitly uses correlations between keywords. Probabilities of objects appearing together in an image are not independent; and the accuracy of annotations can be improved by generating annotation keywords as a set (Moran and Lavrenko, 2011). Recent works in image captioning also use these correlations: explicitly, using graphical models (Kulkarni et al., 2011;Yang et al., 2011); and implicitly, using language models (Feng and Lapata, 2010a). In comparison, sentence extraction is very implicit.
Unsurprisingly, the Text LDA and Mix LDA systems do worse on the include-infrequent evaluation than they do on the standard, because words that do not appear in the training set will not have high probability in the trained topic models. We were unable to reproduce the reported scores for Mix LDA from Feng and Lapata (2010b) where Mix LDA's scores were double the scores of Text LDA (see Footnote 4). We were also unable to reproduce reported scores for tf*idf and Doc Title (Feng and Lapata, 2008). However, we have three reasons why we believe our results are correct. First, BBC has more keywords, and fewer images, than typically seen in CV datasets. The BBC dataset is simply not suited for learning from visual data. Second, a single SIFT descriptor describes which way edges are oriented at a certain point in an image (Lowe, 1999). While certain types of edges may correlate to visual objects also described in the text, we do not expect SIFT features to be as informative as textual features for this task. Third, we refer to the best system scores reported by Leong et al. (2010), who evaluate their text mining system (see section 6.1) on the standard BBC dataset. 9 While their f1 score is slightly worse than our term frequency baseline, they do 4.86% better than tf*idf. But, using the baselines reported in Feng and Lapata (2008), their improvement over tf*idf is 12.06%. Next, we compare their system against frequency baselines using the 10 keyword generation task on the UNT dataset (the oot normal scores in table 5). Their best system performs 4.45% better than term frequency, and 0.55% worse than tf*idf. 10 Although it is difficult to compare different datasets and evaluation metrics, our baselines for BBC seem more reasonable than the reported baselines, given their relative performance to Leong et al's system.

System Comparison
We evaluate against the text mining system from (Leong et al., 2010). Their system generates image keywords by extracting text from the co-occurring text of an image. It uses three features for selecting keywords. Flickr Picturability queries the Flickr API with words from the text in order to find related image tags. Retrieved tags that appear as surface forms in the text are rewarded proportional to their frequency in the text. Wikipedia Salience assigns scores to words based on a graph-based measure of importance that considers each term's document frequency in Wikipedia. Pachinko Allocation Model is a topic model that captures correlations between topics (Li and McCallum, 2006). PAM infers subtopics and supertopics for the text, then retrieves top words from the top topics as annotations. There is also a combined model of these features using an SVM with 10-fold cross-validation.

Evaluation
Evaluation on UNT uses a framework originally developed for the SemEval lexical substitution task (McCarthy and Navigli, 2007). This framework accounts for disagreement between annotators by weighting each generated keyword by the number of human annotators who also selected that keyword. The scoring framework consists of four evaluation measures: best normal, best mode, oot (out-of-ten) normal, and oot mode. 11 The two best evaluations find the accuracy of a single "best" keyword generated by the system 12 .
10 And as we stated earlier, the relative performance of term frequency vs tf*idf is different from dataset to dataset. 11 Both the original framework and its adaptation by Leong et al. (2010) give precision and recall for each of the evaluation measures. However, precision and recall are identical for all baselines and systems, and only slightly different on the upper bound (human) scores. To preserve space, we only present the metric and scores for precision. 12 In contrast to the original SemEval task, where systems can Best normal measures the accuracy for each system annotation a j as the number of times a j appears in the R j , the multi-set union of human tags, and averages over all the test images.
In oot normal, up to ten unordered guesses can be made without penalty.
where A j is the set of ten system annotations for image i j .
The best mode and oot mode metrics are the same as the normal metrics except they only evaluate system annotations for images where R j contains a single most frequent tag. We use the scoring software provided by SemEval 13 with the gold annotation file provided in the UNT Dataset.

Results
The results of the lexical substitution evaluation on the UNT Dataset are shown in Table 5. The results from the normal show support for our earlier idea that the relative performance of term frequency vs tf*idf depends on the dataset. Although the term frequency baseline uses a stopword list, there are other words that appear frequently enough to suggest they are not meaningful to the document -such as copyright disclaimers.
That said, the number of cartoons in the dataset does seem to be strikingly high. Looking at the source of the images, we find that 45 of the 300 images were collected from a single online cartoon library. Predictably, we find that the co-occurring text to these images contains a long list of keywords, and little other text that is relevant to the image. We looked at a small sample of the rest of the dataset and found that many of the other text documents in UNT also have keyword lists.
Including this types of text in a general web corpus is not necessarily a problem, but it's difficult to measure the benefits of using complex techniques like topic modeling and graph similarily to find and extract annotations when in so many cases the annotations have already been found and extracted. This is shown in the normal evaluation results, where the combined system is only slightly better at selecting the single best keyword, and no better than tf*idf for the out-of-ten measure.

Conclusion
The intent of this paper is not to teach researchers how to inflate their own results, but to encourage better practices. With that purpose in mind, we make the following suggestions regarding future work in this area: Get to know your data. The ability to quickly and cheaply collect very large -but very noisy -collections of data from the internet is a great advance for both NLP and CV research. However, there still needs to be an appopriate match betwen the task being performed, the system being proposed, and the dataset being used; and large noisy datasets can hide unintended features or incorrect assumptions about the data.
Use relevant gold annotations. Do not convert other sources of data into annotations. When collecting human annotations, avoid postprocessing steps such as merging or deleting keywords that change the annotators' original intent. Keep an open dialogue with annotators about issues that they find confusing, since that is a sign of an ill-formed task.
Preprocessing should be simple and reproducable. The use of different preprocessing procedures affects the apparent performance of systems and sometimes has unintended consequences.
Use strong baselines and compare to other work only when appropriate. Systems developed for different tasks or datasets can make for misleading comparisons if they don't use all features available. Strong baselines explicitly exploit low-level features that are implicitly exploited by proposed systems, as well as low-level features of the dataset.
Don't remove keywords from gold annotations. Just because keywords are impossible for one system to learn, does not mean they are impossible for all systems to learn. Removing evaluation data artificially inflates system scores and limits comparison to related work. If a proposed system is to learn associations between visual and textual features, then it is necessary to use larger datasets. In general, global annotations, such as scenes, is easiest; identifying specific objects is more difficult; and identification of events, activities, and other abstract qualities has a very low success rate (Fluhr et al., 2006). Alternately, use simpler image features that are known to have a high sucess rate. For example, Deschacht and Moens (2007) used a face detector to determine the number of faces in an image, and then used NLP to determine the names of those people from associated text.