{'fraction_non_alphanumeric': 0.04085603112840467, 'fraction_numerical': 0.011673151750972763, 'mean_word_length': 4.720684039087948, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Annotation-Free Evaluation of Cross-Lingual Image Captioning

Cross-lingual image captioning, with its ability to caption an unlabeled image in a target language other than English, is an emerging topic in the multimedia field. In order to save the precious human resource from re-writing reference sentences per target language, in this paper we make a brave attempt towards annotation-free evaluation of cross-lingual image captioning. Depending on whether we assume the availability of English references, two scenarios are investigated. For the first scenario with the references available, we propose two metrics, i.e., WMDRel and CLinRel. WMDRel measures the semantic relevance between a model-generated caption and machine translation of an English reference using their Word Mover's Distance. By projecting both captions into a deep visual feature space, CLinRel is a visual-oriented cross-lingual relevance measure. As for the second scenario, which has zero reference and is thus more challenging, we propose CMedRel to compute a cross-media relevance between the generated caption and the image content, in the same visual feature space as used by CLinRel. The promising results show high potential of the new metrics for evaluation with no need of references in the target language.

training and test data are described by the same language (English) and (b) cross-lingual image captioning, where training data is described by a source language (English) while the test data is to be annotated by sentencesˆin a distinct target language (Chinese). This paper makes a novel attempt to evaluate the effectiveness of a cross-lingual image captioning model with no need of any reference sentence in the target language. The symbol (+) means the computation of a proposed metric (WMDRel or CLinRel) requires reference in the source language, while (-) means reference-free.

Abstract
Cross-lingual image captioning, with its ability to caption an unlabeled image in a target language other than English, is an emerging topic in the multimedia field. In order to save the precious human resource from re-writing reference sentences per target language, in this paper we make a brave attempt towards annotation-free evaluation of cross-lingual image captioning. Depending on whether we assume the availability of English references, two scenarios are investigated. For the first scenario with the references available, we propose two metrics, i.e., WMDRel and CLinRel. WMDRel measures the semantic relevance between a model-generated caption and machine translation of an English reference using their Word Mover's Distance. By projecting both captions into a deep visual feature space, CLinRel is a visual-oriented cross-lingual relevance measure. As for the second scenario, which has zero reference and is thus more challenging, we

Introduction
Image captioning, which aims to automatically describe the pictorial content of an unlabeled image with a sentence, is being actively studied [2,9,24]. As its subtopic, crosslingual image captioning, with the ability to caption a given image in a target language other than English, is attracting an increasing amount of attention in both multimedia and computer vision fields [6,7,14,15,22,26].
Previous works on topic emphasize novel algorithms that effectively learn image captioning models for the target language from existing English datasets such as Flickr8k [8], Flickr30k [27] and MS-COCO [3]. In [15], for instance, Li et al. use machine translation to automatically translate English captions of Flickr8k into Chinese and subsequently train a Show-Tell model [24] on the translated dataset. Observing the phenomenon that machine-translated sentences can be unreadable, Lan et al. [14] introduce fluency-guided learning, wherein the importance of a training sentence is weighed by its fluency score estimated by a deep language model. Song et al. [22] improve [14] by introducing self-supervised reward with respect to both fluency and visual relevance. Although such a training process requires only a small (or even zero) amount of data in the target language, a large-scale evaluation of the resultant models typically needs thousands of test images associated with manually written captions, known as references, in the same language. Even assisted by an interactive annotation system [10], months of human labor are required to re-annotate a medium-sized testset per target language.
In this paper we contribute to cross-lingual image captioning with a novel approach to its evaluation. More specifically, we make a brave attempt to remove the need of references in the target languages. We propose three metrics that allow us to differentiate between good-performing and badperforming models, when a test image is provided with just one reference in English. Such a prerequisite is valid, as the previous works on cross-lingual image captioning are conducted mostly on established English datasets. Our major conclusions are two-fold: • To the best of our knowledge, this is the first work on evaluating image captioning models in a cross-lingual setting, with no need of any reference in the target language. To that end, we propose three metrics, i.e., WM-DRel, CLinRel and CMedRel, that assess the semantic relevance of auto-generated captions with respect to the image content in varied manners. • We have conducted a number of experiments to evaluate the effectiveness of the three proposed metrics.
Given the varied combinations of image captioning networks, i.e., Show-Tell [24], Up-Down [2] and AoANet [9] and datasets, i.e., COCO-CN [16] and VATEX [25], we build a set of eight Chinese models to be ranked. The combination of WMDRel, CLinRel and CMedRel has Spearman's rank correlation of 0.952 with the sum of the four standard metrics, i.e., BLEU-4, METEOR, ROUGE-L and CIDEr. When no reference in the source language is given, CMedRel alone has Spearman correlation of 0.881 with CIDEr.

Related work
We shall clarify that this paper is not about building a better cross-lingual image captioning model. Rather, we are interested in novel metrics that can be computed without the need of reference sentences in a target language. According to the evaluation protocol used in [14] and its follow-ups, human resources regarding the evaluation of cross-lingual image captioning are spent on two parts. The first part is to manually write references in the target language so that stanard metrics such as BLEU-4 [13], ME-TEOR [4], ROUGE-L [18] and CIDEr [23] can be computed by performing word-level or phrase-level comparison between the auto-generated captions and the references. The second part is to manually assess subjective attributes of sentences such as their readability and fluency. Our proposed approach is to remove the first part so that the relatively limited human resources can be fully spent on the second part. The starting point of our work differs fundamentally from previous efforts on devising better automated metrics [1,12], as they still assume the availability of references in the target language.

Problem Formalization
A cross-lingual image captioning model in its training stage shall learn from training data described in a source language. While in the inference stage, the model generates for a novel image a descriptive sentence in a target language, denoted asˆ:ˆ← ( ).
When come to the evaluation stage, the current setting of cross-lingual image captioning [14,16,21] assumes the availability of at least one ground-truth sentence in the target language, denoted as , w.r.t the image. Similarly, we use to denote a ground-truth sentence in the source language. Accordingly, the quality ofˆis measured based on its wordor phrase-level matching with . Such a matching is typically implemented as (ˆ, ), with ∈ {BLEU-4, METEOR, ROUGE-L, CIDEr}. Given two distinct models ,1 and ,2 , ( ,1 ( ), ) > ( ,2 ( ), ) means the former is better and vice versa. Our goal is to remove the need of .
Depending on whether is available, we consider the following two scenarios: • Scenario-I: Evaluating on an established dataset with available. This scenario applies to the majority of the works on cross-lingual image captioning, as they evaluate on (a subset) of MS-COCO.
• Scenario-II: Evaluating on a novel and fully unlabeled dataset. This scenario is more practical yet much more challenging.
For Scenario-I, a cross-lingual version of , indicated by (ˆ, ) is required to measure to what extent ( ) matches with . As for Scenario-II, a cross-media version of , denoted as (ˆ, ), is needed to measure how ( ) matches with the visual content. Note that when comparing distinct models, their rank matters. Hence, the purpose of and is to approximate the model rank determined by . To that end, we develop three metrics, i.e., WDM Relevance (WDMRel) and Cross-Lingual Relevance (CLin-Rel) to realize , and Cross-Media Relevance (CMedRel) for . The three metrics are illustrated in Fig. 2 and depicted as follows.

Three Proposed Metrics
3.2.1 WMDRel: Word Mover's Distance based Relevance. We re-purpose the Word Mover's Distance (WMD), originally proposed by Kilickaya et al. for measuring document similarity [12], in the new context of cross-lingual image captioning evaluation. In order to deal with synonyms and semantically close words that cannot be modeled by bagof-words based matching, WMD formulates the matching problem between two documents as the classical Earth Mover process, with the goal of moving each word in a document to words in another document. The moving cost between , the reference in a source language (English here), and CMedRel to measure the semantic relevance betweenˆand the visual content. Different from previous works, no reference caption in the target language (Chinese here) is needed.
two words is defined as the Euclidean distance between their word2vec features. Accordingly, WMD between two sentences is defined as the minimum cumulative cost of moving all words in one sentence to successfully match with the other sentence.
Note that WMD is monolingual. Therefore, we have automatically translated to the target language (which is Chinese in this study) by machine translation. We use ( ) to indicate the translated reference, and (ˆ, ( )) as the WMD betweenˆand ( ). Accordingly, we compute WMDRel as the normalized inverse of (ˆ, ( )): where is a normalization factor to ensure a score between 0 to 1. A Chinese word2vec model 1 , pre-trained on 120G text corpus with 6.1 million tokens, is used.

CLinRel:
Cross-Lingual Relevance in Visual Feature Space. It is worth noting that errors in machine translation remain inevitable. As a consequence, ( ) does not fully reflect the semantic meaning of . We therefore look for alternatives that can measure the semantic relevance be-tweenˆand with no need of machine translation. Since a visual feature space is naturally cross-lingual, we consider project bothˆand into such a feature space and consequently compute their relevance in the common space.
In the context of image/video caption retrieval, Dong et al. propose to project a given sentence into a visual feature space by a deep learning model called Word2VisualVec (W2VV) [5]. In particular, the given sentence is first vectorized by three sentence encoders in parallel, i.e., bag-of-words, word2vec and GRU. The output of the encoders is concatenated into a long vector, which is then embedded into the visual feature space by an MLP network. In this work, we adopt W2VV++ [17], a super version of W2VV. We train an English version of W2VV++ and a Chinese version, which are used to project andˆinto the visual feature space, respectively. Given ( ) and (ˆ) as their corresponding vectors, we define CLinRel as their cosine similarity, i.e., We instantiate the visual feature space by extracting 2,048dimensional CNN features using a pre-trained ResNeXt-101 [20], unless stated otherwise.

CMedRel: Cross-Media Relevance. To deal with
Scenario-II where is unavailable, we now introduce CMe-dRel, which assessesˆwith respect to the visual content. We compute such cross-modal relevance as the cosine similarity between (ˆ) and ( ): 4 Evaluation

Experimental Setup
We verify the effectiveness of the proposed metrics by evaluating their consistency with the standard metrics, i.e., BLEU-4, METEOR, ROUGE-L, CIDEr and their combination, which are computed based on references in the target language. Given a set of cross-lingual image captioning models, the consistency between two metrics and is measured in terms of the Spearman's rank correlation coefficient between model ranks given by and . Spearman correlation of +1 means the two metrics are fully consistent.
In what follows, we describe how to build a set of models followed by implementation details.

Model Pool
Construction. An image captioning model is determined by two major factors, i.e., network architecture and training data. By trying varied combinations of the two factors, we construct a pool of eight distinct models as follows.
Choices of Training Data. We use the following bilingual (English-Chinese) datasets, wherein Chinese captions are obtained either by machine translation of the original English captions or by manual annotation: • COCO-CN [16]: A public dataset extending MS-COCO with manually written Chinese sentences. It contains 20,342 images annotated with 27,218 Chinese sentences. We use its development set COCO-CN-dev as training data.
• COCO-MT: Also provided by [16], using the Baidu translation API to automatically translate the original English sentences of MS-COCO to Chinese. COCO-MT contains 123,286 images and 608,873 machinetranslated Chinese sentences. • VATEX [25]. A subset of the kinetics-600 [11] shortvideo collection, showing 600 kinds of human activities. Each video is associated with 10 English sentences and 10 Chinese sentences obtained by crowd sourcing.
Following the notation of [16], we term the dataset with only Chinese annotations as VATEX-CN. We also construct a machine-translated counterpart, which we term VATEX-MT. We use each of the four datasets, i.e., COCO-CN-dev, COCO-MT, VATEX-CN and VATEX-MT, as training data. Basic statistics of the datasets and their usage in our experiments are summarized in Table 1. Choice of Network Architecture. We investigate three representative architectures, namely Show and Tell (Show-Tell) [24], Bottom-up and Top-Down (Up-Down) [2] and Attention on Attention Network (AoANet) [9]: • Show-Tell: Proposed by Vinyals et al. [24], this model generates a caption for a given image in an encodingdecoding manner. The given image is encoded as a feature vector by a pre-trained image CNN model. The feature vector is then used as an input of an LSTM network which iteratively generates a sequence of words as the generated caption. • Up-Down: Proposed by Anderson et al. [2], this model improves Show-Tell by introducing a combined bottomup and top-down visual attention mechanism. In contrast to the global feature used in Show-Tell, Up-Down encodes the given image by a varied number of feature vectors, extracted from objects detected by Faster R-CNN. Such a design not only describes dominant patterns in the image but also capture small-sized objects. In the decoding stage, a weighted average of these features is fed into an LSTM network, with the weights calculated by a self-attention module to adaptively reflect the importance of the individual features for caption generation. In this work, we use visual features provided by Luo et al. [19]. • AoANet: Proposed by Huang et al. [9], this model improves the previous Up-Down model by introducing an Attention on Attention (AoA) module. AoA extends the conventional attention mechanism by adding a second attention layer, allowing the module to take into account the relevance between the query vector (which is the input of the attention module) and the attention result. AoANet is built by applying AoA to Up-Down's encoder and the decoder.
Given the four datasets and the three networks, we shall have 12 models in total. However, as classes and positions of the detected objects vary over frames, Up-Down and AoANet are not directly applicable to video data. Hence, only Show-Tell is trained on all the four datasets. This results in 8 distinct models, see Table 2. Each model is named after the underlying network and training data. E.g., AoANet (COCO-MT) means training AoANet on COCO-MT.

Details of Implementation.
All the image captioning models are trained in a standard supervised manner, with the cross-entropy loss minimized by the Adam optimizer. The initial learning rate of Show-Tell and Up-Down is set to be 0.0005. All hyper-parameters of AoANet follow the original paper [9]. The maximum number of training epochs is 80. Best models are selected based on their CIDEr scores on the validation set of the corresponding dataset.
All models are exclusively tested on the test set of COCO-CN, which has 1,000 images. Each test image is associated with five English sentences originally provided by MS-COCO and on average six Chinese sentences. We use the first English sentence as .
The English version of W2VV++ is trained on paired image and English captions from MS-COCO, with 121k images and 606k captions in total. Note that the images have no overlap with the test set. As for the Chinese version of W2VV++, we pretrain the model using COCO-MT and fine-tune it on COCO-CN-dev.

Experiment 1. Evaluation of the Proposed
Metrics in Scenario-I We summarize the performance of the eight models measured by the varied metrics in Table 2, where BMRC is the sum of BLEU-4, METEOR, ROUGE-L and CIDEr, while WCC is the sum of WMDRel, CLinRel and CMedRel. According to both CIDEr and BMRC, AoANet (COCO-MT) has the top performance, while models using the bottom-up and top-down visual features outperform their Up-Down counterparts. This results is reasonable, in line with the literature that attention mechanisms are helpful. We observe Table 2 that such a model preference is also identified by WCC.
Comparing the individual models, Up-Down (COCO-CN) obtains a higher BMRC than AoANet (COCO-CN), although [9] reports that AoANet is better than Up-Down for English image captioning on MS-COCO. Meanwhile, we notice that AoANet (COCO-MT) has a higher BMRC than Up-Down (COCO-MT). Recall that the amount of training sentences in COCO-MT is around 30 times as large as that of COCO-CN. Hence, the advantage of AoANet is subject to the amount of training data.
Also notice that models trained on COCO-CN obtain higher BLEU-4 than their counterparts trained on COCO-MT. We attribute this result to the reason that the COCO-CN models generate longer sentences, while BLEU-4 adds a brevitypenalty to discourage short sentences. As CIDEr does not take the length of a sentence into account, this explains why some image captioning models have higher CIDEr yet lower BLEU-4.
The effectiveness of the proposed metrics is justified by the Spearman correlation reported in Table 3. Among them, WM-DRel is most correlated with BLEU-4, CLinRel with CIDEr, and CMedRel with CIDEr. We also evaluate varied combinations of the proposed metrics. Among them, WCC has the largest Spearman correlation of 1.0 with CIDEr and 0.952 with BMRC. Thus, WMDRel, CLinRel and CMedRel shall be used together for Scenario-I.

Experiment 2. Evaluation of the Proposed
Metrics in Scenario-II As aforementioned, only CMedRel is applicable in Scenario-II, which is much more difficult by definition. As shown in Table 3, the Spearman correlation coefficients of CMedRel with BLEU-4, METEOR, ROUGH-L, CIDEr and BMRC are 0.714, 0.838, 0.714, 0.881, and 0.786, respectively. All the coefficients are greater than 0.7. This result indicates that CMedRel has good correlations with the standard metrics. Hence, the metric can be used with caution when no reference sentence is available.
For a more intuitive understanding of the results, some generated captions and the corresponding metrics computed upon these captions are presented in Table 4.

Conclusions and Remarks
This paper presents our effort towards annotation-free evaluation of cross-lingual image captioning. Experiments on two cross-lingual datasets (COCO-CN and VATEX) and three representative image captioning networks (Show-Tell, Up-Down and AoANet) allow us to draw conclusions as follows. When each test image is associated with one reference sentence in the source language, the combination of the three proposed metrics (WMDRel, CLinRel and CMedRel) has perfect Spearman correlation of 1 with CIDEr and 0.952 with BMRC. When such cross-lingual references are unavailable, CMedRel still has Spearman correlation of 0.881 with CIDEr and 0.786 with BMRC. These results suggest that the current need of references in the target language can be largely reduced. This will enable a more effective utlization of expensive and thus limited human resources on assessing subjective properties, e.g., readability and fluency, of the auto-generated captions.