{'fraction_non_alphanumeric': 0.034272550062023746, 'fraction_numerical': 0.015346446925394294, 'mean_word_length': 4.711740890688259, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interactive Dual Generative Adversarial Networks for Image Captioning

Image captioning is usually built on either generationbased or retrieval-based approaches. Both ways have certain strengths but suffer from their own limitations. In this paper, we propose an Interactive Dual Generative Adversarial Network (IDGAN) for image captioning, which mutually combines the retrieval-based and generation-based methods to learn a better image captioning ensemble. IDGAN consists of two generators and two discriminators, where the generationand retrieval-based generators mutually benefit from each other’s complementary targets that are learned from two dual adversarial discriminators. Specifically, the generationand retrieval-based generators provide improved synthetic and retrieved candidate captions with informative feedback signals from the two respective discriminators that are trained to distinguish the generated captions from the true captions and assign top rankings to true captions respectively, thus featuring the merits of both retrieval-based and generation-based approaches. Extensive experiments on MSCOCO dataset demonstrate that the proposed IDGAN model significantly outperforms the compared methods for image captioning.


Introduction
Automatic image captioning aims to produce a textual description (usually a sentence) that verbalizes the visual content of an image. Image captioning methods can be roughly divided into two categories: retrieval-based and generationbased methods. Retrieval-based methods describe images by retrieving pre-existing captions from a repository, while generation-based methods synthesize a textual description (typically a sentence) that verbalizes the query image. Both ways have certain advantages but suffer from their own disadvantages.
When a user issues a query image, the retrieval-based methods search a corresponding caption that best matches the query image in a pre-constructed image-caption repository (Ordonez, Kulkarni, and Berg 2011;Hodosh, Young, and Hockenmaier 2013;Gong et al. 2014). For example, (Hodosh, Young, and Hockenmaier 2013) performed image captioning as a ranking or retrieval task, and introduced a ranking-based method to evaluate systems on a sentencebased image description. (Gong et al. 2014) associated images with descriptive sentences by projecting them into a common latent space. Although retrieval-based methods can produce general and syntactically correct captions, the retrieved captions are not tailored for the query images and limited by the capacity of the pre-constructed repository.
To make a caption tailored appropriately for the query image, a better way is to generate a new one accordingly. A typical generation-based image captioning model is encoder-decoder paradigm (Karpathy and Fei-Fei 2015;Vinyals et al. 2015), which consists of two neural networks: a convolutional neural network (CNN) based encoder encodes a given image into a vector representation, based on which a long short-term memory network (LSTM) decoder decodes the vector representation to generate a variablelength image caption word by word. For example, (Xu et al. 2015) proposed an attentive encoder-decoder neural network to dynamically attend to different locations of the images when decoding different words in the captions. (Mun, Cho, and Han 2017) used associated captions that were retrieved from training data to learn visual attention for image captioning. These methods can synthesize a new sentence as the caption, which brings the results of good flexibility and quality. Nevertheless, a well-known problem for generationbased methods is that they are prone to generate universal or non-fluent captions, which do not appropriately reflect the meaning of the given image.
Previously, the retrieval-based and generation-based systems with their own characteristics have been developed separately. In this paper, we are seeking to absorb their merits. We propose an Interactive Dual Generative Adversarial Network (IDGAN) for image captioning, which bridges the communication between generation-and retrieval-based methods by mutually reviewing each other. The dual adversarial training mechanism is established between two generation-and retrieval-based generators, and two generation-and retrieval-based discriminators, to make the generated captions indistinguishable from the groundtruth captions. Specifically, a language model (LM) generator G θ1 synthesizes tailored captions for the query im-age, and a generative ranker G θ2 ranks both retrieved and synthetic captions. A discriminator D φ1 not only differentiates generated captions from human-written captions but also distinguishes bad generated captions from good ones. Another discriminative ranker D φ2 attempts to distinguish the ground-truth captions and the adversarial candidates provided by both generators (G θ1 and G θ2 ), which are trained synchronously with the two generators using the adversarial training framework. This paper has three main contributions listed as follows.
• We introduce an interactive dual generative adversarial framework to mutually enhance both retrieval-based and generation-based image captioning methods, leading to a better ensemble model for image captioning.
• We devise a copy mechanism that naturally incorporates the retrieved guidance captions into the decoding process, enriching the informativeness and diversity of the generated captions.
• Experimental results show that IDGAN model significantly outperforms the state-of-the-art image captioning methods on the widely used MSCOCO dataset.

Related Work
Existing image captioning methods can be roughly divided into two categories: retrieval-based and generationbased methods. Retrieval-based methods first extracted the visually similar images with their captions from a preconstructed image-caption repository, forming a candidate captions pool. The final candidate captions for the input image are then chosen from the captions pool by ranking methods (Ordonez, Kulkarni, and Berg 2011;Hodosh, Young, and Hockenmaier 2013;Gong et al. 2014). For example, (Hodosh, Young, and Hockenmaier 2013) treated image captioning as a ranking or retrieval task, and introduced a ranking-based method to evaluate systems on a sentencebased image description. (Gong et al. 2014) associated images with descriptive sentences by projecting them into a common latent space. Although retrieval-based methods can produce general and syntactically correct captions, the retrieved captions are not tailored for the query images and limited by the capacity of the pre-constructed repository. Inspired by the great success of deep learning algorithms in computer vision and natural language processing, generation-based image captioning methods mainly exploit the encoder-decoder architecture to produce sentences with flexible syntactical structures (Karpathy and Fei-Fei 2015;Vinyals et al. 2015;Gu et al. 2017). For example, Karpathy and Fei-Fei (2015) learned about the inter-model correspondences between language and image data by using the training image-caption pairs. Attention mechanism has been proved to be able to significantly improve the performance of the underlying encoder-decoder based methods (Mun, Cho, and Han 2017;Gu et al. 2018;Yang et al. 2018;Zhao et al. 2018). Mun, Cho, and Han (2017) used associated captions that were retrieved from training data to learn visual attention for image captioning. Chen et al. (2017) encoded the images with multi-layer feature maps, capturing the spatial locations and channels via visual attention mechanisms.
There were also several recent generation-based studies exploring GAN and reinforcement learning techniques for image captioning Rennie et al. 2017). Rennie et al. (2017) introduced an SCST algorithm by using the REINFORCE algorithm to optimize the model. Rather than estimating a "baseline" to reduce the model variance, SCST used its output to normalize the expected rewards. Recently, some works proposed to employ GAN to generate text descriptions for the input images. Xu et al. (2019) proposed an adversarial learning method for image captioning, which enhances the caption generation with retrieved guidance captions.

Problem Definition and Model Overview
Given an image x, image captioning aims to generate a text description y = {w 1 , w 2 , ..., w T } for image x, where T is the length of the text sequence.
As depicted in Figure 1, IDGAN consists of the following components: (1) Generative sequence-to-sequence (seq2seq) model G θ1 , which is responsible for synthesizing M 1 image caption candidates {ŷ M1 m=1 } given an image x by the Monte Carlo (MC) roll-out policy. Such process is also noted as G θ1 (ŷ|x); (2) Generative ranking model G θ2 , which computes a relevance score between each image-caption pair and retrieves M 2 caption candidates { x,ỹ M2 m=1 }. Such process is denoted as G θ2 (ỹ|x); (3) Discriminative classification model D φ1 , which tries to distinguish the human-written captions from adversarial captions generated by G θ1 ; (4) Discriminative ranking model D φ2 , which inherits from the same ranking model as G θ2 , trying to distinguish the true image-caption pairs from adversarial candidates provided by both generators. By learning over symmetric feedback signals from two dual adversarial discriminators, the generation-and retrieval-based models mutually benefit from each other's complementary targets, leading to better image captioning. For each test image, we use the sentences generated by the generative model G θ1 as the final output caption.

Generative Seq2Seq module
The sequence to sequence (seq2seq) (Karpathy and Fei-Fei 2015;Vinyals et al. 2015) framework is used as the backbone of our generation-based image captioning model. In encoding, we retrieve the candidate captions to augment the semantic information of the image and thus learn better representation of the image. In the decoding stage, we integrate the guidance captions into the word generation process by designing a copy mechanism so as to enrich the meaning of the generated captions. Next, we will describe the encoder and decoder in detail.

Pre-retrieval Model for Retrieving Candidate Captions
The candidate captions are defined as the ground-truth captions of k nearest training images for the query image based on visual similarity. The image features are computed for (2). The neighbor images are found by exhaustively computing the cosine similarity between the query image and the training images. We maintain a set of top k captions, denoted as {c 1 , c 2 , . . . , c k }, in terms of the similarity score as candidate captions, which are concatenated into a guidance cap- LSTM network is then employed to extract the semantic meanings of the guidance caption. Formally, given the input word embedding e(w r i ), the hidden state g i is computed from the previous hidden state g i−1 as: where e(w r i ) denotes the embedding of the i-th word in retrieved guidance caption. The hidden states of the guidance caption C is represented as H g = [g 1 , . . . , g m ] and m denotes the length of guidance caption C. We useḡ = 1 m m i=1 g i to represent the average vector of the guidance caption.
Image Encoder Following the similar strategy as in (Karpathy and Fei-Fei 2015;Vinyals et al. 2015;Anderson et al. 2018), in this paper, we use the Faster R-CNN (Ren et al. 2015) that is pre-trained on Visual Genome (Krishna et al. 2017) to compact the raw image x into L vectors (with size D), where each vector represents the features learned at different detection of x. Formally, we refer to these annotated vectors as: An attention mechanism is employed to capture the crucial information from the input image. In particular, we take as input the average representation of the guidance caption C as attention source to learn knowledge-aware image representation z as: where σ is a feed-forward neural network that converts a vector to a real-valued score, α i is attention weight for the i-th image feature z init,i .

Caption Decoder
The generation of image captions is performed by an LSTM decoder based on the learned image representations and the guidance caption representation.
Two-layer Attention Networks Similar to (Anderson et al. 2018), caption decoder contains a two-layer LSTM network. The first LSTM layer (denoted as LSTM (1) ) is characterized as a top-down attention model, and the second LSTM layer (denoted as LSTM (2) ) is a language model. The first LSTM model takes the concatenation of the previous output of the language LSTM (i.e., h t−1 ), the image vector z init , the attentive image feature z, the average feature vector of the guidance captionḡ, and the word embedding of the previous word (i.e., e(w t−1 )) as input: At time step t, the hidden state of LSTM (1) can be calculated as: Given the output of the first LSTM at time step t (i.e., h (1) t ), we learn the attentive image representationz t , which is then fed into LSTM (2) . The attentive image vectorz t ensures that, at each time step, the decoder is capable of getting full information of the initial image representation z init . We computez t when we decode the t-th word as: where σ is a feed-forward neural network, as defined in Eq.
(3). The attention weight β t,i represents the alignment between the i-th location in the image and the t-th generated word. We employ LSTM (2) to generate an image caption word by word. The concatenation of the output of LSTM (1) and the attentive image representation (z t ) is used as the input of LSTM (2) , which is represented as: At time step t, the hidden state of LSTM (2) is calculated as: The generation model is typically a classifier over the vocabulary V c . In particular, we feed the representation h (2) t into a fully connected layer followed by a softmax layer to generate the image caption. Formally, the generation probability of the t-th word is computed by where W c is the parameter to be learned, θ 1 denotes the parameters of the generation-based model.

Copy Mechanism
We also employ copy mechanism to explicitly extract words form the retrieved guidance caption. We assume another set of words V g = {w g 1 , . . . , w g M } for all the unique words in the guidance captions. Since V g may contain words not in V c , copying words in V g enables the decoder to output some out-of-vocaulary (OOV) words.
Given the hidden states H g = [g 1 , . . . , g m ] for the guidance caption C, we computed the context vector for the guidance caption as a weighted sum of the hidden states H g : where h (2) t is defined in Eq. (8), is a multilayer perceptron. Formally, the generator selects a word w g from V g at time step t as follows: where W g indicates the learnable parameter. Finally, at each time step, the caption generator selects a generic word from V c or copies a word from V g with the following distribution: λ t ∈ [0, 1] is a scalar to balance the choice between generating a content word w c and copying a word w g from guidance captions. P θ1 (y t ) is the final word probability distribution. U o is learnable parameter.

Generative Ranking Model
For the retrieval-based approach, we devise a generative ranking model G θ2 to retrieve l competitive candidate captions. We first extract the visually similar images with their captions from the training data set by using the pre-retrieval module defined in Section 3.
where h p i is the i-th hidden state for p, e(w p i ) denotes the word embedding of the i-th word in the caption p, L p is the length of the caption p, μ is the averaging operation.
Instead of an absolute relevance, we optimize IDGAN by using a pair-wise ranking method since relative preference is usually more easily learned. The probability of a caption pair p 1 , p 2 with p 1 more relevant than p 2 being correctly ranked can be measured by the distance of their matching degree to the query image x: where ρ is the sigmoid function, f FC is a fully connected layer, g is any scoring function (i.e., cosine similarity), θ 2 denotes the parameters of the retrieval-based model. z init , o p1 and o p2 are representations of the image x, caption p 1 and caption p 2 . We define a triplet ranking-based loss, which maximizes the relevance between the query image x and the true caption y + , and minimizes the relevance between the query image x and sampled negative caption y − : where denotes the desired margin between the similarities, y − is the negative caption randomly chosen from the entire captions with true and candidate captions excluded.

Discriminative Classification Model
Ideally, a good image caption should be assigned a high adequacy score and contribute more to updating the generator G θ1 . Therefore, we expect the language model discriminator to not only differentiate generated captions from humanwritten captions but also distinguish bad generated captions from good ones. We propose a new objective of the language model discriminator to assign a precise score for each generated caption, which is consistent with their adequacy score: where D binary (x,ŷ) is a binary classifier implemented by LSTM that aims at distinguishing the input caption as originally generated by humans or synthesized by the generator G θ1 . D score (x,ŷ) is computed by an evaluation metric (i.e., CIDEr score) that compares the generated captionŷ to the corresponding ground-truth caption y + .

Discriminative Ranking Model
The discriminator adopts the same ranking model as G θ2 , which aims to distinguish the ground truth captions from the candidate captions produced by both generators (G θ1 and G θ2 ). Concretely, the discriminative model aims to learn the image and caption representations such that the probability that the positive pair x, y + is assigned larger similarity score than the negative pair x, y gen , where y + and y gen indicate the true caption and the caption generated by the two generators. Here, we have y gen ∈ {ỹ ∪ŷ}. Similar to Eq.(15), given the query image x, the probability of a caption pair x, y gen being correctly ranked can be computed using the distance of their relevance scores to the query image x: where ρ, g, and f FC are defined in Eq. (15). We adopt the same training loss as defined in Eq. (16) to optimize the discriminative ranking model D φ2 .

Adversarial Dual Objective
In IDGAN framework, the two generators G θ1 , G θ2 and the two discriminators D φ1 , D φ2 form two interactive dual generative adversarial networks, where the two generators attempt to produce fake captions that achieve high scores so as to fool the two discriminators respectively, while the two discriminators on the contrary are expected to score down the generated and retrieved captions. Their minimax game is summarized as the following objective function L: where E indicates the mathematical expectation, y + is the ground-truth caption for the query image, y − is the negative caption randomly chosen from the entire captions with true and candidate captions excluded.

Optimizing Discriminative Models
The objective of discriminative models D φ1 and D φ2 are to maximize the probability of correctly distinguishing the ground truth captions from the generated captions. For the two generators fixed, we can obtain the optimal parameters for the discriminative models D φ1 and D φ2 with the following formulation: where L g and L r are defined in Eq.(20)-(21). This optimization problem is typically solved with gradient descent since D φ1 and D φ2 are differentiable with respect to φ 1 and φ 2 , respectively.

Optimizing Generative Models
Generative Seq2seq Model Given a query image x, a caption sequenceŷ = [ŵ 0 ,ŵ 1 , ...,ŵ T ] is generated, which can be treated as a decision making process by policy P θ1 (w t |ŵ 1:t−1 , x). It is difficult to back-propagate the gradients from the two discriminators to the generator G θ1 , thus we use the policy gradient method to tackle this problem. With the true caption y + for image x, the reward of the generated image captionŷ is as follows: Generative Ranking Model We train G θ2 to generate competitive image captionỹ that achieves high ranking score from D φ2 . More precisely, when given an image x and a scoring function, the probability of G θ2 choosing a captioñ y from candidate captions pool P is computed by Eq. (15). Formally, we ameliorate G θ2 with the objective function as below: Policy Gradient We apply the policy gradient algorithm (Williams 1992) to update the parameters of the two generators since the sampling process of the generators is nondifferential. Formally, with D φ1 and D φ2 fixed, for each query image x with true caption y + , the minimization of L defined in Eq.(19) in response to θ 1 and θ 2 could be computed as follows: where J θ1 and J θ2 are defined in Eq.(23) and Eq.(24) respectively. T indicates the length of the training sample. Similar to (Rennie et al. 2017), the proximity gradient of the expected rewards of J θ1 can be calculated as follows: where y is a generated caption by the greedy decoding process used as the baseline to reduce the training variance in reinforcement learning. R 1 is the reward function during adversarial training of G θ1 , which is defined as: where γ 1 , γ 2 , γ 3 are parameters that controls the effect of the three kinds of rewards. D score (ŷ|x) is computed by an evaluation metric (i.e., CIDEr score) by comparing the generated captionŷ to the corresponding ground-truth caption.
The gradient of the objective function Eq. (25) with respect to parameters θ 2 is: where R 2 is the reward function during adversarial training of G θ2 , computed as: Methods BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE CIDEr Hard-Attention (Xu et al. 2015) 71.  However, the logarithm may lead to instability of training (Goodfellow et al. 2014). We thus follow  with the reward advantage function:

Experimental Setup
Dataset We adopt the widely used MSCOCO 2014 (denoted as MSCOCO) image captions dataset (Karpathy and Fei-Fei 2015) as the experimental data. In total, MSCOCO is composed of 82,783 training images, 40,504 validation images, and 40,775 testing images. Each image is corresponding to five reference descriptions. For the off-line testing, we use the Karpathy split setting (Karpathy and Fei-Fei 2015), which has been widely adopted in previous studies. There are 113,287 images for training, 5,000 images for validation, and 5,000 images for testing.
Baseline Methods In this study, we compare IDGAN with several state-of-the-art models, and some representative compared methods are Adaptive model (Lu et al. 2017 (14) and Eq. (17) are set to 512. During adversarial training, the two generators (G θ1 and G θ2 ) produce M 1 = M 2 = 5 candidate captions. The value of γ 1 , γ 2 , γ 3 equal to 0.2, 1, 0.8 respectively. We pre-train G θ1 for 30 epochs with maximum likelihood and D φ2 for 5 epochs with triplet loss. After that, we optimize the whole model with interactive adversarial training for 30 epochs.
Automatic Evaluation Metrics We adopt the official evaluation metrics of MSCOCO Image Captioning Challenge that are widely used in previous work (Karpathy and Fei-Fei 2015;Vinyals et al. 2015), including BLEU-N (N=1,2,3,4) (Papineni et al. 2002), METEOR (Banerjee and Lavie 2005), ROUGE (Lin 2004), CIDEr (Vedantam, Lawrence Zitnick, and Parikh 2015). These metrics estimate Ground truth "there is a woman sitting next to a statue on the bench" "an old bench is right on the oceans edge" "the boy is doing a trick on his skate board" "a man with a mask on holding up a phone" TopDown "a man and a woman sitting on a bench" "a bench sitting on top of a beach" "a man riding a skateboard on a street" "a man wearing a hat holding a cell phone" IDGAN "a woman sitting on a bench with a statue" "a wooden bench sitting on the beach next to the ocean" "a man is doing a trick on a skateboard" "a person in a mask holding a cell phone" the consistency between the n-gram existence in the produced text descriptions and the ground truth captions.
6 Experimental Results

Quantitative Evaluation
We firstly report the model comparison from the quantitative perspective. The experimental results on MSCOCO are summarized in Table 1. The results are calculated using the COCO captioning evaluation tool (Lin et al. 2014). Our model achieves statistically significantly better performance than the state-of-the-art competitors on MSCOCO. Specifically, the proposed IDGAN successfully obtains higher scores over all automatic evaluation measures compared to the TopDown model which adopts the same CNN-LSTM framework as IDGAN.
To investigate the impact of different components of the proposed IDGAN for image captioning, we also conduct the ablation study of IDGAN on MSCOCO by removing the retrieval-based generator G θ2 (w/o G θ2 ), the retrieval-based discriminator D φ2 (w/o D φ2 ), the generation-based discriminator D φ1 (w/o D φ1 ), and copy mechanism (w/o copy), respectively. It is no surprise that combining all the factors achieves the best performance for all evaluation metrics. The retrieval-based generator and discriminator (G θ2 and D φ2 ) contribute a great improvement to our model by providing ranking scores to guide the generation-based generator producing better captions.

Human Evaluation
Similar to previous works (Xu et al. 2015;Rennie et al. 2017), we also use human annotation to evaluate the image captioning systems quantitatively. We randomly sample 100 images from the MSCOCO test set and invite three human annotators to score the generated captions based on their Informativeness (whether the caption is appropriate and natural to an image) and Fluency (whether the generated caption is fluent with a proper grammatical structure). The annotators are asked to assign each caption a score of 1 (bad), 2 (poor), 3 (not bad), 4 (satisfactory), 5 (good) for Informativeness and Fluency, respectively. The human evaluation results are summarized in Table 2. According to Table 2, IDGAN substantially outperforms the compared approaches by a noticeable margin on the MSCOCO dataset.

Case Study
To measure the performance of IDGAN from the qualitative perspective, we report several produced image captions in Table 3. We can easily observe from Table 3 that IDGAN is able to generate reasonable and relevant text descriptions of the given images. For example, the sentence "a woman sitting on a bench with a statue" generated by IDGAN precisely describes the content of the image. In contrast, the Top-Down method often fails in such cases.

Error Analysis
To examine the limitations of the proposed model, we additionally carry out an analysis of the errors made by IDGAN. Specifically, we randomly choose 100 images from MSCOCO test set whose captions generated by our model have low human evaluation scores. We reveal several reasons for the low evaluation scores, which can be divided into two primary categories. First, IDGAN fails to generate semantically diverse image captions across visually similar images. For example, IDGAN tends to generate the same caption for two different images that are semantically related or have similar objects, ignoring some details of the two images. One possible solution is to devise a new metric to measure the semantic diversity of image captions, and then the diversity score can be used as a reward in reinforcement learning so as to encourage the model to consider both diversity and accuracy. Second, IDGAN fails to detect some objects in the images that have no high-quality retrieved captions. It suggests that certain object detection strategy needs to be devised in the future so as to generate better captions for specific images.

Conclusion
In this paper, we proposed IDGAN to enhance a generationretrieval ensemble model with dual adversarial learning, allowing for both generation-based and retrieval-based image captioning methods to be mutually enhanced. We integrated retrieved guidance captions into word decoding process by a copy mechanism, which enriched the meaning of the generated captions. Extensive experiments revealed that the proposed IDGAN model significantly outperformed the compared methods by a remarkable margin.