{'fraction_non_alphanumeric': 0.03663442346077077, 'fraction_numerical': 0.010164286810993397, 'mean_word_length': 4.351848808545604, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
3G structure for image caption generation

It is a big challenge of computer vision to make machine automatically describe the content of an image with a natural language sentence. Previous works have made great progress on this task, but they only use the global or local image feature, which may lose some important subtle or global information of an image. In this paper, we propose a model with 3-gated model which fuses the global and local image features together for the task of image caption generation. The model mainly has three gated structures. 1) Gate for the global image feature, which can adaptively decide when and how much the global image feature should be imported into the sentence generator. 2) The gated recurrent neural network (RNN) is used as the sentence generator. 3) The gated feedback method for stacking RNN is employed to increase the capability of nonlinearity fitting. More specially, the global and local image features are combined together in this paper, which makes full use of the image information. The global image feature is controlled by the first gate and the local image feature is selected by the attention mechanism. With the latter two gates, the relationship between image and text can be well explored, which improves the performance of the language part as well as the multi-modal embedding part. Experimental results show that our proposed method outperforms the state-of-the-art for image caption generation.


Introduction
Image caption generation aims to automatically generate a natural language sentence to describe the content of a given image. It is a vital task of scene understanding which is one of the fundamental goals of computer vision and artificial intelligence [1,2]. However, image caption generation is a challenging task. It not only needs to recognize objects in an image, but also needs to capture and express their relationships and attributes with natural language [3].
To address the aforementioned challenge, many methods have been developed and a lot of gratifying results have been achieved in recent years. These methods are roughly divided into two categories: 1) retrieval-based methods [4,5,6] and 2) multi-modal neural network (MMNN) based methods [7,8,9,10].
Although these methods, especially the MMNN-based methods, have attained promising results, further improvements should be got over some limitations.

Motivation and Overview
To generate length-variable and form-variable sentences, we follow the MMNNbased methods. However, in most existing methods, only global image feature or local image feature is used to generate sentences, which may lose some important image information. It is easy to understand that global image feature can catch the overall information of an image and local image information can dig out the fine-grained relationships between image regions and language elements. Therefore, we use a global-local image feature fusing strategy, which can fully mine the image information.
Moreover, the sentence generator in image captioning model should learn both hierarchical and temporal representation very well. While, most of the previous methods use the single-layer "vanilla" RNN [11,12,8] or single-layer LSTM, which cannot learn the hierarchical representation well [13,14]. Meanwhile, the issue of learning multiple adaptive timescales (i.e. the quickly and slowly changing components) [13,14] should also be considered in caption generation, because the sentence is a sequence signal which consists of both fastmoving and slow-moving components [13]. In the natural language processing field, gated feedback RNN (GF-RNN) has attracted the attention of many researchers, because GF-RNN can not only solve the long-term dependency problem in "vanilla" RNN, but also learn multiple adaptive timescales.
Motivated by the aforementioned reasons, we propose a novel 3G model in this paper. More specially, 1) both the global and local image features are input into the multi-modal embedding part for making full use of image information; 2) the gated RNN and gated feedback connecting strategy are used to improve the performance of the language model. This model mainly contains 3 gated modules: gate for global image feature, gated RNN and gated feedback connecting for stacked RNN. So we name it 3G for short. When we fuse the global and local image feature, the first gate is used to control when and how much the global image information should be input into the multi-modal embedding part. The gate is motivated by the gate structure of long-short term memory (LSTM) unit. Recently, RNN is used as the most popular language model. However, "vanilla" RNN is hard to capture the long term dependencies.
So we choose LSTM, one kind of gated RNN, as the language model. It is the second gate. Moreover, in order to explore the nonlinear relationship between the image and text, the most effective strategy for stacked RNN-gated feedback RNN (GF-RNN) [13] is used as the multi-modal part as well as the sentence generator. It is the third gate. With the 3-gated structure, the image information is fully utilized and the performance of the language model is strengthened.

Contributions
Our main contributions are listed as follows: 1. An end-to-end 3G model is proposed to accomplish the image caption generation task. The proposed model contains 3-gated structure, and it can be fully trained with the stochastic gradient descent (SGD) method.
2. The global and local image features are used in this paper, which makes full use of image information to improve the caption quality.
3. The gated feedback connecting strategy is used for stacking the LSTM, which solves the issues of long-term dependency and learning multiple adaptive timescales. In other words, the gated RNN and gated feedback connecting strategy make the multi-modal embedding and language parts much stronger than the previous methods.
A shorter version of this paper appears in [15]. The main extensions in the current work are: 1. In terms of innovation, we have added a new technological novelty in the caption generating part: gated feedback LSTM is first used to generate the description for image and the gated feedback connecting strategy can well explore the nonlinear relationship between text and image.
2. Since the language part in this paper is GF-LSTM, the formulas of the GF-LSTM is introduced in detail in Section 3 and the language generating process modeled by the GF-LSTM is detail stated in Section 4.
3. More experiments and more experimental details has been added in Section 5.

Organization
The rest of this paper is organized as follows. In Section 2, some previous works are briefly introduced. Then the gated feedback LSTM (GF-LSTM) model is introduced in Section 3. Section 4 presents our model for image caption generation. To validate the proposed method, the experimental results are shown in Section 5. At last, Section 6 makes a brief conclusion for this paper.

Related Work
The problem of describing images with natural language sentences has recently attracted increasing interests and many methods have been proposed.
Training image-sentence pairs "A zebra grazing in a green pasture with cows."

 
A double decker bus driving down a street Figure 1: The most popular paradigm for image caption generation. In this diagram, CNN and RNN are used as visual feature extractor and sentence generator, respectively. In the training step (the subfigure on the left of the dotted line), the image feature and the corresponding sentence representation are imported into the multi-modal RNN to learn the mapping relation between image and text. Then in the testing step (the subfigure on the right of the dotted line), only the image feature is imported into RNN, and then the image caption is generated by RNN.
Retrieval-based methods retrieve similar captioned images and generate new descriptions by retrieving a similar sentence from the training dataset [4]. However, the style of the describing sentences generated by these kind methods is lack of variety.
With the wide use of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in computer vision and natural language processing [16,17,18,19,20,21,22,23,24,25], MMNN-based Methods have become the most popular mechanism for image caption generation [11,26,3,27]. As illustrated in Fig. 1, the most common model contains three important parts: a vision part, a language module part and a multi-modal part.  [12,8] also proposed a multi-modal RNN model, and unlike [11], they use the global image feature generated by CNN only at the first time-step. Therefore, one problem is coming: when and how much the global image information should be input into the RNN. These models do not reach a consensus and cannot solve this problem. To address this problem, gating mechanism is used to control the global image feature when and how much be input into the RNN in this paper. The aforementioned models only use the global image feature output by CNN as the whole image information. However, they cannot provide fine-grained modeling of the inter-dependencies between different visual elements and the relationship between the image and text [9]. That is the second problem To address the second problem, attention-based methods has been proposed [9,29,30,31]. Xu et al. [29] exploited attention mechanism for image caption generation. It used the feature map output from the convolutional layer of the CNN as image information. By flattening the feature map into a fixed number vectors, every image was broken into tiles with fixed size. Each vector denotes one tile feature. When predicting the next word, the previous generated word will select the tiles. [30] generates natural language sentence with attention transitioning on the lexical representation. Moreover, attention mechanism is widely used in other computer vision and natural language processing tasks [32,33,34,35].  In this paper, we retain the attention part as the local image feature. To improve the performance of the language part, gated feedback connecting strategy is used for stacking the LSTM.

Gated Feedback LSTM
In this paper, 2-layer GF-LSTM unit is used as sentence generator, so we introduce the GF-LSTM at first.
In previous works, sentence generation models based on RNNs have shown powerful capabilities to generate target sentences. In order to further enhance representing capabilities of RNN, a conventional way is stacking multiple recurrent layers [13]. However, as shown in  [13] has confirmed that GF-RNN shows the best performance on natural language processing among the standard stacking RNN with the same layers and single layer gated RNN (e.g. LSTM or GRU).
We take Fig. 2.(b) as an example and choose LSTM as RNN model, according to [13], the global gates which control previous hidden states to current hidden states are defined as follows: where superscript i = 1, 2 denotes the level of layers, superscript i → j (j = 1, 2) denotes state transiting from layer i to layer j. w i→j g h ∈ R 1×h stands for weight vector for the current input and u i→j g h ∈ R 1×2h (j = 1, 2) represents weight vector for the previous hidden states. h denotes the previous states. Through Eq. (1) we know that gate g i→j h is a single scalar and its value depends on the current lower hidden state and the previous hidden states.
After computing the gates, we describe how to use these gates in LSTM.
When computing the current input gate, output gate, forget gate, memory and the current hidden state, g i→j h is not used, so these formulas are the same as raw LSTM formulas. We rewrite the three gate formula of LSTM unit as follows: where superscript i = 1, 2, W (i) * ∈ R h×h and U (i) * ∈ R h×h denote weights matrixes and b (i) * ∈ R h stands for the biases. i t−1 ∈ R h represent the current and the i-th layer's input gate, forget gate, output gate, the i − 1-th hidden state and the previous i-th hidden state, respectively. When the superscript i = 1, h denotes the sigmoid activation function.
The current memory and the hidden state are computed as follows: Global Feature where c t ∈ R h denotes the updating memory content. " " denotes element multiplication.
The computing formula of updating memory content is different from standard stacked LSTM. The updating memory content of the gated feedback LSTM at the j-th layer is computed as follows: (1). Through this equation, we can know that g i→j h controls how much the previous hidden state h (i) t−1 is used to compute the current updating memory content.

Proposed Approach
Overview. Fig. 3 shows the architecture of our 3G model. To make full use of the image information, the global and local image features are input into the multi-modal embedding module. The first gate controls when and how much the global feature is input into the multi-modal embedding module. While the local feature is selected by the attention mechanism. Another two gates are used in the language and the multi-modal embedding module parts. The gated structure is used to improve the performance of the language model. In this section, we interpret our 3G model in detail.
In the image caption generation task, when given an image, the most wanted sentence should be generated with a maximal probability. In many previous works, probability models are widely used. Similarly, our approach also uses probability generation models for this task. In other words, the LSTM module outputs a probability at each time-step. And we maximize the joint probability conditioned on the given image. So the objective function is written as follows: and we rewrite it as log likelihood function: We compute P (S i |I i ) with chain rule: where θ in these three equations denotes all of the parameters needed to train.
(I i , S i ) denotes the i-th image-sentence pairs and N i is the sentence length. In our model, GF-LSTM is used to model the condition probability (see Section 4.3).

Image Representation
Almost all of the state-of-the-art methods used deep CNN to encode the image, because deep CNN can learn discriminative and representative features from the data such as the given images. Similar to the previous methods, we use the VGG-19 as the image features extractor. More specially, outputs of FC7 and CONV5-4 layers are used as global and local features of the images, respectively.

Image Global Feature Representation
The VGG-19 is pre-trained on ImageNet and used as the image encoder in our model. The global representation of image I is as follows: where I denotes the image I, F c(I) ∈ R l is the output of the FC7 layer. The matrix W I ∈ R h×l is an embedding matrix which projects l-dimension image feature vectors into the embedding space with h-dimension and b I ∈ R h denotes the bias. v ∈ R h is the so-called image global feature representation.

Image Local Feature Representation
When a raw image I ∈ R W ×H×3 is input to VGG-19, the CONV5-4 layer outputs feature map v c ∈ R W ×H ×D . Then, we flatten this feature map into v l ∈ R D×C , where C = W × H . This processing program can be written as where v li ∈ R D , i ∈ {0, 1, · · · , C} denotes the feature of i-th location of image I. In other words, each image I is divided into C regions and every v li represents one region. In the training stage, the proposed method explores the relationship between words and image locations. In other words, when a word is imported, the word will guide which locations should be selected.
t−1 ) need to be controlled by a gate defined in Eq. (1) before they are imported into the input modulator. This is the core idea of the GF-LSTM.

Sentence Representation
In our model, we encode words into one-hot vectors. We denote any sentence as S = (w 1 , w 2 , · · · , w N ), where w i ∈ R N0 denotes the i-th word in the sentence.
We embed these words into embedding space. The concrete formula is as follows: where W s is the embedding matrix of sentences which projects the word vector into the embedding space. So the projection matrix W s is a N 0 × h matrix where N 0 is the size of the dictionary and h is the dimension of the embedding space.

Gated Feedback LSTM for Sentence Generation
Two-layer gated feedback LSTM is used as the sentence generator in our method. In other words, RNN model in Fig. 3 is two-layer gated feedback LSTM. Differing from Fig. 2.(b), every LSTM unit has three inputs: v t ,ẑ t and s t .

LSTM Model
Because the input of model is more complex than the GF-LSTM introduced in Section 3 and GF-LSTM model is core in our model, in this subsection, we introduce our formulas in detail. Fig. 4 shows the diagram of 2-layer GF-LSTM in our 3G model. The function ϕ denotes the hyperbolic tangent function (i.e.
. First, according to the structure shown in Fig. 4, three gates are rewritten as follows: where W * ∈ R h×h and Z (1) * ∈ R h×h denote weights matrixes and b (i) * stands for biases. v t ∈ R h andẑ t ∈ R h are the global and local image features, respectively. Their calculating formulas are introduced in the next two subsections.
Formulas of the current memory and hidden state are the same as Eqs.
(2)-(3). The updating memory contents are computed as follows: The LSTM module outputs a probability at each time-step. We write it as the following formulas: where W y ∈ R N0×h and b y ∈ R N0 denote passing forward parameters. y t ∈ R N0 is an output of LSTM at the t-th time-step. p t+1 ∈ R N0 is a probability vector, of which each element represents the predicting probability of the corresponding word.
Having built the GF-LSTM model, initializing the system is another important thing to do. The memory and the hidden state are initialized by the following formulas: where W c init ∈ R h×h and W h init ∈ R h×h are initial weights. b c init ∈ R h and b h init ∈ R h are initial biases.

Gate for Global Image Feature
v t occurs several times in Section 4.3. And it is an output of global image feature controlled by gate. In previous works [11,28,3,7,12,8], the vast majority of them import the global feature defined in Eq. (6) at the first time-step or at each time-step into the RNN decoder, but Vinyals et al. [3,7] find that global feature imported at the first time-step is better than that at every time-step.
They explain that global feature imported at each time-step may bring more noise to the system. But images in the benchmark image-caption datasets are high quality with little noise. Therefore, this reason may be a little far-fetched.
In this paper, we want to design a robust algorithm that can autonomously decide when and how much the global feature should be imported into the de-coder. Inspired by the gating mechanism exploited in LSTM, we design a gate before the global feature is imported into the multi-modal embedding part. The gate is defined as follows: where w g ∈ R h is the weight vector, b g is the bias. So the t-th gate g t is a scaler and its value correlates with the previous time-step 2-layer hidden state h t−1 . After calculating the gate, the global image feature at time-step t is computed as follows: Through Eq. (18), if we set g t = 1 at t = 0, 1, · · · , T , v is imported into the decoder at each time-step. If we set g t = 1 at t = 0 and g t = 0 at t = 1, 2, · · · , T , v is only imported into the decoder at the first time-step. Theoretically speaking, the 3G model proposed in this paper is more general and more robust.

Attention Mechanism for Local Image Feature
The local image featureẑ t denotes the local information of image. Here we use the attention mechanism as introduced in [29] for local feature. At each time-step, the attention mechanism uses the previous hidden state h (2) t−1 to decide the local feature. The attention model is defined as follows: where w a ∈ R h and U a ∈ R h×h are weights. v l is defined in Eq. (7). b a ∈ R h is the bias. The annotation vector α t ∆ = α t1 · · · α tC T ∈ R C is a probability vector whose each dimension value denotes the probability of the corresponding local image feature. In our algorithm, we use the soft attention model. Therefore,ẑ t is calculated as follows: Through Eq.
where θ is the parameter set which includes parameters of the GF-LSTM, embedding matrixes in Section 4.1 and Section 4.2 and all gate models.
The proposed model is trained with back-propagation through time (BPTT) algorithm [36,37] to minimize the cost function L(θ).

Experiments
In this section, we begin by describing the publicly available datasets used for training and testing the model and the evaluating metrics for image caption generation. Then some typical state-of-the-art models are simply introduced.
Finally, we show the quantitative results compared with the recent state-of-theart methods and analyze the experimental results.

Evaluation Metrics
In order to evaluate the proposed method, two objective metrics are used in this paper. They are BLEU [41] and METEOR [42]. These two metrics are originally designed for evaluating the quality of the automatically machine translation. BLEU score represents the precision ratio of the generated sentence compared with the reference sentences. METEOR score reflects the precision and recall ratio of the generated sentence. It is based on the harmonic mean of uniform precision and recall. For BLEU, we use the scores from BLEU-1 to BLEU-4, which denote the precision of N-gram (N equals to 1, 2, 3 and 4). For both two metrics, the higher score they are, the higher quality of the generated sentences they have.

Comparison Models
In this subsection, some typical state-of-the-art models are briefly introduced. These models are both using CNN + RNN diagram which is the most effective diagram for image caption generation, but they have some differences in detail.
• m-RNN [11]. • Google-NIC [3,7]. Unlike m-RNN, Google-NIC just projects the image feature into the embedding space and imports it into the RNN at the first time-step. Therefore, the RNN here is not only the language model, but also the multi-modal model. DeVS [12,8] is very similar to this model, but they also have a little difference, where LSTM is used as sentence generator in Google-NIC but "vanilla" RNN is used in DeVS.
• NIC-VA [29]. This model has several variations in different tasks such as image caption generation, machine translation, video clip description and speech recognition and has achieved great success in these tasks. In so it gets a great success.

Dataset Processing
Before the experiment, we have preprocessed the datasets as [12] did. At first, we convert all letters of sentences to lowercase and remove non-alphanumeric characters. Then we get rid of words that occur less than five times on the training set. Because some images in MS COCO have more than 5 corresponding sentences, we discard these data to grantee every image has the same number of describing sentences. Since the ground-truth captions of the MS COCO test set are blind to the public, we use the publicly released splits 2 which is used in DeVS within 5,000 testing images.

Image Feature
In the proposed model, deep features generated from the CONV5-4 and

Word Encoding
In our model, we encode words into one-hot vectors. For example, the benchmark dataset has M different words, every word is encoded into a M -dimension vector, in which only one value equals to 1 and others equal to 0. So the location of 1 in the vector denotes the corresponding word in the dictionary. It implies that N 0 in Section 4.2 equals to M .

Training Option
The proposed model is trained with stochastic gradient descent (SGD) using adaptive learning rate algorithms. Similar to [29], the RMSProp algorithm is used for the Flickr8K dataset and for Flickr30K/MS COCO. (LDA) to obtain a "scene vector" for each image. The model also need to train a multilayer perceptron to predict the scene vector. Therefore, (RA+SS)-ENSEMBLE is more complex than the proposed model in this paper. Table 1 shows that the attention-based methods are more effective than the global image feature based methods. This means the attention mechanism is more effective than other models only using the global image feature. However, the 3G model proposed in this paper shows a better or at least comparable performance, which confirms the effectiveness of the proposed model. The quantitative results in Table 1 proves that the global image feature using gating mechanism and the gated feedback LSTM for multi-modal embedding plays an important role on the image caption generation task. Three main points make our 3G model better than the other state-of-the-art models. First, the proposed 3G model in this paper exploits the visual attention mechanism. Second, the introduction of the global image feature with gating control is a good supplement because the visual attention part mainly focuses on the local of the given image. Third, the 2-layer GF-LSTM makes our language model stronger than other language models in the contrast models.  Table 2, the compared methods can be divided into two categories. 1) "CNN-RNN" diagram and 2) Detector+"CNN-RNN" diagram.

Results Evaluation and Analysis
In the first category, only image feature is used to generate the sentence. However, the second category methods need extra image information attained by the extra detector. Therefore, DevS [8], MSR [44], MLBL [46], NIC-VA [29] and our 3G belong to the first category; while RA [9], ATT [30] and Att [47] belong to the second category. Through analyzing the results in Table 2 is trained on the pre-labeled dataset. Therefore, RA, ATT and Att reasonably outperform the proposed method. Meanwhile, it also implies that our method still has potential for improvement if more image information is used. In the experiments, we choose the VGG-Net as the image feature extractor.
The main reasons are as follows: 1) almost all the state-of-the-art contrast methods compared in the experiments use the VGG-Net as image feature extractor [12,11,48,29]; 2) VGG-Net is considered as one of the most effective feature extractor which can extract the discriminative and expressive feature for image.
However, we also add an extra experiment to verify the effectiveness of the different image features and the results are shown in Table 3. Among the models in      In order to verify the effect of the introduction of the image global feature with gating control, we use the 1 layer LSTM as the decoder, which we name it GL-NIC in Table 4. Experimental results in Table 4 prove that the fusion of the global image feature and the local image feature is useful for image caption generation. In fact, the contrast models can be regarded as the special situations of GL-NIC : 1) when setting α t = 0, g t = 1 at every time-step, GL-NIC degenerates as LRCN; 2) when setting g t = 0 for all t, GL-NIC degenerates as A bedroom has many posters on the wall . A bed with a blanket on top of it and a book on the side.
A row of motorcycles parked in front of a building . A row of motorcycles parked next to each other in front of a building.
A bathroom with an enclosed shower next to a sink and a toilet. A bathroom with a white toilet next to a sink.
A bedroom has many posters on the wall . A bed with a blanket on top of it and a book on the side.
A dog running in the grass with a frisbee in his mouth . A dog is playing with a frisbee in the grass.
people on bicycles ride down a busy street A man riding a bike down a street next to a large white building.
A lovely dinner and a bottle of wine A table topped with a plate of food and a bottle of beer.
A kitchen is shown with a variety of items on the counters. A kitchen with a stove, oven, stove, and a sink.
A small blue plane sitting on top of a field . A small airplane is parked on a runway.
A cow standing near a curb in front of a store . A cow is standing in front of a store window at a zoo.
A zebra is standing outside in the snow A zebra standing on snow covered ground next to a forest. NIC-VA; 3) when setting α t = 0 for all t, g t = 1 at t = 0 and g t = 1 for other t, GL-NIC degenerates as Google-NIC. Therefore, fusing the global and local image features with gated mechanism can be more comprehensive and robust to describe the content of the image. ator. So the model is named as NIC-VA (GF-LSTM) in Table 5. We compared NIC-VA (GF-LSTM) model with NIC-VA model. However, the decoder in NIC-VA is 1-layer LSTM, for the sake of fairness, we changed the decoder in NIC-VA into 2-layer stacked LSTM. We mark this model as NIC-VA-2-LSTM. Table 5 shows that the performance of NIC-VA (GF-LSTM) is better than NIC-VA (2layer LSTM). This is because the GF-LSTM can deal with the issue of learning multiple adaptive timescales. The gated feedback collecting method which is a strategy to increase the depth of the LSTM not only uses the previous equal level hidden state, but also uses the previous higher level hidden state. In fact, J. Chung et al. [13] has proven that the GF-RNN outperforms the traditional stacked RNN, especially as the number of nesting levels grows or the length of target sequences increases. proper sentences to describe the main contents of the given images, but also give more elaborate descriptions for the given images. For example, the first image of the second column, the reference describes the relationship of the motorcycles and building ("motorcycles parked in front of a building"), while the sentence generated by our 3G model not only describe the relationship between them, but also describe the relationships between the motorcycles ("next to each other").

The effectiveness of the gated feedback LSTM
Another interesting but not intentional discovery is that our 3G model can generate grammatical correcting sentences, which may be ignored by humans.
For instance, the third image of the second column, the first letter "p" should be capitalized but the reference gives the lower case. Some little errors like this will not be occur in our 3G model.
Some negative examples are also shown in Fig. 7 (b). Such as the third image in column 4, the generated sentence is "A zebra standing on snow covered ground next to a forest", but in fact only a few trees which cannot determine whether it is a forest. Despite that our model makes a small mistake on the scene background, our model always tries to describe all the content of one image. In fact, the sentence generated by 3G model describes the main contents of the image ("A zebra standing on snow covered ground"), but the background gets a little mistake (the zebra may not stand "next to a forest"). The descriptions generated for the first and the second images in column 4 also have some mistakes. We think this may be caused by the repeated scene and reference sentences. That is to say, when a plane in an image, the plane often accompanied by a runway. So the model has learned much knowledge like this, when importing an image similar but having some differences with this scene, the model may generate the wrong description. This problem can be well solved by increasing the number and variety of the dataset.
Generally speaking, two main reasons make the proposed model in this paper able to complete the image caption generation task very well. Firstly, the most advanced language model-GF-LSTM-is used in the proposed 3G model.
Secondly, gated global image feature and attention-based local image feature are fused for image representation, which is beneficial to seize the accurate, comprehensive and meticulous information of images.

Conclusion
In this paper, a 3G model for image caption generation is proposed. The proposed model shows the better performance than other state-of-art model on three benchmark datasets. 3G model mainly has three gating structures: 1) gate for the global image feature, 2) gate for recurrent neural network and 3) gated feedback for multi-layer recurrent neural networks. Through the gated structure, the global image feature can be robustly selected to input into the multi-modal embedding model. We choose the gated recurrent neural network as language model because it solves the long term dependency problem in "vanilla" RNN. Gated feedback collection for multi-layer recurrent neural networks can deal with the problem of learning multiple adaptive timescales, and this makes GF-RNN more proper for language model than the standard stacked RNN. So, for both the image feature information and the language model, the proposed 3G model in this paper reinforces the recent state-of-the-art models for image caption generation.