{'fraction_non_alphanumeric': 0.03637927327456461, 'fraction_numerical': 0.011395398838959363, 'mean_word_length': 4.782197911486822, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

In recent years, transformer structures have been widely applied in image captioning with impressive performance. For good captioning results, the geometry and position relations of different visual objects are often thought of as crucial information. Aiming to further promote image captioning by transformers, this paper proposes an improved Geometry Attention Transformer (GAT) model. In order to further leverage geometric information, two novel geometry-aware architectures are designed respectively for the encoder and decoder in our GAT. Besides, this model includes the two work modules: 1) a geometry gate-controlled self-attention refiner, for explicitly incorporating relative spatial information into image region representations in encoding steps, and 2) a group of position-LSTMs, for precisely informing the decoder of relative word position in generating caption texts. The experiment comparisons on the datasets MS COCO and Flickr30K show that our GAT is efficient, and it could often outperform current state-of-the-art image captioning models.


Introduction
Image Captioning is a challenging problem in computer vision [1]. It aims to automatically describe an image using meaningful text, translating an image into natural language. It often requires to not only recognize what visual objects an image contains but also capture what those objects are doing accurately and even to tell us what the interrelations of different objects are. Image captioning could build a powerful bridge between visual images and human languages. With captions, people could better understand an image [2]. Image captioning has been thought of as one quite useful technology in the image analysis field. In recent years, it has been attracting more and more research attention, and diverse captioning models have been emerging for image captioning [3,4,5,6].
For image captioning, its early models could often be roughly classified into two primary categories [7]. One is usually dependent on image retrieval. It generates image captions by analyzing image correlation and then retrieving candidate texts from existing caption pools [8,9]. In contrast, the other is often summarized as the templatebased category. This kind of methods builds captions typically through the syntactic and semantics analysis on images, with visual concept detecting, sentence template matching and optimizing [10,11,12]. The most obvious characteristic of these two early categories is that hardcoded rules and hand-engineered features are in common use so that the reliability and accuracy of captioning fluctuate heavily [7].
In recent years, facilitated by booming artificial intelligence, neural networks, the third type of method for image captioning, have been becoming one of the most exciting and powerful tools. For example, we could easily see the huge improvements by early work of deep neural models in [13,14,15]. Besides, more neural network based models, such as the multimodal learning [16,17], the encoder−decoder framework [18], the attention mechanism [19,20], the compositional architectures [21], the describing method of novel objects [22] and the deep bifurcation network [23], have been proposed one after another.
Through the survey of a large number of publications, it is easy to figure out that Transformer with an encoder-decoder structure is becoming one of the most main-stream models for image captioning [24]. In general, in a typical transformer model, a given image is often mapped into a set of intermediate vectors by an encoder of CNNbased networks, and then the target caption of this image could be generated word by word through a decoder of RNN-based network [4,16,18,25].
In addition, as mentioned before, the classic attention mechanism has also been widely utilized in image captioning [19,20]. One of its advantages is that it could often effectively guide the decoder to focus on some specific information, such as context correlation, while generating image captions. On traditional attention models, the selfattention mechanism [26] is often thought of as an effective improvement to the classic version. In recent years, this kind of self-attention model has also quickly emerged in image captioning [27,28]. From the task perspective, a self-attention module is often regarded as a feature mapper between a set of queries and key/value pairs.
In image captioning, the geometry and position relations of image objects are often very necessary for accurately describing a given image. For example, "a boy standing on a skateboard" and "a boy raises a skateboard in his hands" indicate two different semantics. However, they are likely to mean the same thing if without relative geometry information. In most existing Transformer-based models with attention modules, we could find that the inherent geometry and position relations between image components have not yet been paid enough attention to and fully utilized [25]. Moreover, although some methods, such as the GSA proposed in [24], carefully considers geometry relations in encoder, more detailed position decoding has not been well solved yet.
Geometry and position relations are so important that accurate image captioning models often have to emphasize them. In order to take full advantage of geometry and position clues, we firstly design an improved encoder model with Geometry selfattention Refiner (GSR) in our transformer structure. This model could explicitly incorporates geometry information into a vanilla self-attention module, implementing the transformation from original appearance queries to geometrical ones. And then, the refined attention weights are calculated to get a mean of weighted appearance values.
The geometrical queries and keys in our model are linearly derived from the bounding box coordinates of each component object. In this way, every object could always ob-tain its appearance features and geometry correlations with each other. In task details, they both are very crucial for a decoder to generate the right caption word sequences with position and semantic.
Furthermore, in the decoder of Transformer, some typical methods often inject the word position information of expected caption texts by adding sine or cosine operation on the top of the word embedding layer, such as the ones in [26,27]. This kind of position injection mechanism has been proved effective, in spite of not always conforming to semantics. To overcome this weakness of position decoding, we design a group of position-LSTMs to model the word order of caption texts. It parses a caption sentence word by word in sequence, ensuring the right order of words. In addition, the hidden layers of position-LSTMs could also remember the information that the decoder has generated. This kind of design is very helpful to self-attention modules to concentrate on particular position parts. In the meanwhile, the information stored in hidden states of LSTMs also contains the geometry information of component objects.
Finally, by the integration and collaborative optimization of GSR and the position-LSTM, our captioning model has achieved better performance than state-of-the-art ones on the two classic datasets: MS COCO and Flickr30k. In summery, the main contributions of this paper could be listed as follows: • We propose an improved image captioning model GAT of transformer structure, which could capture the geometry relations of component objects. Geometry capturing ability could often enable our captioning model to obtain a sense of 'where are target objects' and 'where is the captioning model currently looking at'.
• We design an encoder improved by GSR. It could encapsulate the relative geometry information of objects. Moreover, it takes full advantage of geometry relations to refine object representations.
• We develop a decoder promoted by position-LSTMs. It could transport the order information of generated cation words with relative position relations. Therefore, it could execute position reasoning to decide which word to generate next.

Related Work
In recent years, a king of encoder-decoder transformer architecture with attention modules has been attracting wide research enthusiasm. In the meanwhile, more and more methods have been developed for image captioning, such as the one with soft & hard attention [29], the one with residual connections [30], and the meshed-memory transformer [31].
This type of transformer architecture, shown in Figure 1, can bee clearly seen in [24,27,32,33]. Typically, it consists of an encoder and a decoder, one for extracting visual object features and the other for generating caption text. In detail, both the encoder and the decoder, represented by Fig. 1(a), are designed as a group of multilayer residual networks. Moreover, we could also always see a classic component of 'Self-Attention', shown by Fig. 1(b), in the encoder and decoder [27,28]. In the typical captioning framework illustrated above, besides traditional object appearance features, the input to the encoder contains some novel geometry cues, such as object center, height and width [24]. In the decoder, furthermore, a position encoding operation has also been employed to encode the accurate word sequence of the caption text. And in [24], a simple 'sinusoidal function' is utilized to fulfill this task.
Experiments have also proved that this kind of improvement by geometry and position features is much efficient.
Moreover, in terms of the attention modules in image captioning, there are also all kinds of variants having been explored. For examples, we could easily see the multihead attention in [26], the gate-controlled attention [21], the fully attentive paradigms [28,34], the meshed-connection attention [31], the dual attention [35], etc.
To our knowledge, this kind of encoder-decoder architecture with attention components introduced above is believed to be one of the most potential baselines for image captioning [24,27]. In the future, more ingenious network design, more sophisticated object features, more functional geometry relations, and position encoding strategies are still kept huge research expectations for further promoting image captioning methodology [36].

Proposed Method
Our method generates grounded captions by attending to specific image regions at each step. In terms of structure, it still retains a classic encoder-decoder architecture.
In the encoder, a geometry self-attention refiner (GSR) is designed to optimize image representations. Furthermore, in the decoder, a module for accurately decoding word sequences by LSTM is adopted. More details about it can be seen in Figure 2.
Given an input image I, in figure 2, we use (X A ∈ R N ×d ) to represent a set of image appearance regions, where N is the index of image region, and d is the dimensionality of region data. Moreover, (y = {y 1 , ..., y T }) to indicate the caption word vectors corresponding to I.

Gate-controlled Geometry Self-attention Refiner
Besides appearance information X A , we incorporate geometry information into the vanilla self-attention to refine the representation of the objects for the reason that it is beneficial to comprehend the intrinsic relations among different objects. Therefore, we Figure 2: The detailed architecture of our GAT. The GSR module in encoder generates some weighted average given appearance features X A and geometry features X G , which is further refined by a gatecontrolled unit. Moreover, a group of position-LSTMs precisely injects word order information into the decoder.
propose the Geometry Self-attention Refiner (GSR) to refine the visual information by taking into account the geometry information of the objects.
Given the geometry features of the objects X g ∈ R N ×5 , each row of X g is a 5-dimensional vector: where (x min , y min ), (x max , y max ) and S represent the top-left coordinates, the bottomright coordinates of bounding box, and the relative size to whole image, respectively.
And, all of them are normalized to (0, 1).
Firstly, we obtain X G ∈ R N ×dm by embedding X g to a higher dimension form with an embedding layer followed by a ReLU non-linearity. Then we combine appearance information with geometry information by modifying the queries and keys as follows: where W Q A , W K A are learned appearance matrices and W Q G , W K G are learned geometry matrices, respectively, and are all of dimension R dm×dm . Here [;] indicates concatenation. Q and K combine appearance information with geometrical information, which can be seen as a complementary to acquire fine-grained knowledge of the objects. Then the attention can be calculated as: Then the output of the GSR can be calculated as: Similar to the original Transformer [26], our geometry refiner is also implemented in a multi-head self-attention framework. In this framework, geometry attention will be repeated h times (so, called h heads). In every time of repetition, it always adopts for capturing geometrical features, where k ∈ (1, 2, ..., h). All the results generated from h self-attention heads will also be concatenated together by a linear projection and then fed forward to the next layer.
Moreover, inspired by the work of Huang et al. [20], we further design a parallel Gate-controlled Linear Unit (GLU) to continually refine the output of the geometry self-attention module (the left region of Figure.2). This GLU still takes the current context (c = [X A ; X G ]) as its input to generate a control matrix Gate Ctrl for modulating final attention output by Hadamard product. This group of neural computation can be mathematically expressed by Eqn. (6): where W g , W i ∈ R dm×dm are two groups of weights, and b g , b i ∈ R dm are two groups of neuron biases. Furthermore, denotes a Hadamard product. In terms of structure, our self-attention component and gate-controlled unit can be stacked into multiple layers to further optimize object representations.
Subsequently, the self-attention output is embedded with its original input by the operation of add & Normalization, then transferred to its next layer, i.e., the Feed-Forward sub-lay network shown in the middle of Figure 2. In details, this Feed-Forward sub-layer usually contains two groups of nested affine transformations with the activation function, ReLU (x) = max(0, x). Therefore, this group of processing step could be uniformly formulated by where W 1 and W 2 are two groups of weight matrices to be learned, b 1 and b 2 are two groups of biases. Also following the Transformer architecture in [26], each sub-layer is successively followed by a residual connection and a layer normalization. Therefore, this group of computation could be mathematically expressed by

Position-aware Self-attention Decoder i) Encoding positions via LSTM.
In order to address the order problem of a sequence, existing works usually inject some 'positional encoding' into word embedding by a sinuous function. In our method, we further exploit a position representation strategy and then devise a distinctive LSTM-based position encoding mechanism.
The position-LSTM in our decoder of transformer could model the order of image caption words in decoding process. At each time step t, we define the input of the position-LSTM as: where w t is the word embedding derived by a one-hot vector, andv = 1 k Σ i v i denotes the mean pooling of image features. Therefore, we could obtain: In view of the sequentiality output of LSTM, h t could often be treated as an encoding of sequence order for caption words. Such an encoding could provide its subsequent decoder with precise position information in two aspects. One is what words have been generated by far, and the other is where the decoder is for directing the current relative position of objects. In addition, this position encoding will also be updated at each time step, and meanwhile, it could also guide the decoder to focus on its correlated regions in geometry-aware mode adaptively.
ii) Injecting position encodings into the decoder.
where W p ∈ R |Σ|×m and b p ∈ R |Σ| are also two groups of learnable parameters, and F t is the final output of designed decoder.

Datasets and Metrics
We evaluate our model on the dataset MS COCO [37] and dataset Flickr30k [38].

Implementation Details
We employ a pre-trained Faster R-CNN model with a backbone of ResNet-101 [4], to extract 36 features for each image. The dimension of feature vectors is 2048-D.
Thereafter we project them into 512-D to reduce memory consumption. The hidden state size of LSTM is set to 1024. In addition, the dimensionality of the input to both the GSR and the self-attention module is set to 512. Moreover, the number of selfattention heads is 8. We set the number of layers to 3 in both the encoder and the decoder. In LSTM, the dropout rate is 0.5, and the dropout rate of all self-attention layers is set to 0.1. In the stage of cross-entropy training, we train our model using an initial learning rate of (5 × 10 −4 ) with a decay rate of 0.8 for every three epochs. In the CIDEr optimizing, we train our model in 30 epochs, by a learning rate of (2 × 10 −5 ) with a decay factor of 0.8 also for every three epochs. All compared models are trained with the Adam optimizer with batch size 50. In testing, we always use the same beam size of five for all.

Ablation Experiments
To quantify the performance of our proposed modules, we design a group of ablative experiments on MS COCO. We use a vanilla Transformer (see Figure 1) as our experiment 'base', also similar to the self-attention network [24]. Its encoders do not consider the geometry information, and decoders are only combined with common sinuous position encodings.

i) Effect of Geometry Self-attention
We first apply the GSR to the 'base' model to evaluate its effect on encoders. GSR

ii) Effect of Position LSTM
We further evaluate the impact of the position-LSTM module on captioning performance. We replace the simple sinuous position encoding in [24], with our LSTM-based ones. From the experimental comparisons in Table 1, we could see that our position-LSTM has raised the CIDEr score of the 'base' model by 5.9. Compared with sinuous encoding, our position-LSTM could provide the decoder with an expressive encoding for 'where' to decode and also guide the decoder to capture correlated semantic information from image regions.

iii) Geometry Queries and Keys
To verify the efficiency of concatenating geometrical queries and keys with appearance queries and keys, we investigate different straggles to combine them. Mainly, we directly compare two operations, including 'add' (adding) and 'concat' (concatenating). Table 2 shows the comparison results of these two operations. We could figure out that the performance promotion by concatenating appearance queries and keys with geometry queries and keys is better than by adding them, although they both could surpass the base model in terms of most metrics. For example, the BLEU-1 score of the base model is only 75.0, while it could rise to 76.0 and 77.5 by 'add' and 'concat', respectively.
iv) Gated Linear Unit. In our architecture, GLUs play an important role in refining the output of original self-attention layers. Therefore we also have tested its effect on both encoders and decoders. Table 2 lists the comparison results. We could see that GLUs could always achieve a good performance promotion on the base model.
However, we could also see that the GLU works best when it cooperates alone with encoders. If GLUs are assembled with encoders and decoders at the same time, the model could even get worse performance. For example, in Table 2, the SPICE by the base model without GLUs is 21.2, and it gets to 21.8 when GLUs are integrated only with encoders. However, the SPICE drops to 21.5 when GLUs are combined with all encoders and decoders. This finding is exactly consistent with [20]. It means that stacking too many GLUs on a group of position-LSTM decoder layers could damage the gradient of network training, depressing the capacity of self-attention layers.

Comparison with State-of-the-Art Models
We have also compared our GAT model with several state-of-the-art approaches  [29], the Deep VS [16], the NIC [18], the m-RNN [47], the adaptive model [25], the SEM architecture [48] and the DA framework [30].

i) Off-line Evaluation
We also evaluate our model on the 'Karpathy' test split of MS COCO, as in [16].
All models are firstly trained by cross-entropy loss and then optimized for the CIDEr score. For a fair comparison, the visual features fed to all models are directly extracted by a Faster R-CNN of the same structure. The top half of  In Table 4, we compare the performance of our GAT with state-of-the-art models on dataset Flickr30k. It could be seen that our model outperforms all the compared methods by a large margin. For example, compared with Soft-Attention, which also employs an attention mechanism [29], our GAT obtains an improvement of 7.5 on BLEU-1, 10.9 on BLEU-4 and 4.9 on METEOR. Even compared with DA model [30], the best of the other eight models, our GAT still could often achieve a higher score than DA on all metrics. For example, our GAT could get an increase of about 0.6, 1.4 and 1.4 on BLEU-1, BLEU-4 and CIDEr, respectively. This comparison further demonstrates the superiority of our GAT method.
Therefore, on the whole, our GAT model could be synthetically deemed to be more superior than other compared ones.

Caption Text Comparisons
To qualitatively illustrate the superiority of our method, we present six images with their captions generated respectively by our GAT and the Vanilla Transformer model (i.e., base), as well as three manual captions of ground truth (i.e., GT1∼ GT3) in Figure 3. It can be seen that GAT could generate an accurate image caption with complete semantic structure and clear position relations. For example, the caption by GAT could geometrically describe that "people" are "under umbrellas" and "in front of store" in the top-left sub-image of Figure 3, and "dog laying under the chair" rather than "laying on the chair' in the middle-right sub-image of Figure 3. The novel characteristic that our GAT is able to accurately describe the geometry & position relations mainly owes to the GSR module, which can explicitly incorporate the spatial correlations of image regions into object feature representations. Moreover, the position-LSTM could also keep reminding the decoder, at every decoding step, which visual object should be attended to. On the whole, our GAT could often express almost all exact objects contained in a given image, such as "people", "umbrellas" and "restaurant", etc. in the top-left sub-image of 3. Furthermore, GAT could also capture the geometry relations between any two detected objects and then decide whether to specify the correlations of detected objects, benefited by which it could often generate the highly-reliable image captions with spatial-aware semantic.

Conclusions
In this paper, we propose the Geometry Attention Transformer (GAT), an extension scheme to the well-known Transformer for image captioning in recent years. It is able