{'fraction_non_alphanumeric': 0.028971992668574552, 'fraction_numerical': 0.008135309817035918, 'mean_word_length': 4.956713273319288, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Image Captioning based on Feature Refinement and Reflective Decoding

Image captioning is the process of automatically generating a description of an image in natural language. Image captioning is one of the significant challenges in image understanding since it requires not only recognizing salient objects in the image but also their attributes and the way they interact. The system must then generate a syntactically and semantically correct caption that describes the image content in natural language. With the significant progress in deep learning models and their ability to effectively encode large sets of images and generate correct sentences, several neural-based captioning approaches have been proposed recently, each trying to achieve better accuracy and caption quality. This paper introduces an encoder-decoder-based image captioning system in which the encoder extracts spatial features from the image using ResNet-101. This stage is followed by a refining model, which uses an attention-on-attention mechanism to extract the visual features of the target image objects, then determine their interactions. The decoder consists of an attention-based recurrent module and a reflective attention module, which collaboratively apply attention to the visual and textual features to enhance the decoder's ability to model long-term sequential dependencies. Extensive experiments performed on Flickr30K, show the effectiveness of the proposed approach and the high quality of the generated captions.


Introduction
Visual information is pervasive in our daily life and has arguably become the primary communication medium in the digital age. Images come from various sources, such as social networks, advertisements, news, and digital documents. A small portion of these images have associated description that helps understand, organize and search their content, but the vast majority does not. Image captioning using machine learning is the process of automatic generation of a natural language description for an input image. In order to automatically generate an appropriate caption, the system must detect the various objects in the image and then describe them in natural language.
Image captioning systems have numerous practical applications, such as helping visually impaired individuals better understand the content of images, hence improving their user experience. They can also be integrated into guidance devices that help improve the life quality of the visually impaired. Automatic image indexing can benefit from image captioning since it generates more sophisticated and semantically rich captions than image classification and tagging. Automatic image indexing is essential in many areas like education, medicine, e-commerce, digital libraries, military, and web searching. Furthermore, image captioning can improve the performance of content-based image retrieval applications which search for a specific image based on its content. Social media platforms can use image captioning to generate accurate descriptions of images posted by users and provide better personal recommendation and content filtering. More generally, all domains where humans need to describe or interact with images in any form will benefit from a system capable of automatically generating textual image captions.
Even though effective algorithms for recognizing objects in videos and images have been developed in recent decades [1], describing image visual scenes and features in well-formed sentences remains a very challenging task. Such systems must extract semantic concepts and visual features and transform them into correctly formed sentences in natural language [2]. Generating accurate textual descriptions of images chiefly depends on extracting rich image global visual features. Early work in image captioning mainly used template-based and retrieval-based methods. Both types tried to solve image captioning problems by using training captions or hard-coded language rules. However, these techniques have significant limitations that affect the generated caption quality when used with large, complex, and diverse data that admit various semantic interpretations. With the considerable progress in research on neural networks and their proven capability for handling complex learning tasks, neural network-based approaches have become the dominant approach in image captioning. Deep learning techniques use training data to learn complex and rich image features allowing them to handle large, complex, and diverse datasets. Despite this progress, the challenge remains for researchers to generate better quality captions using more sophisticated visual and language network architectures.
This work proposes an encoder-decoder image captioning system that uses a refining module on the spatial features extracted from ResNet-101 in the encoder and a reflective decoding module in the decoder. Unlike previous approaches that attempt to improve the performance by focusing on improving visual features alone, our model uses the extracted visual features and an attention-on-attention mechanism [3] to model the relationships among objects in the target image. The decoder uses a reflective decoding module [4] that applies two types of attention to the visual and textual information to focus on the most salient objects in the image and obtain more comprehensive information to generate higher-quality captions. The proposed approach is implemented and evaluated using Flickr30k, a benchmark image captioning dataset. The experimental results analysis demonstrate the proposed approach's effectiveness.
This paper is organized as follows. First, we survey the existing literature on image captioning in Section 2 and present a comparative study of related work. The proposed approach is presented in detail in Section 3. Next, we provide a detailed experimental evaluation of the proposed architectures. We finally conclude this paper in Section 5 and give an overview of future work.

Related Work
Automatic generation of image description is known within the research community as image captioning. It is the process of automatically generating a wellformed caption that describes the important objects in an image, including their location, type, properties, and relationships. Following [5,6], we can broadly classify the existing image captioning approaches into two main categories: early work and neural networkbased techniques. Early work approaches mainly used template-based and retrieval-based methods for generating captions. However, following the significant progress in neural networks research in the previous decade, neural-based approaches for image captioning became the dominant trend in the field. Deep learning methods achieved a remarkable improvement in performance and demonstrated excellent capabilities for handling the challenges and complexities of the image captioning task. In what follows, we will present both categories of captioning approaches and highlight their weaknesses and strengths.

Early work on image captioning
Early image captioning systems mainly used retrievalbased and template-based approaches. Authors in [7] and [8] used retrieval-based image captioning by mapping the target image to a space of images and captions retrieved from the dataset. The authors in [7] used Markov Random Field for the mapping and Lin similarity measure to find the best caption. In contrast, [8] used Kernel Canonical Correlation Analysis (KCCA) technique for the mapping and cosine similarity measure for captions ranking. Retrieval-based methods can generate syntactically correct captions. However, since these generated captions are obtained by reusing and composing existing captions, they generally are semantically inaccurate. In addition to the lack of large datasets available in recent years like COCO by Microsoft [9] and Conceptual Captions by Google [10], using relatively small datasets negatively influences the quality of the generated captions.
On the other hand, [11] and [12] used templatebased image captioning by detecting features from the target image and then using templates to generate captions. The authors in [11] used Hidden Markov Model inference (HMM) to fill the template structure. Unlike [11] or any previous approaches that used a fixed template [12] used a computer vision detector system to produce a syntactic tree that will summarize the extracted image features to generate caption, which results in better quality captions. Even though the template-based approach may generate better and more relevant descriptions than the retrieval-based approach, it still has limitations regarding caption creativity, coverage, and complexity. Furthermore, template-based captions lack naturality compared to human written captions due to using predefined templates.
Early work captioning systems, whether retrievalbased or template-based, suffer obvious limitations; They are not flexible enough to generate expressive and meaningful captions. With the significant progress in neural networks research, especially in deep neural networks, many recent works tackled these limitations by adopting neural networks in automatic image captioning using various methods and architectures.

Neural network-based image captioning
The first attempt to use neural networks to solve image captioning problems was in [13]. The authors replaced templates, rules, and constraints with Convolutional Neural Networks (CNN) [14] and Multimodal Neural Language Models (MNLM) to generate better quality captions. They then extended their work in [15] to introduce an encoder-decoder framework that uses deep CNN as a visual model and Long Short-Term Memory (LSTM) as a language model. The Neural Image Caption Generator (NIC) model proposed in [16] is similar to [15] but is trained using maximum likelihood estimation to maximizing the likelihood of observing the ground truth captions. These encoderdecoder frameworks significantly improved the quality of the generated captions grammatically and syntactically. However, their performance depends primarily on the quality of global and high-level features extracted from the target image using CNN and the effectiveness of captions representation by neural language models.
Instead of an end-to-end encoder-decoder image captioning framework, [17] and [18] used compositional architecture-based approaches that consist of a visual model, a language model, and a multimodal similarity model. The authors of [17] used a visual model that focused on image subregions, unlike previous models that operate on the whole image, a Maximum Entropy (ME) language model, and a Deep Multimodal Similarity Model (DMSM) to choose the best caption for the target image. Based on the architecture proposed in, [17] [18] built a vision model that specializes in generating captions for open domain images. However, instead of extracting visual features using CNN, they used a deep Residual Network (ResNet) with entity detection to identify celebrities and landmarks, resulting in improved performance on in-and out-of-domain datasets. However, even though using multiple models can improve the performance and make the image captioning system robust and more powerful, it also increases the system's overall complexity.
Compared to methods that focus on the global features in the whole image, the attention mechanism allows focusing on the most salient objects of the target image before forming a caption. The first to use such a technique in image captioning were [19]. They learn the hidden alignments from scratch instead of detecting image objects like other methods to focus on the important parts of the target image. On the other hand, [20] used a visual sentinel that considers the none visual words in addition to the ones predicted by the language model. Authors in [21] used a spatial transformer network to define the attention areas in the target image without using bounding boxes in the training process. Even though [21] was faster, [20] generates better caption quality. An attention-based mechanism helps improve the quality and relevance of the generated captions and reduces computational complexity since it skips unimportant details in the image. However, it has limitations like neglecting some details in the target image that might be important for caption generation and the difficulty in formulating image captioning as an end-to-end process.
Previously discussed image captioning approaches used either top-down mechanism [22,23] or bottomup mechanism [7,24]. In top-down approaches, one starts by extracting visual image features and then converts them to words. In contrast, bottom-up approaches start by extracting visual image concepts like attributes, regions, and objects and then combining them with words that describe them. Both mechanisms have limitations, such as neglecting some details in the target image that might be important for caption generation in top-down approaches or the problem of formulating an end-to-end process in bottom-up methods. To overcome these limitations, You et al. [25] combined top-down and bottom-up approaches to selectively focus on the image's semantically important concepts and work at any resolution and in any region of the target image. In [26], the authors focus on generating semantically sound captions by extracting the high-level semantic concepts from the target image using an attribute prediction layer. Using a semantic concept-based approach can generate semantically rich image captions. However, focusing on some parts of the image might cause the system to miss some small details in the target image that might be helpful in the caption generation process. Table 1 compares the research works previously mentioned in the literature in terms of architecture, visual model, language model, modeling space, and attention mechanisms. We can observe from the table that most approaches adopt the encode-decoder architecture since it produces high performance with minimum complexity and considerable robustness. In contrast, only a few of them [17], [18] adopt the compositional architecture.

Discussion
In terms of the visual model used to encode the target image, there are five convolutional neural networks that are commonly used which are VGGNet [27], ResNet [28], AlexNet [29], DenseNet and GoogleNet. ResNet performs the best compared to others, but VGGNet is still the most popular among researchers even though it produces the second highest result. This choice is due to the fact that VGGNet is easier to implement, and recent research efforts focused more on simplicity and speed. Regarding the language model, Long-Short-Term Memory (LSTM) [30], which is mainly used as a decoder, is the most popular language model. Researchers tried other models such as Recurrent Neural Networks (RNN) [31], Maximum Entropy Language Model (MELM), Log-Bilinear Language Model (LBL), and Structure-Content Neural Language Model (SC-NLM). However, LSTM remains the model of choice because of the quality of its generated captions. Table 1 also shows that most approaches model the image in a visual space which offers a faster computational speed, but some approaches learn from combining visual and language space.
Finally, using an attention mechanism enables the approaches to dynamically focus on various parts of the image while the captions are being generated or selectively focus on a group of semantic concepts [25], [26]. As a result, attention mechanisms help generate high-quality and relevant captions.

Proposed Approach
We propose an end-to-end image captioning system based on an encoder-decoder framework with attention mechanisms. The encoder-decoder is the most used architecture for image captioning systems, and an end-to-end approach means both the encoder and the decoder are used in a tightly coupled manner. Hence the model is trained as a whole, offering good performance with less engineering effort. In contrast to other approaches that use spatial or global features and focus on the visual features alone, our approach uses the refined visual features extracted using ResNet-101. We then apply two attention mechanisms in the decoder, the first to the visual features to focus on the image's salient parts and the second to the textual features to generate captions with more detailed information.
High-level views of the proposed architecture is shown in Figure 1. The encoder of the proposed architecture consists of ResNet-101 [28] and a refining model [3], which smooths the representations of the extracted object features by modeling relationships among them. The extracted image features are then fed to the decoder.
The decoder part consists of two modules. The first is an attention-based recurrent module [32] used to attend selectively to the visual features extracted by the encoder. The second module is a reflective

The Encoder
As shown in Figure 1, given an image M, we first use a pre-trained ResNet-101 to extract a set of visual feature vectors (spatial features) from the last convolutional layer of the CNN. The extracted features are denoted by A = {a 1 , a 2 , . . . , a k }, a i ∈ R D , where k is the number of the extracted vectors. Inspired by [20], we also extract the global image feature vector A g , which are features taken from the last fully connected layer of CNN.
ResNet-101 is pre-trained on more than a million images from the ImageNet database [33] The residual network stacks layers to fit a residual mapping, resulting in greatly enriched feature vectors. Each layer in the network implements the following transformation: where: • A in is the input vector of the current layer.
• A out is the output vector of the current layer.
• F denotes the residual mapping function of convolutional layers. • W i is the parameters set. Figure 2 gives a detailed view of the encoder. The features A = {a 1 , a 2 , . . . , a k } extracted by ResNet-101 are fed into the refining network to refine their representation by modeling the relationships among them. The refining model consists of three individual linear functions Q (Query), K, and V (Key/Value pair) that perform linear projections of the features vector A. The output is then fed into a multi-head attention function (Att mh ) [34], followed by an attention-onattention (AoA) model [3], a residual connection [28], and finally layer normalization [35].
The multi-head attention module Att mh divides each Q, K, V into slices along the channel dimension, then applies the self attention function f Att dot to every slice Q i , K i , V i . Afterward, the result of each slice is concatenated to produce the final attended vectorV: V = f Att mh (Q, K, V) = Concatenate(head 1 , . . . , head H ).
The attention-on-attention model (AoA) measures how well the objects of the target image are related by measuring the relevance between the multi-head attention resultV and the query Q. After concate-natingV and Q, the model generates the information vector I and the attention gate G conditioned onV and the current context Q. To obtain the final attended information vectorĪ, the model uses element-wise multiplication between the attention gate G and the information vector I: where: • D is the dimension of Q and V.
• σ is the sigmoid activation function.
• is element-wise multiplication. After refining the extracted features, the feature vector A is updated to the refined features vectorĀ, without changing its dimension as follows: where LayerNorm is the layer normalization proposed in [35]: where x i j is the (i, j)-th element of the input.

The Decoder
Given the set of spatial features vector A = {a 1 , a 2 , . . . , a k } produced by the encoder, global image feature vector A g , and the embedding word vector W, the decoder aims to generate a caption S of n words, where S = {s 1 , s 2 , . . . , s n }. As shown in Figure 3, the decoder consists of two modules, an attentionbased recurrent module, and a reflective attention module. The first module accomplishes the main function of the decoder. It contains two LSTM layers and one layer of top-down visual attention (Att vis ), which determines the distribution of attention over all detected regions of the target image. where: • W e is the word embedding matrix for the one-hot Next is Att vis layer, the input is the generated state h 1 t , and the set of visual features A = {a i } k i=1 obtained from the encoder, the Att vis calculates the weight α vis t of the normalized attention distribution over all target image regions as follow: where: rv/hv denotes the learned embedding matrices t is a vector of attention probability scores of visual each feature a i at time step t, α vis t ∈ R 1×D The second layer of LSTM takes the Att vis attention resultâ t which defines the attended features, and the generated hidden state h 1 t from the first LSTM. The updating formula for the second LSTM layer units are Fig. 4 The reflective attention module architecture [4]. as follows: where h 2 t is the LSTM hidden state ∈ R D h . The reflective attention module (RefAM) consists of a reflective attention layer Att re f that selectively attends to the past hidden states and the second LSTM layer. As reflected in Figure 4, the Att re f layer takes the current hidden state h 2 t generated from the second LSTM layer and the set of all previous hidden states {h 2 1 , h 2 2 , . . . , h 2 t−1 } as an input to compute the normalized weight distribution α re f t over all generated hidden states: where: • α re f t is the set of the generated attention probabilities for all hidden states, α re f t ∈ R 1×D h . The weight α re f i,t measures the compatibility of the previously generated hidden state h i and the current hidden state to demonstrate the relevance between them. The reflective decoding then computes the attended hidden stateĥ 2 t ∈ R D h , which is used to predict the next word s t : p(s t |s 1: where: • D o is the captioning vocabulary size. Using the set of all generated hidden states to calculate the next word s t and model the dependencies between each word pair explicitly at different time steps is expected to increase word precision and produce higher quality captions.

Beam Search
Once the model is trained, it can be used to generate new captions in the inference stage. After feeding the <start> token to the LSTM, the generation process continues until the <end> token is reached. Beam search is used to select the next token in the evaluation stage, enabling the model to choose the sequence with the best overall score among several candidate sequences conditioned on the input image. At each decoding step, the search algorithm considers a set of top k sentences at time t as possible candidates to generate a sentence at time t + 1 (k is the beam size). This process is repeated until the sequence with the highest overall score is generated.

Experimental results
This section presents an overview of the datasets used in the experiments and the preparation steps for model input, that is, images and captions. We will then describe the training and evaluation processes, including the metrics used to train, validate, and evaluate the proposed approach. Finally, we present the experimental setup and settings, then discuss the results.

Datasets
We used the popular benchmark dataset Flickr30K dataset [36] to conduct our experiments. For training, validation, and testing, we used the popular 'Karpathy' splits setting [23] for Flickr30K. Flickr30K dataset has a training set of 29K images, a validation set of 1K images, and a test set of 1K images. All images from both datasets are unique and paired with five human-labeled captions.

Data Preparation
In order to train the image captioning model on massive datasets such as Flickr30K, the input data need to be prepared appropriately. To accelerate the training and evaluation process, we preprocessed the images and captions separately into an appropriate format.
We transform captions into a format ready to be fed to the language model. All captions are transformed into a list of tokens with a special start token <start> marking the beginning of the sentence, unknown token <unk> marking unknown words, pad token <pad>, and end token <end> marking the end of the sentence. All these tokens are converted into integers, where every unique word has an associated integer value in a dictionary file called word map. We then combine images and their paired captions into a parallel data structure that supports batching and buffering before starting the training, validation, and evaluation. We also save caption lengths before padding since the actual length is essential to avoid wasting time during the decoding process.

Model Training
Image captioning datasets contain a set of images labeled with captions that describe them in natural language. In the supervised training, the loss function is calculated by comparing the predicted caption with the ground truth caption word by word. Then all losses will be summed to obtain a single value, which needs to be minimized.
The cross-entropy loss L entropy is utilized for optimizing the proposed model in the training stage, which minimizes the negative log-likelihood as follows: log p(s * t |s * 1:t−1 ), (27) where M is the input image, and S * is the ground truth caption. The teacher forcing technique [37] is used in the training stage to speed up the training process of the recurrent neural network. In this method, the expected output from the training dataset (groundtruth caption) at time step y(t) will be used as an input in the next time step x(t + 1) instead of the previously generated word from the language model. This technique is commonly used in sequence-to-sequence generation models in applications such as machine translation, question answering, and image captioning. It solves the problems of instability and slow convergence during the training of recurrent networks that use the previously generated output as an input.

Model Evaluation
After training the model, we can use it to generate new image captions. We use four standard NLP automatic evaluation metrics to assess the quality of the produced captions: Bilingual Evaluation Understudy (BLEU) [38], Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [39], Metric for Evaluation of Translation with Explicit ORdering (METEOR) [40], and Consensus-based Image Description Evaluation (CIDEr) [41].
We implemented the proposed approach in Python and used Pytorch package as computational framework due to its popularity within the deep learning community and its CUDA parallel computing capabilities. Development and experiments were conducted on an Intel Core i7-9700K CPU with 32GB RAM and an NVIDIA RTX GPU 2060 with 8GB RAM. Google Colab cloud platform was used as the main CUDA platform for the computationally intensive parts of the experiments.
In the experiments, we evaluated the performance of several models in addition to the proposed ones. Starting from a simple baseline model, we incrementally added additional modules to test their effect on the model's performance. More precisely, we implemented, trained, and compared the following models: • Baseline: This is the baseline model which has a common encoder-decoder architecture (CNN-RNN). The encoder is ResNet-101, and the decoder uses one layer of LSTM.

Experimental Settings
We trained all models with the following settings: the features dimension for ResNet-101 is 2048. The number of heads in the multi-head attention is set to 8. For each LSTM layer, the hidden and word embedding sizes are set to 1000. The dimension of both attention layers (Visual attention and Reflective attention) is set to 512. The optimizer used is Adamax [42] with a learning rate of 0.002 for the LSTM language model. The number of epochs used for training is 50. We adopt the early stopping method introduced in [19], which uses BLEU evaluation metric to evaluate the model's performance on the validation set. Therefore, the model will stop the training if the value of BLEU score does not improve for 12 consecutive epochs; otherwise, the model will be trained for up to 50 epochs. Weight normalization is used o prevent the model from over-fitting. The dropout value is set to 0.5, and the batch size is set to 64. Finally, the beam search size is set to 5.

Result and Analysis
All experimental results are summarized in Table 3. The higher the value in each column, the better. We reported BLEU with four different N-gram lengths: N=1, N=2, N=3, and N=4. Table 3 shows the performance results on the Flickr30K dataset. First, as we can see, the baseline model has the lowest performance, which is expected since it only uses CNN-RNN without any modifications or additions. The visual attention module (VisAtt) improves the performance of the baseline model in all evaluation metrics (BLUEs, METEOR, ROUGE-L, and CIDEr). Upon adding the reflective attention module (VisAttRefAtt model), the performance improves further compared to using only visual attention. When we add the refinement module (Refin-ingVisAttRefAtt model ), we see a slight degradation in the BLEU score. However, the other evaluation metrics (METEOR, ROUGE-L, and CIDEr) show improvement.    Figure 5 shows the model attention when using feature refinement on a sample image from the Flickr30K test split. We visualize the attention weights used to generate each word in the caption. Each picture shows the generated word and the corersponding attention weights α t at each time step t.

Conclusion
Despite the significant progress made in the field during the last decade, it is clear from the literature that image captioning is still an active research area. Numerous approaches have been proposed recently to improve the performance of captioning systems and address existing issues. The challenges currently tackled by researchers include generating better quality captions, developing more sophisticated and powerful visual and language network architectures, and improving image captioning datasets and evaluation metrics.
In this line of research, we propose an encoderdecoder-based image captioning system with an encoder that extracts the spatial and global features and uses a refining model that models the relationships among objects in the target image. The decoder includes attention-based recurrent and reflective attention modules for better quality caption generation. The proposed approach takes the relationship between the target image objects and focuses on both the visual and the textual information by applying different attention mechanisms. In the experimental evaluation, we developed, trained, evaluated, and compared several models, including the proposed one. The experimental results show the effectiveness of the proposed approach.
As future work, we propose to focus more on the encoder side and try to learn more from the visual features by applying Faster R-CNN to learn the relationships between the different objects in the image.