{'fraction_non_alphanumeric': 0.02687211093990755, 'fraction_numerical': 0.007704160246533128, 'mean_word_length': 4.883248730964467, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Image Description Generation based on Deep Learning

The paper firstly explains the importance and research significance of image description generation and makes a brief research summary on the generation of various algorithms for image description; it further introduces the principle and the algorithm process of the LSTM deep learning algorithm in details, specifically describes establishing steps and the model parameter selection process of the LSTM deep learning algorithm of the Microsoft COCO dataset, implements the deep learning models by employing Python, and finally applies the trained LSTM deep learning model to image description generation by using the Microsoft COCO dataset as an instance.


Introduction
With the rapid development of Internet, the amount of information and complexity of media such as pictures and videos on the Internet have also rapidly increased, which means that it becomes more and more difficult for users to find the information they need in the vast information. If users want to easily acquire required media materials, they need to describe the media in details. However, due to the explosive growth of videos or pictures, the work amount required to manually add corresponding description tags is too huge and has become impossible. Therefore, the computer is required to help the completion of the work. Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. Figure1 is a schematic diagram of image description generation. Deep learning is a field of machine learning that is based on learning several layers of representations, typically by using artificial neural networks. Through the layer hierarchy of a deep learning model, the higher-level concepts are defined from the lower-level concepts [1].
Since Hintonet al. (2006) [2] introduced an efficient way of training deep models and Bengio(2009) [3]showed the capabilities of deep architectures in complicated artificial intelligence tasks, deep learning has become an emerging topic in computer science. Currently, deep learning approaches produce the state-of-the-art solutions to many problems in computer vision, natural language processing, and speech recognition. but it is mostly developed only in a single function in a simple scene. In recent years, researchers have begun to change their research directions to complex cross scenarios. For instance, in the field of image description generation in this paper, the model is required to accurately recognize objects in the picture and describe the relationship between objects using the natural language, which is a cross research direction combining computer vision and natural language processing [4].

Template-based algorithm
According to the template-based algorithm, local features of the image, including the types and attributes of the objects recognized in the image, are firstly extracted. Then the extracted types and attributes of the objects are put into a pre-written image description template, and certain rules are written to select the optimal description content therefrom. The algorithm is advantageous in that the main content of the image can be reflected, but the generated natural language description is over-rigid, with unvaried contents and lacking in detail attributes.

Search-based algorithm
According to the search-based algorithm, the initial step is similar to that of the template-based algorithm. Similarly, local features of the image are extracted, including the types and attributes of objects recognized in the image. Then the extracted features are combined into a feature library, the similarity is measured by employing the similarity algorithm, the image with the highest feature similarity is selected, the image description thereof is merged, and a sentence of description is finally output. The disadvantage is that the algorithm depends too much on the feature library, with low generalization ability.

Encoder-decoder-based algorithm
The encoder-decoder-based method uses the Recurrent Neural Network (RNN) (Yasrab,2018) [5]. According to the method, firstly image data is encoded into eigenvectors with a fixed length through the Convolutional Neural Network (CNN), wherein the CNN acts as the encoder; then these fixed-length eigenvectors are transferred into the RNN as input, and the information transferred by the CNN is decoded and converted into the natural language, wherein the RNN acts as the encoder. This method features strong generalization ability, varied sentences and high performance. This method is used in this paper.

Introduction to LSTM Deep Learning Algorithm
LSTM deep learning algorithm, proposed by Jürgen Schmidhuber and Sepp Hochreiter in 1997., is an improved RNN algorithm which remedies the defect that the RNN is unable to capture the long-term memory. Thus, the algorithm is tremendously suitable for classification and prediction tasks based on time series data.

Basic principle of RNN
A recurrent neural network (RNN) is a class of artificial neural networks that make use of sequential information [6]. An RNN is specialized to process a sequence of values x (0) ,x (1) ,...,x (t) . The same task is performed on every element of a sequence, while the output depends on the previous computations. In other words, RNNs have internal memory that captures information about previous calculations.
The input, output, and A neural network that allows information to be transferred from a certain step to the next in the network. An RNN can be seen as multiple times of copy and paste of the same simple network. The information is continuously transferred in the network, so it can be considered that the information has a certain degree of persistence. For instance, can predict, but if and are far from each other, cannot act on, therefore, the RNN needs to be improved.

LSTM -improved RNN
Despite the fact that RNNs are designed to deal with long-term dependencies, vanilla RNNs tend to suffer from vanishing or exploding gradient When backpropagation trains the network through time, the gradient is passed back through many time steps, and it tends to vanish or explode. The popular solutions to this problem are Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) architectures. LSTM, having a similar structure to that of RNN, is made to solve the problem of long-term memory loss of the RNN. However, LSTM mainly differs from the RNN in the part of the intermediate neural network A. Firstly, the following symbol descriptions are given: LSTM not only has the hide layer of the RNN, but also has another hidden layer cell state, namely The core idea of the LSTM is to control cell state, namely the retention or discarding of through three gates referring to the input gate , the forget gate , and the output gate . Each of the three gates is composed of a sigmoid neural layer and a dot product unit. The gate outputs 1 or 0 through the sigmoid function, wherein 1 represents completely retaining information, and 0 represents completely discarding information.
Step 1: The information firstly flows to the forget gate and updates its output: Step 2: The step is composed of two parts, firstly the input gate determines which values need to be updated to , and secondly the tanh function generates a new candidate value vector , and whether the value is input to depends on the situation. Two parts of the output of the input gate are updated: Step 3: selected through the forget gate is multiplied by , selected by the input gate is multiplied by the candidate value vector , the two products are multiplied to form the current state , and the cell state is updated: Step 4: Finally, determine which information needs to be retained, passes through the output gate and is then multiplied by , and finally the output is formed. The output of the output gate is updated:

Establishment of Image Description Generation Model
The image description generation model in this paper is based on the encoder-decoder method by using the RNN [7]. According to the method, firstly image data is encoded into eigenvectors with fixed length through the CNN, wherein the CNN acts as the encoder. Then the fixed-length eigenvectors are transferred into the RNN as input, and the information transferred by the CNN is decoded and converted into the natural language, wherein the RNN acts as the encoder.
The objective function of the model is to maximize the probability of correct description, wherein is the hyperparameter, is the image eigenvector, is the correct description, and the objective function is as follows: During training, is the input training data pair, and the paper adopts the Adam method to optimize and adjust parameters during maximization of the objective function. The objective of the LSTM neural network is to train it so that when the feature data of the image is input, the sentence corresponding to the image, such as , can be predicted. Considering the LSTM in an expanded form is beneficial to achieve this, therefore, multiple LSTM memory units are copied so that each word of the sentence is received by the corresponding LSTM, and all the LSTMs share the same parameters and the output .

Training and Experiments
Footnotes should be avoided whenever possible. If required they should be used only for brief notes that do not fit conveniently into the text. The content includes the name of the image file, the context relationship between targets, the location coordinates of the targets, etc. Also, the file is a json file and most tag contents are not needed, so the data extraction tool provided by Python is adopted. The relevant content from the dataset can be easily extracted with the tool, and only the file name and the description are retained.  Conversion of uppercase and lowercase letters.The presence of uppercase and lowercase in the original dataset can lead to the condition that "hello" and "Hello" are regarded as two words just because they have different first letters, which is inconvenient for subsequent data conversion. In the case, all letters should be converted to lowercase by the adoption of the built-in uppercase to lowercase script of the open-source OpenCC converter.   Filtering of stop words.A preprocessing method in text analysis is stop word filtering. The function is to filter out some functional words, namely, noise. The functional words are extremely common and have no practical meaning compared with other word. In this way, the storage space can be saved, and the search efficiency can be improved. The specific method of filtering out stop words includes loading functions of the new user-defined dictionary, iterating over the stop word list and performing filtering. The specific implementation is shown in the codes.

About Microsoft COCO Dataset
 Removal of low-frequency words.The description of the image is one sentence, and there are many words in the sentence. If a certain word rarely appears in the descriptions of all images, the word should be regarded as an outlier and removed from the training data. The model in the paper is to be used for prediction so that the probability of model errors can be reduced. There are 8,763 non-repeated words in 72,783 training images. It can be seen that words such as 'in', 'the', and 'on' appear frequently, while words such as 'rockstar', 'foolish', and 'stereoscope' scarcely appear, so we can remove these words as outliers. If the words which appear less than 10 times are removed, we can get 1,651 non-repeated words.  Thesaurus. For convenient passing to the LSTM network, each of the 1,651 non-repeated words after removing the low-frequency words is replaced by an integer, that is, 0-1650 represents 1,651 words.

Image data preprocessing
The field of deep learning depends heavily on data driving. For CNN models such as Inception V3, only a company like Google has enough hash rate to complete the training of such a complex model. The Inception V3 model is obtained by training on the Imagenet dataset, hyperparameters have been persistent, and the parameters can be well applied to other datasets through simple adjustment. The method is called transfer learning. The Inception V3 model is obtained through training of the Imagenet dataset and established with the target of image classification, so the last network of the final Inception V3 model is a 1,000dimensional softmax bottleneck layer which can classify images into 1,000 categories. However, the objective of this paper is not image classification, so the last layer of structure can be removed, and only the input layer to the layer before the output layer is retained. The preprocessing steps are as follows: #  target_size=(299, 299)) # Converting the PIL image to the three-dimensional numpy array x = image.img_to_array(img) # Add another dimension x = np.expand_dims(x, axis=0) # Preprocessing the image by employing preprocess input () from the inception module x = preprocess_input(x) # Converting the vector from the shape (1,2048) to (2048,) x = np.reshape(x, x.shape[1]) Figure 4. Schematic Diagram of Image Data Pre-processing.

Image and description data merging and preprocessing
The section mainly introduces how to merge images and description data, input the merged data into the model in a simple way, and finally train the model through the data.  Figure 6. Training Image 2.
The above two images are two training images, the first one is described as 'The black cat sat on grass', and the second one is described as 'The white cat is walking on road'. The images can be easily segmented into one word through the split function of Python and then put into the vocab list, wherein vocab = {black, cat, endseq, grass, is, on, road, sat, startseq, the, walking, white}, so the non-repeated corpus formed by descriptions of the two images can be obtained. The descriptions of the above pictures are put into the non-repeated word lexicon for retrieval, and the result is shown in Table 6. In order to use the data in the neural network, the problem of image description generation should be regarded as a supervised learning problem to solve, refers to the eigenvector and partial description of the image. The objective of the model is to generate description, so the generated natural language description is the objective function.
is the objective function, namely, the next word. Above is the simplest case, namely, there are only 2 images for combination. In the real test, there are 6,000 training images. Each image has 5 descriptions, which is equivalent to 30,000 images, and the descriptions have an average of 8 words, which is equivalent to merging of 210,000 data into the vector.    The optimal parameters are determined by different iteration numbers. From the above table, it is obvious that the training error is smaller and smaller as the iteration number increases; and when the iteration number of the model reaches 91 rounds, the training error of the model has dropped to 0.076408. It is considered that the time consumption of each iteration is huge and the decreasing training error gradually decreases, so the current parameters are saved as the final parameters of the model, as shown in Table 10.

Conclusion
According to the paper, in the process of establishing the image description generation model, firstly the input corpus data is processed in details, then the image features are extracted through the excellent pretraining CNN model Inception V3, and the problem of slow network convergence is optimized and avoided by employing the Adam method in establishing the model. From the inspection of random network images, the prediction precision is still within a reasonable range so that the prediction results have a certain reference value. The image description generation model in the paper also has a certain reference applicability. However, considering that the training data is not huge enough and the sample input is still inadequate in the paper, so the generation description of the images that appear less frequently in the training set is inaccurate.