{'fraction_non_alphanumeric': 0.029562013231722636, 'fraction_numerical': 0.010216899757139268, 'mean_word_length': 4.914809311540367, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VQA Model Based on Image Descriptive Paragraph and Deep Integration of BERT

Visual Question Answering (VQA) is a fast developing field involving multiple disciplines, and it is constantly challenging more complex tasks. The classic combination of CNN+LSTM can effectively extract images and language representation to complete the VQA task, but there are still many problems, such as excessively long sequence processing, etc. In recent years, BERT model has expanded rapidly from the field of natural language processing to a broader multi-modal field with its strong learning ability. In this paper, we propose a novel way to apply BERT model in the VQA field. We use the descriptive paragraph generation technology to transform the picture into a text paragraph description, and integrate question information and image information on BERT model. Our model achieves an excellent performance on the VQA2.0 dataset with an overall accuracy 5% higher than previous models.


Introduction
Along with the progress of natural language processing and image understanding, more complex and more demanding tasks can already be done. In the Stanford reading comprehension Dateset (SquAD2.0), machines have outperformed human performance. The most basic VQA is the joint application of CNN (Convolutional Neural Networks) and LSTM (Long Short-Term Memory). At present, in the field of natural language processing, transformer's BERT (Bidirectional Encoder Representation from Transformers) pretraining model has better performance than LSTM. The BERT (Bidirectional Encoder Representation from Transformers) model derived from the Transformer model is a general pre-trained language representation model first proposed by Google in 2017 [3]. The full name of BERT is. BERT greatly improved the benchmark performance of each NLP task by virtue of the strong feature learning ability of Transformer and Two-Encoding realized by the mask language model. The performance of BERT model on 11 NLP tasks broke the record [4], including Question Answering (SQuAD v1.1), reasoning Natural Language Inference (MNLI), etc. In addition to handling problem text information, the VQA also needs to handle information about the image. The Image caption technique first appeared in [1][2] to extract Image features by using some operators of Image processing. The emergence of dense captioning technology [14] further makes it more likely to develop the Image caption. We propose a method to integrate BERT model into the visual response field of VQA. Pre-trained language model BERT is added by fine-tuning to enhance the processing of language. With image processing, the visual question-and-answer task could be better completed and higher accuracy could be achieved. In our model, we innovatively combine the descriptive text generation technology with BERT model. Instead of inputting the image feature extraction into BERT for the model to learn, we applied the phased description hierarchy method of the generated image into a reading comprehension problem. The contribution of the article is two-fold: (1) The hierarchical method of phased description of images replaces the input of images to obtain more comprehensive and fine-grained image information. Text-text fusion is faster and more consistent than image-text fusion in the multiple models fusion of VQA. (2) The application of BERT model in VQA achieves faster speed and higher accuracy than LSTM + CNN model.

Visual Question Answering
Visual Question Answering (VQA) is a learning task involving computer vision and natural language processing. Its main goal is to make the computer output a reasonable answer that conforms to the rules of natural language according to the input pictures and questions. Great progress has been made in building VQA models using LSTMs [15] and convolutional networks [16].

Preprocessing for Natural Language Preprocessing
What is interesting about NLP field is that its pre-training technology lags far behind that of computer vision. There is some work beginning to improve word embedding, but it is only a low-level representation of the feature. Most importantly, different NLP tasks design different architectures. In the landmark work of Transformer [3], it is recommended to use the attention module in Transformer as a general module of NLP. Then many methods mainly used for training general representation were proposed such as GPT [7], BERT [4] and XLM [8], etc.

Image Caption and Intensive Captioning Techniques
The early [1,2] approaches of image caption relied heavily on image feature extraction and sentence generation rules. But the effect was not very satisfactory. Inspired by RNN in machine translation, Neural Talk [9] and Show and Tell [10] emerged that used similar techniques to extract image features to implement text generation and description. In recent years, a hierarchical method of generating descriptive paragraphs of images [11], which overcomes the feature that dense captioning technology cannot produce coherent stories, can generate paragraph-based picture descriptions.

Application of BERT in the Field of VQA
The current application of BERT to VQA is divided into single-stream and dual-stream models. VisualBERT [5] and VLBERT [6] are all single-stream models. In single-stream models, text information and visual information are fused at the beginning. And ViLBERT [12] is one of the most representative of the dual-stream model. However, it is unclear whether single-stream is better or double-stream is better. It seems that the result is related to specific tasks and more rigorous comparison experiments are needed for further verification.

Methods
Our basic idea is to use text alignment instead of image-text alignment, and the overview of out method is shown in Figure 1. The detailed method is based on the hierarchical method of image descriptive paragraphs, which converts images into descriptive text paragraphs of images, and then puts them into BERT's text reading model as paragraphs, designing fine-tuning to handle reading comprehension and question answering.

Generating Descriptive Image Paragraphs-Dense Captioning
The paper [11] proposed a hierarchical method model for generating descriptive paragraphs of images. Taking images as input, natural language paragraphs describing it are generated, and the structure is designed to make use of hierarchical text of images and paragraphs. It also has high interpretability. Descriptive paragraphs can be generated by only using a subset of the image. Because it can restore the original image information well, our model uses this method.

BERT's Working Principle:
BERT uses the encoder part of Transformer [3]. Transformer is a model that completely relies on the attention mechanism and can learn the contextual relationship between words in the text and discards the RNN cycle architecture. Transformer's prototype includes two independent mechanisms, an encoder is responsible for receiving text as input, and a decoder is responsible for predicting the results of the task. The goal of BERT is to generate language models, so only the encoder mechanism is needed. Figure. 3 shows the overall structure of BERT. The input is a token sequence. It is first embedding it as a vector, and then input to the neural network. The output is a vector sequence of size H.  Figure 3. The overall structure of Bert. When training a language model, it is difficult to define a prediction target. To overcome this challenge, BERT uses two training strategies, Masked LM and Next Sentence Prediction.

Masked LM(MLM)& Next Sentence Prediction (NSP)
In order to achieve deep bidirectional representation, MLM strategy is adopted. We randomly block some input tokens and then predict them. In order not to damage the language comprehension of the model, only 15% of the words in the sequence are replaced. NSP considers the problem from the perspective of sentences, and pre-trains a binarization next sentence prediction task, which can be easily generated from any monolingual corpus. In the training process, 50% of the input is contextual in the original document. While in the other 50% of the input, it is randomly composed from the corpus and is disconnected from the first sentence. This tactics solves the problem of how to obtain dependencies on sentences required in many NLP tasks. The mission achieved accuracy of 97-98%.

Modification of Fine-tuning End (Fusion of Hierarchical Method of Descriptive Paragraph of Image and BERT Model)
As shown in Figure 4, the input sequence of the model is the embedding corresponding to the sentence pairs. Sentence pairs consist of questions and text containing answers, and are separated by a special separator "[SEP]". As with other downstream tasks, the first token of the input sequence is a special classification embedded in "[CLS]", and the input sequence is the sum of token embedding, segmentation embedding, and position embedding.

Dateset
In a VQA task, given a natural image, the question is asked at the perception level and the correct answer is generated or selected according to the algorithm. We have done the experiments with the most widely used VQA2.0 [13] dataset. The VQA v2.0 dataset is split into train (82,783 images and 443,757 questions), validation (40,504 images and 214,354 questions), and test (81,434 images and 447793 questions) sets.. In VQA2.0, each image has an average of 5 questions and each question has 10 pre-selected answers. There is a one-to-one correspondence between images, questions and answers.

Comparison of Results
In order to verify the effectiveness of the method, several relevant models on the official VQA website [17] were selected and compared with the model in this paper on VQA2.0. The results are shown in Prior means to answer questions on the test set with the most common answers on the training set. Language-only which use a single LSTM architecture just use question to answer to forecast. d-LSTM+n-I is a basic visual question and answer model, Our-1 represents the improved model based on , Our-2 represents the improved model based on . In Table. 1, "yes/no", "number" and "other" correspond to the accuracy rate of the model's prediction under three different answers types respectively, and "overall" is the overall performance on the corresponding data set. As can be seen from Table 1, for questions with the answer of yes/no, the accuracy of the model in our-1 is about 80%. In the question with the answer of number type, the accuracy of our-1 is about 40%. In other type of questions, the accuracy of our-1 is about 57%. The model of our-2, which has a larger amount of pre-training, has been improved more significantly. In general, the accuracy of our models is about 5-10% higher than the accuracy of the model mentioned above. Since the hierarchical method is more able to restore the existence and action of objects in the picture, the accuracy of yes/no type and other type is greatly improved, while that of number type is not significantly improved. It can be seen that the idea of applying BERT to VQA field proposed in this paper is effective, and the improvement on accuracy is significant compared with previous models.

Conclusion
In this paper, we successfully applied BERT model which has an excellent performance in NLP field to VQA field based on the hierarchical method of descriptive paragraphs of images. We have demonstrated that the model performs well in VQA tasks. Compared with the traditional VQA model, the fusion model of BERT and VQA obtains faster speed and higher accuracy. It also has considerable applications in fields related to Natural-Language Question Answering and Computer vision. However, due to the limitation that the image description cannot fully restore the image, how to generate a more problem-oriented text description containing the image context is our further research direction.