{'fraction_non_alphanumeric': 0.04381487641222687, 'fraction_numerical': 0.013993232780868269, 'mean_word_length': 4.8890577507598785, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention

Vision-and-language (V-L) tasks require the system to understand both vision content and natural language, thus learning fine-grained joint representations of vision and language (a.k.a. V-L representations) is of paramount importance. Recently, various pre-trained V-L models are proposed to learn V-L representations and achieve improved results in many tasks. However, the mainstream models process both vision and language inputs with the same set of attention matrices. As a result, the generated V-L representations are entangled in one common latent space. To tackle this problem, we propose DiMBERT (short for Disentangled Multimodal-Attention BERT), which is a novel framework that applies separated attention spaces for vision and language, and the representations of multi-modalities can thus be disentangled explicitly. To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format. In this manner, visual concepts help to bridge the gap between the two modalities. We pre-train DiMBERT on a large amount of image-sentence pairs on two tasks: bidirectional language modeling and sequence-to-sequence language modeling. After pre-train, DiMBERT is further fine-tuned for the downstream tasks. Experiments show that DiMBERT sets new state-of-the-art performance on three tasks (over four datasets), including both generation tasks (image captioning and visual storytelling) and classification tasks (referring expressions). The proposed DiM (short for Disentangled Multimodal-Attention) module can be easily incorporated into existing pre-trained V-L models to boost their performance, up to a 5% increase on the representative task. Finally, we conduct a systematic analysis and demonstrate the effectiveness of our DiM and the introduced visual concepts.


Image RoIs + Visual Concepts
Sentence Words Conceptual Captions [67] BLM + S2SLM generate fine-grained V-L representations [12,52,55,69]. However, many existing systems are task-specific models, focusing on individual tasks only. As a result, learning universal V-L representations and empowering models with the ability to adapt to a wide range of downstream V-L tasks are now becoming the critical topics in current V-L research.
Inspired by the successful pre-training models like ResNet [24] in computer vision and BERT [15] in natural language processing, several attempts [1,55,69,73,97] have been conducted to learn such universal V-L representations. Table 1 summarizes some representative works in image domain. As we can see, most systems adopt the Transformer framework [15,74] as the backbone, feeding both visual and textual features into the same stacked transformers. In the pre-train stage, these systems use large-scale image-sentence pairs to adapt the transformers to the V-L scenarios; In the fine-tune stage, the transformers are further optimized for downstream tasks. Although these pre-training V-L systems receive performance gains across multiple downstream tasks, there are still some potential directions for further improvement: • Entangled Attention: Existing works feed both the visual and textual features into the same set of stacked transformers. In this manner, the same set of attention matrices are used to transform both visual and textual embeddings.
We denote such type of attention mechanism as Entangled Attention, because the visual and textual embeddings are projected to one common latent space. Despite the pre-training on image-sentence pairs, the initial parameters of current systems [12,39,44,69,97] are usually directly inherited from BERT [15] or BERT-based models, e.g., UniLM [16], which is optimized for language modeling only 1 , to boost the performance. On one hand, it could be in-appropriate to apply the parameters that are trained in language modality to the features in visual modality; On the other hand, the ability of language modeling brought by the original BERT could be somewhat affected by the introduced visual modality.
Besides, during parameter optimization, the model needs to consider the intra-relationships of both visual and textual embeddings, as well as the inter-relationships across V-L embeddings. Modeling these three kinds of relations with only one shared set of attention matrices could be insufficient.
• Modality Gap: Most V-L systems only use the region-of-interests (RoIs) / video frames as the visual features.
Although the transformers are proved to be effective in mining correlations, there are still huge gaps between the visual and language modalities.
• Generation Task: Most systems can only be fine-tuned directly on classification tasks, lacking the capability to handle generation tasks. Although VideoBERT [71] and CBT [70] have been proposed to support the generation tasks in video domain, they have to train a separate video-to-text decoder to perform the generation tasks, because they pre-train the V-L systems only as encoders.
To tackle the above three concerns, in this paper, we present the DiMBERT, which takes both visual features (i.e., RoIs and visual concepts) from images and textual features (i.e., sentence words) from sentences as input, and then applies a single cross-modal Transformer to learn vision-language grounded representations. In particular, we propose Disentangled Multimodal-Attention (DiM) module to explicitly disentangle visual and textual modalities. In implementations, DiM module introduces separate projection matrices to project visual and textual modalities into their corresponding visual and textual latent spaces. Following common practice [1,12,39,44,69,97], the weights of textual projection matrices are initialized with the pre-trained parameters from BERT [15], while the weights of visual projection matrices are trained from scratch.
To enhance the correlation between visual and textual modalities, we introduce the visual concepts [18], which capture a wide range of high-level visual semantic information from images [41,63,89]. The visual concepts transform visual features to a set of words describing object (e.g., cat), attribute (e.g., small) and relationship (e.g., standing) of images, which 1) provide a more semantic representation of visual information and thus help shorten the gap between vision and language modalities; 2) contain rich visual semantics and thus help understand vision and language effectively.
Inspired by the work of [97], we pre-train the proposed DiMBERT on the Conceptual Captions [67] with two unsupervised language modeling objectives: bidirectional language modeling [15] and sequence-to-sequence language modeling [16], where the latter enables the direct fine-tuning of DiMBERT on generation tasks. We conduct comprehensive experiments and systematic analysis on three tasks: image captioning [11], visual storytelling [28] and referring expressions [33]. The proposed DiMBERT sets new state-of-the-arts on three tasks (over four benchmark datasets) and the DiM module boosts the performance of various pre-trained V-L models on referring expressions task, which validate our motivation and corroborate the effectiveness and universality of our approach.

RELATED WORK
Our work relates to the vision-and-language problems, the joint representations of vision and language (a.k.a. V-L representations) and the efforts in developing pre-trained models.

Vision-and-Language Problems
Vision-and-language (V-L) problems, which including image captioning [11], visual storytelling [28], referring expression [33] and image caption retrieval [61], and others, have drawn remarkable attention in both natural language processing and computer vision. These tasks combine image and language understanding together at the same time, are tough yet practical. However, current works usually design a task-specific model to deal with one single task at one time. In this paper, we propose a generic framework to conduct various V-L tasks and further improve the performance of each task.

V-L Representations
For a variety of vision-and-language problems, an important goal is to understand the image and language despite their different application scenarios, which justifies the acquisition of fine-grained V-L representations. In the literature, to represent the images, visual features extracted by CNNs or Region-CNNs are most-widely used [3,83], while visual concepts consisting of a set of textual words are also proposed [18]. To represent the language, textual features extracted Manuscript submitted to ACM by RNNs or off-the-shelf NLP models are most-widely used [15,17,25]. Therefore, in previous task-specific V-L models, the features derived from off-the-shelf computer vision and NLP models are combined in an ad-hoc way to acquire the V-L representations for specific tasks. Model training is performed on the dataset for the specific task only, without any generic V-L pre-training. As a result, these task-specific models, which are directly trained for the specific target task, may well suffer from overfitting when the data for the target task is scarce.

Pre-trained Models
In computer vision (CV), pre-trained models, such as ResNet [24], VGG [68] and GoogLeNet [72], pre-trained on ImageNet [14], have achieved early successes in promoting various downstream CV tasks. Transformer-based pretrained NLP models, such as BERT [15], XLNet [87] and RoBERTa [16], have also achieved great success in advancing the state-of-the-arts for a wide range of NLP tasks. Recently, UniLM [16], which is adapted from the BERT architecture, has been proposed to enable BERT to work for both natural language understanding and generation tasks.
Most recently, several pre-trained V-L models [ [71], CBT [70] and VLP [97] proposed recently, are capable of performing generation tasks, and are thus the most relevant works to our approach.
However, they were still entangling visual and textual modalities, and did not attempt to take the visual concepts into consideration. Besides, for VideoBERT [71] and CBT [70], they pre-train the systems only as encoders to learn V-L representations, so they have to train a separate decoder for the generation tasks. For VLP [97], our DiMBERT outperforms it on image captioning tasks and our superiority is further validated on a long text generation task, i.e., visual storytelling [28].

APPROACH
In this section, we first introduce our model in detail. Next, we describe the pre-training tasks to learn vision-language grounded representations. Figure 1 gives an overview of our DiMBERT.

Model Overview
As shown in Figure 1, our DiMBERT consists of three parts: 1) the embeddings of input visual features from images and textual features from sentences; 2) a single cross-modal transformer to learn the alignments and relationships between visual and textual modalities; and 3) the embeddings of learned V-L representations.  [81], which is pre-trained on Visual Genome [37], following [3].
Specifically, the appearance feature ∈ R is the extracted region feature, which is the output of fc6 layer. The visual geometry feature ∈ R is used to encode the geometry location of the RoI in the input image, where a person holding a red umbrella wearing a yellow rain coat. [MASK] umbrella Input Embeddings

Cross-Modal Transformer
Output Embeddings person Visual Concepts: "woman", "umbrella", "holding", "yellow", … , "wearing", "red" Visual Concept Embedding. Visual concepts contain rich visual semantics, and have been used to provide explicit high-level semantic information of an image [78]. Following [18], we adopt a weakly-supervised approach of Multiple Instance Learning [93] to build the visual concepts extractor, which is trained on the MSCOCO caption dataset for 1,000 visual concepts. For each image, only the top = 20 visual concepts are selected. We sort these extracted visual concepts by prediction scores, which means that the position embedding of each visual concept indicates its relevance to the image. Following BERT, the visual concepts use the Word-Piece embeddings [79]. They will then be further combined with position embeddings and a segment token [CEP]. The result is denoted as the visual concept embedding ∈ R model .
Sentence Word Embedding. Similar as in the visual concept embeddings, we tokenize the sentence into WordPieces [79]. After that, a position embedding and a segment token [SEN] embedding are assigned to each sentence word, where the position embedding indicates its order in the input sentence. We denote each sentence word embedding as ∈ R model . and [END] are inserted at the first and last position, representing the start and the end of the sentence, respectively.
[SEP] token is added as the boundary between two different input segments, and [MASK] token indicates the random masked-out word, which need to be predicted by DiMBERT based on all the other available elements, including RoIs, visual concepts and available sentence words. Thus, the input embeddings can be written as: where the , and are the sets of related vectors.

Single Cross-Modal Transformer.
Our DiMBERT is adapted from BERT BASE , thus the backbone of our approach is the 12-layer ( ) Transformer [74], with 768 hidden units ( model ) in each layer. In implementation, we propose the Disentangled Multimodal-Attention (DiM) to replace the Self-Attention (SA) in Transformer. So we hereby describe the difference between DiM and SA. First, we denote the intermediate representations of -th layer as = , , (the representations of four special tokens are omitted for conciseness). And we find that extracted from the image belongs to the visual modality V , while extracted from the image and extracted from the sentence belong to the textual modality T , which we write as = V , T .
Self-Attention (SA). In current pre-trained V-L systems [12,39,44,69,97], to learn the alignments and relationships between the visual modality V and textual modality T , they adopt SA, which consists of = 12 parallel heads with each head SA defined as: where T , T , T are parameters initialized with pre-trained parameters from BERT, which are pre-trained on text data only. Besides, the divisor √︁ ( = model / = 64) is omitted in equations for conciseness, please see [15,74] for details. As we can see, SA first projects visual and textual modalities into one common latent space, resulting in the entanglements between the two modalities. After SA, they use a position-wise feed-forward network (FFN) [15,74], which keeps on processing the features with mixed modalities and thus loses the capability to learn the relationships between the two modalities [95]. After that, the intermediate representations of ( +1)-th layer +1 are obtained by: Disentangled Multimodal-Attention (DiM). As we can see, due to the entanglements between the visual and textual modalities in SA, the models have to devote most of its capability on disentangling them, which makes it hard for systems to learn the relationships between visual and textual modalities efficiently. To this end, we propose Disentangled Multimodal-Attention (DiM) to explicitly disentangle visual and textual modalities. In implementation, the DiM also consists of parallel heads but with each head DiM defined as: where T , T , T are initialized with UniLM [16] parameters pre-trained on text data only; and the V , V , V are new learnable parameters (randomly initialized). After that, following SA, we obtain the intermediate representations ). The reason that we adopt the proposed Disentangled Multimodal-Attention is to learn the alignments and relationships between visual and textual modalities in a more efficient way.

Output Embeddings.
After multiple DiM layers, we use the output of last layer as the output embeddings. ( , and denotes RoIs, concepts and sentence representations, respectively)

Pre-Training Tasks
In our work, we pre-train DiMBERT on the training split [97] of a large scale image-sentence dataset: i.e., Conceptual Captions dataset [67], which contains around 3.3M image-sentence pairs. To pre-train DiMBERT, we introduce two Manuscript submitted to ACM unsupervised language modeling tasks, which are adapted from the masked language modeling (MLM) task: 1) bidirectional language modeling (BLM) [15]: which learns to predict the randomly masked sentence words based on all available input information, i.e., the visual and textual features; 2) sequence-to-sequence language modeling (S2SLM) [16]: which learns to predict the randomly masked sentence words based on partial input information, i.e., all the visual features and the sentence words on the left side of the word to be predicted in the sentence, which satisfies the auto-regressive property and enables our DiMBERT to perform downstream generation tasks. In this section, we will describe these pre-training task in detail.
Masked Language Modeling (MLM). Following BERT, during the pre-training stage, we randomly mask out the input sentence words with 15% probability, replacing the word with 80%, 10% and 10% probabilities of [MASK] token, random word and original word, respectively. Thus, the objective of MLM is to predict the randomly masked sentence word based on the available information. We denote trainable parameters as , and a pair ( m , a ) of input with the masked word as m , and all available elements as a , which are sampled from the training set . The MLM is trained via minimizing negative log likelihood, defined as: where the masked tokens are predicted as a classification problem.
Bidirectional Language Modeling (BLM) and Seq-to-Seq Language Modeling (S2SLM). The main difference between two LM tasks is the different portion of available information that can be used to predict the masked word m . For BLM, as in BERT [15], the model is allowed to use all the input embeddings on both left and right side of the [MASK] token. Thus a consists of all RoIs , all visual concepts and all other sentence words \ m . The loss function is defined as: In implementation, the two LM tasks, which are alternated with random sampling, participate in the pre-training stage at a ratio of 25% and 75%, respectively. We pre-train DiMBERT on 8 GPUs (Tesla V100) with a batch size of 512 for 30 epochs. We use the Adam optimizer [36] with initial learning rates of 3e-4. Through pre-training on Conceptual Captions dataset, our model is capable to learn vision-language grounded representations. Following VisualBERT [44], to let our DiMBERT better adapt to the downstream target domains, we further pre-train DiMBERT using the data from downstream tasks. Eventually, we get the final pre-trained model by averaging the last 20 checkpoints.

EXPERIMENTS
We evaluate DiMBERT on three representative vision-and-language tasks, i.e., two generation tasks (image captioning [11] and visual storytelling [28]) and a classification task (referring expressions [33]

Image Captioning
The task of image captioning aims to generate a descriptive sentence for an input image and has received extensive research interests.
Datasets and Metrics. We use the popular Flickr30k [90] and MSCOCO [11] datasets to evaluate our reported results.
The datasets contain 31,783 images and 123,287 images, respectively, with 5 sentences paired to each image. To make fair comparisons [27,86,97], we use the widely-used splits in the work of Karpathy and Li [32] to report our results.
As a result, there are 5,000 images each in the validation set and the test set for MSCOCO, and 1,000 images as for Flickr30k.
We test the model performance with MSCOCO captioning evaluation toolkit [11], which reports the widely-used automatic evaluation metrics SPICE [2], CIDEr [75], ROUGE [49], METEOR [5,48] and BLEU [64]. SPICE is based on scene graph matching and CIDEr is based on n-gram matching. They both incorporate the consensus of a reference set for an example. These two metrics are specifically designed for the evaluation of image captioning systems. ROUGE is proposed for automatic evaluation of the extracted text summarization. METEOR and BLEU are originally designed for machine translation evaluation.
Fine-Tuning and Inference. Figure 2 illustrates the details of fine-tuning. As we can see, we apply the Seq-to-Seq  Table 2. Comparisons with the state-of-the-art task-specific models and pre-training models on various downstream tasks, i.e., image captioning (Flickr30k and MSCOCO datasets (with CIDEr optimization)), visual storytelling (VIST dataset) and referring expression (RefCOCO+ dataset). B-4, M, R, C and S are short for BLEU-4, METEOR, ROUGE-L, CIDEr and SPICE, respectively. For referring expression task, following common practice [12,69], we evaluate on both ground-truth RoIs (val, testA, testB) and detected boxes (val , testA , testB ) provided by [92]. The models of referring expression are evaluated in terms of accuracy (%).  Results. We compare our DiMBERT with two types of existing works: 1) The state-of-the-art task-specific models like GVD [96] on Flickr30k and AoANet [27] on MSCOCO image captioning datasets.

RefCOCO+
2) The pre-training based models, like VLP [97]. The results are shown in Table 2. As we can see, the proposed DiMBERT outperforms all baselines across all metrics over the board, which demonstrates the capability of DiMBERT to achieve consistent performance gains over different datasets.
We also evaluate our DiMBERT on the online MSCOCO evaluation server 2 , where the ground truth captions are not available. We compare with the top-performing entries on the leaderboard whose methods are published, which including AoANet [27], SGAE [86], ETA [40]. For online evaluation, nearly all of the recent submitted systems use model ensemble [3,27,31,86,88]. From Table 3, we can find that our DiMBERT is able to outperform these state-of-the-art models across all metrics over the board, in a single model submission. It further demonstrates the effectiveness of the proposed DiMBERT.

Visual Storytelling
Visual storytelling task belongs to long text generation tasks. The goal is to generate a reasonable and coherent paragraph-level story based on the image stream.
Datasets and Metrics. In visual storytelling, our reported results are evaluated on the VIST dataset [28], which contains 210,819 unique images in 10,117 Flickr albums. Each sample contains one story, describing five selected images. We follow the standard split [77] for fair comparisons. There are 40,098 / 4,988 / 5,050 images for training / validation / testing, respectively. Following common practice [41,77,85], we adopt four evaluation metrics, including BLEU, ROUGE, METEOR and CIDEr, for the evaluation of our approach.
Fine-Tuning and Inference. For the visual storytelling task, we adopt the same fine-tuning and inference strategy as the image captioning task. These two tasks differ solely in the input format. As illustrated in Figure 2, the visual storytelling task takes five images as input. For each image, we first extract corresponding RoIs and concepts. Then we feed the embeddings of these RoIs and concepts into DiMBERT from Image 1 to Image 5.
Results. We choose two recently proposed state-of-the-art models AREL [77] and VSCMR [41] for comparison. As shown in Table 2, the DiMBERT outperforms the AREL and VSCMR by a large margin of 46.8% and 53.3% in terms of CIDEr scores, respectively, and sets a new state-of-the-art, which shows that the proposed DiMBERT also works well on generating long texts.

Referring Expressions
Referring Expressions belongs to classification tasks which aims to locate a target image region given a textual query.
Datasets and Metrics. We evaluate our model on RefCOCO+ dataset [33], which consists of 141k expressions for 50k referred objects in 20k images in the MSCOCO dataset [50]. The referring expressions in RefCOCO+ are forbidden from using absolute location words, e.g. right car. Therefore the referring expressions focus on purely appearance-based descriptions. RefCOCO+ is split into train, validation and two test sets (testA and testB). Specifically, images containing multiple people are in testA set, while images containing multiple objects of other categories are in testB set. Following common practice [12,69], we evaluate on both ground-truth RoIs (val, testA, testB) and detected boxes (val , testA , testB ) provided by [92]. We choose MAttNet [92], ViLBERT [55], VL-BERT [69] and UNITER [12] for comparison, where the first one MAttNet is the state-of-the-art task-specific model, while the rest are pre-training models. The models are evaluated in terms of accuracy (%).
Fine-Tuning and Inference. As illustrated in Figure 2, we add a linear layer on the to output the classification scores for all the input RoIs, and take the RoI with the highest score as the final prediction. DiMBERT is fine-tuned under a binary cross-entropy loss on 8 GPUs with a batch size of 256 for 20 epochs.
Results. Table 2 shows that our DiMBERT outperforms all models, which validates the effectiveness of DiMBERT in referring expressions task. In particular, DiMBERT outperforms VL-BERT LARGE and UNITER LARGE which are developed from a larger model BERT LARGE [15], while DiMBERT is adapted from BERT BASE [15]. In other word, DiMBERT can achieve higher accuracy with much less parameters. Furthermore, in addition to Conceptual Captions dataset [67], UNITER LARGE uses extra a large amount of image-sentence pairs (see Table 1), e.g., VG Captions [37] and SBU Captions [62], to pre-train the model. This proves the efficiency of DiMBERT in pre-training datasets and parameters.
In all, these results suggest that the DiMBERT can be applied to a wide range of downstream tasks, no matter what the type of task is (e.g., classification task and generation task). More encouragingly, our approach outperforms existing Manuscript submitted to ACM published state-of-the-art models, including the task-specific models and pre-training models, across all aforementioned tasks, which further confirms the effectiveness and universality of the proposed DiMBERT.

ANALYSIS
In this section, we conduct a series of analysis on a generation task, i.e., image captioning (MSCOCO), and a classification task, i.e., referring expressions (RefCOCO+), to provide some insights and answer the following questions: (1) What is the contribution of each component in DiMBERT? (2) What is the effect of Disentangled Multimodal-Attention module?
(3) What is the contribution of visual concepts? (4) Why DiMBERT can adapt effectively to a wide range of downstream tasks? (5) Where does the actual improvement in the evaluation scores comes from? Please note that the performance on MSCOCO image captioning dataset reported in this section is different from the ones reported in Section 4: As for fair comparisons with state-of-art works [27,86], we further perform CIDEr-based training objective using reinforcement training [66]. While in this section, for simplicity, we directly do fine-tuning with cross entropy loss.

Ablation Study
In this section, we conduct the ablation analysis on RefCOCO+ and MSCOCO image captioning datasets. To analyze the effect of each component, we evaluate from the following three perspectives: • Parameter Initialization: the initial parameters inherited from BERT [15] or UniLM [16].  Table 4 shows the results in which the base model denote the one trained from scratch on downstream tasks without parameter initialization and pre-train, DiMBERT denotes the full model proposed in this paper. We can find that: • Initializing the parameters from existing models like BERT is beneficial, as the pre-trained language models could better capture the contextual representations and the structure of sentences.
Manuscript submitted to ACM • Comparing the results of (1) and (2) in

Effect of DiM Module
In this subsection, we evaluate the effectiveness of the proposed Disentangled Multimodal-Attention (DiM) mechanism.
We compare DiM with Entangled Self-Attention (ESA) which applies the same set of attention matrices to sentences, RoIs and concepts. As shown in Table 5 DiM can also be easily integrated into existing pre-trained models. In this section, we further equip UNITER BASE [12], VL-BERT BASE [69] and ViLBERT [55] with DiM. Specifically, we pre-train these models on the Conceptual Captions dataset and evaluate the accuracy on the referring expression task. As shown in Table 6, DiM can successfully boost all baselines, with the most significant improvement up to relatively 4%, 5% and 5% for val , testA and testB , respectively.
The significant improvements demonstrate the effectiveness and universality of the proposed DiM module.
Manuscript submitted to ACM

Effect of Visual Concepts
We conduct some experiments to investigate the contribution of visual concepts (ViCo) in our model. Table 7 shows that the visual concepts promote the baselines over all metrics, especially in Attributes and Color. The reason is that the visual concepts contain more attribute words and color words than the sentence. The abundant visual semantics in visual concepts can greatly enrich semantic representations of image regions. It can be confirmed in Figure 3 (c), which illustrates the image representations refined by DiMBERT w/o ViCo. Compared with the image representations refined by DiMBERT (Figure 3 (b)), it is obvious that the DiMBERT w/o ViCo is insufficient in providing suitable semantic information for image regions. It is worth noticing that given the absence of input sentence, most existing models won't work well, but our model can still refine the image representations. The Figure 3 (d) shows the semantic-grounded image representations [52] refined by our DiMBERT without input sentence. Table 8 shows that when the number of visual concepts is 20, the visual concept extractor and the model get the highest F1 score and highest performance, respectively, which is the reason why the value of is set to 20 in our DiMBERT. For other variants, we speculate that when is set to small values (lower recall score), the model will suffer from the inadequacy of information. When is set to large values (lower precision score), the module will introduce more noisy information.

Visualization of DiMBERT
To evaluate the effectiveness of DiMBERT, we visualize the alignments between visual regions and semantic words according to the attention weights in the last Disentangled Multimodal-Attention layer. Figure 3 (a) and (b) show the examples of vision-language representations learned by our DiMBERT. As we can see, the model provides visual references for the input sentence, e.g., the sentence word man is aligned to the blue, yellow and red regions. The visual-referred sentence representations play an important role in understanding the sentence correctly, because the visual references help alleviate semantic ambiguity, e.g., the word bank can either refer to a financial organization or Sentence A man with an umbrella walks in the rain.

DiMBERT
Sentence A man with an umbrella walks in the rain.

Visual Concepts
"umbrella", "rain", "wearing", "man", "walking", "blue", "holding", "tie" Specifically, (a) and (b) show that DiMBERT can provide visual-referred sentence representations and semantic-grounded image representations, respectively; (c) shows that the model is insufficient in providing suitable semantic information for regions without visual concepts; (d) shows that DiMBERT still works well even without input sentence.

Reference:
A pretty woman in a white bikini holding a surfboard over her head.
DiMBERT: A man is holding a white surfboard and standing on the beach.
Reference: A large dog sitting in a living room in front of a fireplace.
DiMBERT: A dog laying on a living room floor near a clock.

1) Baseline:
A table standing next to a green chair.

2) DiMBERT:
A bunch of pink flowers on wall with a wooden table and chairs nearby.

3) w/o ViCo:
A table on a field with chairs and flowers nearby.

4) w/o DiM:
A bunch of pink flowers sitting on a wooden table.

1) Baseline:
A dog holding a frisbee standing on a field.

2) DiMBERT:
A brown dog walking across a green field carrying a white frisbee in its mouth.

3) w/o ViCo:
A dog carrying a frisbee in its mouth on top of a grass field.

4) w/o DiM:
A brown dog catching a white frisbee. the side of a river, and the word mouse can either refer to a mammal or an electronic device. Similarly, the model can provide a clearer semantic information for image regions, e.g., the red region is aligned to the words wearing, blue and tie. Thus, the original image representations are refined to semantic-grounded image representations [52]. Note that there is no suitable semantic word in the sentence to align with the red region, thus DiMBERT does not assign too much attention weights for any sentence word, which also indicates the effectiveness of our model when it comes to insufficient sentence words.
The refined semantic-grounded image representations and visual-referred sentence representations are beneficial for understanding both image and sentence, providing a solid bias for vision-and-language tasks.

Examples and Bad Case Analysis
In the left plot of Figure 4, we list some intuitive examples on MSCOCO image captioning task to find the the actual improvement. Compared the 1 st and 2 nd lines, we find that the pre-training procedure helps the base model to generate more complete and accurate captions. Compared the 2 nd , 3 rd and 4 th lines, we find that the visual concepts are bringing more details in colors and attributes, such as brown, white, pink and wooden than the "w/o ViCo" model, and the Disentangled Multimodal-Attention is bringing more comprehensive in objects, such as field, mouth, wall and chairs than the "w/o DiM" model, which corroborate the effectiveness of our approach.
We also analyze some bad cases to provide insights on how the DiMBERT may be improved. We find that there are mainly two types of errors, i.e., visual similarity and noise. In the right plot of Figure 4, we give some examples. In the first example, our model dis-identify the woman as a man due to their visual similarity. However, humans can find that the person in the image wearing a bra. In the second example, the DiMBERT mistakes the incorrect visual concept, i.e., clock, for an appropriate one when it generates caption. A more powerful ViCo extractor may be helpful in solving the problem, but it is unlikely to be completely avoided.

CONCLUSIONS
We present a visual concepts and proposed Disentangled Multimodal-Attention based pre-training model, i.e., DiMBERT, which is pre-trained on a large amount of image-sentence pairs to learn vision-language grounded representations.