{'fraction_non_alphanumeric': 0.03821313240043057, 'fraction_numerical': 0.010375553163497189, 'mean_word_length': 4.960613081447157, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cross- and intra-modal knowledge in a unified scene graph to enhance the semantic alignments. Specifically, we introduce a novel structural knowledge masking (SKM) strategy to use the scene graph structure as a priori to perform masked language (region) modeling, which enhances the semantic alignments by eliminating the interference information within and across modalities. Extensive ablation studies and comprehensive analysis verifies the effectiveness of ROSITA in semantic alignments. Pretrained with both in-domain and out-of-domain datasets, ROSITA significantly outperforms existing state-of-the-art VLP methods on three typical vision-and-language tasks over six benchmark datasets.

: Schematic of the knowledge integration strategies of three VLP methods, i.e., OSCAR [23], ERNIE-ViL [45], and our ROSITA. OSCAR and ERNIE-ViL only exploit the intra-modal knowledge from the image and text modalities, respectively. In contrast, ROSITA simultaneously encodes the cross-modal knowledge (pink line) and intra-modal knowledge (blue line) in a unified scene graph centered at specific anchor objects, which is used to enhance the learning of fine-grained semantic alignments across modalities.

INTRODUCTION
Motivated by the success of the pretrain-then-finetune paradigm of BERT in natural language understanding [8], there has been an increasing interest in developing vision-and-language pretraining (VLP) models [7,23,27,38] to address a wide range of vision-andlanguage (V+L) tasks. In particular, these approaches first pretrain transformer-based models on large image-text corpus to learn taskagnostic representations, and then finetune the models on downstream V+L tasks, e.g., visual question answering [50,51], image text retrieval [18,31], and referring expression comprehension [16,52]. Compared to earlier methods that are only adapted to one V+L task [46,47,49], VLP models is generalizable across multiple tasks and also achieves significantly better performance on respective tasks. Learning fine-grained semantic alignments between image regions and text words plays a key role in V+L tasks. However, manually annotating such dense alignment between regions and words is expensive and is unrealistic under the large-scale scenario. Therefore, most existing VLP approaches [7,21,27] use a weaklysupervised learning strategy to model the alignments implicitly. Taking the image regions and text words as inputs, they adopt multilayer Transformers [39] as their backbones to learn fine-grained semantic alignments from coarse-grained image-text matching supervision. Moreover, the interference within and across modalities makes the learning of semantic alignments even more challenging.
To facilitate the learning of semantic alignments, two recent VLP approaches OSCAR [23] and ERNIE-ViL [45] introduce extra knowledge in different ways. Specifically, OSCAR additionally extracts the predicted region tags from images and uses these tags as anchor points to align with text words implicitly. ERNIE-ViL explicitly constructs a scene graph from text and puts more emphasis on the keywords (e.g., objects along with their attributes and relations) in the scene graph in its pretraining objectives. In terms of knowledge source, both of them use the intra-modal knowledge from a single modality to enhance the semantic alignments: OSCAR models the intra-modal knowledge in the image modality while ERNIE-ViL models the intra-modal knowledge in the text modality. The success of the two methods above raises a question: Is it possible to utilize the intra-modal knowledge from both modalities along with the crossmodal knowledge to further enhance the semantic alignments?
In this paper, we present a new VLP method called ROSITA, which encodes the cROSs-and InTrA-modal knowledge simultaneously in a unified scene graph. As shown in Figure 1, the graph consists of a set of knowledge entries, where each entry corresponds to an anchor object along with its associated cross-and intra-modal knowledge. The intra-modal knowledge refers to the relationships between the anchor object and its intra-modal contexts (e.g., spatially related regions or contextually related words). The crossmodal knowledge corresponds to the relationships between the anchor object and its semantically similar objects from the opposite modality (e.g., the region predicted as "grass" and the word "steppe").
Although we have obtained a set of knowledge entries, how to effectively use them to enhance semantic alignments is still nontrivial. We propose a novel structural knowledge masking (SKM) strategy that can be seamlessly integrated with the masked language (region) modeling tasks, which are commonly used in existing VLP methods [7,27]. In principle, SKM determinately masks the anchor object while selectively masking its cross-and intra-modal contexts in a knowledge entry. This strategy effectively eliminates the interference information within and across modalities and enhances the semantic alignments by enforcing the model to acquire accurate information from the opposite modality.
The contributions of this work are three-fold: (1) We present a new VLP method ROSITA, which incorporates cross-and intra-modal knowledge simultaneously to enhance the semantic alignments across different modalities. (2) We introduce a novel structural knowledge masking strategy to use the scene graph structure as a priori to be integrated with the commonly used masked language (region) modeling tasks in existing VLP methods. (3) We achieve the best results on three typical V+L tasks over six benchmark datasets, outperforming existing state-of-theart VLP methods.

RELATED WORK
We briefly review previous studies on unimodal pretraining and vision-and-language pretraining, especially those studies on knowledge enhanced pretraining.
Unimodal Pretraining. The pretraining technique has been widely used in computer vision (CV) tasks. Deep convolutional neural networks like VGGNet [35] or ResNet [12] pretrained on ImageNet can well generalize to various downstream tasks [11,26,33]. In contrast to CV tasks, the popularization of pretraining in the natural language processing (NLP) community is relatively late. Based on the multi-layer Transformer architecture [39], many famous pretraining approaches (e.g., BERT [8], GPT [32], and XLNet [42]) have been put forward. Different from the supervised pretraining paradigm in CV tasks, the pretraining paradigm in NLP tasks is self-supervised that aims to train a model to predict words based on their contexts without introducing human annotations. In particular, BERT introduces a novel masking language modeling (MLM) task that randomly masks the input words and predicts these masked words based on their contexts. This MLM strategy is naturally inherited by the VLP methods.
Vision-and-Language Pretraining (VLP). Different from the purely self-supervised paradigm in NLP tasks, VLP models are pretrained on large-scale paired image-text corpus, e.g., image captioning datasets like [6,29,34]. Mirroring the success of BERT, recent studies naturally extend its framework to the vision-andlanguage domain to pretrain VLP models for a wide range of V+L tasks [7,13,23,27,38,45,53]. ViLBERT [27] and LXMERT [38] are two pioneering works in this field, where the twostream architectures are adopted to encode the image features and textual features with two separate Transformers and then perform multimodal fusion via a third Transformer. Recent works tend to use the single-stream architectures, where the multimodal features are directly fused using one Transformer [7,19,21,36]. Moreover, other techniques like knowledge integration [23,45], multilingual enhancement [55], contrastive learning [22], and adversarial training [9] are introduced to further improve the performance of the pretrained models.
Knowledge-Enhanced Pretraining. Incorporating prior knowledge (e.g., external knowledge graph) to enhance model pretraining has been investigated earlier by two ERNIE methods [37,54] and widely explored in recent years [25,40,41]. The introduced prior knowledge enables the model to better understand the syntactic and semantic structure of the text, thus facilitating model pretraining by an improved structural MLM task. In the VLP task, prior knowledge can be acquired from both the image and text modalities. ERNIE-ViL constructs a scene graph from text and puts more emphasis on the discovered keywords [45]. OSCAR exploits the predicted tags of image regions to enhance the semantic alignment across the two modalities [23]. A concurrent work UC2 utilizes off-the-shelf machine translation model to construct aligned multilingual dataset for texts and regard this extra information as prior knowledge to enhance the learning of cross-modal semantic alignment [55]. Despite the success of these knowledge-enhanced VLP methods, they only utilize the intra-modal knowledge from a single modality, which restricts their effectiveness in learning semantic alignments.
To the best of our knowledge, our ROSITA is the first VLP method to integrate the cross-modal and intra-modal knowledge simultaneously in order to enhance the learning of semantic alignments across different modalities.

KNOWLEDGE EXTRACTION
In this section, we introduce the procedure of extracting knowledge entries from an image-text pair. We first construct a unified graph to model the intra-and cross-modal knowledge from an imagetext pair. On top of the established graph, we select anchor objects to obtain a set of knowledge entries. The process of knowledge extraction is illustrated in Figure 2.
Unified Scene Graph Construction. Given an image-text pair, we resort to a unified scene graph structure =< , , > to encode its intra-and cross-modal knowledge simultaneously [48]. The vertex set includes the words and regions from the text and image, respectively. The edge set and similarity set contain pairwise relationships and their corresponding similarities between vertices (i.e., edge weights), respectively.
The intra-modal knowledge within the image and text are first represented as an image scene graph and a text scene graph, respectively. For the image scene graph, regions extracted from a pretrained object detector are considered as the vertices in . Inspired by [14,20,43], we calculate the similarity between each paired regions by their Intersection over Union (IoU) score. The region pairs with IoU scores larger than zero are considered to have edges in and their IoU scores are regarded as their similarities in . For the text graph, we use an off-the-shelf scene graph parser provided by [1] to obtain a text scene graph from a text. The text scene graph explicitly encodes the keywords of objects, attributes, and relations found in the text while discarding the rest of uninformative words. These mentioned keywords in the scene graph are regarded as the vertices in . The word-word relationships in the scene graph (i.e., object-attribute or objectrelation) correspond to the edges in . The similarity between two vertices is the co-occurrence frequencies of the referred object-attribute (or object-relation) pair calculated on the whole dataset. Since the similarity distributions of the image and text modalities may vary widely, we normalize the similarities within each modality, respectively.
As we have modeled the intra-modal knowledge in the graph, we further integrate cross-modal knowledge to align the image regions to their semantically related words. Since such crossmodal alignment supervision is not available, we establish pseudo semantic alignments between region-word pairs as follows. For the image regions, the predicted region tags are aligned to the object words with respect to their semantic similarities on words. We adopt a pretrained word embedding model [30] to calculate the pairwise similarities between object tags and object words 1 . We set a minimum confidence threshold of 0.5 to the similarity scores to make a trade-off between precision and recall. The region-word pairs surpass the threshold will form cross-modal edges in and their corresponding scores represent the similarities in .
Knowledge Representation. Based on the constructed unified scene graph , we illustrate the procedure of extracting knowledge entries from the scene graph in detail. Note that each knowledge entry is associated with an anchor object, we first select all possible anchor objects from the graph. We define an anchor object as the vertex (an image region or a text word) in the graph that is referred to by at least one cross-modal edge. Since the attribute and relation words are not directly connected to any image region, they cannot be anchor objects according to our definition.
After obtaining the anchor objects, we integrate the intra-modal knowledge and cross-modal knowledge in to obtain a knowledge entry. Given an anchor object ∈ , its corresponding knowledge entry is represented as a subgraph ( ) ⊆ and is obtained by the union of three subgraphs of as follows: where cross ( ) contains the relationships between and its directly connected contexts by cross-modal edges. intra ( ) models the relationships between and its directly connected contexts by intramodal edges. intra ( cross ( )) includes the relationships between the vertices in cross ( ) and their corresponding intra-modal contexts. It is worth noting that the anchor object can reach every vertex in ( ) within two hops.

THE ROSITA FRAMEWORK
Based on the extracted knowledge entries from image-text pairs, we introduce the ROSITA framework in this section. We first describe the image and text feature representations and the network architecture. Then, we introduce a structural knowledge masking (SKM) strategy, which takes the knowledge entries as a priori to perform the masked language (region) modeling. Finally, we describe the whole pretraining objective with multi-task learning. The overall framework is illustrated in Figure 3. Image and Text Feature Representations. Following the commonly used strategy in existing VLP methods [6,23], the input image is represented as a set of regional features, which are extracted from a Faster R-CNN model pretrained on Visual Genome [2]. More specifically, we extract regions with the highest confidence probabilities from the image. For the -th region, it is represented as a visual feature ∈ R 2048 and a positional feature ∈ R 5 [45]. The two features are fused into a -dimensional image representation ∈ R using two linear projections as follows: where ∈ R 2048× and ∈ R 5× . Finally, the image is represented as a feature matrix ∈ R × .
For its paired text, we adopt the word processing method similar to [8]. The input text is first tokenized into words and trimmed (or padded) to a maximum of words. Each word and its index (i.e., the absolute position of in the text) are projected to vectors by two individual embedding layers, then added to obtain the positionaware text representation as follows: where is -dimensional to match the image representation. The text is finally represented as a feature matrix ∈ R × .
Network Architecture. The image features = [ 1 , ..., ] and text features = [ 1 , ..., ] are first concatenated before feeding to the network. We insert two special tokens to the concatenated features to obtain the multimodal input features : where the [SEP] token marks the boundary between the image and text features. The [CLS] token is used to predict whether the given image and text are paired or not. The multimodal features are fed into a single-stream Transformer with layers [8]. Each layer consists of a multi-head selfattention (MSA) block and a feed-forward networks (FFN) block.
Structural Knowledge Masking. The masked language modeling (MLM) [8] and masked region modeling (MRM) [7] tasks are commonly used in almost all the VLP methods [7,27,45]. They randomly mask the input tokens (i.e., words or regions) and predict these masked tokens based on their contextual tokens. Since the random masking based MLM and MRM tasks are not aware of the keywords and key regions to be aligned, their efficacy in the alignment learning is weak. To this end, we present an alternative structural knowledge masking (SKM) strategy to selectively mask the tokens referred to by the extracted knowledge entry. Accordingly, the MLM and MRM tasks are respectively modified to the SKMLM and SKMRM tasks to adapt to the SKM strategy.
Let an image be represented as a set of regions R = { 1 , ..., } and a text be represented as a sequence of words W = { 1 , ..., }, we construct a unified scene graph on top of R and W, and extract a set of knowledge entries from . Let ( ) =<ˆ,ˆ,ˆ> be one of the knowledge entries, where ∈ˆis the anchor object. The vertices are represented asˆ= { 1 , ..., } and the similarities between the vertices are represented asˆ∈ R × , where is the number of vertices in this entry.
The strategy of SKM is to determinately mask the anchor object while probabilistically masking its intra-modal contexts and cross-modal contexts with respect to the graph structure of the knowledge entry. Since the similarities between and its contexts are different, we assign independent masking probabilities to each of the contexts with respect to their similarities to , rather than simply using an identical masking probability for all the contexts. To obtain the masking probabilities for the contexts, we introduce a masking strategy that satisfies the following principles: for the intramodal contexts, a larger similarity score refers to a higher masking probability. For the cross-modal contexts, a larger score leads to a lower masking probability. The reasons behind this masking strategy will be explained hereinafter.
Note that not all contexts have direct connections to the anchor object. Therefore, we calculate the transmission probabilities = [ 1 , ..., ] ∈ [0, 1] from the anchor object to its contexts inˆ based on the normalized similarities defined inˆ. Since can reach all the vertices inˆwithin two hops, is defined as follows: where ( ) ∈ {0, 1} is a one-hot vector with the -th element to be 1. The two terms correspond to the transmission probabilities between the anchor object and its one-hop and two-hop contexts, respectively. After that, we convert the transmission probabilities to the masking probabilities = [ 1 , ..., ] ∈ [0, 1] using the following rules to satisfy our masking strategy above: if is the anchor object , if is within the intra-modal contexts if is within the cross-modal contexts (7) where is a hyper-parameter to balance the masking probabilities of the intra-modal and cross-modal contexts. and denote the transmission and masking probability of the vertex , respectively.
Given a knowledge entry, we use the calculated masking probabilities to obtain two groups of mask indices and , indicating the words and regions to be masked, respectively. The partially masked input features are passed through the network and then fed into the SKMLM and SKMRM tasks.
In particular, if the anchor point refers to an object word in the text, we resort to the SKMLM task to reconstruct the masked words W as follows: where is the trainable parameters. Each pair (W, R) is sampled from the whole training set . W \ and R \ refer to the remaining words in W and the remaining regions in R with excluding the masked tokens from their modalities, respectively.
Analogously, if refers to a region in the image, we resort to the SKMRM task to reconstruct the masked regions R as follows: where (·) refers to some loss functions. Similar to [7], we use the regression-based loss and classification-based loss jointly. The motivations of SKM can be explained as follows: (i) The intra-modal contexts may contain interference information (e.g., the word "sky" is frequently associated with an attribute "blue" and a visual object of "wheel" is usually within an object of "car"). Such interference information may leak out the semantics of the anchor object and reduce the difficulty of anchor object reconstruction, leading to a degradation of the pretrained model. Therefore, when masking an anchor object, its intra-modal contexts with high similarities will have high probabilities to be masked simultaneously. This operation reduces the risk of information leakage and enforces the model to acquire precise information from the opposite modality, which implicitly enhance the semantic alignments; (ii) The cross-modal contexts with low similarities may contain irrelevant or noisy information. Therefore, when masking an anchor object, its cross-modal contexts with high similarities will have low probabilities to be masked at the same time, which explicitly excludes potential noise thus benefiting the semantic alignments. As a result, the synergy of the masking operations above significantly facilitates the semantic alignments. Multi-task Learning. Similar to [7], we adopt a multi-task learning objective to pretrain our model. Besides the proposed SKMLM and SKMRM tasks, we also include the image-text matching (ITM) task. Moreover, since the SKMLM and SKMRM tasks only focus on the key tokens included in the knowledge entry, we still retain the original random masking-based MLM and MRM tasks to guarantee a good coverage of the remaining tokens in the image and text 2 .

EXPERIMENTS
We evaluate ROSITA on three V+L tasks and perform thorough comparative analysis to the state-of-the-art VLP methods on six datasets. Furthermore, we conduct comprehensive ablation experiments to explore its effectiveness in learning fine-grained semantic alignments.

Pretraining Setup
Datasets. Following the strategy in [7], we construct the pretraining dataset consisting of 9.5M train and 155K validation image-text pairs from four existing datasets, namely the COCO Captions [6], Visual Genome Captions [17], Conceptual Captions [34], and SBU Captions [29]. The four datasets are categorized into the in-domain and out-of-domain datasets based on whether they share the same images with the downstream tasks. The statistics of the pretraining dataset are shown in Table 1.
Implementation Details. For the input image-text pairs, we extract a fixed number of 36 region features from a pre-trained Faster R-CNN model [2] and adopt the BPE strategy to tokenize the sentence into a maximum of 50 words following [8]. Our ROSITA model adopts a 12-layer Transformer encoder architecture with 768 hidden units and 12 attention heads. The hyper-parameter in Eq. (7) is set to 0.9. The masking probabilities in the original MRM and MLM tasks are set to 15% [45]. The model is initialized with the parameters from a pretrained BERT-base model [8], and then trained up to 40 epochs with a batch size of 512.

Downstream Tasks
After obtaining the pretrained ROSITA model, we finetune it on three downstream V+L tasks as follows.
Visual Question Answering (VQA) is a task that requires the model to answer natural language questions about an image. We adopt the widely used VQAv2 dataset [3,10], which is manually built on the images from the MSCOCO dataset [24]. The dataset is split into train (83k images and 444k questions), validation (41k images and 214k questions), and test (81k images and 448k questions) sets. Following the strategy in [7], we feed the representation of the [CLS] token to a linear classifier to predict the corresponding answer from a vocabulary of size 3129 [49].
Referring Expression Comprehension (REC) is a task that requires to localize an image region referred to by a natural language query. We evaluate the performance on RefCOCO [16], RefCOCO+ [16] and RefCOCOg [28] datasets. All the three datasets are collected from COCO images [31]. RefCOCO and RefCOCO+ are split into four subsets, including train (120k queries), validation (11k queries), testA (6k queries about people), and testB (6k queries about objects), while RefCOCOg is split into three subsets, including train (81k queries), validation (5k queries), and test (10k queries). The representation for each image region is used to predict a ranking score and a refined bounding box.
Image-Text Retrieval (ITR) is a task that requires the model to calculate a similarity score between an image and a sentence and then perform cross-modal retrieval. We conduct experiments on the COCO Captions [6] and Flickr30K [44] datasets, respectively. Following the partition strategy by [15], the COCO dataset is split into 82k/5k/5k train/validation/test images, while the Flickr30K dataset is split into 29k/1k/1k train/validation/test images. Similar to [7], we use an offline hard sample mining strategy to obtain 128 negative samples per each positive sample, and use the representation of the [CLS] token to predict a matching score.

Main Results
We compare the proposed ROSITA model against existing stateof-the-art VLP methods. As shown in Table 2, ROSITA achieves the overall best performance on all downstream tasks, which verifies the effectiveness of the integrated cross-and intra-modal knowledge and the corresponding SKM strategy 3 . It is worth noting that some methods like ViLBERT, LXMERT, and ERNIE-ViL adopt the two-stream architecture, which have much more parameters (ROSITA: 116M, VilBERT: 221M, LXMERT: 183M, ERNIE-ViL: 228M). Some methods like UNITER and VILLA use a larger number of image features (up to 100 regions), which has been verified to benefit the performance at the expense of much higher computational cost. In contrast, ROSITA uses a fixed number Table 3: Ablations of ROSITA variants without the cross-and intra-modal knowledge. All models are pretrained on the indomain datasets and then finetuned on specific downstream tasks. For each model, we report the accuracies on the pretraining tasks an downstream tasks, respectively. As we only have positive image-text pairs in the pretraining datasets, we use the offline hard sample mining strategy to generate an equal number of negative samples for the evaluation of the ITM task.

Ablation Studies
We run a number of ablations to investigate the reasons of ROSITA's effectiveness. The results show in Table 3-4 and Figure 4-5 are discussed in detail below.
Cross-and Intra-modal Knowledge. In Table 3, we show the effects of the intra-modal knowledge and cross-modal knowledge based on the performance on the pretraining and downstream tasks. Taking the full ROSITA as the reference model (Line #1), we obtain the different variants by removing the cross-modal knowledge or the intra-modal knowledge. The variant without cross-modal knowledge (Line #2) indicates that the model is not aware of the anchor objects and the SKM strategy is performed only on a single modality using the intra-modal knowledge. In contrast, the variant without intra-modal knowledge (Line #3) indicates that the model is aware of the anchor objects but is not aware of the intra-modal contexts. Finally, by removing both the cross-and intra-modal knowledge, we obtain a baseline variant nearly identical to UNITER [7] (Line #4) 4 . Given the pretrained models of the four variants above (i.e., without finetuning on downstream tasks), we evaluate their performance on three pretraining tasks. The ITM task examines the ability of semantic alignment between image-text pairs. From the results, we can see that both types of knowledge bring performance improvement to the ITM task (#4 vs. #3 and #2). Moreover, the two types of knowledge are complementary that their synergy brings 4 Our model has slight performance deviations compared with the original UNITER model since we use different visual features and pretraining hyper-parameters.
2.1 points improvement compared to the baseline model without any knowledge (#4 vs. #1). Although the ITM task is the most straightforward metric for semantic alignment, it only measures the coarse-grained alignments on the image-text level, thus cannot fully reveal the capability of ROSITA. As a complement, we resort to the SKMLM and SKMRM tasks to evaluate the fine-grained alignments on the region-word level. Compared with the baseline model in #4, the full ROSITA model improves the accuracies by 7.0 and 7.9 points on the SKMLM and SKMRM tasks, respectively.
Next, we report the performance of these variants on different downstream tasks. From the demonstrated results, we obtain similar observations to those on the pretraining tasks. The full ROSITA model consistently outperforms all the counterparts, verifying the effectiveness of the cross-and intra-modal knowledge.
SKM Strategy. After extracting knowledge entries from imagetext pairs, we have two alternative masking strategies in SKM, i.e., the independent probabilities and the identical probability. For the masking strategy with identical probability, we evaluate the choices of different probabilities within {15%, 30%, 45%}. The results in Table 4 show that the model pretrained with independent probabilities steadily outperforms all the counterparts with the identical probability. For the models pretrained with the identical probability strategy, their performance is sensitive to the choices of the predefined probability. A small masking probability (e.g., 15%) may degrade the model towards the baseline without any knowledge. A large masking probability (e.g., 45%) may shield the essential information that is necessary to learn the semantic alignments. In comparison, the masking strategy with independent probabilities provides a more fine-grained understanding of the knowledge structure, leading to a more robust pretrained model.
Cross-modal Semantic Alignments. The effect of fine-grained semantic alignments across modalities can be inferred from the attention maps of the learned Transformer model [5]. We visualize the learned cross-modal attentions (i.e., region-to-words and wordto-regions attentions) from the pretrained UNITER [7] and our ROSITA models, as shown in Figure 4. Taking the image-text pair as inputs with exactly one token (a region or a word) being masked at a time, we pass the multimodal features through the pretrained model and extract the attention map from the last MSA block 5 . The region-to-words and word-to-regions attentions of the masked    Figure 5: Visualizations of the region-to-words attentions (left) and word-to-regions attentions (right) from a pretrained ROSITA model with masking multiple regions (or words) at one time.
token correspond to one specific row in the bottom-left and topright area of the attention map, respectively. From the visualized cross-modal attentions, we can see that ROSITA learns significantly better semantic alignments than UNITER. ROSITA can precisely align the masked object to its reference object in the opposite modality while UNITER fails to establish such cross-modal alignments. For example, when the region of "ramp" is masked, ROSITA activates the word "ramp" precisely while UNITER obtains the largest attention value on the word "skate". When another region of "person" is masked, ROSITA precisely activates the word "person" while UNITER still activates the incorrect word "skate". Similar phenomena are observed in the opposite direction. ROSITA activates the accurate regions to the masked words while UNITER fails to do it.
To step further, we conduct a more challenging task as follows. We mask multiple regions (or words) at the same time to examine whether the semantic alignments can still be achieved. The visualized results in Figure 5 show that ROSITA works surprisingly well to establish accurate semantic alignment for each masked token. For example, when the regions of "bowl" and "bowl & carrots" are masked simultaneously, the region of "carrot" is precisely aligned to the word "carrots", and the region of "bowl & carrots" is aligned to the two words "bowl" and "carrots" uniformly. In the opposite direction, when the words "man", "candles", and "cake" are masked at the same time, their corresponding regions are highlighted in the learned attentions, respectively.

CONCLUSION
In this paper, we present a new VLP method called ROSITA, which integrates the cross-and intra-modal knowledge in a unified scene graph to enhance the learning of cross-modal semantic alignment. We introduce a novel structural knowledge masking (SKM) strategy to perform masked language (region) modeling with respect to the knowledge entries extracted from the unified scene graph. Extensive ablations, comparative experiments, and comprehensive analysis show that ROSITA significantly outperforms existing state-of-theart VLP approaches on three typical V+L tasks over six benchmark datasets. We hope our study will be helpful to inspire future research in the vision-and-language community and beyond.