{'fraction_non_alphanumeric': 0.03501160022899153, 'fraction_numerical': 0.015969146403929012, 'mean_word_length': 4.821785651640063, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FLAVA: A Foundational Language And Vision Alignment Model

State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a"foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.


Introduction
Large-scale pre-training of vision and language transformers has led to impressive performance gains in a wide variety of downstream tasks. In particular, contrastive methods such as CLIP [83] and ALIGN [50] have shown that natural language supervision can lead to very high quality visual models for transfer learning.
Purely contrastive methods, however, also have important shortcomings. Their cross-modal nature does not make them easily usable on multimodal problems that require dealing with both modalities at the same time. They require large corpora, which for both CLIP and ALIGN have not been made accessible to the research community and the details of which remain shrouded in mystery, notwithstanding well-known issues with the construction of such datasets [9].
In contrast, the recent literature is rich with transformer models that explicitly target the multimodal vision-andlanguage domain by having earlier fusion and shared selfattention across modalities. For those cases, however, the unimodal vision-only or language-only performance of the model is often either glossed over or ignored completely.
If the future of our field lies in generalized "foundation models" [10] or "universal" transformers [72] with many * Equal contribution. (e.g. GLUE) Figure 1. We present FLAVA, a language and vision alignment model that learns strong representations from multimodal (imagetext pairs) and unimodal data (unpaired images and text) and can be applied to target a broad scope of tasks from three domains (visual recognition, language understanding, and multimodal reasoning) under a common transformer model architecture.
different capabilities, then the following limitation should be overcome: a true foundation model in the vision and language space should not only be good at vision, or language, or vision-and-language problems-it should be good at all three, at the same time.
Combining information from different modalities into one universal architecture holds promise not only because it is similar to how humans make sense of the world, but also because it may lead to better sample efficiency and much richer representations.
Generally, models in the vision-and-language space can be divided into two categories: (i) dual encoders where the image and text are encoded separately followed by a shallow interaction layer for downstream tasks [50,83]; and (ii) fusion encoder(s) with self-attention spanning across the modalities [16, 34, 44, 45, 62-65, 70, 71, 101, 102, 109, 118, 120]. The dual encoder approach works well for unimodal [107,108] and cross-modal retrieval tasks [66,81] but their lack of fusion usually causes them to underperform on tasks that involve visual reasoning and question answering [39,53,93,96] which is where models based on fusion encoder(s) shine.
Within the fusion encoder category, a further distinction can be made as to whether the model uses a single transformer for early and unconstrained fusion between modalities (e.g., VisualBERT, UNITER, VLBERT, OSCAR [16,63,65,101,120]) or allows cross-attention only in specific co-attention transformer layers while having some modality specific layers (e.g., LXMERT, ViLBERT, ERNIE-ViL [70,71,102,116]. Another distinguishing factor between different models lies in the image features that are used, ranging from region features [63,70,118], to patch embeddings [54,62,109], to convolution or grid features [46,51]. Dual encoder models use contrastive pretraining to predict the correct N paired combinations among N 2 possibilities. On the other hand, with fusion encoders, inspired by unimodal pretraining schemes such as masked language modeling [28,68], masked image modeling [5], and causal language modeling [84], numerous pretraining tasks have been explored: (i) Masked Language Modeling (MLM) for V&L where masked words in the caption are predicted with help of the paired image [63,70,102]; (ii) prefixLM, where with the help of an image, the model tries to complete a caption [26,109]; (iii) image-text matching, where the model predicts whether given pair of image and text match or not; and (iv) masked region modeling, where the model regresses onto the image features or predicts its object class.
Compared to previous work, our model FLAVA works on a wide range of tasks in each of the vision, language, and vision-and-language domains. FLAVA uses a shared trunk which was pretrained on only openly available public paired data. FLAVA combines dual and fusion encoder approaches into one holistic model that can be pretrained with our novel FLAVA pretraining scheme that leverages pretraining objectives from both categories. FLAVA is designed to be able to take advantage of unpaired unimodal data along with multimodal paired data, resulting in a model that can handle unimodal and retrieval tasks as well as cross-modal and multimodal vision-and-language tasks.

FLAVA: A Foundational Language And Vision Alignment Model
The goal of this work is to learn a foundational language and vision representation that enables unimodal vision and language understanding as well as multimodal reasoning, all within a single pre-trained model. We show how this can be achieved with a simple and elegant architecture

The model architecture
The FLAVA model architecture is shown in Figure 2. The model involves an image encoder to extract unimodal image representations, a text encoder to obtain unimodal text representations, and a multimodal encoder to fuse and align the image and text representations for multimodal reasoning, all of which are based on transformers. Image encoder. We adopt the ViT architecture [31] for the image encoder. Given an input image, we resize it to a fixed image size and split the image into patches, which are then linearly embedded and fed into a transformer model (along with positional embeddings and an extra image classification token [CLS I]). The image encoder output is a list of image hidden state vectors {h I }, each corresponding to an image patch, plus an additional h CLS,I for [CLS I]. We use the ViT-B/16 architecture for our image encoder. Text encoder. Given an input piece of text (e.g., a sentence or a pair of sentences), we first tokenize and embed it into a list of word vectors following [28]. Then, we apply a transformer model over the word vectors to encode them into a list of hidden state vectors {h T }, including h CLS,T for the text classification [CLS T] token. Importantly, different from prior work, our text encoder has exactly the same ar-chitecture as the visual encoder, i.e., we use the same ViT architecture (but with different parameters) for both the visual and textual encoder, i.e. ViT-B/16. Multimodal encoder. We use a separate transformer to fuse the image and text hidden states. Specifically, we apply two learned linear projections over each hidden state vector in Applying to downstream tasks. The FLAVA model can be applied to both unimodal and multimodal tasks in a straightforward manner. For visual recognition tasks (e.g. Ima-geNet classification), we apply a classifier head (e.g. a linear layer or a multi-layer perceptron) on top of the unimodal h CLS,I from the image encoder. Similarly, for language understanding and multimodal reasoning tasks, we apply a classifier head on top of h CLS,T from the text encoder or h CLS,M from the multimodal encoder, respectively. We pretrain the FLAVA model once, and evaluate it separately on each downstream task. More details about finetuning, linear, and zero-shot evaluation on specific tasks can be found in the supplemental.

Multimodal pretraining objectives
We aim to obtain strong representations through pretraining on both multimodal data (paired image and text) as well as unimodal data (unpaired images or text). FLAVA pretraining involves the following multimodal objectives. Global contrastive (GC) loss. Our image-text contrastive loss resembles that of CLIP [83]. Given a batch of images and text, we maximize the cosine similarities between matched image and text pairs and minimize those for the unmatched pairs. This is accomplished by linearly projecting each h CLS,I and h CLS,T into an embedding space, followed by L2-normalization, dot-product, and a softmax loss scaled by temperature.
Large models are often trained using multiple GPUs data parallelism, where the samples in a batch are split across GPUs. When gathering embeddings for the image and text contrastive objective, the open-source CLIP implementation [48] only back-propagates the gradients of the contrastive loss to the embeddings from the local GPU where the dot-product is performed. In contrast, through experiments that can be found in the supplemental, we observe a noticeable performance gain by performing full backpropagation across GPUs compared to only doing backpropagation locally. We call our loss "global contrastive" L GC to distinguish it from "local contrastive" approaches. Masked multimodal modeling (MMM). While a number of previous vision-and-language pretraining approaches (e.g. [63]) have focused on masked modeling of the text modality by reconstructing masked tokens from the multimodal input, most of them do not involve masked learning on image modality directly at the image pixel level in an end-to-end manner. Here, we introduce a novel masked multimodal modeling (MMM) pretraining objective L MMM that masks both the image patches and the text tokens and jointly works on both modalities.
Specifically, given an image and text input, we first tokenize the input image patches using a pretrained dVAE tokenizer [89], which maps each image patch into an index in a visual codebook similar to a word dictionary (we use the same dVAE tokenizer as in [5]). Then, we replace a subset of image patches based on rectangular block image regions following BEiT [5] and 15% of text tokens following BERT [28] with a special [MASK] token. Then, from the multimodal encoder's output {h M }, we apply a multilayer perceptron to predict the visual codebook index of the masked image patches, or the word vocabulary index of the masked text tokens.
This objective can be seen as an extension of the multimodal masked language modeling such that it incorporates masking on the image side. In our experiments, we find that our MMM pretraining leads to improvements over and in addition to the contrastive loss pretraining, especially for multimodal downstream tasks such as VQA. Note that we apply global contrastive loss on image patches and text tokens without any masking, which are forwarded through the image and text encoders separately from the MMM loss.
Image-text matching (ITM). Finally, we add an image-text matching loss L ITM following prior vision-and-language pretraining literature [16,70,102]. During pretraining, we feed a batch of samples including both matched and unmatched image-text pairs. Then, on top of h CLS,M from the multimodal encoder, we apply a classifier to decide if an input image and text match each other.

Unimodal pretraining objectives
While the objectives in Sec. 3.2 allow pretraining the FLAVA model on paired image-and-text data, the vast majority of datasets (such as ImageNet for images and CC-News for text) are unimodal without paired data from the other modality. To efficiently learn a representation for a wide range of downstream tasks, we would also like to leverage these datasets and incorporate unimodal and unaligned information into our representations.
In this work, we introduce knowledge and information from these unimodal datasets through 1) pretraining the image encoder and text encoder on unimodal datasets; 2) pretraining the entire FLAVA model jointly on both unimodal and multimodal datasets; or 3) a combination of both by starting from pretrained encoders and then jointly training. When applied to stand-alone image or text data, we adopt masked image modeling (MIM) and masked language modeling (MLM) losses over the image and text encoders respectively, as described in what follows.
Masked image modeling (MIM). On unimodal image datasets, we mask a set of image patches following the rectangular block-wise masking in BEiT [5] and reconstruct them from other image patches. The input image is first tokenized using a pretrained dVAE tokenizer [89] (same as the one used in the MMM objective in Sec. 3.2), and then a classifier is applied on the image encoder outputs {h I } to predict the dVAE tokens of the masked patches.
Masked language modeling (MLM). We apply a masked language modeling loss [28] on top of the text encoder to pretrain on stand-alone text datasets. A fraction (15%) of the text tokens are masked in the input, and reconstructed from the other tokens using a classifier over the unimodal text hidden states output {h T }.
Encoder initialization from unimodal pretraining. We use three sources of data for pretraining: unimodal image data (ImageNet-1K [90]), unimodal text data (CCNews [68] and BookCorpus [121]), and multimodal image-text paired data (Sec. 3.5). We first pretrain the text encoder with the MLM objective on the unimodal text dataset. We experiment with different ways for pretraining the image encoder: we pretrain the image encoder on unpaired image datasets with either MIM or the DINO objective [13], before joint training on both unimodal and multimodal datasets. We empirically found the latter to work quite well, despite the  switch to an MIM objective on images post-initialization (more details in supplemental). Then, we initialize the whole FLAVA model with the two respective unimodallypretrained encoders, or when we train from scratch, we initialize randomly. We always initialize the multimodal encoder randomly for pretraining.
Joint unimodal and multimodal training. After unimodal pretraining of the image and text encoders, we continue training the entire FLAVA model jointly on the three types of datasets with round-robin sampling. In each training iteration, we choose one of the datasets according to a sampling ratio that we determine empirically (see supplemental) and obtain a batch of samples. Then, depending on the dataset type, we apply unimodal MIM on image data, unimodal MLM on text data, or the multimodal losses (contrastive, MMM, and ITM) in Sec. 3.2 on image-text pairs.

Implementation details
We find that the optimizer hyperparameters play a critical role in effective pretraining. A large batch size, a large weight decay, and a long warm-up are all important for preventing divergence with a large learning rate (we use 8,192 batch size, 1e-3 learning rate, 0.1 weight decay, and 10,000 iteration warm-up in our pretraining tasks together with the AdamW optimizer [55,69]). In addition, the ViT transformer architecture (which applies layer norm [3] before the multi-head attention rather than after [115]) provides more robust learning for the text encoder under large learning rate than the BERT [28] transformer architecture. FLAVA is implemented using the open-source MMF [94] and fairseq [78] libraries. We use Fully-Sharded Data Parallel (FSDP) [86,87] and train in full FP16 precision except the layer norm [3] to reduce GPU memory consumption.

Data: Public Multimodal Datasets (PMD)
For multimodal pretraining, we constructed a corpus out of publicly available sources of image-text data, which are presented in Table 2 with examples in Fig. 3. The total count of text-image pairs is 70M, including 68M unique images, and the average caption length is 12.1 words. For the YFCC100M dataset [103], we filter the image-text data by discarding non-English captions and only keeping captions that contain more than two words. We first consider the
We compare our joint pretraining method (FLAVA in Table 3 and 4) with other settings, on this diverse array of 35 tasks. We report the average performance on the NLP, vision, and multimodal tasks, and an additional macro average across all the three modalities in Table 3, and also the detailed the performance on each task in Table 4. Full FLAVA pretraining achieves the best results. Table 3 shows baselines and different ablation settings of FLAVA, including: models trained with unimodal MIM and MLM losses, FLAVA C trained with only image-text contrastive loss, FLAVA MM trained only on multimodal data, models without unimodal initialization, and the full model (each setting is detailed in the paragraphs below). The full FLAVA model in row 6 outperforms all other settings in average performance over NLP, vision, and multimodal tasks.  Table 3. Our full FLAVA pretraining (row 6) achieves the best average scores on vision, language, and multimodal tasks compared to ablations. Row 1 to 4 are pretrained on PMD while row 5 and 6 also involve unimodal IN-1k, CCNews, and BookCorpus datasets.
Effective global contrastive loss in FLAVA. We next perform a step-by-step ablation of our model ( In a more detailed analysis, we find that the latter improves our macro average over vision, NLP, and multi-modal tasks by +1.65% with only minor additional computation overhead, indicating the global back-propagation implementation in contrastive loss are critical to effective pretraining. MMM and ITM objectives benefit multimodal tasks. Next, we include the other multimodal objectives from Sec. 3.2 into our pretraining, using L MMM and L ITM along with L GC . The results are denoted as FLAVA MM in Table 4 column 4. Compared to FLAVA C with only the contrastive loss L GC (column 3 vs 4), this setting improves multimodal average score by +2.86%, NLP average score by +9%, and also vision average score slightly by +0.3%. We additionally compare FLAVA MM with two other baseline settings -the FLAVA model trained with only unimodal MIM or MLM losses in Sec. 3.3, respectively over the images or the text in PMD. These two baselines are shown in Table 4 column 1 and 2, which are largely outperformed by FLAVA MM . These results indicate that the combined multimodal objectives (contrastive, MMM, ITM) allow FLAVA to learn powerful representations for both unimodal and multimodal downstream tasks. 1 We fine-tune CLIP on multimodal downstream tasks (VQAv2, SNLI-VE, and HM) by applying a classifier on the concatenation of the two output vectors from its image and text encoders (details in supplemental).

Joint unimodal & multimodal pretraining helps NLP.
For the full FLAVA pretraining, we introduce unimodal image data from ImageNet-1k (IN-1k) and text data from CCNews and BookCorpus (BC). In this setting, we apply FLAVA MM losses on PMD data batches, MIM loss on IN-1k unimodal image data and MLM loss on CCNews text data, following Sec. 3.3, shown in Table 4 column 5. Comparing it to FLAVA MM in column 4 with only multimodal pretraining, this joint unimodal and multimodal pretraining improves the NLP average score from 74.22 to 75.55, which suggests that the additional text data from CCNews and BookCorpus benefits language understanding through the MLM objective.
However, we also observe from column 4 vs 5 that the macro average over all tasks decreases slightly. We suspect that this is because adding different tasks to the mix makes the optimization problem much harder, especially when the whole model is randomly initialized. Also, the round-robin sampling of tasks does not follow any particular curriculum to order the learning sequence of these tasks. Naturally, having some vision and language understanding is important before learning multimodal tasks, which motivates us to explore first leveraging unimodal pretraining before the joint training, as described below. Better image and text encoders via unimodal pretraining. As detailed in Section 3.3, in order to leverage unimodal learning before joint training, we initialize the model from pretrained self-supervised weights for both vision and language encoders. For vision encoder, we initialize from an off-the-shelf DINO model pretrained on ImageNet-1k [90]. For the language encoder, we pretrain a ViT model with MLM loss on CCNews and BookCorpus datasets and use its model weights. Comparing column 5 vs 6, we observe pretrained encoders boost the performance of FLAVA on all tasks. We empirically find that initializing the vision encoder from a DINO self-supervised model gives better performance compared to a BEiT self-supervised model (see supplemental for additional details).

Comparison to state-of-the-art models
We compare our full FLAVA model (Table 4 column 6) with several state-of-the-art models on multimodal tasks, language tasks, and ImageNet linear evaluation, in Table 5. FLAVA largely outperforms previous multimodal approaches pretrained on public data (row 4 to 11) on both language and multimodal tasks and approaches the wellestablished BERT model on several GLUE tasks.
FLAVA combines unimodal and multimodal losses and learns more generic representations which are transferable to vision, language, and multimodal tasks. We evaluate the best released CLIP [83] ViT-B/16 model (pretrained on 400M image-text pairs in [83] with the same image encoder architecture as in FLAVA) on our task benchmark, shown in Table 5 row 2. Compared to CLIP, we train FLAVA on just 70M data which is ∼6x smaller. In Fig. 4, we observe that FLAVA works significantly better on language and multimodal tasks while slightly worse than CLIP on some vision-only tasks. In addition, we note that FLAVA outperforms the variant of the CLIP model pretrained only on the PMD dataset (Table 5 row 10).  [47]. The results on V&L tasks are from original papers. For UniT, we use "shared, (COCO init.)" version. Note that SimVLM is pretrained on an order of magnitude more data than FLAVA (1.8B vs 70M). † : taken from [95]; ‡ : taken from [53]. The overall best result among the multimodal approaches is underlined while bold signifies the best model trained on public data.
forming it on multimodal tasks and ImageNet linear evaluation. FLAVA is pretrained using a much smaller dataset compared to 1.8B image-text pairs in [109], and we anticipate that FLAVA's performance will further heavily improve as the pretraining dataset size increases.

Conclusion
In this work, we have presented a foundational vision and language alignment model that performs well on all three target modalities: 1) vision, 2) language, and 3) vision & language. We introduced a novel set of objectives to achieve this goal and conducted experiments on a wide variety of 35 tasks to analyze the model's performance. FLAVA was trained on a corpus of publicly available datasets that is several orders of magnitude smaller than similar recent models, but still obtained better or competitive performance. Our work points the way forward towards generalized but open models that perform well on a wide variety of multimodal tasks. Broader impacts and limitations. The models in this work are trained on public datasets widely used in the community. This enables reproducibility and we hope that our work will motivate others to compare models across a wide area of tasks and domains with the same data. However, like all natural data, these datasets have biases, potentially affecting our models. We partly mitigate this by combining several public datasets to increase the diversity and evaluating on an even larger set of target datasets. Still, further study is needed to identify and reduce potentially harmful biases.

A. Hyperparameters and details of FLAVA
We summarize the hyperparameters in our FLAVA model in Table A.1. We also list the sampling probabilities of the datasets for joint pretraining in Table A We find that a large batch size, a large weight decay, and a long warmup are helpful to stabilize training and prevent divergence under a large learning rate. Based on this finding, we performed a hyperparameter search based by monitoring the learning curve as well as monitoring the zero-shot image classification accuracy based on the image-text contrastive loss on using the text templates from CLIP [83] to obtain the hyperparameters above.

B. Training and evaluation details B.1. Pretraining details
Language encoder pretraining. We follow RoBERTa base pretraining hyperparameters to train our pre-norm ViTbased text encoder [68]. Specifically, we pretrain our text encoder using masked language modeling (MLM) [28] on CCNews and BookCorpus for 125K iterations with a batch size of 2048 and a learning rate of 5e-4. We pick the best checkpoint based on the MLM loss without any further hyperparameter sweeps over RoBERTa's default configuration. Vision encoder pretraining. We pretrain the image encoder in FLAVA on the ImageNet-1k dataset following either BEiT [5] or DINO [13]. When pretraining a ViT-B/16 image encoder with BEiT, we adopt the hyperparameters and training details in [5] with a masked image modeling loss by predicting the dVAE visual tokens of the masked image patches. We also follow the training protocols in [13] to pretrain a DINO ViT-B/16 model as our image encoder. As discussed in Sec. C, we empirically find that the DINOpretrained image encoder gives better final performance. Full FLAVA pretraining. We pretrain jointly on the unimodal and multimodal datasets, following the sampling probabilities of these datasets as provided in Table A.2. Specifically, for each update, we pick a dataset based on its sampling probability and obtain a complete batch from it. In all our ablations, we use a training schedule such that the PMD dataset is sampled for a total of 150K iterations. We monitor the zero-shot accuracy on ImageNet classification [90] every 8K updates and select the best checkpoint based on the ImageNet-1k zero-shot accuracy. We follow [83] to calculate the zero-shot accuracy.

B.2. Vision, language and multimodal evaluation
We evaluate the pretrained FLAVA model across a broad range of vision, natural language, and multimodal tasks. We discuss our evaluation details of these tasks below. Linear probing on vision tasks. We perform linear probe evaluations on the datasets by closely following the setup described in [83]. We extract image features from the final layer of the image encoder (before the multi-modal encoder) and train a logistic regression classifier (L-BFGS implementation from [80]) on the extracted image features. We follow the hyperparameters similar to [83] : 1000 iterations, logistic regression λ parameter sweep from 1e-6 to 1e6. Fine-tuning on NLP tasks. For NLP tasks, we finetune the language encoder end to end for all the GLUE tasks. We add a classification head on top of the language encoder for all the tasks, except for the STS-B task, where we use a regression head. The hyperparameters we use for finetuning follow the setup of RoBERTa [68]. 2 Fine-tuning on multimodal VQA, SNLI-VE, and Hateful Memes. We adopt the following settings when finetuning on VQA, SNLI-VE, and Hateful Memes, adding a 2-layer classifier head with a hidden dimension of 1536 on top of h CLS,M from the multimodal encoder (corresponding to [CLS M]). For VQAv2, we use 1e-4 learning rate, 44000 updates, and an input image size of 480×480. For SNLI-VE and Hateful Memes, we use 1e-5 learning rate, a total iteration number of 24000, and an input image size of 224 × 224 (we use the OCR tokens extracted from the images as the textual input for Hateful Memes). On all these three tasks, we use the AdamW optimizer with a batch size of 256, 1e-2 weight decay, and 2000 warm-up iterations followed by a cosine decay schedule.
We use the same approach above to also evaluate the CLIP model on VQAv2, SNLI-VE, and Hateful Memes datasets. Since CLIP does not have a multimodal encoder, we concatenate the image feature vector from its image encoder and the text feature vector from its text encoder, apply a 2-layer classifier head (with the same hidden dimension of 1536) over the concatenated feature, and finetune the model following the same hyperparameters as for FLAVA. Zero-shot multimodal text and image retrieval. We also evaluate the FLAVA model on the multimodal zero-shot retrieval tasks over the Flickr30K and COCO datasets, where the model needs to select a text caption based on a query image or select an image based on a query caption. We use the cosine similarities between the image and text feature computed in the global contrastive loss in FLAVA as the matching scores between the image and text modalities. Then, the text caption (or image) with the highest matching score to the query is retrieved. Similarly, we also evaluate the zero-shot text and image retrieval performance of the CLIP model using the cosine similarities between its image and text features.

C. Additional ablations and analyses
Unimodal-pretrained vision encoders. We experiment with initializing our model from different pretrained vision encoders (while keeping the language encoder the same). We study two different self-supervised ViT-B/16 models trained on ImageNet-1k: i) BEiT and ii) DINO. Under three FLAVA pretraining settings, FLAVA C , FLAVA MM and FLAVA (full pretraining), initializing from any of the two pretrained vision encoders (along with pretrained language encoders) leads to significant improvement in all tasks. In Table C.1, comparing columns 5 vs 6, 8 vs 9, and 11 vs 12 between BEiT and DINO initialization, DINO gives better performance on vision and multimodal tasks. On NLP tasks, the results are mixed and comparable, as the language encoder is initialized from the same pretrained weights. Global vs. local contrastive losses. In our FLAVA model, we apply a global contrastive loss, where the image and text features are gathered across GPUs and the loss is backpropagated through the gathering operation to all GPUs. This is in contrast with the implementation in [48], where the loss is only back-propagated to local features from the same GPU. It can be seen from Table C.1 (columns 3 vs 4) that the global contrastive loss (column 4) leads to a noticeable gain in the average vision and NLP performance compared to its local contrastive counterpart and also provides a slight boost in multimodal performance.
Observations on SST and VQA. Some of our vision tasks involve classifying an image using the text written on the image pixels, and require the model to perform OCR to read text from images. For example, in the SST task in Table C.1 (which is also evaluated as an image classification task in [83]), the model is asked to classify the sentiment of a natural language sentence by printing the sentence words onto an image and providing the image pixels to the model. It can be seen from   Table 4 in the main paper) with additional ablations (see Sec. C for details). The overall best result is underlined while bold signifies the best on public data (PMD and unimodal).
text pairs (column 14), and we anticipate that FLAVA will also be able to perform scene text reading when pretrained on a larger dataset with enough scene text information.
Our FLAVA model reaches a final accuracy of 72.49 on the VQAv2 dataset. While this accuracy is below the stateof-the-art on VQAv2, we note that this is a reasonable performance given the amount of data used in FLAVA pretraining. Recent models such as SimVLM [109] often use a much larger dataset (e.g. 1.8B image-text pairs [109]), and we believe more pretraining data will also benefit FLAVA. FLAVA and CLIP [83] use transformers [105] as the image and text encoders in their comparable variations (column 3, FLAVA C -local contrastive and column 13, CLIP-ViT-B/16 in Table C.1). Compared to CLIP which uses a text vocabulary of size 49152, in FLAVA we use BERT's text vocabulary with a size of 30522. CLIP uses lowercased byte pair encoding similar to [84,91] whereas we use BERT's tokenizer from [112] to tokenize the text. Furthermore, we use a hidden size of 768 instead of 512 and use the ViT architecture (based on the implementation in Hugging Face [112]) instead of the GPT-style transformer architecture in CLIP for both text and image encoders [115]. Table D.1 shows the comparison of macro averages for the three domains between the original CLIP architecture and our optimized FLAVA architecture trained on PMD under the same settings with local contrastive loss (corresponding to columns 13 and 3 in Table C.1, respectively). A comparison between rows 1 and 2 in Table D.1 shows that our architecture optimizations help achieve a better macro average overall.