Data Efficient Masked Language Modeling for Vision and Language
  Masked language modeling (MLM) is one of the key sub-tasks in vision-language
pretraining. In the cross-modal setting, tokens in the sentence are masked at
random, and the model predicts the masked tokens given the image and the text.
In this paper, we observe several key disadvantages of MLM in this setting.
First, as captions tend to be short, in a third of the sentences no token is
sampled. Second, the majority of masked tokens are stop-words and punctuation,
leading to under-utilization of the image. We investigate a range of
alternative masking strategies specific to the cross-modal setting that address
these shortcomings, aiming for better fusion of text and image in the learned
representation. When pre-training the LXMERT model, our alternative masking
strategies consistently improve over the original masking strategy on three
downstream tasks, especially in low resource settings. Further, our
pre-training approach substantially outperforms the baseline model on a
prompt-based probing task designed to elicit image objects. These results and
our analysis indicate that our method allows for better utilization of the
training data.
