{'fraction_non_alphanumeric': 0.034360844105623685, 'fraction_numerical': 0.017898574743122307, 'mean_word_length': 4.819029251044681, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation

Based on large-scale pretrained networks and the liability to be easily overfitting with limited labelled training data of multimodal translation (MMT) is a critical issue in MMT. To this end and we propose a transfer learning solution. Specifically and 1) A vanilla Transformer is pre-trained on massive bilingual text-only corpus to obtain prior knowledge; 2) A multimodal Transformer named VLTransformer is proposed with several components incorporated visual contexts; and 3) The parameters of VLTransformer are initialized with the pre-trained vanilla Transformer and then being fine-tuned on MMT tasks with a newly proposed method named cross-modal masking which forces the model to learn from both modalities. We evaluated on the Multi30k en-de and en-fr dataset and improving up to 8% BLEU score compared with the SOTA performance. The experimental result demonstrates that performing transfer learning with monomodal pre-trained NMT model on multimodal NMT tasks can obtain considerable boosts.


Introduction
Transformer-based models using large-scale parallel corpora have significantly improved the performance of neural machine translation (NMT), marking an important milestone (Vaswani et al., 2017). Additionally, multimodal machine translation (MMT) incorporating image signals into RNN-based encoder-decoder shows improvements on translation quality due to the forceful disambiguation (Specia et al., 2016a). In this paper, we aim to investigate, on top of Transformer, whether the paradigm of first pretraining and then fine-tuning can be effectively applied to MMT, concretely transferring from monomodal to multimodal tasks.
Constant attention has been paid on MMT task (Specia et al., 2016a) in the Conference of Machine Translation (WMT) in recent years (2016)(2017)(2018). Formally, it aims to learn a function mapping: X × I → Y, which takes source text and an image as input and translate them into the target text as shown in Figure 1. Additional modality is to disambiguate the source sentence, with the reference of image. However, the effectiveness of the visual context has been questioned by prior work (Specia et al., 2016b;Barrault et al., 2018;Caglayan et al., 2019). They show that visual context is not convincingly useful and the marginal gain is pretty modest, which is speculated to be resulted from the limitation of available datasets -the scale of parallel dataset of MMT task is not enough to train a robust MMT model. Compared with the translation corpus on news such as Common Crawl and UN corpus, commonly-used MMT dataset Multi30k (Elliott et al., 2016) is too small to train large-capacity models with millions of parameters. Therefore, it is imperative to put efforts on methods in low-resource MMT.
For the text-only NMT tasks, the Transformer (Vaswani et al., 2017) provides a novel architecture on language generation which supersedes RNN architectures rapidly with enhanced parallelizability. Meanwhile, the framework of pre-training and fine-tuning becomes a standard pipeline since BERT (Devlin et al., 2019) achieved the SOTA performances over a bunch of natural language understanding tasks. This to some extent suggests that transfer learning could effectively solve NLP tasks which requires deep understanding on the semantics but have limited size of in-domain data. Therefore, in this paper, we will investigate whether it's feasible to apply transfer learning to MMT task, i.e. transferring the prior knowledge learned from monomodal task into a multimodal task, as shown in Figure 2. The contribution of our work can be summarized as follows: • We propose the Visual Language Transformer (VLTransformer) which is compatible for both monomodal and multimodal inputs. The model achieves competitive results on Multi30k En-De and En-Fr tasks.
• We present a method of fine-tuning a pretrained monomodal MT model in the multimodal MT task, which is implemented by appropriately masking elements in both modalities to encourage the model to make full use of the input information.

Related Work
There are a spectrum of prior works investigating MMT. (Caglayan et al., 2016;Calixto and Liu, 2017) used standard RNN encoder-decoder with attention (Bahdanau et al., 2015) to fuse textual and visual features. Both of them employed pretrained image classification models like VGG and ResNet to extract visual features and combine with textual features with different schemes of attentions. Imaginet is proposed to predict the visual feature conditioned on textual inputs, which is used to improve the quality of the representation of contexts (Elliott and Kádár, 2017), where they decompose the MMT task into two sub-tasks where each can be trained separately with large external corpus. Hirasawa et al. (2019) extends the work of Imagination by converting the decoding process into a similarity based searching between the predicted embedding and the embedding of the vocabulary, which is achieved by optimizing a marginal loss on pre-trained word embeddings with predicted word embeddings. Besides, (Specia et al., 2016b;Barrault et al., 2018) make comprehensive summaries on the MMT tasks from MMT 2016 to 2018, which shows two major findings from the task: 1). The effectiveness of the additional modality is still questionable or limited, which encourages researchers to go further on the usage of visual information. 2). Fine-grained evaluation metrics have to be adopted to evaluate the true impact of the multimodality.
There are still some impressive works built upon Transformer-based architecture. MeMAD (Grönroos et al., 2018) achieves the best performance on flickr16 and flickr17 test sets with a multimodal Transformer model, which is pre-trained on massive out of domain data including OpenSubtitles and MS-COCO captions. They perform comprehensive experiments on the model with different data and model settings. (Zhang et al., 2020) proposes the method named universal visual retrieval which builds a look up table from topic word and image with TF-IDF. Before translation, m images are retrieved from the image set. Then, visual features will be aggregated with textual features to produce the hidden states. The UMNMT proposed in (Su et al., 2019) makes it possible to train a MMT model with bilingual but non-paired corpus and images. In their work, each language has an encoder and a decoder but shares one image encoder. They use the cycle-consistency loss to train the model by translating the text into target language, then, recover it back.
In summary, many approaches are proposed to tackle the MMT task from following two direction: • Improve the architecture of the model to make better use of visual modality.
• Leveraging external resources, monolingual or monomodal resources to enhance the performance.
However, we find that the pre-training and fine-tuning framework is under-investigated for MMT tasks, especially the cross-modal pre-training, which motivates us to explore in this work.

VLTransformer
First of all, we briefly review the architecture of Transformer (Vaswani et al., 2017). In the transformer, source texts are fed into the encoder and transformed into vectors with the word embedding and positional embedding, then, N layers of multi-head attention blocks are applied  to produce the hidden states H. For the decoder, the previously generated tokens until step t will be fed into the decoder to interact with the context H to predict the token of step t + 1. More formally, the encoding and decoding process is denoted as follows: where X and Y are source and target tokens, E S , H S and E T , H T represent for embeddings and hidden states of source and target texts respectively. We and Pe are word embeddings and positional embeddings. MHA represents for the Multi-head Attention blocks. y t+1 is the predicted token comes from the transformation of the last hidden state h T,t .

Image Embedding
To create high quality visual features, we use the Bottom-Up and Top-Down Attention (BUTD) (Anderson et al., 2018) to extract image features. Specifically, the Bottom Up attention of BUTD is based on Faster R-CNN (Ren et al., 2015) for object detection. They pre-train the model on the Visual Genome (Krishna et al., 2017) dataset which has fine-grained labels of objects with 1600 object classes and 400 object attributes. The extracted features are used as follows in the MMT model: where the pooled ROI features are represented by V ROI ∈ R m×dROI , d ROI = 2048 in the experiment, m is the number of detected objects. V c ∈ R m×1600 are predicted class one-hot vectors which will be multiplied with an embedding matrix in the experiment. V a ∈ R m×400 are attribute class one-hot vectors, and the bounding boxes V bbox ∈ R m×4 represents for normalized coordinates (x 0 , y 0 , x 1 , y 1 ) of detected objects. Coordinates are normalized into [0, 1] with the size of the image, i.e. x/x img , y/y img . φ represents for linear transformations to scale the dimensionality along with the original Transformer d model . The summation of 4 types of features simultaneously encodes most of necessary visual information, which is more fine-grained and informative comparing with previous works (Elliott and Kádár, 2017;Zhou et al., 2018;Caglayan et al., 2016) which only uses pooled ResNet (He et al., 2016) features or pooled object embeddings (Grönroos et al., 2018).

Fusion of Image and Text
In order to take the advantage of pre-trained NMT models and avoid overfitting using largecapacity network with limited multimodal labelled training data, we introduce parameters that needs to be trained from scratch as few as possible into the model. Therefore, instead of using architectures like LXMERT (Tan and Bansal, 2019) and the model proposed in (Zhang et al., 2020), where large sets of newly initialized parameters will be introduced into an independent image encoder, we share the original encoder layers of the Transformer to encode both modalities by directly concatenating the visual and the textual features. More specifically: where the Te represents for newly introduced type embedding inspired by the Next sentence prediction (NSP) of BERT (Devlin et al., 2019), which uses 0 for text and 1 for vision. E S is the replacement of Eq. 1. Finally, we concatenate embeddings of tokens and objects along the length dimension, as described in Figure 3. The sequence length becomes the summation of token number and detected objects number, |E S,V | = |V| + |E S |.
In such case, we only introduce a few amount of parameters to incorporate vision features, which reduces the perturbation on the Transformer Encoder and Decoder. In the experiment, we find that this can significantly improve the training efficiency on the small dataset. In addition, compared with the cross-attention method (i.e. H=SelfAttn(Token, Vision, Vision) which maps visual information onto token representations), concatenation reserves complete contexts in both modalities for the decoder, which is not limited by the length of source sentence.

Cross Modal Masking
In experiment, compared to using text-only inputs, we find that directly fine-tuning the pretrained transformer on multimodality inputs can't obtain extra performance boosts, which motivates us to investigate the reason behind that. Observing the attention map of encoder-decoder attention weights, we find that the model only assigns weights to text representations and entirely ignores visual information.
To force the model fully exploit both two modalities: text and image, we propose a cross modal masking (CMM) method to train the model with complementary information by partially masking out some inputs in one of any modality. Specifically, we randomly choose a modality to mask following the Bernoulli distribution, and then, randomly mask q tokens or q objects within specific modality. The masked token will be replaced by special token "¡unk¿" and the masked image region will be replaced by a noisy vector sampled from the standard normal distribution. This method is inspired by the masked language model (Devlin et al., 2019) and . Differently, they use the masking for unsupervised pre-training, while we use it directly in the translation task without predicting the masked place. Thus, masking here  only acts like the noise introduced in denoising autoencoder, it forces the model to learn by predicting unknown tokens and recover the corrupted vectors. We find it effectively prevents the model from neglecting visual contexts by CMM in training. See Figure 3 for more intuitive details.

Dataset
In the experiment, we use the Multi30k (Elliott et al., 2016) dataset to evaluate our method. The sizes of the dataset are 29000:1014:1000:1000 for training, validation, test2016 and test2017 set, each instance in form of triples (source, target, image). English descriptions are provided as source texts, German and French corpus are provided as target texts. All corresponding images are from Flickr30k (Young et al., 2014) dataset. We use the Moses toolkit (Hoang and Koehn, 2008) to pre-process the data with lowercasing, tokenizing and punctuation normalization.
For image features, we use BUTD (Anderson et al., 2018) to extract 4 groups of features for each object, including pooled ROI feature vector, object class, object attribute and bounding box. Maximum of 36 detected objects are reserved with the prediction probability higher than 0.5. The BUTD model is not fine-tuned in the translation task.

Setup
We use the pre-trained transformer model provided by fairseq (Ott et al., 2019) which is implemented with PyTorch (Paszke et al., 2019). The En-De model (Transformer-Large) is trained on WMT'19 corpus and En-Fr (Transformer-Big) model is trained on WMT'14 corpus. Both models share the vocabulary between source and target language, resulting in sizes of 42020 and 44508 for En-De and En-Fr vocabularies. The parameters of the embedding layer as well as the output projection layer are also shared for the encoder and the decoder in both models. The BPE (Sennrich et al., 2016)  The order for visual and text inputs are changed for more clearance). The difference between the left and the middle plot is that the cross-modal masking is performed on the middle one where "dog" is deliberated replaced by < unk >.
We can see that when the "dog" is masked, the model pays more attention on the visual features of two detected dogs. During fine-tuning, we use the learning-rate of 1e-4 with 4000 steps of warm-up and inverse-sqrt warm-up strategy. We use 0.3 for dropout probability, 0.1 for label smoothing (Pereyra et al., 2017), Adam (Kingma and Ba, 2015) is used as the optimizer. For the VL-Transformer, we use the parameter of fairseq pre-trained Transformer to initialize the backbone and text related embeddings, vision related parameters are initialized randomly. The model is fine-tuned on a Tesla V100 GPU with fp16 enabled and converges in less than 20 minutes for 10 epochs.
The baseline method is the pre-trained Transformer without fine-tuning. We use BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) as evaluation metrics with lowercased text.

Analysis
We compare our results with another six latest methods in Table 1. As the goal of newlyproposed NMTUVR (Zhang et al., 2020) is to improve universal NMT with multimodality, direct comparison with ours is unfair. As expectation, the pre-trained Transformer set a very strong baseline, which demonstrate that a well-trained text-only NMT model has been able to produce satisfying translations in the absence of word and phrase ambiguitity. At the same time, the profit of fine-tuning the Transformer is significant, even with only textual inputs. For the VLTransformer, the model trained without CMM is already better than the text-only method, which could demonstrates the effectiveness of visual contexts, in addition, the model trained with CMM is consistently better than the model without CMM, which demonstrates that CMM is a key point to improve the cross-modal interaction. Comparing with the MeMAD (Grönroos et al., 2018) which uses massive of external multimodal corpus (OpenSubtitles and MS-COCO), we only use the officially published training set for fine-tuning which is more efficiency. CMM. We filter 5 high score objects to investigate the alignment between target tokens and source inputs. There is evidence showing that the model is able to attend correct objects (i.e. two dogs) no matter the word "dog" is appeared in source texts or not (replaced by the < unk > or not), which means it could translate the sentence with both modality.
Although the attention map looks good, we actually manually amplify the score of visual features, in the experiment, we find that the model is more inclined to get contextual information from text instead of image although we have already used cross-modal masking. Some reasons can be speculated: 1) The size of training data is relatively small which means the newly initialized visual related parameters can not be fully trained. 2) We investigate the extracted detected objects and find out that there are mistakes in the detection which actually leads noise into the model.

Conclusion
We propose a cross-modal transfer learning solution to take full advantage of pre-trained monomodal model in the multimodal task. The approach of CMM to incorporate visual information into translation achieves remarkable results in the MMT tasks evaluated on Multi30k dataset, which reveals that prior knowledge of monomodal data can be transferred in a multimodal model even if fine-tuning on limited multimodal data. Furthermore, the shared encoder demonstrates perfect compatibility with the newly introduced visual features, which encourages us to dig into methods for visual and textual alignment with Transformer architectures. To sum, we show the evidence that our model is able to decode from both modalities after fine-tuning with the cross-modal masking method.