{'fraction_non_alphanumeric': 0.03340720676352294, 'fraction_numerical': 0.014733645243371911, 'mean_word_length': 5.032928942807626, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training

In this paper, we present a cross-modal recipe retrieval framework, Transformer-based Network for Large Batch Training (TNLBT), which is inspired by ACME~(Adversarial Cross-Modal Embedding) and H-T~(Hierarchical Transformer). TNLBT aims to accomplish retrieval tasks while generating images from recipe embeddings. We apply the Hierarchical Transformer-based recipe text encoder, the Vision Transformer~(ViT)-based recipe image encoder, and an adversarial network architecture to enable better cross-modal embedding learning for recipe texts and images. In addition, we use self-supervised learning to exploit the rich information in the recipe texts having no corresponding images. Since contrastive learning could benefit from a larger batch size according to the recent literature on self-supervised learning, we adopt a large batch size during training and have validated its effectiveness. In the experiments, the proposed framework significantly outperformed the current state-of-the-art frameworks in both cross-modal recipe retrieval and image generation tasks on the benchmark Recipe1M. This is the first work which confirmed the effectiveness of large batch training on cross-modal recipe embeddings.


INTRODUCTION
Cross-modal recipe retrieval investigates the relation between recipe texts and recipe images to enable mutual retrieval between them. Recipe1M dataset [20,16] is frequently used as a benchmark to evaluate the performance of cross-modal recipe retrieval frameworks. One challenge of the recipe retrieval task is that food images usually contain many non-food parts like plates and different backgrounds as noise. The recipe texts are perplexing, making them difficult to encode, since there are three components in the recipe text: title, ingredients, and instructions, the last two of which are long and structured texts. Most existing works focus just on recipe or image embedding learning, though we believe that a framework that focuses on both recipe and image embedding learning is necessary.
As one of the early works Salvador et al. [20] proposed Joint Embedding (JE), projecting embeddings from both modalities into a common shared space and minimizing the cosine similarity between them to enable cross-modal recipe retrieval. GAN [7] was introduced in some recent works [27,24,22] for food image synthesis from embeddings to obtain more reliable cross-modal embeddings. By confirming if the same or similar embeddings can be extracted from the food images generated from cross-modal embeddings, the retrieval performance were improved. Furthermore, modality adversarial loss was introduced in ACME [24] which can narrow the modality gap between image and text embedding. Inspired by the success of GAN-based image synthesis in recipe retrieval, we also adopt GAN-based image synthesis in this work. However, regarding recipe text embedding, LSTM [8] was simply applied for encoding recipe texts in these works [27,24,22], which sometimes limits leveraging the information in long and structured recipe texts. In order to address this issue, some other methods [6,26,19] focused on improving recipe embedding learning. Authors of MCEN [6] applied attention mechanism, and Zan et al. [26] applied BERT [4] for recipe encoding. With the trend of Transformer [23] in natural language processing, X-MRS [9] and H-T [19] were recently proposed to adopt this technique in recipe retrieval. Authors of H-T in particular introduced a simple but effective framework with a Transformer-based structure recipe encoder and self-supervised learning, allowing the model to explore complementary information among recipe texts. Inspired by H-T, we adopt a Transformer-based recipe encoder and self-supervised learning for recipe embedding learning.
In this paper, we propose a frustratingly straightforward Transformer-based framework that uses all the current state-of-the-art techniques. That is, we use a hierarchical transformer architecture recipe encoder for recipe embedding learning, and an adversarial network to investigate the complementary information between recipe and image embedding. In the experiments, we adopt ViT [5] and CLIP-ViT [18] as the image encoder backbone to validate the effectiveness the proposed framework. Since contrastive learning benefits from larger batch sizes according to the recent work on selfsupervised learning [2] and self-supervised learning and triplet losses are used in the proposed framework, we adopt large batch training in our experiments. Furthermore, we conducted extensive experiments and ablation studies to further validate the effectiveness of our proposed framework. We discovered that large batch training was surprisingly effective for Transformer-based cross-modal recipe embedding. The results showed that our proposed framework outperformed the state-of-the-arts both in recipe retrieval (medR 1.0, R1 56.5 in 10k test set size) and image generation tasks (FID score 16.5) by a large margin. More specifically, we summarize the contributions of this work as follows: 1. We proposed a frustratingly straightforward Transformer-based cross-modal recipe retrieval framework, Transformer-based Network for Large Batch Training (TNLBT), which achieves the state-of-the-art performance on both recipe retrieval and image generation tasks. 2. We conducted experiments with large batch inspired by the effectiveness of large batch training on contrastive learning [2]. 3. Through the comprehensive experiments, we confirmed that the proposed framework outperforms the current state-of-the-arts with a large margin, especially in the case of a large batch size, 768. This is the first work which confirmed the effectiveness of large batch training on cross-modal recipe retrieval as far as we know.

Cross-Modal Recipe Retrieval
Cross-modal retrieval aims to enable mutual retrieval between two different modalities, often image and text. The common idea of cross-modal retrieval is to embed features from two different modalities into a shared common space while keeping the distribution of corresponding embedded features close to enable mutual retrieval. A substantial issue in this task is how to narrow the gap between the various modalities [11]. Salvador et al., who was the first to propose the cross-modal recipe retrieval task and the Recipe1M dataset, proposed joint embedding [20] to enable cross-modal recipe retrieval. This method was modified in AdaMine [1] by using a triplet loss [21] to improve the retrieval accuracy. In order to further exploit the information in the Recipe1M dataset, some state-of-the-art techniques like Transformer [23] and Generative Adversarial Networks (GAN) [7] are used in recipe retrieval. GAN, enables image generation conditioned on recipe embedding in recipe retrieval framework. GAN was introduced in several previous works [27,24,22], which improves retrieval accuracy while enabling image generation conditioned on recipe text. The issues of these previous works are, the retrieval accuracy and quality of generated images are very limited since they only adopted simple LSTM [8] for text embedding learning. However, in our proposed method we obtain generated images of better quality using Transformer.
Rather than incorporating complex networks like GAN, some research [6,26,19] focus on recipe embedding learning to enhance the performance of retrieval tasks. Authors of MCEN [6] introduced cross-modal attention and consistency and Zan et al. [26] introduced BERT [4] as a recipe encoder to enable cross-modal retrieval. Authors of X-MRS [9] introduced a Transformer [23] encoder to gain recipe embedding, further proposed the use of imperfect multilingual translations, and achieved state-ofthe-art performances on retrieval tasks. Salvador et al. proposed a simply but effective framework H-T [19], to facilitate the power of Transformer. However, authors of these works [6,26,19] just simply applied bi-directional triplet loss on image and recipe features without image synthesis and reconstruction, which somewhat limited the crossmodal information learning between recipe and image features. In order to address this issue, we introduce GAN-based architecture to enhance the cross-modal embedding learning in our architecture in addition to adopting Transformer-based recipe encoders and self-supervised learning on recipe-only samples in the Recipe1M.

Food Image Synthesis
GAN [7] has been introduced and proven its effectiveness for improving recipe retrieval performance in some recent works [27,24,22,9]. Authors of R2GAN [27], ACME [24] and X-MRS [9] applied text-conditioned image synthesis which generates recipe images from recipe embeddings, while authors of RDE-GAN [22] proposed to disentangle image features into dish shape features, which contained only non-recipe information, and recipe embeddings and to integrate both of them to generate recipe images. Similar to ACME and RDE-GAN, we also leverage GAN in our proposed framework to improve retrieval accuracy while enabling image generation conditioned on recipe text. The biggest difference between our work and the previous works is, we applied Transformer-based encoders to further improve the quality of generated images compared to the previous works [24,22].

Overview
We propose a frustratingly straightforward Transformer-based framework TNLBT for cross-modal recipe retrieval which has the hierarchical Transformers for text encoding, and Vision Transformer (ViT) [5] as an image encoder. Furthermore, we propose to adopt large batch training which is confirmed to be beneficial for contrastive training and Transformer-based networks. Figure 1 shows the architecture of the proposed framework. Similar to H-T [19], our proposed framework applies hierarchical Transformer encoders and a triplet loss L rec with self-supervised learning to explore complementary information in the recipe text. ViT-B [5] or CLIP-ViT [18] is used as the backbone of image encoder. The biggest difference between our work and H-T [19] is, we introduce a more sophisticated adversarial network to leverage the learned Transformer-based embeddings. Rather than just a bi-directional triplet loss is adopted in H-T for the distance learning of image and recipe embeddings, we further introduce several loss functions to enhance the cross-modal embedding learning and enable image generation conditioned on recipe text. Inspired by the success of triplet loss [21], a triplet loss L ret for distance learning in image-recipe retrieval is applied in the retrieval space. In order to mitigate the modality gap problem, we use the modality alignment loss function L M A as in ACME [24]. Furthermore, we introduce image-to-recipe and recipe-to-image information recover, and a translation consistency loss function L trans to keep modality-specific information. We expect that the embeddings learned in the above process correctly retains the original information and enables image generation from recipe embeddings. Finally the overall loss function can be formulated as follows: where λ 1 , λ 2 , λ 3 are hyperparameters to control the loss balance.

Recipe Encoder and Self-Supervised Learning
The purpose of the recipe encoder, E R (even though we adopt several encoders here, we note these as a whole recipe encoder E R ), is to encode recipe text, t, to recipe embeddings, R. In order to better explore and make use of the huge information among recipe text with components of title, ingredients, and instructions, we adopt the hierarchical Transformers as the recipe encoder similar to H-T [19]. Note that we introduced an adversarial network for image synthesis and modality alignment better to exploit the correlation in image-recipe pairs, while H-T simply applied a bi-directional triplet loss function to image-recipe pairs. For the three components (title, ingredients, instructions) in the recipe texts, the proposed framework TNLBT encodes them separately using hierarchical Transformer encoders (E ttl , E ing , E ins ), with 2 layers and 4 attention heads, to obtain embeddings of title, ingredients and instructions (e ttl , e ing , e ins ) initially. Next, we apply self-supervised recipe loss L rec as in H-T, on e ttl , e ing , e ins to explore the complementary information among them. The introduction of this loss allows us to use recipe-only image-recipe pairs to further leverage the complementary information among the three components of the recipe text. Finally, we apply a merging encoder E mrg to project three components embeddings (e ttl , e ing , e ins ) to an unified recipe embedding, R.

Image Encoder
The purpose of the image encoder E V is to encode recipe image i to image embedding V. We adopt the base size model of Vision Transformer (ViT-B) [5] pre-trained on ImageNet-21k [3] as the image encoder backbone. In order to better leverage the benefit of large batch training, we also adopt CLIP-ViT [18] as image encoder backbone where the proposed framework achieved superior performance on retrieval task.

Modality Alignment Loss
For the purpose of narrowing the gap between recipe and image embeddings, which often results in bad generalization or slow convergence, we applied modality alignment loss L M A as in ACME [24]. We adopt a discriminator D M here to achieve this goal, which aligns the distribution of recipe embeddings R = E R (t) and image embeddings V = E V (i) during training. An adversarial loss is applied by controlling the recipe and image embeddings aligned to each other resulting in the discriminator D M cannot distinguish the source of the given embeddings empirically. Following ACME, we also adopt WGAN-GP [10] here and the loss function is formulated as follows:

Retrieval Loss
Following the success of triplet loss [21] in the recent works [1,24,22,19], we adopt a triplet loss for distance learning. Here we obtain the recipe and image embeddings R = E R (t), V = E V (i). We obtain an anchor recipe embedding R a and an anchor image embedding V a , and obtain a negative sample with subscript n and a positive sample with subscript p to process this distance learning. The loss function is formulated as follows: where margin α = 0.3, d(.) is the Euclidean distance, [x] + = max(x, 0). In addition, we adopt a hard sample mining strategy [12] to further facilitate the distance learning.

Translation Consistency Loss
The learned embeddings are useful for training but sometimes lost the modality-specific information which is meaningful and important. To alleviate this information loss, we adopt translation consistency loss following ACME [24], which ensures the learned embeddings preserve the original information across modalities. We accomplish this goal by forcing the recipe and image embeddings to recover the information in the other modalities: recipe image generation and ingredients prediction conditioned on recipe embedding and image embedding respectively. Hence, the total translation consistency loss is composed of the losses for the recipe and the image as follows: Image Generation from Recipe Embeddings In order to preserve the modalityspecific information in recipe embeddings, we aim to recover the information in image modality to ensure this property, and we set two losses as for two-fold goal: (1) we expect the generated image is as realistic as possible, (2) we expect the generated realistic image matches the target recipe. To accomplish these two goals, we adopt GAN [7] to generate images from recipe embeddings and introduce a loss L r2i for goal (1) and a loss L clsr2i for goal (2) following ACME [24]. During training, the discriminator D r2i is used to distinguish the generated image and real image, and the generator G is used to generate food images which is conditioned on the recipe embedding F C(E R (r)). The loss L r2i for ensuring the generated images realistic is formulated as follows: While ensuring the generated image is as realistic as possible, we introduce loss L clsr2i to ensure that the generated image matches the target recipe. During training, a classifier cls r2i is used to encourage the generator to generate a food image with the corresponding food category to recipe embedding. L clsr2i is simply a cross-entropy loss. Combining these two loss functions for two goals separately, the translation consistency loss for the recipe is formulated as follows: Classification and Prediction from Image Embeddings We adopt ingredients prediction and title classification on image embeddings here for ensuring the translation consistency of images. We leverage a multi-label network here to predict the ingredients from image embeddings using a 4,102-d one-hot vector representing the existence of 4,102 different ingredients. We denote this multi-label objective as L i2r . The same as ensuring the generated image is with the correct food category in defining L clsr2i , we propose to ensure the predicted ingredients are from the food with the correct category, which means the image embeddings can be classified into the correct food category. We use L clsi2r here to make sure the image embeddings can be classified into one correct food category of 1,047 recipe categories, where L clsi2r is a cross-entropy loss. Combining these two loss functions, the translation consistency loss for the image is formulated as follows:

EXPERIMENTS
In this section, we present the experiments to validate the effectiveness of our proposed framework TNLBT both in recipe retrieval and image generation tasks, including ablation studies and comparison with the previous works. Extensive experiments are also performed to further validate the effectiveness of TNLBT.

Implementation Details
Data set. We evaluated the performance of our proposed method on Recipe1M [20,16] following the previous works. Following the official dataset splits, we used 238,999 image-recipe pairs for training, 51,119 pairs for validation, and 51,303 pairs for testing. Evaluation Metrics. Following previous works, we evaluated retrieval performance with the median rank (medR) and R@{1,5,10} on two different test set sizes 1k and 10k. We reported the average metrics of 10 groups which are randomly chosen from the test set. We also evaluated the performance of image generation conditioned on Table 1: Comparison with the existing works on retrieval performance. The performance of recipe retrieval is evaluated on the criteria of medR(↓) and R@{1,5,10}(↑). Especially all the R@K metrics were all improved by around 10.0% in TNLBT-C compared to TNLBT-V.  Method FID ACME [24] 30.7 CHEF [17] 23.0 X-MRS [9] 28.6 TNLBT-V 17.9 TNLBT-C 16.5 Table 3: Evaluation of the importance of different components on image-to-recipe retrieval in TNLBT. * means training without recipe-only data.  Table 4: Comparison on image-to-recipe retrieval performance with different batch sizes adopted in TNLBT. Results are reported on rankings of size 10k.
TNLBT-V TNLBT-C #batch medR R@1 R@5 R@10 medR R@1 R@5 R@10  recipe embeddings using Fréchet Inception Distance (FID) [13] score following previous works [22,9] measuring the similarity of the distribution between generated images and ground-truth images. Fig. 2: Generated images conditioned on different recipe embeddings. The first row are recipes used to generate images, the second row are the ground-truth images which recipe embeddings aim to recover. The third row are the generated images.
Training Details. The objective is to minimize the total loss function Eq. 1. We empirically decided the hyperparameters in Eq. 1 as follows: λ 1 = 0.05, λ 2 = 0.005, λ 3 = 0.002. For ViT-B [5] image encoder backbone framework, We trained the model for 50 epochs using Adam [14] with a learning rate of 10 −4 . For CLIP-ViT [18] image encoder backbone framework, we first trained the model for 20 epochs while freezing the image encoder with a learning rate of 10 −4 and then we trained the model for another 100 epochs with a learning rate of 10 −6 where the image encoder were not frozen. Since contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning [2], batch size was set to 768 during training.

Cross-Modal Recipe Retrieval
We evaluated and compared the performance of the proposed framework TNLBT with the previous works in Table 1. TNLBT with ViT-B [5] image encoder backbone (TNLBT-V) outperforms all the existing works across all metrics with large margins. We also tested the performance of our proposed framework with CLIP-ViT [18] image encoder backbone (TNLBT-C), which outperforms all the existing works across metrics with even much larger margins. When the image encoder backbone was changed from ViT-B to CLIP-ViT, all the R@K metrics were all improved by around 10.0%, and medR on 10k test set size was improved to 1.0, which has never been achieved before.

Image Generation
The proposed model generates images of 128 × 128 resolutions conditioned on recipe embeddings. We computed the FID [13] score to evaluate the image generation performance of our proposed method 1 . The results are reported in Table 2, where TNLBT outperformed the previous works [24,17,9] with a large margin on image generation 2 . Figure 2 shows qualitative results on image generation conditioned on recipe embeddings, showing that TNLBT could generate appropriate recipe images corresponding to the given ingredients in recipes.

Ablation Studies
We also performed ablation studies to evaluate the importance of the applied components in Table 3. When L M A was applied, the accuracy decreased incrementally while with more loss functions applied the accuracy increased again. However, when only L M A was taken, the retrieval accuracy decreased compared to the combination of all loss functions. Hence we empirically think there is a complementary relationship among L M A and the other loss terms except L ret , contributing to the retrieval performance. The loss combination without L transi applied outperforms all the other combinations by an incremental margin. Finally, when the proposed large batch training strategy and CLIP-ViT [18] applied, the retrieval accuracy was further improved.

Batch Size in Cross-Modal Embedding Learning
Since contrastive learning benefits from large batch size was reported in [2], we performed extensive experiments here to validate the influence of batch size on both recipe retrieval and image generation, where we are the first to validate this influence.
Recipe Retrieval. Table 4 shows the image-to-recipe retrieval performance of TNLBT with different batch sizes. Retrieval performance was improved substantially when the batch size increased from 64 to 512, indicating the importance of adopting large batch during training. The improvement by increasing batch size is limited afterwards and the performance was hurt incrementally when batch size increased to 1024. Image Generation. We also investigated the influence of batch size on image generation performance in Table 5, where the performance of image generation hurt with batch size increasing, while the performance of recipe retrieval improved according to Table 4. Hence, we emperically believe that batch size serves as a trade-off here.

CONCLUSIONS
In this research, we proposed a frustratingly straightforward Transformer-based Network for Large Batch Training (TNLBT) using the Hierarchical Transformer-based recipe encoder, the Vision Transformer-based image encoder, and a sophisticated adversarial network for cross-modal recipe embedding learning. We further adopted selfsupervised learning to investigate the complementary information in recipe texts. Through the experiments, it was confirmed that TNLBT outperformed the state-of-the-arts on both cross-modal recipe retrieval and food image generation tasks by large margins. We also found that CLIP-ViT [18] achieved much better performance than ViT-B [5] as an image encoder backbone in TNLBT. With the extensive experiments, we found that the retrieval performance could benefit from a large batch size while the performance of image generation conditioned on recipe embeddings sometimes got hurt by a large batch size, where we were the first to validate the influence of batch size in recipe retrieval and image generation tasks.