{'fraction_non_alphanumeric': 0.03330018286227583, 'fraction_numerical': 0.013057008116518559, 'mean_word_length': 4.428770463253222, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation

Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space


Introduction
Learning cross-modal visual and textual representation is essential for bridging the semantic gap between images and languages. It is the cornerstone for a wide range of visionlanguage (V+L) tasks, such as image-text cross-modal retrieval, visual question answering (VQA) [2], image captioning [2], and so on.
Inspired by the success of BERT [9] and XLNet [48] using self-supervised learning on natural language processing, there has been a surging research interest in vision-language pre-training on image-text pairs. The learned task-agnostic representation is shown to be effective for many imagelanguage applications after fine-tuning on specific down-* equal contributions.  Figure 1: An overview of the end-to-end LAViTeR network. VTA module is assisted by ITM and TIM modules, which in-turn learns to better align the corresponding visual and textual counterparts. The bidirectional arrows indicate the alignment between words and their respective objects in the given image. The intra-word arrows indicate the relationships between the input words that the network learns. stream tasks. Self-supervised learning is designed to explore the organization of the data as its own source of supervision. This promising approach releases the burden of annotating data with ground truth labels, provides an opportunity to explore a large amount of unlabeled data such as image-text pairs, video-text pairs in free form format from online platforms. This approach has been applied to radiology images combined with their associated reports [24,5] to leverage the abundance of unlabeled medical data. This

Related Work
In this section, we discuss related work about multimodel representation learning and alignment. We also briefly review two main tasks in our architecture to pre-train the joint representation, the GAN-based image generation and image captioning.

Multi Model Representation Learning
VilBERT [26] and LXMERT [37] are the two pioneering works in image-text joint representation learning, utilizing two streams of Transformers to images and text independently. Those image and text representations then fused by a self-attention mechanism in the later stage. After those two pioneered work, single-stream architecture has also been proposed using a single Transformer to jointly learn imagetext embedding, such as UNITER [6], VisualBERT [22], Unicoder-VL [21], VL-VERT [35], B2T2 [1]. Typical selfsupervised learning tasks, such as masked language modeling, masked region modeling, image-text matching, and word-region alignment are applied to pre-train the models. More recently, VILLA has been proposed using adversarial training as a general framework can be applied to any V+L models [11].
There are quite a lot of work trying to explicitly enforce the word and region alignment, such as VisualBERT [22], UNITER [6], Oscar [23], MMAC [44]. For example, UNITER [6] uses the Optimal Transport [30] to explicitly calculate the minimum cost of transporting between the image embedding to word embedding.

Text to Image Generation
Image synthesis from text is a fundamental task in multimodal learning across vision and language. Most proposed works in image generation are based on conditional Generative Adversarial Networks (GAN). A common approach utilizes a text encoder, mostly RNN-based text encoder before the popularity of Transformer, to encode the text description to guide the image generator [32,50]. The attention mechanism is widely used to guide the generator to focus on certain words when generating specific regions [47]. The attention mechanism is used to capture the similarity between the generated images and the sentences in both the global level and fine-grained word/region level. MirrorGAN [31], tries to learn better text-to-image generation by re-describing the generated images.

Image Captioning
Image Captioning is the reverse process of text-to-image generation. It typically consists of a CNN encoder and an RNN decoder to transfer the information from images to the generated text description [41]. Attention mechanism has been shown very effective focusing on salient objects while generating the corresponding words [46]. Following the success of Faster-RCNN [33] in object detection, bottom-up features provide informative regions in the image, which are used for region level attention [2,17] and visual scene graph modeling [49]. Recently, transformer architecture [39] is also used for image captioning, which further boosts the captioning performance with its implicit self-attention mechanism [14,15,51].

Learning Aligned Visual and Textual Representations
As shown in Figure 2, our proposed Learning Aligned Visual and Textual Representations (LAViTeR) model consists of three modules: the visual text alignment (VTA) module, which contains image encoder F I and text encoder F T ; the text-to-image (TIM) module, which is the image generator G; and image-to-text (ITM) module with the text generator C. VTA learns visual-text representations via matching real image-text pairs as our main task. In addition, we introduce two novel assisting tasks, illustrated in blue and green dashed boxes, for a better representation learning: fake-image-real-text matching with fake images generated from TIM; and real-image-fake-text matching where fake texts are converted from ITM using VTA image representations. Meanwhile, both TIM and ITM are also trained on their own losses. We will introduce each of them in the following part of this section.

VTA: Visual Text Alignment
Given an image-text tuple I i , T i , we want to learn the alignment between the words in T i and the parts of image I i . For this, we first extract the global features v and local features r using an InceptionV3 [36] based image encoder, where r ∈ R D×M is flattened from the intermediate feature map of 'mixed 6e' layer and v ∈ R D from the last average pooling layer. Both of them are projected to the representation feature space with a projection layer. This is denoted as a function F I in Figure 2, such that r, v = F I (I). Next, we extract sentence and word level features s and w respectively using a Transformer [39] based text encoder. In Figure 2, this text encoder is denoted as a function F T , such that, w, s = F T (T ), where w ∈ R D×N and s ∈ R D .
We define the F T as follows: Given text T ∈ R N , we use word-based token to embed it as e ∈ R D×N , which is summed with positional encoding as the input of F T . In transformer encoder layer, e is first transformed into queries Q = W T Q e, keys K = W T K e and values V = W T V e within each attention head, where W Q , W K , Q V ∈ R D×D k in our setting. Then "scaled dot-product self-attention" is applied to e as follows: where f s is the softmax function.
Multi-head attention is applied to the self-attention sublayer and the outputs from h heads are concatenated: where The output from the multi-head attention is then sent to a feed-forward network. The residual mechanism [13] and layer normalization are applied to multi-head attention and FFN outputs: and s =w is used as the sentence representation feature. Similar to [47] we align the word to image regions by implementing a word-level attention mechanism. First, the word-region attention score α is obtained by multiplying the query, w with context, r, and then normalizing the product using softmax. Next α is multiplied with the context r to obtain the contextual vector c.
Where m ∈ R N ×M is the match vector; f s N is the soft-max operation along N words in text T i ; f s M is the soft-max operation along M sub-regions of image I i ; γ 1 is a hyperparameter to tune the required amount of visual attention for a word and is a matrix multiplication operation. Next, we calculate the element wise cosine similarity between c and w as cos = (c T w)/(||c||||w||) and compute the image to text matching score S by following the work done in [3,45]: where γ 2 is the importance magnification factor of the most relevant word and image sub-region in the given pair Finally, inspired by to [10,18], we calculate the the posterior probability P of image I i matching with text T i in a batch of B paired samples:  where f s B is the soft-max of matching score S over B paired samples and γ 3 is a hyper-parameter and L IT mw is the loss when features of image sub-regions r are matched to features of words w in text. Here, the text and image samples at different index are considered as negative pairs and the samples at same index are considered positive pairs. To maintain the symmetry, we also calculate L T I mw where the image and text variables are switched: Furthermore, we calculate the sentence level matching loss L IT ms by computing cosine similarity between global vectors v and s as cos = (v T s)/(||v||||s||) and substituting the value of cos in Equation 5, 6 and 8. Similar to before we can calculate L T I ms by switching s and v. Thus we compute the total matching loss by adding all the losses.
During the first phase of training the objective is to reduce L m for pairs of real image I and corresponding text T . For this training we preset γ 1 , γ 2 , γ3 as per the settings defined in [47] and the batch size B = 8.

TIM: Text to Image Module
To learn the parameters for transforming the textual domain to visual counterpart we use Conditional Generative Adversarial Networks [12,28] with the sentence vector s as the conditional input. Inspired by the AttnGAN [47], we employ a cascade of GANs. Formally, our TIM task has k discriminators with coupled generators G ∈ {G 1 . . . G k } that generate imagesÎ ∈ {I i1 . . . I ik } of different scales, where the suffix i indicate the i th datapoint. We utilize only the output of the k th generator for calculating the assisted losses explained in Equation 15.
We first transform a real text sample T i using the transformer [39] based text encoder F T to output a sentence vector s and word vector w ∈ R D×N . We denote the entire image generation system, including generators and discriminators as a function G as displayed in Figure 2. A 1-D uniformly sampled random noise vector z along with w as condition, are merged and fed as input to G which outputs an RGB imageÎ i corresponding to T i The image generative loss L G is thus calculated in an adversarial setup and is defined as follows: We train the discriminators, to learn to distinguish between the real I and fakeÎ samples, alternately with the generators while reducing the cross-entropy loss as below: (10) where I is from the read data distribution P data andÎ is from the generated data distribution P G . In Equation 10 the first and the second terms are unconditional and conditional losses as defined in [50]. Furthermore, we calculate the matching loss L I m between fake image vectorsr,v and real text vectors w, s respectively, similar to the process outlined in Subsection 3.1. Hence the total loss for the TIM can be defined as: where λÎ is the hyper-parameter which is tuned to get better performance and the matching loss L I m between the generated imageÎ i to real text T i , as explained in Subsection 3.4.

ITM: Image to Text Module
The Image-to-Text branch (ITM) aims to generate fake text to assist VTA training in 3.1. It has been shown that the attention mechanism leverages the performance of image captioning models [46]. Inspired by the power and success of the transformer layer in various vision tasks [17,15,14,4], we decide to use a transformer based image captioning model C as ITM.
Following the implementation of [39], we use a stack of transformer layers for the transformer encoder C e and decoder C d . Similar to [46], in order to pass image regional features into the transformer, we use the flattened regional feature sequence r ∈ R D×M from F I as the input to C e , where each column of r is a representation corresponding to a certain part of the image. As for the decoder, given the caption T 1:N of length N , all the words prior to the target position p are embedded as e 1:p−1 ∈ R D×(p−1) , which is used as the decoder inputs to predict the pth wordT p .
According to [4], when passing the image features into the transformer, in order to supplement the permutationinvariant issue, it's better to add positional encodings to the input of each self-attention sub-layer in C e and encoderdecoder cross-attention sub-layer in C d instead of only applied at the bottom of encoder. Similarly, the positional embeddings for text input are also added to the input of each self-attention sub-layer in C d .
Taking r as input, C e applies the same operation as Eq. (1-3) in each transformer layer and refines the regional visual features via self-attention mechanism. Then it's top output r e = C e (r) is passed to the encoder-decoder crossattention sub-layer within each transformer layer of C d to further introduce the visual-language cross-attention mechanism for image-to-text generation. The sub-layer takes r e as keys along with the self-attention sub-layer outputẽ 1:p−1 as queries: The decoder output is sent to MLP to predict the probability of word at position p: Given the ground truth caption T 1:N , we train the ITM with cross-entropy loss:

Assisting losses
Given the generated imagesÎ from TIM in Subsection 3.2 and generated textsT from ITM in Subsection 3.3, we introduce two assisting matching losses in the model training: fake-image-real-text matching loss L I m and fake-textreal-image matching loss L T m . Similar to L m in Subsection 3.1, we inputÎ to F I and Along with L G and L C , our final multimodal loss for joint training is: where λ m , λÎ , λT , λ G , λ C are hyper-parameters to add weights for each loss above.

Experimental Results
In this section, we explain our experiment settings, evaluation metrics and results to evaluate our proposed model.

Datasets
Our model is evaluated on two public datasets including CUB [42] and MS-COCO [25], which are widely used in text-to-image generation and image captioning tasks.
CUB is a dataset contains 200 bird species and is popular for classification, text-based image generation and image captioning tasks. It has 8855 training images and 2933 test images, where each image has 10 text descriptions. Since CUB only contains bird images with various attributes, the semantic domain is relative simple and narrow for image text representation, which is suitable for evaluating our model in a specific semantic task. We preprocess the dataset according to the method in [47].
MS-COCO is a challenging dataset which is popular for various image-text related tasks including image captioning and image-text matching. It has 82783 training images which paired with 5 captions per image and 40504 images for testing. It provides large amounts of common object classes in the images, which can show the representation performance of our model in generalized semantic space.

Evaluation Metric
We employ R-precision a technique proposed in [47] to rank the retrieval results. To evaluate the task of reducing the heterogeneity gap between relevant word and image representation, we also propose an Attribute to Image Matching Cosine Score (AIMCoS) which matches the similarity of specific textual attributes which are supposed to be found in the paired image. Specifically, first for CUB dataset, we extract the attributes corresponding to each image file from the validation dataset. Each attribute contains text entries that list the features like color, bill length, shape, etc. of a bird. For each experiment, we first extract the global features v of the image I i using F I . Next, we extract the features s of textual input of each attribute associated with image. Afterwards, we compute the average of cosine similarity f cos of v with the representations of each attribute corresponding to I i . Finally, we find the mean of match scores for all the images in the validation set. Mathematically if there are K attributes present in image I i then where U is the number of images in the validation set of CUB dataset. For computing the AIMCoS using F I and F T trained on COCO dataset, each of the 80 classes are considered as attributes. We then create a smaller evaluation set (LAViTeR cocoeval ), which contains 100 images, extracted from ImageNet [7] dataset queried on the keyword. Next, similar to Eq. 17 we calculate the mean of f cos between the representations of each 100 images and the representation of text the respective class name. Lastly, this score is averaged over all the classes to calculate AIMCoS for COCO dataset. In Figure 3, the larger point with label present within each cluster is the representations of the textual labels. The other points of the cluster are the representations of image data-point. The LAViTeR model is able to bridge the representational gap between textual and visual data effectively while still performing at par with other techniques.

Implementation Details
The LAViTeR model is implemented in Pytorch [29] and all the experiments are carried out on NVIDIA GTX 1080ti Three training phases are designed for LAViTeR in order to make the joint training more stable and easier to converge. In phase 1, we pretrain VTA on the training data so that F I and F T can generate acceptable image and text representations for the following phases. We freeze F I except the output layers for r, v, and train the transformer layer in F T and the output layers in F I with a learning rate 0.0002. As for phase 2, C in ITM and G in TIM are pretrained separately with the r, v, w, s features from the pretrained VTA as inputs. The learning rate is set to 0.0002 for G, and for pretraining C it is first set to 0.0001 and then decay by 0.1 after 20 epochs. When the pretrainings are completed, we freeze the first 5 layers of F I and jointly train all the modules together as our final LAViTeR model in phase 3, with a small learning rate of 10 −6 . All the training are optimized with the Adam optimizer [20] with a weight decay 0.0001. The batch size is set to 96 for phase 1, 32 and 14 for C and G pretraining in phase 2, and 8 for the joint training due to the limitation of our GPU memory. In all experiments, γ 1 = 4, γ 2 = 5, γ 3 = 10 are used for the matching losses.

Performance
Hyper-parameter Selection We first adjust the λ in our multi-model loss (see Eq. 16) in order to test the effect of different loss terms in our joint training and select the best λ setting for our multi-modal loss. The results on CUB and COCO are shown in Table 1 Table 1: The best top-3 R-precision score of each LAViTeR model setting on COCO (top nine rows) and CUB (bottom 6 rows) test set. AIMCoS score is also reported.
As we can see, different λ settings have influenced the performance of the model. The best setting on CUB is from the bottom row in Table 1, which gives the best top-3 Rprecision as 66.8%, where we choose a relative large λ m = 5 for loss L m while keeping small weights as λ G = 0.01 and λ C = 0.1 for L G and L C . Meanwhile, row 9 in Table  1 with a similar setting (λ m = 10, λ G = 0.01, λ C = 0.1) also gives the best top-3 R-precision on COCO as 90.79%, and row 3 in Table 1 which adjusts λ m to 50 gives the best AIMCoS score as 0.5063. These results show that L m requires higher weight among all the loss terms, since it's a credible objective for real-image-real-text matching. On the other hand, L I m and L T m are calculated from the fake image or fake text matching, which may induce some noises in the gradients due to some low quality generated images and captions, thus smaller λÎ and λT gives better results. Moreover, since L G and L C is only used for necessary updates in G and C during joint training and has limited contribution to VTA, thus we just keep tiny weights for them.
Ablation Studies and Analysis We also conduct necessary ablation studies in Table 2 to further inspect the effect of each modules of LAViTeR. First, we set all the λs to 0 except λ m , which results in the worst R-precision among all the settings on COCO dataset (see the top row in Table  1). This case shows the importance of our assisting losses during the joint training. Since L m trains the image and text encoders but has no effect on C and G, the updated feature outputs from F I and F T are no longer working well for the captioning model and image generator, hence the quality of generated images and captions degraded which also affects the matching performance.
In Table 2, we test the performance of LAViTeR when different modules are removed. Since symmetric globallocal matching loss L m is also used to train the image-text matching model (called DAMSM in their work) in [47,31], so we also evaluate their performance using our metrics, which refers to AttnGAN/MirrorGAN in Table 2 and 3. DAMSM uses RNN-based text encoder in their paper instead of the transformer-based model like ours, so it also plays a role as one of our baseline. In our ablation settings, LAViTeR is our final model. L M (F ) and L M (T ) mean that only the VTA module is trained in phase 3 as our baseline without TIM and ITM branches, and the image encoder backbone is frozen (F , same training method as DAMSM) and trainable (T ) during the joint training, respectively. LAViTeR-Img2Txt only keeps the ITM module in the joint training to generate image captions for L T m and the image generator G with L I m is removed; As a complementary setting, LAViTeR-Txt2Img keeps the image generator and L I m and deducts the captioning module C along with L T m . As it shows in Table 2, L M (T ) improves the performance for 0.66% in R-precision and 0.005 in AIMCoS, which shows that the image encoder requires update during training for a better performance. In LAViTeR-Img2Txt where G is trimmed, the R-precision drops by 0.57% from the full model and even lower than the baseline F M (T ). This might be related to the large amount of unseen fake images no longer being generated and used for VTA training, thus reduces the diversity of the training set and the matching accuracy. Meanwhile, if C is removed as in LAViTeR-Txt2Img setting, both the R-precision and AIMCoS are reduced by 0.28% and 0.0023 respectively, which could be caused by lack of fake captions. When no captions are generated for matching, the training sentences and words become less various and thus affect the image-text matching performance shown in R-precision as well as the word level matching accuracy suggested by AIMCoS. On both CUB and COCO datasets, our full model LAViTeR works much better than DAMSM in [47,31], which improves R-precision by 0.74% in COCO and 4.2% in CUB, and boosts AIMCoS from 0.44 to 0.5063 in COCO and 0.072 to 0.201 in CUB, almost 3 times better. This result shows that image-to-text and text-to-image generation branches can assist the image-text matching/representation model training by aligning generated image with real text and generated text with real image.
We also find an interesting phenomenon in our baselines. When replacing the original RNN-based text encoder in DAMSM with our transformer-based text encoder, L M (F ) has a slightly lower R-precision but much higher AIMCoS than DAMSM baseline in both Tables. This actually shows one of the main difference between transformer layer and RNN layer: due to the recurrence feature and short memory of RNN, one layer transformer may have no obvious advantage over RNN for captions with normal length around 15, which is implied in R-precision for image-caption matching; however, RNN cannot get good representations of short phrases with only 1-3 words, since it requires enough context to give reasonable outputs. In contrast, transformer's self-attention mechanism is able to effectively handle any text length and align even a single word with the corresponding image regions, which is observed in the substantial increase in AIMCoS.   Visualization Analysis In addition, we illustrate some qualitative results from image-to-text and text-to-image matching on COCO in Figure 4 and Figure 5 respectively. For image queries, the captions with top-5 similarity scores from our model are retrieved. Most captions are correct matches. It is observed that sentences that have "incorrect" match labels actually share similar semantics with the image queries. In Figure 5, text queries with the top-3 image matches from our model are listed. The correct matching is retrieved with other similar images, and we find that all the high ranked images are quite reasonable. These "incorrect" matching pairs with close semantics data expose a drawback in the current evaluation metrics for multi-modal matching: many images along with their paired captions in the test set share similar semantics and are close to each other in the common semantic space, however, these neighbor samples are treated as "mismatch" in the image-to-text and text-to-image retrieval evaluation metrics and show no difference with those distinctly mismatched samples with large semantic gap. This evaluation defect motivates us to propose the new evaluation metric AIMCoS.

Conclusion
In this paper, we introduce a novel architecture for visual and textual representation learning assisted by two auxiliary tasks, image-to-text generation and text-to-image generation. The generated images and texts are matched with real text and images to jointly train the representation model with two assisting matching losses. A new evaluation metric AIMCoS is proposed for measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets demonstrate the effectiveness of the proposed architecture and evaluation metric.

Appendices
Below we mention some analysis and downstream tasks performed using our pre-trained encoders.

A. VTA Qualitative Results
Apart from the T-SNE visualization in main paper, we also plot the Image representation vs Token representation similarity map in Figure 6. The x-axis denotes the image representations and the yaxis denotes the category word token representation. Since there are 80 classes in COCO dataset [25], the matrix size is 80 × 80. There are 80 Tokens and 8000 images, i.e. 100 images per token. We first compute similarity of each token representation with all the 8000 images in LaViTeR cocoeval dataset. Then, the average similarity value of match with each token plotted in Figure 6. The dark color diagonal shows a high similarity between true matches, indicating the efficacy of LAViTeR model.
From Figure 6, we observe that, the token "apple" matches highest with the images of apple but also contains high similarity with images of another fruits like "orange" because they are usually kept together. Similarly, the token "microwave" is highly matched with images of "microwave" and also with "oven" and in reality most of the times they appear together. More interestingly, the token "person" is not matched with other images showing no correlation, but is matched highly with images of "person" and also "skateboard" as in many "skateboard" images there is a person in the frame. Also, the token "spoon" is highest matched with "spoon" images and also slightly matched with images of "fork" and "knife" as these items usually appear together naturally. Thus, the joint training is able to reduce the diversity gap between the nature of textual and visual representations meaningfully.

B. TIM Qualitative Results
Although our primary goal is not to improve GAN, we compare the quality of generated images from LAViTeR model with the ground truth and the generated images by [31,47]. Those examples are selected from the visualization of the previous state-of-the-art papers directly.
As can be seen from Figure 7 and 8, the LAViTeR model is able to perform at-par and in some cases better than the previous state-of-the-art models. We do not report these images in the main paper as the major goal of LaViTeR is for representation joint learning and alignment. The model is able to perform better due to continuous training of VTA, TIM and ITM models cooperatively. The generated images during the training process are able to provide a diverse set of images, which further assists the joint learning and also the GAN module.

Model
IS ↑ FID ↓   We also compute the Inception Score (IS) [34] and Fréchet Inception Distance (FID) [16] to compare the GAN module trained while joint learning. Table 4 shows that GAN in LaViTeR model performs better than all the previous models in terms of IS Score while the FID score is still comparable to other text to image network as [50] that report FID scores. However, better GAN performance is a by-product of LAViTeR model, not it's goal.     [41] and [46] are also listed here for reference.
ITM; for the last image, TV and coffee table are shown in the background corner of the image and are ignored by the ground truth texts, but are captured by the generated caption. Even only a part of the object appears in an image, it still has a chance to show up in our generated caption, e.g. in the image in the middle of the last row, a tiny part of a fork tip appears on the left, our ITM can generate 'a fork' given such a small clue in the image. Missing attributes are also predicted such as the attribute 'smiling' is generated in our generated text, which is never described in the ground truths. These generated texts from ITM with the missing objects and attributes in the original images pro-vide further useful information outside the training dataset and help the matching model learn more word-region level matching pairs. This is exactly one of our motivations for our LAViTeR method. We also evaluate the captioning ability of our ITM branch from the joint training, listed in Table 5. As we can see, our ITM can get a comparable result with other captioning models, such as [46] and [41], which shows that our ITM has the ability to generate reasonable captions for realimage-fake-text matching training. Note that we are not aiming at coming up with a novel and state-of-the-art captioning model, the main function of ITM branch is to provide more reasonable fake text for training the VTA matching model.

D. Code
Our code referenced from libraries open sourced by the work done in CATR [38] and AttnGAN [47].