{'fraction_non_alphanumeric': 0.03818360978814255, 'fraction_numerical': 0.012645754639513878, 'mean_word_length': 5.048423143779488, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis

Synthesizing high-quality realistic images from text descriptions is a challenging task. Existing text-to-image Generative Adversarial Networks generally employ a stacked architecture as the backbone yet still remain three flaws. First, the stacked architecture introduces the entanglements between generators of different image scales. Second, existing studies prefer to apply and fix extra networks in adversarial learning for text-image semantic consistency, which limits the supervision capability of these networks. Third, the cross-modal attention-based text-image fusion that widely adopted by previous works is limited on several special image scales because of the computational cost. To these ends, we propose a simpler but more effective Deep Fusion Generative Adversarial Networks (DF-GAN). To be specific, we propose: (i) a novel one-stage text-to-image backbone that directly synthesizes high-resolution images without entanglements between different generators, (ii) a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty and One-Way Output, which enhances the text-image semantic consistency without introducing extra networks, (iii) a novel deep text-image fusion block, which deepens the fusion process to make a full fusion between text and visual features. Compared with current state-of-the-art methods, our proposed DF-GAN is simpler but more efficient to synthesize realistic and text-matching images and achieves better performance on widely used datasets.


Introduction
The last few years have witnessed the great success of Generative Adversarial Networks (GANs) [8] for a variety of applications [4,27,48]. Among them, text-to-image synthesis is one of the most important applications of GANs. It aims to generate realistic and text-consistent images from the given natural language descriptions. Due to its practical value, text-to-image synthesis has become an active research area recently [3, 9, 13, 19-21, 32, 33, 35, 51, 53, 60].
Two major challenges for text-to-image synthesis are the authenticity of the generated image, and the semantic consistency between the given text and the generated image. Due to the instability of the GAN model, most recent models adopt the stacked architecture [56,57] as the backbone to generate high-resolution images. They employ cross-modal attention to fuse text and image features [37,50,56,57,60] and then introduce DAMSM network [50], cycle consistency [33], or Siamese network [51] to ensure the textimage semantic consistency by extra networks.
Although impressive results have been presented by previous works [9,19,21,32,33,51,60], there still remain three problems. First, the stacked architecture [56] introduces entanglements between different generators, and this makes the final refined images look like a simple combination of fuzzy shape and some details. As shown in Figure 1(a), the final refined image has a fuzzy shape synthesized by G 0 , coarse attributes (e.g., eye and beak) synthesized by G 1 , and fine-grained details (e.g., eye reflection) added by G 2 . The final synthesized image looks like a simple combination of visual features from different image scales. Second, existing studies usually fix the extra networks [33,50] during the adversarial training, making these networks easily fooled by the generator to synthesize adversarial features [30,52], thereby weakening their supervision power on semantic consistency. Third, cross-modal attention [50] can not make full use of text information. They can only be applied two times on 64×64 and 128×128 image features due to its high computational cost. It limits the effectiveness of the text-image fusion process and makes the model hard to extend to higher-resolution image synthesis.
To address the above issues, we propose a novel textto-image generation method named Deep Fusion Generative Adversarial Network (DF-GAN). For the first issue, we replace the stacked backbone with a one-stage backbone. It is composed of hinge loss [54] and residual networks [11] which stabilizes the GAN training process to synthesize high-resolution images directly. Since there is only one generator in the one-stage backbone, it avoids the entanglements between different generators.
For the second issue, we design a Target-Aware Discriminator composed of Matching-Aware Gradient Penalty (MA-GP) and One-Way Output to enhance the text-image semantic consistency. MA-GP is a regularization strategy on the discriminator. It pursues the gradient of discriminator on target data (real and text-matching image) to be zero. Thereby, the MA-GP constructs a smooth loss surface at real and matching data points which further promotes the generator to synthesize text-matching images. Moreover, considering that the previous Two-Way Output slows down the convergence process of the generator under MA-GP, we replace it with a more effective One-Way Output.
For the third issue, we propose a Deep text-image Fusion Block (DFBlock) to fuse the text information into image features more effectively. The DFBlock consists of several Affine Transformations [31]. The Affine Transformation is a lightweight module that manipulates the visual feature maps through channel-wise scaling and shifting operation. Stacking multiple DFBlocks at all image scales deepens the text-image fusion process and makes a full fusion between text and visual features.
Overall, our contributions can be summarized as follows: • We propose a novel one-stage text-to-image backbone that can synthesize high-resolution images directly without entanglements between different generators.
• We propose a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty (MA-GP) and One-Way Output. It significantly enhances the text-image semantic consistency without introducing extra networks.
• We propose a novel Deep text-image Fusion Block (DFBlock), which fully fuses text and visual features more effectively and deeply.
• Extensive qualitative and quantitative experiments on two challenging datasets demonstrate that the proposed DF-GAN outperforms existing state-of-the-art text-to-image models.

Related Work
Generative Adversarial Networks (GANs) [8] are an attractive framework that can be used to mimic complex real-world distributions by solving a min-max optimization problem between a generator and discriminator [16,17,43,54]. For instance, Reed et al. first applied the conditional GAN to generate plausible images from text descriptions [37,38]. StackGAN [56,57] generates high-resolution images by stacking multiple generators and discriminators and provides the text information to the generator by concatenating text vectors as well as the input noises. Next, AttnGAN [50] introduces the cross-modal attention mechanism to help the generator synthesize images with more details. MirrorGAN [33] regenerates text descriptions from generated images for text-image semantic consistency [59]. SD-GAN [51] employs the Siamese structure [45,46] to distill the semantic commons from texts for image generation consistency. DM-GAN [60] introduces the Memory Network [10,49] to refine fuzzy image contents when the initial images are not well generated in stacked architecture. Recently, some large transformer-based text-to-image methods [7,24,35] show excellent performance on complex image synthesis. They tokenize the images and take the image tokens and word tokens to make auto-regressive training by a unidirectional Transformer [2,34].
Our DF-GAN is much different from previous methods. First, it generates high-resolution images directly by a onestage backbone. Second, it adopts a Target-Aware Discriminator to enhance text-image semantic consistency without introducing extra networks. Third, it fuses text and image features more deeply and effectively through a sequence of DFBlocks. Compared with previous models, our DF-GAN is much simpler but more effective in synthesizing realistic and text-matching images.

The Proposed DF-GAN
In this paper, we propose a simple model for text-toimage synthesis named Deep Fusion GAN (DF-GAN).To synthesize more realistic and text-matching images, we propose: (i) a novel one-stage text-to-image backbone that can synthesize high-resolution images directly without visual feature entanglements. (ii) a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty

Model Overview
The proposed DF-GAN is composed of a generator, a discriminator, and a pre-trained text encoder as shown in Figure 2. The generator has two inputs, a sentence vector encoded by text encoder and a noise vector sampled from the Gaussian distribution to ensure the diversity of the generated images. The noise vector is first fed into a fully connected layer and reshaped.We then apply a series of UP-Blocks to upsample the image features. The UPBlock is composed of an upsample layer, a residual block, and DF-Blocks to fuse the text and image features during the image generation process. Finally, a convolution layer converts image features into images.
The discriminator converts images into image features through a series of DownBlocks. Then the sentence vector will be replicated and concatenated with image features. An adversarial loss will be predicted to evaluate the visual realism and semantic consistency of inputs. By distinguishing generated images from real samples, the discriminator promotes the generator to synthesize images with higher quality and text-image semantic consistency.
The text encoder is a bi-directional Long Short-Term Memory (LSTM) [41] that extracts semantic vectors from the text description. We directly use the pre-trained model provided by AttnGAN [50].

One-Stage Text-to-Image Backbone
Since the instability of the GAN model, previous text-toimage GANs usually employ stacked architecture [56,57] to generate high-resolution images from low-resolution ones. However, the stacked architecture introduces entanglements between different generators, and it makes the final refined images look like a simple combination of fuzzy shape and some details (see Figure 1(a)).
Inspired by recent studies on unconditional image generation [23,54], we propose a one-stage text-to-image backbone that can synthesize high-resolution images directly by a single pair of generator and discriminator. We employ the hinge loss [23] to stabilize the adversarial training process. Since there is only one generator in the one-stage backbone, it avoids the entanglements between different generators. As the single generator in our one-stage framework needs to synthesize high-resolution images from noise vectors directly, it must contain more layers than previous generators in stacked architecture. To train these layers effectively, we introduce residual networks [11] to stabilize the training of deeper networks. The formulation of our one-stage method with hinge loss [23] is as follows: where z is the noise vector sampled from Gaussian distribution; e is the sentence vector; P g , P r , P mis denote the synthetic data distribution, real data distribution, and mismatching data distribution, respectively.

Target-Aware Discriminator
In this section, we detailed the proposed Target-Aware Discriminator, which is composed of Matching-Aware Gradient Penalty (MA-GP) and One-Way Output. The Target-Aware Discriminator promotes the generator to synthesize more realistic and text-image semantic-consistent images.

Matching-Aware Gradient Penalty
The Matching-Aware zero-centered Gradient Penalty (MA-GP) is our newly designed strategy to enhance text-image semantic consistency. In this subsection, we first show the unconditional gradient penalty [28] from a novel and clear perspective, then extend it to our MA-GP for the text-toimage generation task.
As shown in Figure 3(a), in unconditional image generation, the target data (real images) correspond to a low discriminator loss. Correspondingly, the synthetic images correspond to a high discriminator loss. The hinge loss limits the range of discriminator loss between -1 and 1. The gradient penalty on real data will reduce the gradient of the real data point and its vicinity. The surface of the loss function around the real data point is then smoothed which is helpful for the synthetic data point to converge to the real data point.
Based on the above analysis, we find that the gradient penalty on target data constructs a better loss landscape to help the generator converge. By leveraging the view into the text-to-image generation. As shown in Figure 3(b), in text-to-image generation, the discriminator ob-serves four kinds of inputs: synthetic images with matching text (fake, match), synthetic images with mismatched text (fake, mismatch), real images with matching text (real, match), real images with mismatched text (real, mismatch). For text-visual semantic consistency, we tend to apply gradient penalty on the text-matching real data, the target of text-to-image synthesis. Therefore, in MA-GP, the gradient penalty should be applied on real images with matching text. The whole formulation of our model with MA-GP is as follows: where k and p are two hyper-parameters to balance the effectiveness of gradient penalty.
By using the MA-GP loss as a regularization on the discriminator, our model can better converge to the textmatching real data, therefore synthesizing more textmatching images. Besides, since the discriminator is jointly trained in our network, it prevents the generator from synthesizing adversarial features of the fixed extra network. Moreover, since MA-GP does not incorporate any extra networks for text-image consistency and the gradients are already computed by back propagation process, the only computation introduced by our proposed MA-GP is the gradient summation, which is more computational friendly than extra networks.

One-Way Output
In the previous text-to-image GANs [50,56,57], image features extracted by discriminator are usually used in two ways (Figure 4(a)): one determines whether the image is real or fake, the other concatenates the image feature and sentence vector to evaluate text-image semantic consistency. Correspondingly, the unconditional loss and the conditional loss are computed in these models.
However, it is shown that the Two-Way Output weakens the effectiveness of MA-GP and slows down the convergence of the generator. Concretely, as depicted in Figure  3(b), the conditional loss gives a gradient α pointing to the real and matching inputs after back propagation, while the unconditional loss gives a gradient β only pointing to the real images. However, the direction of the final gradient which just simply sums up α and β does not point to the real and matching data points as we expected. Since the target of the generator is to synthesize real and text-matching images, the final gradient with deviation cannot well achieve text-image semantic consistency and slows down the convergence process of the generator. Therefore, we propose the One-Way Output for text-toimage synthesis. As shown in Figure 4(b), our discriminator concatenates the image feature and sentence vector, then outputs only one adversarial loss through two convolution layers. Through the One-Way Output, we are able to make the single gradient α pointed to the target data points (real and match) directly, which optimize and accelerate the convergence of the generator.
By combining the MA-GP and the One-Way Output, our Target-Aware Discriminator can guide the generator to synthesize more real and text-matching images.

Efficient Text-Image Fusion
To fuse text and image features efficiently, we propose a novel Deep text-image Fusion Block (DFBlock). Compared with previous text-image fusion modules, our DF-Block deepens the text-image fusion process to make a full text-image fusion.
As shown in Figure 2, the generator of our DF-GAN consists of 7 UPBlocks. A UPBlock contains two Text-Image Fusion blocks. To fully utilize the text information in fusion, we propose the Deep text-image Fusion Block (DF-Block) which stacks multiple Affine Transformations and ReLU layers in Fusion Block. For Affine transformation, as shown in Figure 5(c), we adopt two MLPs (Multilayer Perceptron) to predict the language-conditioned channel-wise scaling parameters γ and shifting parameters θ from sentence vector e, respectively: For a given input feature map X∈R B×C×H×W , we first conduct the channel-wise scaling operation on X with the scaling parameter γ, then apply the channel-wise shifting operation with the shifting parameter θ. Such a process can be expressed as follows: where AF F denotes the Affine Transformation; x i is the i th channel of visual feature maps; e is the sentence vector; γ i and θ i are scaling parameter and shifting parameter for the i th channel of visual feature maps. The Affine layer expands the conditional representation space of the generator. However, the Affine transformation is a linear transformation for each channel. It limits the effectiveness of text-image fusion process. Thereby, we add a ReLU layer between two Affine layers which brings the nonlinearity into the fusion process. It enlarges the conditional representation space compared with only one Affine layer. A larger representation space is helpful for the generator to map different images to different representations according to text descriptions.
Our DFBlock is partly inspired by Conditional Batch Normalization (CBN) [5] and Adaptive Instance Normalization (AdaIN) [14,16] which contain the Affine transformation. However, both CBN and AdaIN employ normalization layers [15,44] which transform the feature maps into the normal distribution. It generates an opposite effect to the Affine Transformation which is expected to increase the distance between different samples. It is then unhelpful for the conditional generation process. To this end, we remove the normalization process. Furthermore, our DFBlock deepens the text-image fusion process. We stack multiple Affine layers and add a ReLU layer between. It promotes the diversity of visual features and enlarges the representation spaces to With the deepening of the fusion process, the DFBlock brings two main benefits for text-to-image generation: First, it makes the generator more fully exploit the text information when fusing text and image features. Second, deepening the fusion process enlargers the representation space of the fusion module, which is beneficial to generate semantic consistent images from different text descriptions.
Furthermore, compared with previous text-to-image GANs [50,56,57,60], the proposed DFBlock makes our model no longer consider the limitation from image scales when fusing the text and image features. This is because existing text-to-image GANs generally employ the crossmodal attention mechanism which suffers a rapid growth of computation cost along with the increase of image size.

Experiments
In this section, we first introduce the datasets, training details, and evaluation metrics used in our experiments, then evaluate DF-GAN and its variants quantitatively and qualitatively. Datasets. We follow previous work [33,50,51,56,57,60] and evaluate the proposed model on two challenging datasets, i.e., CUB bird [47] and COCO [25]. The CUB dataset contains 11,788 images belonging to 200 bird species. Each bird image has ten language descriptions. The COCO dataset contains 80k images for training and 40k images for testing. Each image in this dataset has five language descriptions. Training Details. We optimize our network using Adam [18] with β 1 =0.0 and β 2 =0.9. The learning rate is set to 0.0001 for the generator and 0.0004 for the discriminator according to Two Timescale Update Rule (TTUR) [12]. Evaluation Details. Following previous works [50,60], we choose the Inception Score (IS) [40] and Fréchet Inception Distance (FID) [12] to evaluate the performance of our net-work. Specifically, IS computes the Kullback-Leibler (KL) divergence between a conditional distribution and marginal distribution. Higher IS means higher quality of the generated images, and each image clearly belongs to a specific class. FID [12] computes the Fréchet distance between the distribution of the synthetic images and real-world images in the feature space of a pre-trained Inception v3 network. Contrary to IS, more realistic images have a lower FID. To compute both IS and FID, each model generates 30,000 images (256×256 resolution) from text descriptions randomly selected from the test dataset.
As stated in the recent works [21,58], the IS cannot evaluate the image quality well on the COCO dataset, which also exists in our proposed method. Moreover, we find that some GAN-based models [50,60] achieve significant higher IS than Transformer-based large text-to-image models [7,35] on the COCO dataset, but the visual quality of synthesized images is obviously lower than Transformerbased models [7,35]. Thus, we do not compare IS on the COCO dataset. In contrast, FID is more robust and aligns human qualitative evaluation on the COCO dataset.
Moreover, we evaluate the number of parameters (NoP) to compare the model size with current methods.

Qualitative Evaluation
We also compare the visualization results synthesized by AttnGAN [50], DM-GAN [60], and the proposed DF-GAN.
It can be seen that images synthesized by AttnGAN [50] and DM-GAN [60] in Figure 6 look like a simple combination of fuzzy shape and some visual details (1 st , 3 rd , 5 th , 7 th , and 8 th columns). As shown in the 5 th , 7 th , and 8 th columns, the birds synthesized by AttnGAN [50] and DM-GAN [60] contain wrong shapes. Moreover, the images synthesized by our DF-GAN have better object shapes and realistic fine-grained details (e.g., 1 st , 3 rd , 7 th , and 8 th columns). Besides, the posture of the bird in our DF-GAN result is also more natural (e.g., 7 th and 8 th columns).
Comparing the text-image semantic consistency with other models, we find that our DF-GAN can also capture more fine-grained details in text descriptions. For example, as the results shown in 1 st , 2 th , 6 th columns in Figure 6, other models cannot synthesize the "holding ski poles", "train track", and "a black stripe by its eyes" described in the text well, but the proposed DF-GAN can synthesize them more correctly.

Ablation Study
In this section, we conduct ablation studies on the testing set of the CUB dataset to verify the effectiveness of each component in the proposed DF-GAN. The com-  Table 3, we find that, compared with other fusion methods, concatenation cannot efficiently fuse text and image fea- tures. The comparison among CBN, AdaIN, and AFFBlock proves that Normalization is not essential in Fusion Block, and removing normalization even slightly improves the results. The comparison between DFBlock and AFFBlock demonstrates the effectiveness of deepening the text-image fusion process. In sum, the comparison results prove the effectiveness of our proposed DFBlock.

Limitations
Although DF-GAN shows superiority in text-to-image synthesis, some limitations must be taken into consideration in future studies. First, our model only introduces the sentence-level text information, which limits the ability of fine-grained visual feature synthesis. Second, introducing pre-trained large language models [6,34] to provide additional knowledge may further improve the performance. We will try to address these limitations in our future work.

Conclusion and Future Work
In this paper, we propose a novel DF-GAN for the textto-image generation tasks. We present a one-stage text-toimage Backbone that can synthesize high-resolution images directly without entanglements between different generators. We also propose a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty (MA-GP) and One-Way Output. It can further enhance the textimage semantic consistency without introducing extra networks. Besides, we introduce a novel Deep text-image Fusion Block (DFBlock) which fully fuses text and image features more effectively and deeply. Extensive experiment results demonstrate that our proposed DF-GAN significantly outperforms current state-of-the-art models on the CUB dataset and more challenging COCO dataset.