{'fraction_non_alphanumeric': 0.03614306525342844, 'fraction_numerical': 0.011980710302908525, 'mean_word_length': 4.764441870566093, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach

Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as"change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.


INTRODUCTION
Editing image attributes on portable devices is an uncomfortable operation for most people, but especially for impaired users. On smartphones, changing the hair color on a picture requires to manually select the pixels to be altered and color them properly. Inspired by the emergence of voice assistants, which provide us a more convenient way to interact with machines through human language, we make pilot attempts to manipulate images through textual commands. Meeting this goal requires a deep integration of vision and language techniques, as previously achieved for other tasks, namely: image captioning [23,52], text-to-image generation [25,26,38,42,53], text-based video editing [13] and drawingbased image editing [22,39]. However, modifying parts of the image through natural language is still a challenging research problem.
Existing approaches, based on Generative Adversarial Networks (GANs) [1,15], require either to describe all the characteristics of the desired image [42,59] (e.g. "young oval-faced lady with blond and black hair, smiling"), or collecting a richly detailed dataset of supervised actions [11,12] or captions [26,38,42,43,51]. For example, TAGAN [38] uses text input to change an image in order to have the attributes explained in the provided text description, but this method relies on detailed human-written captions. Similarly, El-Nouby et al. [11,12] rely on annotated datasets to interact with the user and to generate semantically-consistent images from textual instructions. Two main issues have hindered the possibility to have an effective and unsupervised approach that follows human-written commands (e.g. "make him smile"). On the one hand, the literature on text-based image manipulation does not explicitly model image attributes, which would indicate if a specific part of the image is going to be changed or not. On the other hand, most GAN models assume a deterministic mapping, strongly limiting the capacity to handle the inherent ambiguity of human textual commands (e.g. "blond hair" might mean different shades of blond colors).
In this paper, we propose Describe What to Change (DWC), a novel method based on GANs to manipulate visual attributes from a human-written textual description of the required changes, thus named DWC-GAN. To explicitly model visual attributes and cope with the text ambiguity, our model translates images from one visual domain to other visual domains in a multi-modal (stochastic) fashion. Our contribution is three-fold: • We propose the use of textual commands instead of image captions. As shown in Table 1, this leads to three advantages: (1) the commands can be more flexible than image captions and we can compose them for progressive manipulation; (2) the commands can be automatically generated, and (3) users are not required to know and mention all the modeled visual attributes in an input image, only the desired changes. DWC-GAN thus does not rely on human-annotated captioning text compared with state-of-the-art approaches [26,38,42,43]. • Our model disentangles the content features and the attribute representation from the input image making the manipulation more controllable and robust. The attribute space follows a Gaussian Mixture Model (GMM), where the attributes of the source and the modified image follow the Gaussian distribution associated to their respective domains. • To the best of our knowledge, we are the first to manipulate images through textual commands in a multi-modal fashion, allowing multiple versions of the same translation.

RELATED WORK
Our work is best placed in the literature of image-to-image translation and text-to-image generation. The former aims to transform

Input text Features
Captioning text: This woman is old and smiling, with black hair, eyeglasses and no beard.
-Enumerates all the attributes; -No need to understand attributes from the input image; -Often trained with positive and negative annotation for each translation pair [26,38].
Textual command: Change the hair to be black, increase her age.
-Focuses only on the differences; -Requires the extraction of attributes from the input image; -Trained without ground truth for each translation pair. an input image belonging to a visual domain (e.g. young people) to another domain (e.g. elderly people), while the latter tries to generate images from a textual description provided as input. These two fields have witnessed many improvements in quality and realism, thanks to the advent of GANs [15,35] and, in particular, conditional GANs (cGANs) [35], which are conditioned by extra information (e.g. the hair colour in face generation). GANs aim to synthesize images through a min-max game between a discriminator, trying to discriminate between real and fake data, and a generator, seeking to generate data that resemble the real ones. Image-to-image translation: Conditional GANs were first employed for image-to-image translation in pix2pix [21] for learning a mapping from one domain to another by minimizing the L 1 loss between the generated and the target image. However, pix2pix requires a large amount of paired data (e.g. colored-greyscale images), which is often unrealistic to collect. Thus, the interest had shifted to unsupervised models that learn to translate images with unpaired data, for which the image content might vary (e.g. daylight-night images can have different people and cars). These models require additional constraints to better differentiate the domain-dependent parts of the image from the domain-independent ones. For example, Liu et al. [31] assume that the two domains share a common latent space. CycleGAN [57] requires an image translated from a domain A to a domain B to be translated back to A, and applies a consistency loss, which was later employed by many other works [7,20,30,36]. MUNIT [20] assumes that images share a domain-invariant content latent space and a style latent space, which is specific to a domain. This choice allows for generating multiple samples of the translation by drawing from the style distribution. Most of these approaches are, however, limited to one-to-one domain translation, which requires training a large number of models in the case of multiple domains. StarGAN [7] proposed a unified approach for multi-domain translation by employing a binary domain vector as input, which specifies the target domain, and a domain classification loss that helps the translation. Recently, GMM-UNIT [29] proposed a unified approach for multi-domain and multi-modal translation by modeling attributes through a Gaussian mixture, in which each Gaussian component represents a domain. In this paper, inspired by GMM-UNIT, we build a translation system where both the original and the manipulated attributes follow a GMM.
Text-to-image generation: cGANs can be conditioned with complex information, such as human-written descriptions, and then can generate accurate images, as shown by Reed et al. [42,43]. For example, StackGAN [53] uses two stages of GANs in order to generate high-resolution images. The first stage generates a low-resolution image with the primitive shape and colour of the desired object, while the second one further refines the image and synthesize 256x256 images. The same authors improved the model with multiple generators through StackGAN++ [54]. Qiao et al. [41] focused instead on semantic consistency, enforcing the generated images to have the same semantics with the input text description. In other words, the caption synthesized from the generated image should have the same meaning as the input text. Recently, Li et al.
proposed StoryGAN [28], which generates a series of images that are contextually coherent with previously generated images and with the sequence of text descriptions provided by the user.
These approaches, however, require to accurately describe the picture to be generated, without allowing to start from an existing image and modify it through text.
Image manipulation: Modifying images through some userdefined conditions is a challenging task. Most of the previous approaches rely on conditional inpaiting [3,22,56], in which the network fills a user-selected area with pixel values coherent with the user preferences and the context. However, image editing does not necessarily require to select the exact pixels that have to be changed. Wang et al. [49] learned to change global image characteristics such as brightness and white balance from textual information. Zou et al. [59] proposed a network that colourizes sketches following the instructions provided by the input text specifications. El-Nouby et al. [11,12] introduced a network that generates images from a dialogue with the user. A first image is generated from text, then objects are added according to the user preference provided as free text. Chen et al. [5] learned with a recurrent model and an attention module to colourize greyscale images from a description of the colours or the scene (e.g. "the afternoon light flooded the little room from the window"). Recently, Cheng et al. [6] proposed a network, tailored for dialogue systems, that gets as inputs an image and a series of textual descriptions that are apt to modify the image. The network is encouraged to synthesize images that are similar to the input and the previously generated images. However, examples show that the content of the image might vary substantially. Nam et al. [38] and Li et al. [26] introduced instead TAGAN and ManiGAN that modify fine-grained attributes such as the colours of birds' wings through text. The key idea of their methods is reconstructing the images with positive and negative sentences, where the positive/negative refers to captioning text matching/mismatching the corresponding image. The recurrent network is trained to learn the words that refer to the attributes in the images and it allows to change words to manipulate multiple attributes at once.
Current image manipulation methods based on text often rely on human-annotated captioning text [26,38,42,43,51], which describes the scene in the image. Moreover, they do not explicitly model the concept of attributes (i.e. domains), sometimes failing at balancing the trade-off between keeping the old content and changing it [38], which is a well-known issue in image-to-image translation. Last but not least, existing solutions are deterministic, thus limiting the diversity of the translated outputs, even if the provided text might have different meanings. For example, "wear a beard" might mean to have a goatee, short or long beard. Therefore, we build on state-of-the-art approaches for image-toimage translation and on the image manipulation literature and propose to explicitly model the visual attributes of an image and learn how to translate them through automatically generated commands. We are, up to the best of our knowledge, the first proposing a multi-domain and multi-modal method that translates images from one domain to another one by means of the user descriptions of the attributes to be changed.

METHODOLOGY
Our model is designed to modify the attributes of an image x through a human-written text t, which describes the changes to be made in the image. In other words, we want to generate a new imagex that has the visual content of x and all its attributes but those attributes that should be altered accordingly to the text t. Different from previous works, t is not a description of all the attributes of the generated image, but it is a direct command describing the modifications to be made (e.g. change the hair color to black). Thus, the proposed method should be able to model the attributes of the original image, the modifications of (some of) the attributes described in the text, and the attributes of the generated image. Moreover, it has to deal with the ambiguity of input text, which does not describe all the attributes of the target image, nor it is direct and clear as one-hot vector inputs.
The architecture, as shown in Figure 2, processes the input as follows. Given N attributes of a set of images, we first disentangle the visual content c ∈ C C C ⊂ R C and the attributes a ∈ A A A ⊂ R N of the input image x through the encoders E c and E a , respectively. Then, E t , which is a Recurrent Neural Network (RNN) module, : Architecture of our model. First, we disentangle the attributes and the content of the input image. Then, we modify the visual attributes of the original image using a text encoder. The generator uses a MLP to produce a set of AdaIN [19] parameters from the attribute representation. The content features is then processed by the parameterized generator to predict a realistic image with required attributes. Finally, the discriminator classifies whether the generated image is real or fake.
uses the input sentence t and the extracted attributes a to infer the desired (target) attributes a ′ = E t (t, a). The inferred target attributes are then used along with the extracted content c, by the generator G, to output the image G(c, a ′ ). Finally, the discriminator D discerns between "real" or "fake" images (D r eal ) and recognizes the domain of the generated image (D cl s ). Here, a domain is a combination of visual attributes. The model (content and attribute extractors, generator and discriminators) is learned in an end-toend manner.

Assumptions
The basic assumption of DWC-GAN is that each image can be decomposed in a domain-invariant content space and a domainspecific attribute space [14,20,29]. We let the network model the high-dimension content features, while we represent the attributes of the image through Gaussian components in a mixture. This representation allows to model the combinations of attributes in a continuous space. We can then exploit this space to work with attribute combinations that have never (or little) been observed in the data. Formally and similarly to GMM-UNIT [29], we model the attributes with a K-component d-dimensional GMM: where ϕ k , µ µ µ k and Σ Σ Σ k denote respectively the weight, mean vector and covariance matrix of the k-th ×d is symmetric and positive definite). Therefore, we model the attributes in a domain through a corresponding GMM component. Thus, for an image x from domain X X X k (i.e. x ∼ p X X X k ), its latent attribute a is assumed to follow the k-th Gaussian component:

Multi-modal image generation
In order to have a multi-modal (non-deterministic) translation, we enforce both the sampled and extracted attributes to follow the image domain distribution, that is ∀ℓ, k: We ensure these two properties with the following losses. Reconstruction losses. These losses, introduced at first in [20,57], force the output to be consistent with the original content and attributes. Specifically, the self-reconstruction loss ensures that the original image is recovered if its attribute and content is used in the generator. If c x = E c (x) and a x = E a (x), then for all k: The content and attribute reconstruction losses [20] are used to constrain and learn the content and attribute extractors: The cycle reconstruction loss [57] enforces consistency when translating an image into a new domain and then back to the original one: where the use of the L 1 loss inside the expectation is motivated by previous works [21] showing that L 1 produces sharper results than L 2 . To encourage the generator to produce diverse images, we explicitly regularize G with the diversity sensitive loss [8,32]: Domain losses. Similarly to StarGAN [7], for any given input image x, we would like the method to classify it as its original domain, and to be able to generate an image in any domain from its content. Therefore, we need two different losses, one directly applied to the original images, and a second one applied to the generated images: where d X X X ℓ is the label of ℓ-th domain. Importantly, the discriminator D is trained using the first loss, while the generator G is trained using the second loss. Adversarial losses. These terms enforce the generated images to be indistinguishable from the real images by following the formulation of LSGAN [33]:

Attribute manipulation losses
As mentioned above, the underlying statistical assumption for the attributes is to follow a GMM -one component per domain. Very importantly, both the attribute representations extracted from image by E a and the attribute representations obtained from E t should follow the assumption and correspond to the correct component. We do that by imposing that the KullbackâĂŞLeibler (KL) divergence of the extracted and manipulated attributes correspond to the Gaussian component of the original and targetted domain respectively. Recalling that a x = E a (x), we write: q(t ) dt is the KL. Intuitively, the second KL divergence enforces the images generated from the manipulated attributes to follow the distribution of the target domain: Finally, the full objective function of our network is: where {λ r ec_s , λ cyc , λ K L } are hyper-parameters of weights for corresponding loss terms. The value of most of these parameters come from the literature. We refer to Supplementary Material for the details. Note that these losses are required to constrain down the difficult problem of image to image translation, disentangle content and style, and achieve stochastic results by sampling from the GMM distribution of attributes. Without these losses, the problem would be much less constrained and difficult.

Domain Sampling
We propose a strategy to obtain diverse manipulated results during testing by following the assumption introduced in Section 3.1. After extracting the target attributes a ′ from the original image x and input text t, we assign it to a closest component (µ µ µ * , Σ Σ Σ * ) of the GMM by following: k * = arg max k ∈ {1, ..., K } {Φ(a ′ ; µ µ µ k , Σ Σ Σ k )}, where Φ is the Gaussian probability density function. Then, we can randomly sample several a * from the component N (µ µ µ k * , Σ Σ Σ k * ). With such different sampled a * 's and the trained non-linear generator G, our model achieves multi-modality in any domain, which is different from state-of-the-art approaches [38,54] that use random noise sampled from a standard normal distribution N (0, 1) to generate multiple diverse results independently of the domain.
Sampling all the target attributes might hurt translation performance. Thus, we propose an additional constraint that the ideal attribute representation is a mixture of domain sampling a * and extracted representation E a (x), where a * copies the attribute representation from E a (x) when a target attribute is the same as the attribute in the input image.

Unsupervised Attention
Attention mechanisms have proven successful in preserving the visual information that should not be modified in an image-to-image translation [29,30,34,40]. For example, GMM-UNIT [29] claims that models are inclined to manipulate the intensity and details of pixels that are not related to the desired attribute transformation. Therefore, we follow the state-of-the-art approaches and add a convolutional layer followed by a sigmoid layer at the end of G to learn a single channel attention mask M in an unsupervised manner. Then, the final predictionx is obtained from a convex combination of the input and the initial translation:x =x · M + x · (1 − M).

EXPERIMENTAL SETUP 4.1 Datasets
CelebA. The CelebFaces Attributes (CelebA) dataset [31] contains 202,599 face images of celebrities where each face is annotated with 40 binary attributes. This dataset is composed of some attributes that are mutually exclusive (e.g. either male or female) and those that are mutually inclusive (e.g. people could have both blond and black hair). To be consistent with previous papers [7,29], we select a subset of attributes, namely black hair, blond hair, brown hair, male/female, young/old, smile, eyeglasses, beard. As we model these attributes as different GMM components, our model allows the generation of images with up to 2 8 different combinations of the attributes. Note that in the dataset only 100 combinations exist. As preprocessing, we center crop the initial 178×218 images to 178×178 and resize them to 128×128. We randomly select 2,000 images for testing and use all remaining images for training. CUB Birds. The CUB dataset [48] contains 11,788 images of 200 bird species, and provides hundreds of annotated attributes for each bird (e.g. color, shape). We select some color attributes {grey, brown, red, yellow, white, buff } of three body parts {crown, wing, breast} of the birds, yielding a total of K = 18 attributes used to represent around one thousand domains appearing in the dataset. Differently from the CelebA dataset, the selected colors could appear in all selected body parts. Hence, the colors and body parts follow a hierarchical structure, requiring the model to modify the color of a specific body part. We crop all images by using the annotated bounding boxes of the birds and resize images to 128×128.
Additional details can be found in the Supplementary Material.

Automatic Text Description
The text encoder E t has to be trained with a dataset describing the changes to be made in the visual attributes of the input image. Therefore, we here propose a protocol to generate the corpus automatically for the datasets provided multiple annotated attributes or labels. Given an image x, we can collect a set of attributes {p 1 , . . . , p N } and obtain the corresponding real-valued vector p ∈ {0, 1} N for generating the description text. We denote p ′ as the randomly assigned real-valued vector of a target image. After observing the human-used commands of manipulating images, we propose an automatic description text generation method through three different strategies: Step-by-step actions. We can compare the differences between p and p ′ element-by-element. For example, if the hair colors in the source image and target image are "brown" and "black", respectively. We can describe the action like "change the brown hair to black", "change the hair color to black", "black hair". Similarly, we can describe all actions that change the image one by one. For the same attributes in p and p ′ , we can provide empty description or the no-change commands (e.g. "do not change the hair color", "keep his hair color unchanged"). Then, the sequence of step-by-step actions are disrupted randomly. A full example for this strategy is "make the face older, translate the face to be smiling, remove the eyeglasses, do nothing on the gender, change the hair color to black, wear a beard". Overall description. In practice, it's feasible to describe the attributes in the target images to manipulate images, which is similar to the text-to-image task [26,38,42,43]. Here, we completely describe the appeared attributes in p ′ . For example, "a smiling young man with black hair, wearing a beard and without eyeglasses". In addition, the untouched attributes are not regarded as existing attributes in the target image by default. Mixed description. To increase the generalization, we mix the two previous strategies by randomly mentioning some attributes in p and specifying the exact change of attributes in p ′ For example, translate the young miss to be an old smiling man with black hair and wearing beards and eyeglasses.". In this case, the attributes of both the input image and input text are mentioned, which is more complex for the language model. During training, these three strategies are randomly selected for each input image, and are used for both the CelebA and the CUB Birds datasets. More details are provided in the supplementary material. This dataset is released along with the paper to allow future work and better comparisons with our results.

Metrics
We quantitatively evaluate our model through the image quality, diversity and the accuracy of generated images. We evaluate the image quality through the Fréchet Inception Distance (FID) [17] and the Inception Score (IS) [44], the diversity is measured by the Learned Perceptual Image Patch Similarity (LPIPS) [55], the mean accuracy (Mean Acc) of multi-label classification, while we also measure the accuracy through a user study. FID and IS. We randomly select 10000 source images and 10000 target images. The models transfer all source images with the same attributes of target images through textual commands, as shown in Section 4.2. Hence, the FID is estimated using 10000 generated images vs the selected target 10000 images. The IS is evaluated using Inception-v3 [46] and 10000 generated images for FID. LPIPS. The LPIPS distance is defined as the L 2 distance between the features extracted by a deep learning model of two images. This distance has been demonstrated to match well the human perceptual similarity [55]. Thus, following [20,58], we randomly select 100 input images and translate them to different domains. For each domain translation, we generate 10 images for each input image and evaluate the average LPIPS distance between the 10 generated images. Finally, we get the average of all distances. Higher LPIPS distance indicates better diversity among the generated images. We show in the results both the mean and standard deviation. Accuracy and Realism. Through these two metrics we evaluate how humans perceive generated images. Accuracy measures whether the attributes of the generated images are coherent with the target attributes (0: incorrect; 1: correct). Realism quantifies how realistic a generated image is perceived (0: bad, unrecognizable; 1: neutral, recognizable but with obvious defects; 2: good, recognizable but with slight defects; 3: perfect, recognizable and without any defect). We test Accuracy and Realism by randomly sampling 50 images with the same translation condition and collect more than 20 surveys from different people with various background.

Baseline Models
StackGAN++ [54] is a text-to-image model that takes text as input and encodes the text into embedding for the decoding network. Here, we train/test this model by using the embedding of the manipulation text as input. TAGAN [38] and ManiGAN [26] are two image-conditioned textto-image models that take the original image and detailed description text of the target image as input. We follow the original method and train/test the model on our automatic description text. StarGAN [7] is a unified multi-domain translation model that takes the original image and target attribute vector as input. Here, we add the same RNN module (i.e. the text understanding module) used in our framework to StarGAN to extract attributes from the input text. After that, we feed the embedding of the input text and original image as input. We call this modified version as StarGAN*.
For all the baseline models, we use the same pre-trained word embeddings in 300 dimensions for the input description text. The embeddings are trained by the skip-gram model described in [4] on Wikipedia using fastText 1 .

Input Image
"change the blond hair to black" "more serious" "make the face older" "male, wear a beard" Input Text

RESULTS
We begin by quantitatively comparing our model with state of the art. Table 2 shows that our model generates better quality images than all competing methods, using around half the parameters and achieving comparable training speed. Specifically, DWC-GAN outperforms our most similar competitor, TAGAN, attaining higher IS (3.069 vs 1.178) and lower FID (32.14 vs 421.84) with less required parameters (32.75M vs 44.20M). Surprisingly, the diversity of generated images is lower than StackGAN++. However, the qualitative results show that StackGAN++ generates diverse but very noisy images, in which the attributes are not easily understandable (see the Supplementary Material for details). Contrarily, Figure 3 shows that DWC-GAN generates high-quality images that are also consistent with the manipulation described by the text. Our approach outperforms state of the art in all the metrics in the CUB dataset as well, as shown in both Figure 3 and Table 3. Additional results can be seen in Figure 1 and in the Supplementary Material. Differently from state of the art, DWC-GAN explicitly models the attributes of images in a smooth and continuous space through a GMM. This allows to translate images in multiple domains, but also to have multi-modal results. As Figure 1 shows, our model generates different realizations of the same manipulation. For example, in "add beards" the model synthesizes various versions of the subjects' beards, as the command is open to more than one interpretation. State-of-the-art models such as TAGAN and StarGAN* do instead generate results with no, or low, diversity (see the Supplementary Material for additional details).
Our model uses the source attributes and the human-written sentence to generate the desired attributes. The results of StackGAN++ show that it is not feasible to only encode textual commands to generate meaningful target images, while StarGAN* shows that taking into account only the text attributes as an additional condition is not enough. In particular, StarGAN* exhibits a visible mode collapse in this setting. These results show that applying and modifying the existing baselines is not feasible to solve our task. We trained our model with automatically generated text, thus without relying on human-annotated datasets as state of the art does [38,42,43]. We observe that while TAGAN does not perform well with our corpus, we have consistent results with different strategies of automatic text generation, and even with a text strategy that is similar to their dataset (the Overall strategy). This result proves that annotating the dataset with automatically generated textual commands is a feasible strategy to train a model that manipulates images through text. Human evaluation We evaluated StarGAN* and DWC-GAN by a user study where more than 20 people where asked to judge the Accuracy and Realism of generated results. On average, 82.0% of the images generated by DWC-GAN were judged as correct, while StarGAN* generated only 25.6% of corrected results. Regarding Realism, DWC-GAN had an average score of 2.25, while StarGAN* had 1.17. Specifically, in 50.50% of the people said DWC-GAN were perfect, recognizable and without defects while only 2.82% of the people said the same for StarGAN*. StarGAN* results are indeed often judged "neutral, recognizable but with obvious defects" (52.91% of times). StackGAN++ and TAGAN do not achieve accuracy and realism higher than zero because they do not generate recognizable images. We refer to the Supplementary Material for additional details.
These results show that DWC-GAN generates correct images, which can be mistaken with the real images at higher rates than competitors.  Progressive Manipulation A few state of the art approaches for text-to-image generation and image manipulation are expressly designed for conversational systems [6,11,12,28,37]. In Figure 15 we show that we can repeatedly apply our method to generated images to have a manipulation in multiples steps. This result highlights the consistency of our generated results, which can be used as input of our model, and that DWC-GAN might be applied in an interactive environment. Domain Interpolation Our method generates interpolated images from two sentences. Figure 6 shows that by manipulating an image in two different ways, it is possible to generate the intermediate steps that go from one textual manipulation to the other. User Study on Textual Commands Our training corpus is automatically generated, thus it might suffer limited generalizability.  Input Image "change the blond hair to brown" ←→ "blond hair, make her older" "grey breast" ←→ "don't change anything" Figure 6: Domain interpolation given an input image.
We conduct an experiment to verify these possible issues by asking 15 different people to describe in natural language how to edit an image A to make it similar to another image B. We asked people to do this experiment for 15 randomly chosen images. The resulting collected textual commands and their corresponding images are fed into our model to evaluate the results. We found that our model can be generalized to real human-written descriptions well. Details are presented in the Supplementary Material. Attention Visualization We visualize the unsupervised attention mask in Figure 7. It shows the localization area of different manipulations, which indeed helps to focus on the attribute-related pixels and preserves the background pixels.

Input Image Attention Output
"remove the beard" Attention Output Ablation Study We here investigate the effect of removing some components from our architecture. Results are showed in Table 4. First, we remove the Cycle consistency, which is widely employed by the literature of image-to-image translation [20,57,58]. We observe that both the image quality and diversity significantly performs worse without the cycle consistency, which indicates that the Cycle consistency loss indeed constraints the network on the translation. Then, when we remove the Attribute reconstruction, which is applied to learn the attribute extractor E a . Here, we use an attribute classifier to measure the F1 score. It shows that the constraint on the attribute latent variable slightly decreases the classification performance (see the Supplementary Material for details on the attribute classifier). However, both image quality and diversity are slightly improved. We also evaluate the contribution of pre-trained word embeddings by using our model without them. We observe that the use of pre-trained word embeddings improves the quality of generated images and converge the training faster. We observe that diversity sensitive loss indeed increases the diversities during the sampling, while slightly reduces the performances on FID and IS. In additional, L cyc , L r ec_a and using pre-trained embeddings can also have effects on the diversities. We investigate whether the contribution of our model relies only on the use of the GMM to model the attributes. Thus, we constrain down our model to a StarGAN*-like model by setting the covariance matrix of the GMM to zero (∀k Σ Σ Σ k = 0). As expected, we observe that our model behaves similarly but with no diversity (LPIPS: .000±.000). Moreover, this proves that our model outperforms StarGAN* even in this case (see Table 2 for a comparison).

CONCLUSION
In this paper, we presented a novel method to manipulate visual attributes through text. Our model builds upon recent literature of image to image translation, using unpaired images and modeling visual attributes through a GMM. Users interact with the model by feeding an image and a text that describes the desired alteration to the original image. The model extracts from the input image the visual attributes, understand the text and the desired transformation, and sample multiple images having the content of the input image and the style of the desired output. By sampling multiple synthesized images, our model deals with the inherent ambiguity of natural language and shows the user multiple transformations (e.g. multiple shadows and styles of blonde hair) from which she/he might select the best output. To the best of our knowledge, we are the first at exploiting automatically generated sentences and unpaired images to manipulate image attributes with textual commands. We showed that our method can model multiple attributes, and that can be applied in a wide range of situations such as interpolation and progressive manipulation. We foresee that our work will stimulate further research on attribute-based image manipulation and generation and will stimulate software integration in virtual assistants such as Alexa or image editors such as Photoshop.
We use the Adam optimizer [24] with β 1 = 0.5, β 2 = 0.999, and an initial learning rate of 0.001. The learning rate is divided by 10 whenever the training loss does not decrease for two epochs. The training is early stopped when the validation loss does not decrease after 10 epochs.
We apply a central crop and random rotation until at most 10 degrees data augmentation techniques during training. The images are then resized to 299 × 299 pixels. The accuracy is 0.91 and 0.91 in the training and test set respectively.
We use this classifier to evaluate the accuracy of our transformations through the Micro-F1 (the F1 score that weights equally all test samples) score between the desired attributes and the generated ones.
To automatically collect manipulation text, we provide some common templates and phrases to collect the text, such as "change ... to ...", "with ... hair", "make the face/bird ...", "a man/woman/bird with/has ...". We randomly permute the description sequence of the attributes and replace some words with synonymy (e.g. "change", "translate", "modify" and "make") to improve the generalization. Similarly, we randomly select the synonymy named entities during generate text (e.g., "man", "male", "boy", "gentleman" and "sir" for the attribute Male). As shown in Figure 9, it indicates that our model generates consistent results with various textual commands for the same manipulation. We also test the sensitivity of the strategy, as the choice of the strategy might influence the results of the model. Table 6 shows that our model is robust to the choice of the textual strategy.

D USER STUDY
We ask different people with various backgrounds to describe how to change the "image A" to be similar to "image B", as shown in Figure 10 and Figure 11. In Figure 8 we show the result of the human evaluation on the realism.

E DOMAIN INTERPOLATION
We show more results in Figure 13 and Figure 14 on various face images. Figure 16 show examples of domain interpolation.

F DOMAIN SAMPLING
As shown in Figure 12, we apply the domain sampling to obtain diverse results that are consistent with the same textual commands.
We also evaluate the ability of DWC-GAN to synthesize new images with attributes that are extremely scarce or non present in