{'fraction_non_alphanumeric': 0.03461223375204806, 'fraction_numerical': 0.013175860185690879, 'mean_word_length': 4.824652087475149, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Image Generation and Editing with Variational Info Generative AdversarialNetworks

Recently there has been an enormous interest in generative models for images in deep learning. In pursuit of this, Generative Adversarial Networks (GAN) and Variational Auto-Encoder (VAE) have surfaced as two most prominent and popular models. While VAEs tend to produce excellent reconstructions but blurry samples, GANs generate sharp but slightly distorted images. In this paper we propose a new model called Variational InfoGAN (ViGAN). Our aim is two fold: (i) To generated new images conditioned on visual descriptions, and (ii) modify the image, by fixing the latent representation of image and varying the visual description. We evaluate our model on Labeled Faces in the Wild (LFW), celebA and a modified version of MNIST datasets and demonstrate the ability of our model to generate new images as well as to modify a given image by changing attributes.


Introduction
Generative models, that are essential part of unsupervised learning, capture the structure/patterns in data by learning to generate samples that resemble the training data. Earlier work in generative modeling focused on graphical models and energy based models with latent variables, for example, Restricted Boltzmann Machines (RBM) (Bengio et al., 2011;Hinton, 2010), Deep Belief Networks (DBN) (Hinton et al., 2006). In these models, exact inference, normalization constant and its gradients are intractable and samples are usually obtained with expensive Markov Chain Monte Carlo (MCMC) techniques.
Recent advances in deep learning have enabled us to by-pass above mentioned challenges by using deep neural networks as parametrized functions that generate samples. Few dominant approaches that emerged in recent years are Variational AutoEncoders (VAEs) (Kingma & Welling, 2013), Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), Generative Stochastic Networks (Bengio et al., 2014), Deep Recurrent Attention Writer (Gregor et al., 2015), Pixel Recurrent Neural Networks (van den Oord et al., 2016b) and Pixel Convolutional Neural Networks (van den Oord et al., 2016a). Generative models have also helped to set benchmark results in semisupervised learning Radford et al., 2015). Lately there have been attempts to use the advances in deep generative models for semi-supervised learning Chen et al., 2016) and joint modeling of images and their visual descriptions (Reed et al., 2016;Yan et al., 2015).
In this work, we address the problem of jointly modeling images and their visual descriptions through deep generative models. Multimodal learning requires learning a correspondence between data across different modalities. In deep learning context, the first works towards multimodal learning is by N. Srivatsava et al. (Srivastava & Salakhutdinov, 2012) and J. Nigam et al. (Ngiam et al., 2011). These models pose this problem as learning cross-modal representations at latent feature level and some models even try to learn a shared representation across modalities. This often becomes complex, because one modality rarely contains all the information about the other modality. In the case of images and text, the information present in image modality is much more dense than the information in text modality.
VAEs are trained by maximizing the lower bound to the log-likelihood of the data and they include an inference network as part of the training procedure. Extensions such as (Burda et al., 2015) are proposed to increase the tightness of the lower bound. On the other hand, GANs are trained as a mini-max game between the generator and discriminator. Training of GANs only require a differentiable mapping from a latent space to the data space, without any requirement for inference network. Recently proposed InfoGAN (Chen et al., 2016) framework, a variation of GAN (Goodfellow et al., 2014), defines image generation as a function of two sets of variables and is shown to learn interpretable representations of images. This is achieved by including a mutual information objective between a small subset of interpretable latent variables (denoted by c) and the observations. But as in GANs, it also lacks an inference mechanism to infer the latent state given an image.
In this paper, we propose a new model, Variational Info GAN (ViGAN) that combines the InfoGAN model with VAE by using a subset of latent variables to represent the description of images. We also extend the architecture to include an inference mechanism and present a training procedure to jointly train the inference network and the model. We provide a training and sampling procedure for the proposed model. We denonstrate ViGAN's ability to generate new images conditioned on attributes, as well as modifying given images by changing attributes on three datasets namely, MNIST (LeCun et al., 1998), LFW (Huang et al., 2007), celebA (Liu et al., 2015).

Background Material
VAEs and GANs are based on the idea of using a differentiable neural network to transform latent variables to data samples. While the training procedure for VAE includes an approximate inference network, GAN is trained via an auxiliary discriminator network, that learns to discriminate real data samples from generated samples. In this section, we briefly describe previous work on VAE, GAN and In-foGAN on which our model builds upon. In Section 3, we develop our proposed model using the notations introduced in this section.

Variational Auto-encoder (VAE)
A VAE (Kingma & Welling, 2013) works on the principle of maximizing lower bound to the marginal likelihood of the data. In simple terms, VAE can be viewed as an autoencoder with a prior over latent variables. It consists of two networks, encoder, a mapping from data space X to latent space Z and decoder, a reverse mapping from latent space Z to data space X .
Let p(x|z) represent the probability distribution parametrized by the decoder, p(z) represent the prior over the latent space and p(x) represent probability of the data point x. Then upper bound on the negative log-likelihood of a data point x can be written as where q(z|x) is an approximation to the true posterior p(z|x). The loss function for VAE is given as where L recon represent reconstruction loss and L prior represents KL divergence between the approximate posterior q(z|x) and the prior p(z).

Generative Adversarial Network (GAN)
GAN (Goodfellow et al., 2014) consist of two networks, namely, generator (G) and discriminator (D). Generator network takes noise z drawn from a prior distribution p(z) as input and give an image as output. Discriminator network gives the probability that the input image is real.
The GAN objective is to train discriminator to distinguish between real and fake data samples, while simultaneously training the generator to fool discriminator. Losses for the model are given as where L gen and L dis represent generator and discriminator losses respectively. During training, parameters of generator and discriminator are updated alternatively.
InfoGAN (Chen et al., 2016) extends GAN by maximizing mutual information between a small set of latent variables and observations. This can be incorporated into GAN objective by adding a loss term representing the mutual information. This ensures that the information represented by a set of "interpretable latent variables" (c), is not lost during generation process. Let [z, c] be the input to the generator. Then mutual information between c and G(z, c) is where Q(c|x) is approximation to P (c|x). This lower bound can be maximized directly. Hence recognition loss can be defined as (7) Now the modified GAN objective can be written as The recognizer Q(c|x) is realized by a neural network. It can be made to share parameters with the discriminator. The set of interpretable latent variables c can either be continuous, categorical or a combination of both.

Motivation
Mutual information objective of InfoGAN ensures that the information represented by the interpretable latent variables (c), is preserved during the generation process. But it has no control over the information that c represents. In this paper, we propose a method to jointly model images and their descriptions by using c to represent the known aspects of the data, and z to represent all the remaining factors of variation that are not captured by the attributes c. This can done by providing (z, c) as an input to the generator, wherez is the representation of image obtained by passing it through an encoder. In this setup, we assume that the description c is available for all the images. This approach differs from that of InfoGAN, where no labelled data is used and other semi-supervised approaches , in which only a subset of data is assumed to be labelled.
The encoder-generator combination with a prior overz can be thought of as an VAE, where as generator-discriminator combination can be taken to be GAN. Schematic representation of the network architecture is presented in Figure 1. Expectations over the approximate posterior q(z|x) are computed using the reparametrization trick presented in (Kingma & Welling, 2013).

Training and Sampling
The loss for the encoder network L enc is sum of reconstruction loss L recon and the KL divergence L prior as described in the previous section: L prior = KL(q(z|x)||p(z)), and (11) L recon = −E z∼q(z|x),c∼P (c|x) log P (x|z, c).
Loss for the recognition network Q, denoted as L recog is computed using only the samples from the actual dataset.
Loss for the generator/decoder is comprised of three parts. GAN loss for reconstructed samples and generated samples, recognition loss for reconstructed samples and generated samples, and the reconstruction loss.
where λ 1 and λ 2 are the hyper parameters, that quantify the relative importance between reconstruction and recognition objectives.
Finally, the discriminator is trained with three sets of examples with images from the actual data set treated as real examples and generated and reconstructed images treated as fake examples. Discriminator loss, L dis is Given these losses, steps for training the ViGAN is as follows: 1. minimize L enc w.r.t to parameters of encoder, 2. minimize L gen w.r.t to parameters of generator, 3. minimize L recog w.r.t to parameters of recognizer, and 4. minimize L dis w.r.t to parameters of discriminator.
This setup with VAE and GAN sharing common decoder/generator is similar to the model described in (Larsen et al., 2015). However, the decoder/generator conditioned on encoded representation of the image and the attribute vector. Further, to improve the perceptual quality of images, we consider the reconstruction loss in the hidden space of the discriminator instead of the pixel space as in (Larsen et al., 2015).
One of the main advantage of ViGAN is that the same network can be used to generate new images conditioned on attributes as well modify a given image by fixing the image representation and changing attributes. The procedure for this is described below.
• To generate new images, sample z from p(z)and c from P (c|x). Then pass the concatenated vector [z, c] through generator.
• To modify a image, first get representation of image z by by passing it through encoder. Then sample the desired attributes c. Then pass the concatenated vector [z, c] through generator.

Datasets
We trained the proposed model on following three datasets. 3. LFW: This dataset consists of around 13,000 images of human faces paired 73 real valued visual attributed. We cropped and scaled each image to 64 x 64 pixels.

Training Details
We used a similar architecture for all three datasets, with appropriated changes to the dimension of the attribute vector and number of image channels (one channel for modified MNIST and three for other two datasets). Detailed architectures for encoder, generator and discriminator are presented in the following tables For optimization, Adam optimizer (Kingma & Ba, 2014) is used for all the networks, with learning rates of 0.001, 0.001, 0.0002, 0.0002 for encoder, generator, discriminator and recognizer respectively.
We observed the ability of the network to modify images, to be very sensitive to the choice of hyper parameters λ 1 and λ 2 . Suitable choice of λ 1 and λ 2 is very important to strike a balance between ability of network to accurately reconstruct the input and at the same time be sensitive to changes in the attributes.

Results
Below we demontrate the ability of network to generate new images, as well as to modify a given image by changing the attributes. Figure  In Figure 3 and Figure 4, we demonstrate the capability of the proposed model to modify images according to the changes in attributes, while keeping other features intact. In Figure 3. the middle row shows the actual images, while in top and bottom rows, first digit is replaced by 0 and 1 respectively. Similarly in Figure 4, the middle row shows the actual images, while in top and bottom rows, second digit is replaced by 2 and 7 respectively.
In Figure 6, eyeglasses are added to the base image from celebA dataset by changing the corresponding attribute value to 1. Figure 7. shows the modification results on LFW dataset with real valued attributes. In Figure 4. we present four variations of two base images, generated by the network, corresponding to the attributes frowning, smiling with closed mouth, smiling with open mouth and wide eyes. Further some extensions of above mentioned models are proposed to generate images from text. Some examples of such models are (Mansimov et al., 2015;Reed et al., 2016;Mirza & Osindero, 2014) . These models can generate images given attributes, but cannot modify a given image. (Larsen et al., 2015) proposed a method to modify faces, by adding or subtracting a mean visual attribute vector. But the method is applicable for only binary attributes. (Pandey & Dukkipati, 2016) proposes another method based on embedding visual representations and attribute descriptions in a common space.

Concluding Remarks
Recently Reed et.al (Reed et al., 2016) also proposed a similar method based for conditional image generation. They too demonstrate image modification with their network. But they learn the posterior separately from the generation process, while our method learns posterior along with the generation network In the work, we proposed a new architecture for joint modeling of images and their visual descriptions by combining the ideas from Variational Auto Encoders (VAEs) and Info-GAN. We demonstrate the ability of the proposed model to generate new images as well as to modify existing images.