{'fraction_non_alphanumeric': 0.027481542247744052, 'fraction_numerical': 0.004019688269073011, 'mean_word_length': 4.576852698993596, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Conditional MoCoGAN for Zero-Shot Video Generation

We propose a conditional generative adversarial network (GAN) model for zero-shot video generation. In this study, we have explored zero-shot conditional generation setting. In other words, we generate unseen videos from training samples with missing classes. The task is an extension of conditional data generation. The key idea is to learn disentangled representations in the latent space of a GAN. To realize this objective, we base our model on the motion and content decomposed GAN and conditional GAN for image generation. We build the model to find better-disentangled representations and to generate good-quality videos. We demonstrate the effectiveness of our proposed model through experiments on the Weizmann action database and the MUG facial expression database.


INTRODUCTION
Video recognition using deep learning has been actively researched and is being used in the real world. Human action recognition from videos has been applied in various fields, and further development is expected. Deep learning requires a rich and accurate dataset. However, when we try to form a large amount of data, we often encounter the problems of insufficient data and class imbalance. One method for solving these problems is using deep generative models. Generative adversarial networks (GANs) [1] are deep generative models that can generate realistic data based on a learned dataset. The generation of realistic static images has progressed significantly by using deep generative models such as GANs. In recent years, GANs have been applied to video generation. Typical deep video generation models include VideoGAN [2] and the motion and content decomposed GAN (MoCoGAN) [3].
Conditional generation, which generates data of a specified class, is crucial to solving the problems mentioned above using deep generative models, such as GANs. Zero-shot generation, which generates the data of a class that has never been seen in training from the information contained in the training data, can be considered as an extension of conditional generation. For example, if the training data contain only a walking video of person A and a dancing video of person B, this task generates a dancing video of person A.
There have been several studies on zero-shot generative models. DistillGAN [4] showed that it is possible to generate zero-shot images by providing captions of the classes that are not included in the training data to a model trained on a domain. Research on zero-shot video generation includes HOIGAN [5] and Everybody dance now [6]. In the latter, the video is converted from the source video to the target video through frame-by-frame image conversion using the skeletal information of the video containing the person. HOIGAN achieved zero-shot video generation by conditioning on a combination of actions and objects that were not included in the training data. However, its architecture is too complex because it uses word embedding vectors and masking images, and it has multiple discriminators with different roles. The same is true for Everybody dance now [6], which requires frame-by-frame skeletal information of the source video and a dedicated image transformation network from the skeleton to the target. The conditioning for video generation in these methods is very strong.
In this study, we propose a class-conditional video generative model and tackle the task of zero-shot video generation by conditioning on classes that are not included in the training data. To achieve class-conditional video generation, we incorporated the learning method of conditional GAN (CGAN) [7], which is a class-conditional generation model, into the training of MoCoGAN. To summarize, the contributions of this study are as follows: • The class-conditional generation method for static images is also effective in GANs for video generation.
• When the class of a video is decomposed into two classes, namely, motion and content, it can correctly generate a video that satisfies both classes simultaneously, even if the combination of classes is not included in the training data.

RELATED WORK
GANs have attracted significant attention as deep generative models, and various derived models have been proposed. The GAN architecture consists of two networks: a generator and a discriminator. The generator G takes a noise vector z as input and tries to generate data that resemble the training data. Meanwhile, the discriminator D takes data as input and tries to determine whether they are the real training data, or the fake data generated by G. The GAN trains these two networks via an adversarial process given by the following objective: wherex = G(z), and x represent the training data.

Conditional GAN (CGAN)
CGAN is a conditional generative model that is based on the basic GAN structure. The main difference between the CGAN and the original GAN is that both the noise vector and the condition vector are given as input to the generator and discriminator. The objective function is expressed as follows: where y andỹ are the class labels of x andx, respectively.

Motion and Content Decomposed GAN (MoCoGAN)
MoCoGAN is a model that applies a GAN to video generation. It assumes that a video can be decomposed into two features, namely, motion and content, and it generates a video by sampling each input noise vector from a different latent space. The generator G I generates a single imagex (t) from the given input vector. This image is one frame of the video, and by inputting T vectors,x, a T -frame video is generated. The input vector is divided into z c , which is a vector representing the content of the video, and [z (1) m , · · · , z (T ) m ], which are vectors representing the motion of the video. Because it is more natural to have uniform content within a single video, the content vector is sampled only once per video, and it is combined with motion vectors generated by the recurrent model for each m ]] to G I and generate frames from each vector.
The whole discriminator consists of two discriminators, D I and D V . D I takes a randomly sampled frame from the video as input, and D V takes the entire video as input.
The objective function is expressed as follows: where x andx represent the real video and the generated video, respectively.

PROPOSED MODEL
In this study, we propose a conditional-MoCoGAN, which is a GAN architecture based on MoCoGAN, and it learns a conditional generative model that subdivides the latent space into classes by providing more detailed class information on motion and content. An overview of the conditional-MoCoGAN architecture is given in Fig.1. In the manner of the CGAN, we input the class label into the generator and the discriminator. Through this approach, we believe that class-conditional video generation is possible.

Generator
The generator consists of a Gated Recurrent Unit (GRU) R M and an image generator G I . In conditional-MoCoGAN, the one-hot vector of the motion class label y m , a noise vector where the operator ⊕ indicates the concatenation of vectors or tensors. Additionally, the one-hot vector of the content class label y c , a noise vector c , and z (t) m are input into G I .
(t) m , c are sampled from Gaussian distribution. A generated video is equal to the sequence of frames and is represented bỹ

Discriminator
The whole discriminator consists of two discriminators, D I and D V , similar to MoCoGAN. Our model conditions the motion and content classes on a single video. Therefore, every video has a motion class label and a content class label. Following the CGAN learning method, our discriminators receive the video and the class label concatenated as an input. The inputs of the two discriminators are the sampled frame x (t) and the one-hot tensor of the content class label y c , and the full video x and the one-hot tensor of the motion class y m , When an unknown combination of classes that are not included in the training data is specified, and a video of the corresponding class is correctly generated, we can say that zero-shot video generation has been achieved.

EXPERIMENTS
We train the proposed model using the standard back propagation algorithm. We use rectified linear unit (ReLU) as the activation function for G I and leaky-ReLU for D I and D V . We employ the ADAM optimizer and set the learning rate to 2 × 10 −4 for all networks, and we apply spectral normalization [8] to D I and D V . Dimensions of latent vectors z c and z (t) m are both set to 30, following MoCoGAN [3]. Our source code is available at GitHub 1 .

Dataset
We evaluate our model on the following datasets. The length of videos is fixed at 16 frames.  The Weizmann action database [9] contains videos of nine different actions performed by nine people. Motion classes include actions, such as jumping and waving-hands. In this experiment, only the four action classes where the person does not move are used as training data. We resized the videos to 64 × 64 pixels.
The MUG facial expression database [10] contains videos of seven different expressions performed by 86 people. Motion classes include facial expressions, such as sad and surprised. In this experiment, we use only the four action classes and nine content classes to maintain the same conditions as  those in the experiment with the other dataset. We resized the videos to 96 × 96 pixels.
Additionally, one motion class from each content class is removed from the training data to evaluate zero-shot video generation.

Quantitative Evaluation
We used the classification accuracy of the generated videos by a classifier and the Fréchet inception distance (FID) [11] as the quantitative evaluation metrics.
The FID is calculated as follows: where µ and Σ represent the mean and covariance matrix computed from the real videos, andμ andΣ represent the mean and covariance matrix computed from the generated videos. We computed the FID between 100 generated videos and the videos in the dataset for all classes.
To calculate the classification accuracy, we trained the classifier separately on each dataset. Next, we calculated the accuracy of motion and content class predicted by the classifier. We trained three classifiers: C m for only the motion class, C c for only the content class, and C d for both the motion and content classes.
The comparison results are shown in Table 1 and Fig.2. In Fig.2(a), classes with a value of 0 are the classes that are not included in the training data. Although the FID does not improve, the balance of classes in the generated video is clearly better than that of the original MoCoGAN.

Qualitative Evaluation
In MoCoGAN, a video is generated from two vectors, z m and z c . As a qualitative evaluation, Fig.3 and Fig.4 show the generated videos when one of the vectors is fixed.
The latent space is separated into motion and content in MoCoGAN. However, as shown in Fig.4(b), when z m was fixed and only z c was changed, both the motion and the content of the generated video changed. On the contrary, when z c was fixed and only z m was changed, the motion of the generated video did not change. This might be due to the fact that the separation of latent space is too simple, and in fact, the latent space has not been fully disentangled. In our model, we solved this problem by further subdividing the latent space of motion and content into classes. As shown in Fig.3(a) and Fig.4(a), when only one of the vectors, z c or z m , is changed, the generated video changes correspondingly. In particular, the video in the second row of Fig.3(a) corresponds to zeroshot video generation.

CONCLUSION
In this study, we propose a conditional video generative model that subdivides the latent space into classes by incorporating the learning method of CGAN into MoCoGAN. The MoCoGAN is a video generative model that divides the latent space into motion and content. We trained our model on two datasets and evaluated the generated videos using quantitative and qualitative methods. From the results, we found that class-conditional generation is possible in video generation, and that zero-shot generation, which generates a class with a