{'fraction_non_alphanumeric': 0.026099345897286446, 'fraction_numerical': 0.008613431772553592, 'mean_word_length': 4.487562189054726, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hierarchical Model for Long-term Video Prediction

Video prediction has been an active topic of research in the past few years. Many algorithms focus on pixel-level predictions, which generates results that blur and disintegrate within a few frames. In this project, we use a hierarchical approach for long-term video prediction. We aim at estimating high-level structure in the input frame first, then predict how that structure grows in the future. Finally, we use an image analogy network to recover a realistic image from the predicted structure. Our method is largely adopted from the work by Villegas et al. The method is built with a combination of LSTMs and analogy-based convolutional auto-encoder networks. Additionally, in order to generate more realistic frame predictions, we also adopt adversarial loss. We evaluate our method on the Penn Action dataset, and demonstrate good results on high-level long-term structure prediction.


Introduction
Learning to predict the future is an important research topic in computer vision and artificial intelligence. [2] Humans make predictions and inferences about the real-world all the time. These predictions focus on the semantics of the action (i.e. where a ball will land, expected overall shape of a figure, etc.), rather than the exact location or pixel intensity. In addition, the rough semantics generally provides more value to the prediction than raw pixel values would anyways.
To better reflect how humans make predictions, we seek to develop a hierarchical approach for video prediction that first predicts some higher-level features of future frames, then regenerate realistic video from the predicted higherlevel features. We initially hoped to generate motion segmentations of objects within the videos as the higher-level intermediate features. A recent work by Villegas et al. [10] predicts future poses of humans and then uses image analogies to map the high frequency details of input images to the predicted poses. Since this is very similar to our proposed approach, we decided to first adopt and implement the methods within this paper, then proceed onto replacing pose with motion segmentation in the future.
We evaluate our method on the Penn Action dataset [12], and demonstrate good results on high-level long-term structure prediction.
The rest of the paper is organized as follows: A review of the related work is presented in Section 2. The deep convolutional model we use is presented in Section 3. The experimental results and analysis are shown in Section 4 and 5. Finally we conclude our work with discussion of future work in Section 6.

Related Work
There has been numerous recent work on pixel-wise video prediction.   [5] uses a multiscale convolutional generative adversarial network (GAN) approach to recursively generate the next frame given the past few frames. Such pixel-based generations are usually reasonable for the first few predicted frames, but the quality degrades quickly. Vondrick et al. (2016) [11] separately generates a moving foreground representation of a video and a static background and combines the two. Similar approaches include [7], [4] and [9]. Their results, while not photo-realistic, captures the general motion of the foreground object rather well.
The approaches mentioned above use pixel-wise prediction on video frames, but we hope that a hierarchical approach would produce consistent video for a longer duration. Our current hierarchical approach obtains the current pose of the human in the video, predict future poses, and generate realistic video corresponding to the poses. In the future, we will explore replacing pose estimations with motion segmentation, which could generalize to nonhuman objects.

Overview
Our model is a three step process: 1) pose estimation, 2) pose prediction, and 3) image analogy generation. First, given the input frame x t , our model first get the corresponding pose heatmap p t via pose estimation. Second, our pose prediction network predicts the pose heatmap for fu-ture frames. Finally, our image analogy network recovers a realistic image from the corresponding pose heatmap and input frame.

Pose Estimation
We represent each pose as a list of joint locations. There is work by Zhe Cao et al. [1] and Newell et al. [6] to extract joint locations for different poses. For the purposes of focusing on pose prediction and image analogy, we use the poses annotated by the Penn Action Dataset for our pose predictions.

Pose Prediction
We create a sequence-to-sequence LSTM network, illustrated in Fig. 1, to predict the future joint locations given a sequence of input joint locations.
where p t ∈ R 2L is a vector containing the coordinates of all the joint locations at time t, c t is the LSTM memory cell vector at time t, and h t is the latent output of the LSTM at time t. Initially, we feed k frames of pose estimation (p 1 to p k ) into the LSTM to give the LSTM context on the video to be predicted.
We generate the future latent outputs h t for T frames after the k frames of initial input by feeding in 0 as the input into the LSTM at each time step.
We then predict the pose in these T frames by learning a two-layer neural network mapping from the latent output of the LSTM to pose joint locations.
where σ is the sigmoid activation function. Notice that we normalized the x and y coordinates in p to be between 0 and 1. Also, we do not feed in the predicted posesp t as input into the LSTM, so that the errors in the predictions does not propagate further into future frames. Instead, this encourages the LSTM model to predict all the future frames only given the original frames. Our loss function for training the LSTM aims to enforce accuracy of the predictions equally for all T future time points.

Image Analogy
We use Image Analogy network [8] to synthesize the future frame from its pose structure. As is illustrated in Fig. 2,   the relationship between p t and p t+n is the same as the relationship between x t and x t+n . More specifically, the future frame x t+n can be generated by transferring the structure transformation from p t to p t+n to the observed frame x t . To promote more realistic generations, we adopt a generative adversarial network [3] for our image analogy network.
Generator: As is shown in Fig. 3, our image generator uses a pose encoder f pose , an image encoder f img and an image decoder f dec . Specifically, f pose is a convolutional encoder that specializes on identifying key pose features from the pose input that reflects high-level human structure. f img is also a convolutional encoder that acts on an image input by mapping the observed appearance into a feature space where the pose feature transformations can be easily imposed to synthesize the future frame using the deconvolutional decoder f dec . The image analogy is then performed bŷ wherex t+n andp t+n are the generated image and corresponding predicted pose at time t + n, x t and p t are the input image and corresponding estimated pose at time t, g(:) is a function that maps a set of (x, y) joint coordinates to a stack of L heatmaps, and n is the difference in time between the frame to be generated and the input frame.
For the network architecture, our f img encoder is a VGG-16, f pose is a simplified VGG with many layers removed, while f dec is a reversed VGG-16 where all convolutions are replaced by deconvolutions. Our reasoning for removing some VGG-16 hidden layers in f pose is that the pose heatmaps are relatively simple and do not need as much representative power (and also our GPUs are dying).
The loss of generator is written as follows: is the -2 distance between ground-truth and predicted in image space, L f eat = AlexNet(x t+n ) − AlexNet(x t+n ) 2 2 is -2 distance between ground-truth and predicted in the feature space defined by the last convolution layer of a pre-trained AlexNet, and L adv = − log D([p t+n ,x t+n ]) is the adversarial loss. Finally, λ img , λ f eat , and λ adv are hyperparameters denoting how much each loss function will contribute to our overall loss function.
Discriminator: We input pose structure and video pairs to the discriminator, which outputs 1 if pose and video are real and from the same time slice and outputs 0 otherwise. The network architecture of the discriminator is also VGG-16. Our discriminator loss is defined as follows: Note that the first term incentivizes the discriminator to predict "real" given a real image and pose pair, the second term incentivizes the discriminator the predict "fake" given a real pose and fake image, and the third term incentivizes the discriminator to predict "fake" given an image at time t and a pose at a different time (so the discriminator can become sensitive to pose differences).

Experiments
Our experiments largely focus on human pose predictions and creating image analogies.

Dataset
We use the Penn Action Dataset [12] for training and testing. The dataset consists of 2325 videos of human actions. The 15 actions in the dataset are baseball pitches, baseball swings, bench press, and some others. Each image has a 13 annotated joint locations stored as xy coordinates denoting joint coordinate position in image. The joints include the head, left shoulder, right shoulder, left elbow, right elbow, and some others. Most videos are 480x270 and consist around 50-150 frames. The details of the dataset is shown in Table 1.

Pose Estimation
Using the annotated joint locations from the Penn Action Dataset, we create a corresponding heatmap by drawing a circle of radius 5 at each location and applying a Gaussian blur with σ = 5 (illustrated in Fig. 4). This heatmap will represent our pose estimation input into the image analogy network.

Pose Prediction
We apply a 2-layer neural network function to the output of the LSTM architecture described above. The network consists of a 1024-node hidden layer followed by a 100node hidden layer, both with sigmoid activation function. We use k = 15 input images to encode our LSTM network and decode T = 45 prediction images. We trained the network on 73 squats video and ran tests on 5 videos.

Image Analogy
For the image analogy network, we trained the network on squats videos. We converted all videos to grayscale and resize to 224x224. Our network parameters are λ img = 1, λ f eat = 1 and λ adv = 0.1. We define each training epoch The learning rate for the generator network and the discriminator network is 1e −6 . We did the majority of the training on a GTX 1070 and the above parameters took around 12 hours to train. We used a stochastic gradient descent optimizer with a minibatch size of 4. Because the network is so large (several VGGs), 4 was the maximum mini-batch size we could work with.

Results on Pose Prediction
Our results show that the LSTM network is able to capture the movement of squats accurately. There is high variation in the form and speed of how people do squats. The LSTM model is able to capture an "average" prediction model based on the trained videos. Thus, though the predicted joint locations do not align perfectly in our tests, the predictions show a convincing alternative to the ground truth, as shown in Fig. 6 and 7.

Analysis of Pose Prediction
Our initial test with a simple sequence-to-sequence network with 128 hidden units and no fully connect output layer did not work well in predicting poses for test videos. The simple sequence-to-sequence network was able to overfit to training data, but was unable to make meaningful predictions for test images. By increasing the number of hidden units in the LSTM block and adding two fully connected layers to the output of the LSTM, we were able to create a working prediction model.

Results on Image Analogy
Our image analogy network partially captures the motion of the human. From the results in Fig. 8, we can see that the network easily converged on a mean image prediction of the training images and only partially used the pose information to generate accurate predictions. We tried contacting the authors of the original paper for insight into how they trained their network and what hyperparameters they used, but got no response. Thus, in an attempt to get better performance, we chose to tune hyperparameters (details shown in Section 5.4). However, hyperparameter tuning does not led to very significant improvement in results.

Analysis of Image Analogy
In our initial attempt to create the image analogy network, we had a bug that scrambled the heatmaps (i.e. inputted heatmaps did not correspond to the correct frame). The result is that the image outputted is a mean image of the training frames for each video. Interestingly, after fixing this heatmap correspondences bug, the network still produces the same result.
Our first instinct was that our hyperparameters λ img , λ f eat and λ adv were not tuned properly. We did cross validation across 9 different combinations of parameters, all producing similar or worse results. . We noticed that the l2 loss of the images contributed the most to the loss and to the overall quality of the image. With multiple VGG networks (totaling 4GB in weights), running numerous tests to tune hyperparameters is very time consuming. Hence, we stopped after 9 variations.
The second observation is the information from the pose seems to be ignored. We believe putting more emphasis on  the pose in the latent space may help fix our issue. This is something we have not tried and can be considered for future work.
One notable part is that there is no released source code for the architectures proposed and we wrote all the code from scratch in Tensorflow.
Finally, finding an optimal learning rate proved to be very crucial for proper convergence. Because our network is very large, we were unable to load optimizers like Adam, Momentum, or Adagrad in Tensorflow. These optimization methods store additional variables for each weight, which exceed our GPU vram limit of 8GB. We resorted to stan-dard GradientDescentOptimizer with a very small batch size, which requires a more precisely defined learning rate. In Fig. 5. we show the effect of our training loss on two learning rates: = 1e −5 and = 1e −6 . We can see that = 1e −5 is a suboptimal learning rate that is too high and was not able to converge as quickly as = 1e −6 . Notice the instability associated with = 1e −5 loss. Training the network can take many hours, making it an important task to find a good learning rate to speed up training and converge to an optimal value.

Conclusion and Future Work
In this paper, we present a hierarchical model of pixellevel video prediction. Using human action video dataset as benchmark, we demonstrate our hierarchical prediction approach that our hierarchical model is able to predict pose heatmaps.
The success of our work is that the model can predict long-term pose structure from video frames. This is significant because it supports the success of hierarchical model for video frame prediction. The limitation of our work is that the analogy network does not yield very realistic result.
For future work, we will work on making long-term RGB video frame prediction. This work only generates a single feature trajectory and work can be done on generating predictions for multiple features. Finally, we can work on generalizing pose prediction to segmentation predictions to work for arbitrary objects (and not just human poses).