{'fraction_non_alphanumeric': 0.03612584598500092, 'fraction_numerical': 0.012956527041034083, 'mean_word_length': 4.690026019080659, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Bi-directional Flow-based Video Generation from one Snapshot

Imagining multiple consecutive frames given one single snapshot is challenging, since it is difficult to simultaneously predict diverse motions from a single image and faithfully generate novel frames without visual distortions. In this work, we leverage an unsupervised variational model to learn rich motion patterns in the form of long-term bi-directional flow fields, and apply the predicted flows to generate high-quality video sequences. In contrast to the state-of-the-art approach, our method does not require external flow supervisions for learning. This is achieved through a novel module that performs bi-directional flows prediction from a single image. In addition, with the bi-directional flow consistency check, our method can handle occlusion and warping artifacts in a principled manner. Our method can be trained end-to-end based on arbitrarily sampled natural video clips, and it is able to capture multi-modal motion uncertainty and synthesizes photo-realistic novel sequences. Quantitative and qualitative evaluations over synthetic and real-world datasets demonstrate the effectiveness of the proposed approach over the state-of-the-art methods.


Introduction
We wish to address the problem of training a deep generation model for imagining photorealistic videos from just a single image. The problem is a non-trivial task as the model cannot only guess plausible dynamics conditioned on static contents. The problem is thus significantly harder than motion estimation task or video prediction problem in which paired consecutive images are assumed. Under the singleimage constraint, choosing a suitable representation learning method becomes critical to the final visual quality and plausibility of the rendered novel sequences. Existing methods such as MoCoGAN [26], VGAN [28], Visual Dynamics [35], and FRGAN [37] either directly render RGB pixel values or residual images for the modeling of dynamics in novel sequences, but they usually distort appearance patterns and are thus short for preserving visual quality. A recent method proposed by Li et al. [12] learned dense flows to propagate pixels from the reference image directly to the novel sequences, offering a better chance to generate visually plausible videos. However, Li et al. [12] require external and accurate flow supervisions (generated from SpyNet [38] to train the flow generation component, and the inherent warping artifacts were not handled in a principle way. Warping artifacts frequently occur in the generated novel frames, such as the examples in Fig. 1. In this paper, we adopt the same notion of 'per-pixel flow propagation' as in Li et al. [12] but with the following consideration: (1) how to learn robust content-aware flow distributions in an unsupervised manner, without any external flow supervision; (2) how to synthesize photorealistic frames while eliminating warping artifacts such as occlusions and warping duplicates in a principle manner.
To this end, we propose an unsupervised bi-directional flow-based video generation framework solely based on one single image, named as ImagineFlow, which tackles the aforementioned challenges in a unified way. Our model has three appealing properties: (i) End-to-End Unsupervised Learning -It allows end-toend unsupervised learning to generate novel video from a single snapshot based on per-pixel flow propagation. The unsupervised learning module is new in the literature. It relaxes the need of external flows for supervision. The proposed learning is incredibly convenient and powerful in our task as it does not require laborious ground-truth annotations or preparation and the training data is nearly infinite and rich in motion patterns.
(ii) Bi-directional Flow Generation -We formulate a bidirectional flow generator, which outputs forward flows (input image → target images), and the backward flows (target frames → input image) simultaneously. The bi-directional flows can self-regularize each other according to a wellknown cycle consistency criteria, i.e., valid flows always have corresponding inverse flows back to their original locations. The resultant flow distributions are within reasonable flow manifolds even they are learned without explicit flow supervision. In contrast, Li et al.
[12] only generate the backward flows, whose reliability is governed by explicit flow supervisions. (iii) Occlusion-aware Image Synthesis -Another merit of the proposed bi-directional flows generation is that the cycle consistency of flows allows us to detect occlusions in a robust and principle way. Occlusions can be detected in areas where cycle consistency is violated. Our model can leverage the occlusions inferred to help determine between per-pixel flow propagation or pixel hallucination for novel frame generation. Specifically, our occlusion-aware image synthesis inputs bilinear warped [38] novel frames overlaid with corresponded visibility (i.e., non-occluded) masks, and then employs a learnable mapping module that projects the warped frames onto the space of natural images. This module generates reasonable contents in the disoccluded area and seamlessly repairs warping duplicates simultaneously.
Ablation study validates the effectiveness of our Imag-ineFlow model. Extensive experimental evaluations also demonstrate its superior qualitative and quantitative performance over state-of-the-art methods [35,28,1,26,12].

Related Work
Motion Prediction. Single image based video generation is closely related to the motion prediction problem. Given an observed image or a short video clip, various methods have been proposed to predict dense future motions by optical flows [21,32,30], object trajectories [31], difference or residual images [35], or deep visual representations [27].
While most methods follow a deterministic manner, a few seminal works also try to characterize the uncertainties in the predicted motions, based on probabilistic models such as conditional variational autoencoders [35,30] or generative adversarial networks [13]. Our work falls into a probabilistic motion modeling framework. We differ to aforementioned studies in that we aim at employing the predicted motion distribution to synthesize structurally coherent novel frames with reasonable motions. This requires new formulation for occlusion reasoning and frame generation.
Motion-based Video Generation Video generation can be roughly categorized into two classes according to whether it takes condition or not. A series of unconditioned video generation methods synthesize novel videos from scratch, with different learning techniques such as adversarial learning, motion/content separation, recurrent neural networks [25,26,22]. The visual qualities of their outputs are still not satisfactory to generate photorealistic videos. Conditioned video generation appears to be a more promising choice to generate visually plausible videos. A category of approaches predict novel frames from consecutive input frames [9,25,20,16,17,3]. Another category, which matches our problem setting, predicts future frames just based on one still image [37,12,1,35]. Some single image based methods characterize motions as feature filters or middle-level transformations [1,29,35]. These methods usually fail to preserve appearance patterns. To achieve photorealistic generation quality, Li et al.
[12] also apply flows as the medium, but the method requires ground-truth flow supervisions and no explicit occlusion handling is introduced. By contrast, our approach is unsupervised, as it learns to generate bi-directional flows directly through the constrain of cycle consistency. The flows are reliable even without explicit supervisions, and they permit occlusion detection in a principle manner. With the structurally coherent flow fields, our method is more effective in rendering realistic video clips. Flow-based warping artifacts are eliminated by an additional occlusion-aware synthesis module.
Applications and Regularizations for Flows. Flows have been adopted for various tasks, such as video enhancement by task-oriented flow [34], video interpolation and extrapolation by voxel flow [15], gaze direction detection [5] and novel view synthesis [38]. Reliable flows often require regularizations to strengthen its structural coherence. Some approaches regularize the structures explicitly by flow supervisions [2,12] or implicitly by adversarial networks [13]. In this work, inspired by the cross validation in stereo and optical flow estimations, we found that simply with cycle flow consistency, the learned flow space can already be effectively enforced with structural coherence without laborious human labeling or preprocessing.

Problem Definition
Our task is to learn a probabilistic distribution p(I T |I 0 ) conditioned on a reference frame I 0 , and then sample novel sequencesĨ T from this conditioned distribution, where T = {1, . . . , T } are the frame indices.
In this work, we aim at simultaneously predicting a set of backward flows W b T = {W b t } t∈T pointing from tentative novel framesĨ T to the reference frame I 0 , and corresponding forward flows W f T = {W f t } t∈T inversely from I 0 tõ I T , and then leveraging the predicted bi-directional flow fields to generate novel sequencesĨ T . More specifically, this task is equivalent to learning 1) a bi-directional flow generator p φ (W b T , W f T |z, I 0 ) that is also conditioned on a motion code z, and 2) an occlusionaware image synthesis module R ω (·). The random motion code z may be sampled from a standard Gaussian distribution. φ and ω are network parameters. The complete model is learned from a training set of sequence pairs {(I

Recap: Image Synthesis by Backward Warping
Given the backward flows W b T from the tentative target frames to the reference frame I 0 , the synthesized frames are usually warped via bilinear sampling from I 0 [6]: Existing flow-based models, such as Li et al.
[12], only apply the backward flows to generate novel frames. However, in the framework of unsupervised learning, using backward flows alone is less favored due to three issues: Warping Artifacts. Backward warping operations will produce artifacts due to occlusions. Occlusions result in unfilled holes and generate warping duplicates in the warped images.
Motion Inconsistency. Since unsupervised backward flow learning usually applies photometric consistency to learn the flow space, the learned backward flow distributions may Figure 3. Bi-directional flow generation. This network outputs bidirectional flows and further their visibility masks by cycle consistency. Content features from E θ (I0) are compositionally fused into the flow generator.
not align with the real flow space, especially when the sequences contain plain area or repeated patterns. Reliable flows should be cross-consistent outside the occlusion regions, i.e., an object in the target frames should always be able to predict reliable inverse flows back to the object in the reference frame, as illustrated in Fig. 2, similar to the concerns arised in optical flow and stereo estimation [4].
Structure Inconsistency. Moreover, W b t and novel frames (not the reference frame) are co-aligned in their spatial distributions, as shown in Fig. 2. It means that the backward flows do not only have to capture pixel-wise motions but also need to present the spatial structures in the target frames. Unfortunately, naïve unsupervised learning usually tends to generate flows aligned with the condition I 0 rather than novel sequences, thus neither the pixel-wise motion nor the spatial alignment can be well discovered.

Bi-directional Flow Generation
Different from the backward flows, the forward flows W f T are otherwise consistent with the spatial structure of I 0 , as shown in Fig. 2. And ideally W f T and W b T should be cross-consistent except the occlusion regions. Thus we also learn the forward flows W f T as an auxiliary output concurrently with the backward flows W b T , and exploit these flows to regularize the spatial structures and enforce motion consistency of the backward flows. Moreover, the crossconsistency between the bi-directional flows also give cues for the occlusion detection.
We generate bi-directional flows from a bi-directional flow generator , with two parallel output branches, as visualized in Fig. 3. The paired flows are constrained by the cross consistency that valid pixel-wise paths by W b T and W f T form loop closure and bi-directional visual consistency between a training pair I 0 and I T = {I t } t∈T .
Occlusion Detection Occluded regions are usually the regions where the bi-directional flows are inconsistent. We define the visibility mask M 0←t indicating pixels in I 0 that are also visible in I t , according to the backward-to-forward Figure 4. ilarly as the conditions [36,18] that Similarly, we also obtain the visibility mask M t←0 about pixels in I t that are also visible in I 0 , based on the same condition for the forward-to-backward flow difference ∆W b←f t . The hyper-parameters are set as α = 1.0, β = 0.1 in our experiments. Note that M t←0 corresponds to the target frame I t and M 0←t is with the reference frame I 0 .
Cycle-consistent Flow Learning The bi-directional flows are learned to enforce their internal cycle consistency, where the valid (i.e., in non-occluded regions) forward (or backward) flows pointing from I 0 (or I t ) to I t (or I 0 ) are mirrored by the backward (forward) flows from the warped locations in I t (or I 0 ) to the original locations in I 0 (or I t ). We define the cycle consistency objective L cc by 1 norm as The learned flows should also be constrained by the bidirectional photometric consistency in the valid regions as The bi-directional flows in the occlusion regions are otherwise naïvely guessed with a smoothness prior within their neighborhood, as we only apply the non-occluded flows to generate the warped frames, as Eq. (1), while leaving the occluded regions undefined.
Compositional Condition Fusion In addition to cycle consistency, the proposed bi-directional flow genera- are required to generate contentaware flows that are semantically corresponded to the con-tent structures in I 0 . It is achieved by introducing a compositional condition fusion scheme into the main branch of bidirectional flow generator, which looks like the Hourglass structure [19] that gradually adapts the sampled motion features with multi-level content features {c m } M m=1 extracted from the image encoder E θ (I 0 ), as illustrated in Fig. 4.
Our flow generator starts from fusing the sampled random variable z with the top-level content feature vector c 1 , by treating the content features as a depth-wise convolution kernel. The fused motion features are upscaled by a deconvolution layer consisting of an upsampling operation and a 3D convolution layer. 3D convolution layers are employed throughout the main branch of to learn spatiotemporal features and offer more complex motion patterns in the generated flows.
The subsequent network repeats several stacked network blocks composed by a 2D-to-3D fusion block, an aforementioned deconvolution layer and an additional 3D convolution layer, before split into two branches of forward and backward flow subnets, as shown in Fig. 4(a). The proposed 2D-to-3D fusion blocks fuses a content feature map c m and a corresponding 3D motion feature volume V m , by at first concatenating c m along the channel axis of each time slice of V m , and then being convoluted by one 3D convolution layer for a seamless feature fusion. It suggests that the motion features in any time stamp should explicitly share the same content features with each other, similar as [26].

Occlusion-aware Image Synthesis
Backward bilinear warping operation F(x, W b t |I 0 ) inherently suffers from warping artifacts, thus it could not produce visually plausible novel videos. Li et al.
[12] applied an image refinement module to remove any warping artifacts. We argue that explicit occlusion handling would be more effective in removing artifacts and inpainting contents in the occluded regions.
Unlike the frame interpolation studies [15,7] that require at least two frames to infer occluded regions, our method only needs one reference image I 0 to simultaneously infer bi-directional flows {W f T , W b T }. These flows infer the visibility masks {M 0←t , M t←0 } t∈T , according to Eq. (2). Consequently, we propose an occlusion-aware image synthesis module R ω that accepts the visibility mask M t←0 and the naïvely warped frame I t←0 . It outputs a refined novel frameĨ t←0 with the suppression of warping artifacts.
The network for the image synthesis module is similar as those for the inpainting task [14], which applies multi-level skip connections in an autoencoder. The encoder borrows the same architecture of the VGG-19 up to the ers in the decoder.
In the training stage, the proposed synthesis module also refines the naïvely warped frame I 0←t from the target frame I t to the reference frame I 0 with the help of the visibility mask M 0←t . We train this network using a perceptual loss [8] for bi-directional image synthesis: where Φ k (·) denote the features of a pretrained VGG-19 network at the layer ReLUk 1, and λ is to balance the terms. The proposed occlusion-aware image synthesis module is able to fill unreliable regions with semantically meaningful contents, and hallucinate fine details to overcome the blurring artifacts caused by the bilinear warping operation. This model is learned in an unsupervised fashion and can be incorporated with the aforementioned bi-directional flow generator for an end-to-end system for flow generation and image synthesis.

Network Training and Implementation
We apply the CVAE [23] to learn our conditioned generative model. The complete system includes (1) Motion Encoder q ψ (z|I 0 , I T ) that produces a latent motion prior distribution encoding the underlying motions for the input sequences I T w.r.t. the reference frame I 0 ; (2) Image Encoder E θ (I 0 ) that extracts content features of I 0 and it is included in the bi-directional flow generator; (3) Bi-directional Flow Generator p φ (W f T , W b T |z, I 0 ) that generates bi-directional flow fields from sampled motion variables z and correlated with the content features in E θ (I 0 ); (4) Occlusion-aware Image Synthesis R ω (·) that synthesizes the final sequences based on a combined warped frames and their visibility masks.
Training Phase. In the training phase, the motion encoder encodes a stack of adjacent frames {I T , I 0 } as a 3D volume and produces mean and variance vectors to model the posterior q ψ (z|I 0 , I T ). The image encoder E θ (I 0 ) extracts multi-level content features {c m } M m=1 . The bi-directional flow generator p φ (W f T , W b T |z, I 0 ) uses the sampled motion variables z from q ψ (z|I 0 , I T ) as the input. At the end of the flow generator, we produce the initial bi-directional warped frames {I t←0 , I 0←t } t∈T and their visibility masks {M t←0 , M 0←t } t∈T . They are then inputted into the proposed occlusion-aware image synthesis module R ω (·) to obtain the final synthesized frames {Ĩ t←0 ,Ĩ 0←t } t∈T .
The objective of our ImagineFlow model (Fig. 5) extends the variational upper bound [10]  λ bi-vc L bi-vc (z (s) ) + λ cc L cc (z (s) ) + λ pp L pp (z (s) ), where the bi-directional photometric consistency L bi-vc , cycle flow consistency L cc and the perceptual loss L pp are monte-carlo integrated to serve as the negative loglikelihood for this generation model. The KL-divergence aims at constraining the discrepancy between the posterior q ψ (z|I 0 , I T ) and the naïve motion prior. In addition to the above objective, we add a small amount of TV-1 norm to enhance the smoothness of the final images and flows. λ bi-vc = 1.0, λ cc = 0.05 and λ pp = 1.0.
Test Phase. In the testing phase, we just require a plain motion prior p(z) = N (z|0, I) to replace q ψ (z|I 0 , I T ) for the sampling of motion variable z. We just employ the backward flows W b T and their visibility masks {M t←0 } t∈T for the final sequence generation. The novel sequence {Ĩ t←0 } t∈T is generated by inputting {I t←0 , M t←0 } t∈T into the occlusion-aware image synthesis module.

Settings
Datasets. The proposed model is trained and evaluated on three popular video datasets: UCF-101 dataset [24], Moving MNIST dataset [25] and Exercises dataset [35]. UCF-101 contains 13, 320 real video clips from 101 action classes with substantial background movements. Moving MNIST dataset is a synthetic dataset constructed by warping the digits in MINST dataset [11] with affine transformations. The Exercises dataset includes around 60k pairs of frames from real workout videos with a static background. Implementation Details. Our system was implemented in PyTorch. It is end-to-end trained by Adam optimizer, with a small learning rate of 0.001, β 1 = 0.9 and β 2 = 0.999. The batch size is 32 and the train/test images are cropped and resized to 64 × 64 for the Moving MNIST dataset and 128 × 128 for the UCF-101 and Exercise Datasets. We use two settings to show the superiority of the proposed method. (1) For long-term video generation centered around a single frame, we set the number of rendering frames to 8. (2) To compare with prior arts on long-term and short-term future frame predictions, we modify our network by predicting the next 4 frames and the next 1 frame, accordingly. Moreover, the baseline model copies the main structure but only preserves the backward flow branch and the highest fusion block, and removes the occlusion-aware image synthesis module. Evaluation Metrics. We also quantitatively evaluate the models in addition to subjective tests. We sample 100 sequences for one reference frame and report the best PSNR and SSIM [33], named as PSNR@100 and SSIM@100. A good model should synthesize at least one sequence that is similar to the ground-truth. We also apply the recent Frechet Inception Distance (FID) based on I3D model to evaluate the perceptual performance of the generated sequences.

Ablation Study
The unique advantage of the proposed ImagineFlow model is its capability of learning robust flow distributions and synthesizing visually plausible videos. It includes several pivotal components that contribute to this feature.
Motion Representation. We show that flows are more reliable than difference images and pixel values in synthesizing novel frames. To verify it, we add two more baselines by changing the output of our baseline network to RGB pixel values and difference images (named as Flow, Pixel and Diff, respectively). As shown in Fig 6(a), in the exercise dataset, Diff blurs out the face details, but Pixel is often too blur to capture upper torsos and arms. In contrast, Flow preserves the detailed contents and produces fewer visual artifacts around body parts. In addition, in the UCF-101 dataset ( Fig. 6(b)), Pixel fails to capture large motions and Diff produces severe aliasing around the legs of the athletes. Evaluations in Fig. 6(c) also quantitatively prove that the flow-based baseline outperforms the rest counterparts. image encoder and flow decoder. As shown in Fig. 7(a), compared to the baseline model, Cyc. Consist encourages more physically consistent flows around the upper body so that there are much fewer visual distortions in synthesizing arms. But the generated flows are blur and cannot capture with the content well. Comp. Fusion applies multilevel content features to regularize the spatial structure of the predicted flows (e.g., small flows in the background and homogeneous flows aligned with the upper body). But this structure does not generate physically reasonable flows, so the synthesized arms suffer from warping artifacts (e.g., the arms are much thinner). Their combination (i.e., the flow generator in the ImagineFlow model) performs the best and the predicted flows are not only physically reliable but also consistent with the captured contents. Apart from the qualitative results, either PSNR @100 or SSIM@100 reports similar performance gains in Fig. 7(c), which suggest that the proposed components are complementary to each other. Our flow-based frame synthesis faithfully inpaints unreliable regions in the occlusion-aware warped frames. For instance, the violinist in Fig. 8(a) is moving right, thus the occluded white board should be revealed in the target frame. Our flow generator successfully discovers these unreliable regions (see Fig. 8(b)), and our frame synthesis module completes these regions and guesses the structure of the white board as what is desired. As for comparison, we also show the backward warped target frame in Fig. 8(c). Since it renders novel frames solely based on pixels in the reference image, the occluded background cannot be discovered, even though the underlying flows are estimated accurately.
Motion Diversity. The proposed method shows sufficient diversity in generating novel sequences. In Fig. 9, different samples for one reference frame in the Exercise and Moving MNIST datasets demonstrate diverse motion variations. The motion is illustrated by creating a RGB image where the magenta channels are from the sampled frame and the green channel from the reference frame, as suggested in [35]. For examples, Fig. 9(a-1) shows diversified motions around the legs and Fig. 9(a-2)   term motion patterns for two digits. Sample 1 in Fig. 9(b-1) shows contractive and clockwise motions, while sample 2 in Fig. 9(b-2) depicts an ascending digit pair. Motion Complexity. The proposed ImagineFlow model can capture complex and long-term motions. In Fig. 10(a), we demonstrate sampled 9-frame sequences given the 5 th frames as the reference. The sequence "surfing" has varying tidal waves over time nearby the surfing board, and the athlete is blending over. In the sequence "violinist", our ImagineFlow model covers occluded regions and preserves content structures with meaningful motions in playing violin. The long-term ImagineFlow model samples more complex motion patterns than the short-term one, as shown in Fig. 10(b) and (c). The example in Fig. 10(b) shows that the long-term prediction is capable of guessing the rotation of the female dancer while the iterative short-term prediction will gradually distort and blur the contents. The example in Fig. 10(b) also finds that the iterative variant fails to preserve the spatial structure of the digits.

Experimental Comparisons
Our ImagineFlow model is compared with various single image based video synthesis methods such as Visual Dynamics [35], Video GAN [28], Video Imagination [1], MoCoGAN [26] and Li et al. [12], in which Visual Dynamics is designed for the next frame prediction but Video GAN, Video Imagination, MoCoGAN and Li et al. can be applied for longer video generation.
Exercise Dataset: As shown in Fig. 12(a), the proposed model and Visual Dynamics are compared by sampling two future frames given a same reference frame. Our results do not only present distinct motions but also preserve struc- tural coherence without blurring and distortions. However, Visual Dynamics, which depends on difference images as the motion representation, produced blurry future frames with mixed arm patterns (shown in the first row) or distorted hand structures (as in the second row).
Moving MNIST Dataset: Similar as our method, Video Imagination is able to render reliable predictions on the Moving MNIST dataset, as shown in Fig. 12(b). It is because these digits are generated by affine transformations and thus their motions fit the mechanism of Video Imagination. However, Video GAN produces large motions but the shapes of the digits are severely distorted.
UCF-101 Dataset: Fig. 12(c) shows the visual comparison with Video Imagination, Video GAN and Li et al. on the ice dancing sequence about synthesizing 4-frame sequences. These reference GAN based methods, such as Video GAN and MoCoGAN usually fail when the videos contain over-complex spatio-temporal patterns (i.e., ice dancing) that either the motion cannot be well captured or the novel contents will be distorted tremendously. On the other hand, Video Imagination produces noisy future contents with aliasing artifacts from multiple warped contents from the reference frame. Li et al.
[12] also employ the flows as the medium for motion representation, but the pre-   dicted motions are too subtle and the rendered novel frames are distorted out of the space of real images. Our prediction is able to render photorealistic "ice dancing" moves of the athletes without structural distortions.
For all three datasets, the proposed ImagineFlow model finds reliable motion patterns, and remarkably preserve the structure coherence in the rendered novel frames. We show the FID score vs. MoCoGAN [26] and Li et al.
[12], our method receives the best quantitative performance. Moreover, although no adversarial training is introduced in our system, our model is still able to generate photo-realistic results with fine details.

Conclusion
In this paper, we propose a novel probabilistic framework that samples video sequences from a single image. Our model improves the CVAE framework with a bidirectional flow generator and a compositional fusion structure, thus is able to learn content-aware and structural coherent flow distributions. The involved flow-based frame synthesis module renders high-quality novel sequences and solves the rendering artifacts inherently in the warpingbased operation. We have shown that the proposed model performs well on both on synthetic and real-world videos.

Appendix: Detailed Network Architecture
The complete network consists of (a) a motion encoder q ψ (z|I 0 , I T ), (b) a bi-directional flow generator p φ (W f T , W)T b |z, I 0 ), (c) an image encoder E θ (I 0 ) and (d) an occlusion-aware synthesis module R ω (·). In the following subsections, we will specify the network architectures for each component.
Note that batch normalization is applied in the entire network. Leaky ReLU with λ = 0.2 is the activation function used in each convolution layer. But linear activation function is applied in the output layers. Input images are normalized in the range [0, 1]. We only illustrate the network for the input size of 128 × 128 and sequence length of 8.
Motion Encoder. The motion encoder uses an image volume of size N × 9 × 128 × 128 × 3 as its input. For a sample in one batch, it stacks an image sequence I T and its reference frame I 0 along the time dimension. N is the batch size and |I T | = 8. The output has two branches, one indicates the means µ and the other is log σ, as the parameters for the Gaussian posterior q ψ (z|I 0 , I T ). This network is shown in Tab. 2.
Image Encoder. The image encoder takes the reference image I 0 as the input, and outputs intermediate content features {c m } 5 m=0 through six consecutive 2D convolution layers. The highest level of the content features is c 1 , which is a N × 1 × 1 × 1 × 1024 tensor. The network specification is shown in Tab. 3.
Bi-directional Flow Generator. The bi-directional flow generator starts from sampling the motion variables z either by the reparameterization trick as z = µ + σ • , where ∼ N (0, I), or the standard Gaussian distribution z ∼ N (0, I).
The sampled motion tensor of size N × 1 × 1 × 1 × 1024 is then inputted into the flow generator. The first fusion is similar as the depthwise convolution with the convolution kernel as the content feature c 1 . The subsequent fusions are operated as: at first concatenation of the motion features and the content features along the time dimension, and then a 3D convolution layer to fuse across all dimen-sions. We use the nearest neighboring upsampling with a 3D convolution layer to deconvolute the preceding motion features. The network outputs are flow volumes of size N × 8 × 128 × 128 × 4, i.e., the number of bi-directional flows is 8. The forward and backward flows are split along the last dimension.
The network architecture of the flow generator is shown in Tab. 4, its main branch actually mirrored the structure of the motion encoder.
Occlusion-aware Image Synthesis. The structure of the occlusion-aware frame synthesis is briefly depicted in the main article. It uses the network layers up to conv4 1 of VGG-19 as the encoder. Its decoder mirrors the encoder with nearest neighbor upsampling to replace the max pooling operation. The skip connections link encoding layers convk 1, k = 1, 2, 3 to their corresponding decoding layers. The input of this network is a stacked tensor about the warped frame and its visibility map, along the channel dimension.