{'fraction_non_alphanumeric': 0.03575514874141877, 'fraction_numerical': 0.010655034324942792, 'mean_word_length': 4.516765285996056, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Disentangling Video with Independent Prediction

We propose an unsupervised variational model for disentangling video into independent factors, i.e. each factor's future can be predicted from its past without considering the others. We show that our approach often learns factors which are interpretable as objects in a scene.


Introduction
Deep neural networks have delivered impressive performance on a range of perceptual tasks, but their distributed representation is difficult to interpret and poses a challenge for problems involving reasoning. Motivated by this, the deep learning community has recently explored methods [1][2][3][4][6][7][8]12,[15][16][17]20,21] for learning distributed representations which are disentangled, i.e. a unit (or small group) within the latent feature vector are exclusively responsible for capturing distinct concepts within the input signal. This work proposes such an approach for the video domain, where the temporal structure of the signal is leveraged to automatically separate the input into factors that vary independently of one another over time. Our results demonstrate that these factors correspond to distinct objects within the video, thus providing a natural representation for making predictions about future motions of the objects and subsequent high-level reasoning tasks.
Related work. [22] leverage structure at different time-scales to factor signals into independent components. Our approach can handle multiple factors at the same time-scale, instead relying on prediction as the factoring mechanism. Outside of the video domain, [1] proposed to disentangle factors that tend to change independently and sparsely in real-world inputs, while preserving information about them. [15] learned to disentangle factors of variation in synthetic images using weak supervision, and [21] extended the method to be fully unsupervised. Similar to our work, both [3] and [4] propose unsupervised schemes for disentangling video, the latter using a variational approach. However, ours differs in that it uncovers general factors, rather than specific ones like identity and pose or static/dynamic features as these approaches do.

Generative model and inference
The intuition behind our model is simple: in any real-world scene, most objects do not physically interact with one another, so their motion can be modeled independently. To find a representation of videos with these same independences, we introduce an approach based on a temporal version [14] of the variational auto-encoder [10], shown in Figure 1. In this model, each video X comprises a sequence of frames x 1 ...x n . Z t = z 1 t ...z k t represents all the latent factors at time t, where each factor z i t is represented by a vector. Each of these factors evolve independently from one another and combine to produce the observation x t for timestep t. Instead of directly maximizing the likelihood p(X; θ), we use variational inference [5] to optimize the evidence lower bound (ELBO), that is θ * , φ * = arg max θ,φ L(θ, φ; X) : distribution parameterization parameter sharing?  [14]. For more details, please see Appendix A.
For a derivation, see Appendix B. This lower bound naturally splits into two factors. The first, p θ (X|Z), is the log-likelihood of the data X under our model when sampling from the approximate posterior, that is the "reconstruction" of X from Z. The second factor is the KL divergence between the learned prior p θ (Z) and the approximate posterior q φ (Z|X). It represents how far the predictions given by p are from the inferred values given by q; it is the prediction error in the latent space. Our goal is to optimize the space of Z to make both reconstruction and prediction possible.
For the variational approximation, we choose q φ (Z|X) = n t=1 q(Z t |Z t−1 , x t ; φ) which, analogous to a Kalman filter, considers only the past state and current observation. This approximation marginalizes over the future, in that q(Z t |Z t−1 , x t ) must encode sufficient information to allow correct predictions of the future as well as fit the prior and the current observation. This allows us to do inference on a single frame and ensure our representation retains as much information about the future as possible. If we wish to use our learned representation for a downstream task, we may discard the generative model and use the inference network alone. This inference network can provide a factorized representation given a single image or use a sequence of images to produce increasingly tight estimates of the latent variables.
Neural network parameterization. We choose all of the distributions in our model to be Gaussian with diagonal covariance. To allow our model to scale to complex nonlinear observations and dynamics, we parameterize each distribution with a neural network. Table 1 describes each of these parameterizations. As each distribution is diagonal Gaussian, each network produces two outputs, corresponding to the mean and the variance of each distribution.
Training. This model can be thought of as a series of variational autoencoders with a learned prior, and the training procedure is largely similar to [10]. At each timestep t in a sequence, we compute the prior p θ (Z t |Z t−1 ) over the latent space, using N (0, I) for t = 0. We infer the approximate posterior q φ (Z t |Z t−1 , x t ) by observing x t and compute the KL divergence D KL (q||p). We then sample Z t ∼ q φ (Z t |Z t−1 , x t ) using the reparameterization trick and compute log p θ (x t |Z t ). At the end of a sequence we update our parameters θ, φ by backprop to maximize the ELBO (defined above).  Figure 2: Heatmap of correlation between latent variables (shown as blocks) at time t (y-axis) vs. at time t + 1 (x-axis) on the 5th Avenue dataset. The block-diagonal structure exhibited here reflects that z i t+1 largely depends on its own previous state z i t rather than on the other latent factors. The entangled model does not appear to have structure in its latent space. These plots were generated using models with two-layer transition networks; with deeper transitions there is almost zero linear dependence between any pair of latent units.

Experiments
Datasets. We apply this model to two video datasets: the widely-used moving MNIST [19] and a new dataset of real-world videos of 5th Avenue recorded from above, which was collected by the authors. More details about these datasets, including a sample frame from 5th Avenue, can be found in Appendix C.
Baseline. In each of these evaluations we compare with a model which is identical to the factored model except that its transition function p(Z t |Z t−1 ) is not factored (referred to as the entangled model in our experiments). It can be viewed as a special case of our factored model which has a single high-dimensional latent factor, and its latent space has the same total dimension as the corresponding factored model in each experiment. This comparison is intended to be as tight as possible, with any differences between the factored model and the baseline coming exclusively from the factorization in the latent space.

Evaluation
Lower bound. We compare the variational lower bound achieved by our factorized model with that of a non-factorized but otherwise identical model. These experiments reveal the price in terms of data fidelity that we pay for representing the data as independently-changing factors. Table 2 shows that the factored model achieves a lower bound on par with the entangled model.
Correlation structure. By plotting the correlation between samples from the approximate posterior over the latent variables at time t and at time t + 1, we may observe whether our model has been able to learn a representation for which z i t+1 really does only depend on z i t and not on the other latent factors. Note that this only captures the linear dependence of these variables; this analysis helps to illustrate the structure of our latent space, but should not be considered definitive. Figure 2 shows that each latent factor is much more correlated with its own previous state than with other latent factors.
Approximate mutual information. We use Kraskov's method for estimating mutual information [13] to approximate I(z i t+1 ; z i t ) and compare it to I(z i t+1 ; z j =i t ). A latent factor z i t should be much more informative about its own future z i t+1 than a different factor z j =i t is. The results, shown in Table  2, reveal that the evolution of each factor in the factored models depends almost exclusively on their past.
For the entangled models, which do not have separate factors, there is no a priori subdivision into high-and low-mutual-information segments of units. The reported scores were generated by creating 20 random partitionings of the latent units into k factors, then reporting the mutual information numbers for the partitioning that had the greatest difference between same-factor and cross-factor information.  The entangled models show much more cross-factor information than the factored models in our tests on moving MNIST, where we use two latent factors. However, as the number of factors increases, the average information between a pair of factors naturally diminishes. As a result on 5th Avenue, where we subdivide the latent units into eight factors, the entangled model shows cross-information almost as low as the factored model.

Independent generations.
Finally, we evaluate qualitatively the representations learned by our model. We infer the approximate posterior q(Z|X), then set all of the latent variables fixed at their t = 2 values given by q(Z 2 |x 1,2 ). We then produce a sequence of generations by picking a single variable z i to vary, then for each timestep t = 2...n drawing a sample from p(x|z 1 2 ...z i−1 2 , z i t , z i+1 2 ...z k 2 ) (note the single bolded factor z i t varying with t). That is, we hold all but one of the latent variables fixed and allow the single one to vary with the posterior. This allows us to see exactly what that single variable represents in this video. The images generated by this process are shown in Figure 3.

Discussion
By taking advantage of the structure present in video, our model can pull apart latent factors which change independently and produce a representation composed of semantically meaningful variables. The approach is conceptually simple and based on the insight that if two objects do not interact, they can be predicted independently. In the future we hope to apply a richer family of approximations to scale to more complex data.
where z_dim is the number of latent factors times the dimensionality of each factor. Each layer is followed by a ReLU activation. We use a fixed variance in our Normal observation model of 0.25 for moving MNIST and 0.05 for 5th Avenue. This variance hyperparameter may be tuned to balance the tradeoff between fitting the predictions and making tight reconstructions.

Transition network
Each latent factor has its own transition function p θ (z i t |z i t−1 ). Each of these has the following form: Linear (latent_dim -> 128) Linear (128 -> 128) Linear (128 -> 128) Linear (128 -> 128) Linear (128 -> latent_dim * 2) where latent_dim is the dimensionality of a single latent factor and each layer is followed by a SELU activation [11]. The latent_dim * 2 output is for the mean and (diagonal) variance vectors for the Normal distribution. The variance is produced as log σ 2 for numerical reasons.

Appendix B: Deriving the ELBO
For this section the factored form of the transitions is not relevant. As such our development here will use the more general non-factored form, and we can substitute in our factored special case later. For simplicity of notation we will use p(·) in place of p θ (·). Likewise we use q(·) in place of q φ (·) to represent our variational approximation function with parameters φ.
We begin with the form of our latent-variable generative model. Since the series Z is Markov, We introduce our variational auxiliary functions: We may now convert this integral to an expectation with respect to q(Z|X) = q(Z 1 |X)q(Z 2 |Z 1 , X)...q(Z tmax |Z tmax−1 , X): By Jensen's inequality, Realizing that the expectations E log q p are KL divergences gives us an objective we can optimize: This lower bound lies below the true log-probability by an additive term of D KL (q(Z|X)||p(Z|X)) [10]. As the variational approximation q(Z|X) improves (i.e., approaches the true posterior), this lower bound approaches the true log-likelihood of the data.
This bound would hold for any function p θ (Z t |Z t−1 ). Our model factors this general transition function: which corresponds to a hidden Markov model with multiple Markov chains running in parallel in the latent space.

Appendix C: Details on datasets
Moving MNIST This dataset consists of two digits from the MNIST dataset bouncing in a 64x64 pixel frame. Each digit is on a separate plane of the input (i.e., one is red and the other is green). The digits have randomized starting location and velocity vector for each sequence, but their motion is deterministic over the course of the sequence and the digits do not interact. 5th Avenue The 5th Avenue dataset has greater complexity in its visuals and its dynamics than moving MNIST, but was designed to be simple enough to model with some fidelity using contemporary techniques. It consists of around 20 hours of video sampled at 2 frames per second. The videos were recorded from the 5th floor of a building overlooking 5th Avenue in Manhattan and show the busy street scene below including pedestrians and passing cars. Each video was recorded with a fixed camera position; between videos the camera position is nearly the same but may vary slightly. The data includes global variations such as time of day and weather. It was recorded on an iPhone 7 at 1080p resolution, though in our experiments we resize it to 64x64. A representative example image is shown in Figure  4.