{'fraction_non_alphanumeric': 0.047802911228046197, 'fraction_numerical': 0.01957903369097246, 'mean_word_length': 4.600267891312668, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Higher-order Recurrent Neural Networks (RNNs) are effective for long-term forecasting since such architectures can model higher-order correlations and long-term dynamics more effectively. However, higher-order models are expensive and require exponentially more parameters and operations compared with their first-order counterparts. This problem is particularly pronounced in multidimensional data such as videos. To address this issue, we propose Convolutional Tensor-Train Decomposition (CTTD), a novel tensor decomposition with convolutional operations. With CTTD, we construct Convolutional Tensor-Train LSTM (Conv-TT-LSTM) to capture higher-order space-time correlations in videos. We demonstrate that the proposed model outperforms the conventional (first-order) Convolutional LSTM (ConvLSTM) as well as the state-of-the-art ConvLSTM-based approaches in pixel-level video prediction tasks on Moving-MNIST and KTH action datasets, but with much fewer parameters.


Introduction
Video understanding is a challenging problem, as it requires a model to learn complex representation of spatial and temporal dynamics. The problem arises in a wide range of applications, including autonomous driving, robot control , and other visual perception tasks like action recognition or object tracking (Alahi et al., 2016 Transformer are common choices to learn temporal dynamics (Hochreiter & Schmidhuber, 1997;Vaswani et al., 2017). These models are extended to learn spatio-temporal data by incorporating convolution operations (Convolutional LSTM (Xingjian et al., 2015)) or attention in spatiotemporal volumes (Video Transformer (Weissenborn et al., 2019)). While Transformer directly maps input and output sequences using attention mechanism, RNNs encodes the sequential dependencies by the interactions of consecutive two steps (first-order Markovian model).
Higher-order RNNs (Soltani & Jiang, 2016;Yu et al., 2017) are their higher-order Markovian generalization that explicitly characterize long-term temporal dependencies. These models are effective in long-term forecasting problems by incorporating higher-order states (longer history) in RNNs: at each time step, a model learns longer-term correlations from multiple past steps. Although Transformer can also be used to capture long-term dependencies, it is hard to design a model especially for multi-dimensional data. Higher-order RNNs learn underlying sequential correlations which makes Markovian models more natural for sequence forecasting problems.
The transition dynamics in higher-order RNNs is naturally characterized by higher-order tensors (instead of transition matrices as in first-order models) with the order equal to the number of past steps for prediction. Therefore, these models typically require exponentially more parameters and operations than their first-order counterparts, making the learning harder and prone to overfitting.
A principled approach to address the curse of dimensionality is tensor decomposition, where a higher-order tensor is compressed into smaller core tensors (Anandkumar et al., 2014). Tensor representations are powerful since they retain rich expressivity even with a small number of parameters. The synergy of tensor method and higher-order RNNs has led to compressed higher-order tensor RNNs (Yu et al., 2017), where the transition tensor is factorized by Tensor-Train Decomposition (TTD) (Oseledets, 2011). However, this approach is not directly compatible to learning in videos since classic decompositions do not preserve the spatio-temporal structure. To extend higher-order RNNs to video applications, we propose a novel Convolutional Tensor-Train Decomposition (CTTD), which is capable of factorizing an intractably large convolutional operator into a set of smaller components.
Many recent video models use Convolutional LSTM (Con-vLSTM) as a basic block in RNNs (Xingjian et al., 2015), where spatio-temporal information is encoded as a tensor explicitly in each cell. Like the traditional RNNs, the model is in nature a first-order Markovian model. Therefore, the model cannot easily capture higher-order temporal correlations. Most of first-order ConvLSTM-based approaches focus on next or first few frames prediction (Lotter et al., 2016;Finn et al., 2016;Byeon et al., 2018).
With CTTD, we propose a higher-order generalization to ConvLSTM, Convolutional Tensor-Train LSTM (Conv-TT-LSTM), that is able to learn long-term spatio-temporal structure in videos. We show experimentally that our proposed Conv-TT-LSTM outperforms the plain ConvLSTM and other ConvLSTM-based models augmented by other techniques (Finn et al., 2016;Wang et al., 2017;2018a) and achieve the state-of-the-art in long-term video prediction.
• We introduce a novel Convolutional Tensor-Train Decomposition (CTTD) that is capable to factorize a large convolutional kernel into a chain of smaller tensors.
• We integrate CTTD into convolutional LSTM (ConvL-STM) and propose Conv-TT-LSTM, which learns higherorder spatio-temporal correlations in video sequence with a small number of model parameters.
• We address the problem of gradient instability in training higher-order tensor RNNs, by carefully designing learning scheduling and gradient clipping.
• In the experiments, we show our proposed Conv-TT-LSTM consistently produces sharper prediction over a long period of time than first-order ConvLSTM for both Moving-MNIST-2 and KTH action datasets. In addition, Conv-TT-LSTM outperforms the state-of-the-art PredRNN++ (Wang et al., 2018a) in LPIPS (Zhang et al., 2018) by 0.050 on the Moving-MNIST-2 and 0.071 on the KTH action dataset, with 5.6 times fewer parameters.
In summary, we obtain best of both worlds: capturing higherorder spatio-temporal correlations and model compression.
Tensor methods have been used to compresse recurrent networks in diverse aspects: (1) Sutskever et al. (2011) introduces multiplicative RNNs, where the weights tensor for inputs-states interactions are factorized into three matrices.
(2) Yu et al. (2017) uses tensor-train decomposition to constrain the complexity of higher-order LSTM, where each step is computed based on the outer product of previous steps. While this work only considers vector input at each step, we generalize their approach to higher-order ConvL-STM to cope with video input using our proposed Convolutional Tensor-Train Decomposition.
(3) Yang et al. (2017) has proposed to compress both inputs-states and states-states matrices within each cell with Tensor-Train decomposition by reshaping the matrices into tensors, and showed improvement in video classification. Different from Yang et al.
(2017), we aim to compress higher-order ConvLSTM (instead of first-order fully-connected LSTM). Furthermore, we propose Convolutional Tensor-Train decomposition to deal with a dense prediction problem, which requires the decomposition to preserve spatial structure after compression.
Higher-order RNNs Zhang et al. (2016) and Soltani & Jiang (2016) introduce connections cross multiple previous time steps to better learn long-term dynamics. These models require excessively more parameters than first-order RNNs. Yu et al. (2017) addressed the problem by compressing the model parameters using Tensor-Train Decomposition.
Video Prediction. Prior works on video prediction have focused on several directions: predicting short-term video (Lotter et al., 2016;Byeon et al., 2018), decomposing motion and contents (Finn et al., 2016;Villegas et al., 2017;Denton et al., 2017;Hsieh et al., 2018), improving the objective function (Mathieu et al., 2015), and handling diversity of the future (Denton & Fergus, 2018;Babaeizadeh et al., 2017;Lee et al., 2018). Many of these works use ConvL-STMas a base module, which deploys 2D convolutional operations in LSTM to efficiently exploit spatio-temporal information. Some works modified the standard ConvL-STM to better capture spatio-temporal correlations (Wang et al., 2017;2018a). Wang et al. (2018b) integrated 3D convolutions into ConvLSTM. In addition, current cell states are combined with its historical records using self-attention to efficiently recall the history information. Byeon et al. (2018) applied ConvLSTM in all possible directions to cap-ture full contexts in video. They also demonstrated strong performance using a deep ConvLSTM network as a baseline, which is used as the base architecture in the present paper.

Tensor Diagrams and Tensor Operations
In this section, we introduce the basic concepts of tensor diagrams and tensor operations. These concepts will serve as building blocks for higher-order models (in particular, tensor-train models) in the next section.
Tensor Diagrams. Following the convention in quantum physics (Cichocki et al., 2016), we introduce tensor diagrams in Figure 1, graphical representations of multidimensional arrays. In these diagrams, an array is represented as a vertex (node), whose order is denoted by the number of edges (links) connected to the node, where each edge corresponding to a mode (axis). The dimension of a mode is denoted by a number with the corresponding edge.  Tensor Operations. With these diagrams, we can represent the operations between (higher-order) tensors graphically. In Figure 2, we illustrate three tensor operations commonly used in neural networks, namely tensor contraction, tensor convolution and batch product. In these figures, an operation is denoted by connecting the edges from both input tensors, where the type of operation is distinguished by the shape/color of the edges: solid line stands for tensor contraction or batch product, and dotted line represents tensor convolution. Notice that a tensor operation can be arbitrarily complicated by linking multiple edges from multiple tensors simultaneously, see Figure 4 for an example.

Tensor-Trains and Sequence Modeling
The goal of tensor decomposition is to represent a higherorder tensor in a set of smaller and lower-order core tensors, with fewer parameters while preserving essential information. Yu et al. (2017) used tensor-train decomposition (Oseledets, 2011) to reduce both parameters and computations in higher-order recurrent model. We will review the standard model in the first part of this section. However, the approach by Yu et al. (2017) only considers recurrent models with vector inputs. Since spatial structure is not preserved by standard tensor-train decomposition, their approach cannot be tensor-train cannot be extended to cope with video/image inputs directly. In the second part, we propose a novel Convolutional Tensor-Train De-composition (CTTD). With CTTD, a large convolutional kernel is factorized into a chain of smaller kernels. We show that such decomposition can significantly reduce both parameters and operations of higher-order spatio-temporal recurrent models.
Throughout this section, We will use tensor diagrams to illustrate the operations in all models. The exact mathematical expression for these models are included in Appendix A.
Standard Tensor-Train Decomposition. Given an morder tensor T ∈ R I1×···×Im with I l as the dimension of its l-th mode, a standard tensor-train decomposition (TTD) factorizes the tensor T into a set of m core tensors {T l } m l=1 with T l ∈ R I l ×R l ×R l+1 , illustrated in Figure 3a. In TTD, the ranks {R l } m−1 l=1 control the number of parameters in the tensor-train format, and the original tensor T of size ( the complexity of TTD only grows linearly with the order m (assuming R l 's are constants). Therefore, TTD is commonly used to approximate higher-order tensors with fewer parameters.
As shown in Figure 3a, the tensor diagram for TTD takes a "train" shape (and therefore its name). The sequential structure in TTD makes it particularly suitable for sequence modeling. Consider a higher-order model that predicts a scalar output y ∈ R based on the outer product of a sequence This model is intractable in practice since the number of parameters in T ∈ R I1×···Im (and therefore computational complexity) grows exponentially with the order m. Now suppose T takes a tensor-train format as in Figure 3a, Equation (1) can be efficiently evaluated sequentially from left to right as in Figure 3c: x 1 first interacts with T 1 , and the intermediate result interacts with T 2 and x 2 simultaneously, and so on. Notice that the higher-order tensor T is never reconstructed in the process, therefore both space and computational complexities grow linearly (not exponentially as in Equation (1)) with the order m.
Convolutional Tensor-Train Decomposition. A convolutional layer (in neural networks) is typically parameterized by a tensor T ∈ R K×R1×Rm+1 , where K is the kernel size, and R 1 , R m+1 are the number of input and output channels. Suppose the kernel size K takes the form K = ( m l=1 K l ) − m + 1, we propose convolutional tensortrain decomposition (CTTD) that factorizes T into a set of m core tensors {T l } m l=1 with T l ∈ R K l ×R l ×R l−1 as in Figure 3b, and denote T = CTTD({T l } m l=1 ). As in TTD, the ranks of CTTD {R l } m l=1 control the complexity of the convolutional tensor-train format, and the number of parameters reduces from KR 1 R m+1 to m l=1 K l R l R l+1 . Similar to standard TTD, its convolutional counterpart can Tensor convolution. T :,j,k,m,n = T 1 :,j,k * T 2 :,m,n For all examples, the inputs are two third-order tensors T 1 ∈ R I×J×K and T 2 ∈ R I×M ×N , and the operations are on mode I. (a) In tensor contraction, the edges for operation must share the same dimension, and will vanish after contraction; (b) In tensor convolution, a new edge will emerge after the operation. Different from contraction, the dimensions for the operating edges may be different depending on the type of convolution. (c) In batch product, a new edge will also emerge after the operation. Different from convolution, the dimensions for all operating edges must be the same.
(d) Convolutional tensor-train for spatial-temporal modeling. Notice that in convolutional TTD, the edges {R l } m l=1 for contraction is not directly linked to any vertex in order to avoid an extra dimension in T l . Both models can be evaluated by interacting the tensors from left to right to obtain the final output and take Figure 3d for an example: U 1 is first interacted with T 1 , and the resulted tensor is further interacted with T 2 and U 2 . The process repeats until all tensors are merged into one. be used to compress higher-order spatio-temporal recurrent models with convolutional operations. Consider a model that predicts a feature V ∈ R I m ×Rm+1 based on a sequence of input features {U l } m l=1 with U l ∈ R I l ×R l (where I l is the feature length, and R l is the number of channels in U l ), where W l is the corresponding weights tensor for U l , which itself takes takes a convolutional tensor-train format in Figure 3b. The structure also admits an efficient algorithm: U 1 is first interacted with T 1 , and the result is further interacted with T 2 and U 2 simultaneously. The process is illustrated in Figure 3d. Again, the original weights W l 's are never reconstructed in such a process. The overall operations of a third-order CTTD (Equation (2)) for two-dimensional features are illustrated in Figure 4a. In this paper, we denote Equation (2)

Convolutional Tensor-Train LSTM
Convolutional LSTM (ConvLSTM) is a basic building block for most recent video forecasting models (Xingjian et al., 2015), where the spatial information is encoded explicitly as tensors in the LSTM cells. In a ConvLSTM network, each cell is a first-order Markov model, where the hidden state is updated based on its previous step. In this section, we propose Convolutional Tensor-Train LSTM (Conv-TT-LSTM), where ConvLSTM is integrated with convolutional tensortrain to model higher-order spatio-temporal correlations explicitly. The proposed model is illustrated in Figure 4.
Notations. In this section, * is overloaded to denote convolution between higher-order tensors. For instance, given a 4-th order weights tensor W ∈ R K×K×S×C and a 3-rd order input tensor X ∈ R H×W ×S , Y = W * X computes a 3-rd output tensor Y ∈ R H×W ×T as Y :,:,c = s=1 W :,:,s,c * X :,:,s . The symbol • is used to denote element-wise product between two tensors, and σ represents a function that performs  Figure 4. Illustration of (a) convolutional tensor-train decomposition (Equation (2)) and (b) convolutional tensor-train LSTM (Equation (6)). The original frames to Conv-TT-LSTM are first grouped by a sliding window before fed into the convolutional tensor-train.
In Conv-TT-LSTM, the CTTD is used with two minor modifications: (1) The edges for convolutions are extended to two-dimensional, i.e. I l and K l in Figure 3d are tuples of two dimensions I l = (H, W ) and K l = (k, k); (2) The indices are named reversely such that they reflect the number of steps from the current output, e.g. U (3) = U 1 and U (1) = U 3 in Figure 3d.
element-wise (nonlinear) transformation on a tensor.

Convolutional Long Short-Term Memory (ConvLSTM).
Xingjian et al. (2015) extended fully-connected LSTM to Convolutional LSTM to model spatio-temporal structures within each recurrent unit, where all features are encoded as third-order tensors with dimensions (height H × width W × channels C) and operations are replaced by convolutions between tensors. In each cell, the parameters are characterized by two 4-th order tensors W ∈ R K×K×S×4C and T ∈ R K×K×C×4C , where K is the kernel size for convolutions, S, C are the numbers of channels of the inputs X (t) ∈ R H×W ×S and hidden states H (t) ∈ R H×W ×C . At each time step t, a cell updates its hidden states H (t) based on the previous step H (t−1) and the current input X (t) . [ where σ(·) applies sigmoid on the input gate I (t) , forget gate F (t) , output gate O (t) , and tanh(·) on memory cell C (t) . Notice that C (t) , I (t) , F (t) , O (t) ∈ R H×W ×C are all 3-rd order tensors.
We introduce a higher-order recurrent unit to capture multisteps spatio-temporal correlations in ConvLSTM, where the hidden state H (t) is updated based on its (1) Since CCT({T (l) } m l=1 , ·) takes a series of m tensors as inputs, the first step in (5) (6). In this work, we devise a sliding window strategy to compute Equation (5). With this strategy, a sliding subset of {H (l) } are concatenated intoĤ (t,o) , which is then transformed into an inputH (t,o) to the convolutional tensor-train (See Figure 4b). Concretely, the Conv-TT-LSTM model computes eachH (t,o) by the following equation: The n previous states {H (l) } n l=1 are first concatenated (over time axis) into m tensors {Ĥ (t,o) } m o=1 by a sliding window, each of which has sizeĤ (t,o) ∈ R H×W ×D×C (with D = n − m + 1) and thereafter mapped toH (t,o) ∈ R H×W ×R by 3D-convolution with a kernel K (l) ∈ R k×k×D×R .
We note that a fixing window strategy is also feasible here, where all Ul's are concatenated into a single tensor, which repeatedly used in every single input. The discussion of fixed window and the empirical comparison against sliding window are shown in Appendix B.

Experiments
We first evaluate our approach on the synthetic Moving-MNIST-2 dataset (Srivastava et al., 2015). In addition, we use KTH human action dataset (Laptev et al., 2004) to test the performance of our proposed models in more realistic scenario.
Model Architecture. All experiments use a stack of 12layers of ConvLSTM or Conv-TT-LSTM with 32 channels for the first and last 3 layers, and 48 channels for the 6 layers in the middle. A convolutional layer is applied on top of all recurrent layers to compute the predicted frames, followed by an extra sigmoid layer for colored videos. Following Byeon et al. (2018), two skip connections performing concatenation over channels are added between (3, 9) and (6, 12) layers. Illustration of the network architecture is included in Appendix D.All parameters are initialized by Xavier's normalized initializer (Glorot & Bengio, 2010) and initial states in ConvLSTM or Conv-TT-LSTM are initialized as zeros.
Evaluation Metrics. We use two traditional metrics MSE (or PSNR) and SSIM (Wang et al., 2004), and a recently proposed deep-learning based metric LPIPS (Zhang et al., 2018), which measures the similarity between deep features. Since MSE (or PSNR) is based on pixel-wise difference, it favors vague and blurry predictions, which is not a proper measurement of perceptual similarity. While SSIM was originally proposed to address the problem, Zhang et al. (2018) shows that their proposed LPIPS metric aligns better to human perception.
Learning Strategy. All models are trained with ADAM optimizer (Kingma & Ba, 2014) with L 1 +L 2 loss. Learning rate decay and scheduled sampling (Bengio et al., 2015) are used to ease training. Scheduled sampling is started once the model does not improve in 20 epochs (in term of validation loss), and the sampling ratio is decreased linearly from 1 until it reaches zero (by 2 × 10 −4 each epoch for Moving-MNIST-2 and 5 × 10 −4 for KTH). Learning rate decay is further activated if the loss does not drop in 20 epochs, and the rate is decreased exponentially by 0.98 every 5 epochs.
Hyper-parameters Selection. We perform a wide range of hyper-parameters search for Conv-TT-LSTM to identify the best model. The full list of search values are summarized in Table 4 (Appendix C).The initial learning rate of 10 −3 is found for baseline ConvLSTM and 10 −4 for our proposed Conv-TT-LSTM. We found that the Conv-TT-LSTM models suffer from exploding gradients when learning rate is high, therefore we also explore various gradient clipping values and select 1 for all models. All other hyper-parameters are selected using the best validation performance.

Moving-MNIST-2 Dataset
The Moving-MNIST-2 dataset is generated by moving two digits of size 28 × 28 in MNIST dataset within a 64 × 64 black canvas. These digits are placed at a random initial location, and move with constant velocity in the canvas and bounce when they reach the boundary. Following Wang et al. (2018a), we generate 10,000 videos for training, 3,000 for validation, and 5,000 for test with default parameters in the generator 5 . All our models are trained to predict 10 frames given 10 input frames.
Multi-Steps Prediction. Table 1 reports the average statistics for 10 and 30 frames prediction, and Figure 5 shows comparison of per-frame statistics for PredRNN++ model, ConvLSTM baseline and our proposed Conv-TT-LSTM model. (1) Our Conv-TT-LSTM model consistently outperform the 12-layer ConvLSTM baseline for both 10 and 30 frames prediction with fewer parameters; (2) The Conv-TT-LSTM outperform previous approaches in terms of SSIM and LPIPS (especially on 30 frames prediction), with less than one fifth of the model parameters.
We reproduce the PredRNN++ model (Wang et al., 2018a) from their source code 2 , and we find that (1) The Pre-dRNN++ model tends to output vague and blurry results in long-term prediction (especially after 20 steps).
(2) and our Conv-TT-LSTMs are able to produce sharp and realistic digits over all steps. An example of comparison for different models is shown in Figure 6. The visualization is consistent with the results in Table 1 and Figure 5.
Ablation Studies on CTTD The core component in our Conv-TT-LSTM is a higher-order convolutional TTD. In the following ablation studies, we present the necessity of (1) higher-order model and (2) convolutional operations in the decomposition for capturing long-term spatio-temporal information. We compare the performance of two ablated models against our Conv-TT-LSTM in Table 3. The single  Table 3. Ablation studies of higher-order Conv-TT-LSTM model.
In these experiments, we evaluate the necessity of (1) higher-order model and (2)  order means that the higher-order model is replaced to a firstorder model (Tensor order=1). By replacing 3 × 3 filters to 1×1 in CTTD, the effect of convolutions in CTTD is demonstrated compared to the standard TTD. The results show that the ablated models at best achieve similar performance of ConvLSTM baseline, which shows both higher-order model and convolutional operations are necessary for long-term video prediction.

KTH Action Dataset
KTH action dataset (Laptev et al., 2004) contains videos of 25 individuals performing 6 types of actions on a simple background. Our experimental setup follows Wang et al.
(2018a), which uses persons 1-16 for training and 17-25 for testing, and each frame is resized to 128 × 128 pixels. All our models are trained to predict 10 frames given 10 input frames. During training, we randomly select 20 contiguous frames from the training videos as a sample and group every 10,000 samples into one epoch to apply the learning strategy as explained at the beginning of this section.
Results. In Table 2, we report the evaluation on both 20 and 40 frames prediction.
(1) Our models are consistently better than the ConvLSTM baseline for both 20 and 40 frames prediction.
(2) While our proposed Conv-TT-LSTMs achieve lower SSIM value compared to the state-of-the-art models in 20 frames prediction, they outperform all previous models in LPIPS for both 20 and 40 frames prediction. An example of the predictions by the baseline and Conv-TT-LSTMs is shown in Figure 6. . 20 frames prediction on KTH given 10 input frames. Every 2 frames are shown. The first frames (t = 1 and 11) are animations.
To view the animation, Adobe reader is required.

Conclusion
In this paper, we proposed Convolutional Tensor-Train Decomposition (CTTD) to factorize a large convolutional kernel into a set of smaller core tensors. We applied CTTD to efficiently construct convolutional tensor-train LSTM (Conv-TT-LSTM), a higher-order recurrent neural network, that is capable to effectively capture long-term spatio-temporal correlations. We demonstrated the capacity of our proposed Conv-TT-LSTM on video prediction, and showed our model outperforms standard ConvLSTM and produce better results compared to other state-of-the-art models with fewer parameters. Utilizing the proposed model for high-resolution videos is still challenging due to gradient vanishing or explosion. Future direction will include investigating other training strategies or model designs to ease the training.  Appendix: Convolutional Tensor-Train LSTM for Spatio-temporal Learning

A. Mathematical Expressions for Tensor-Trains
In Section 4, we introduced the concepts of standard and convolutional tensor-trains and their applications in sequence modeling in tensor diagrams. In this section, we present their equivalent forms in mathematical expressions.

A.1. Standard Tensor-Train Decomposition and Temporal Modeling
Notice that tensor diagram of standard tensor-train in Figure 3a can be expressed as and the higher-order regressive model in Equation 1 can be rewritten as Then the sequential algorithm illustrated in Figure 3c is equivalent to where the vectors {v l } m l=1 (with v l ∈ R R l ) are intermediate steps, with initial input v 0 = 1, and final output y = v m . We will prove the correctness of this algorithm by induction.

Proof of Equation 10
We denote the standard tensor-train decomposition in Equation 8 as T = TTD({T l } m l=1 ), then Equation 9 can be rewritten as Equation 11 since R 0 = 1 and v where R 0 = 1, v 0 1 = 1 and the sequential algorithm in Equation 10 is performed at Equation 13.

A.2. Convolutional Tensor-Train Decomposition and Spatio-Temporal Modeling
Notice that the convolutional tensor-train in Figure 3b can be expressed as T 1 :,r1,r2 * · · · * T m :,rm,rm+1 And the sequential algorithm illustrated in Figure 3d can be equivalently stated as T l :,r l ,r l+1 * V l :,r l + U l :,r l where {V l } m l=1 (with V l ∈ R H×W ×R l ) are intermediate results, and V 1 ∈ R H×W ×Rm is initialized as all zeros and final prediction is returned as V = V m+1 .

Proof of Equation 16
We denote the convolutional tensor-train decomposition in Equation 15 as T = CTTD(T l ) m l=1 , then Equation 16 can be rewritten as Equation 17 since V 1 is an all zeros tensor.
which completes the induction.

B.1. Model Details
Details of the Architecture. All experiments use a stack of 12-layers of ConvLSTM or Conv-TT-LSTM with 32 channels for the first and last 3 layers, and 48 channels for the 6 layers in the middle. A convolutional layer is applied on top of all LSTM layers to compute the predicted frames, followed by an optional sigmoid function (In the experiments, we add sigmoid for KTH dataset but not for Moving-MNIST-2). Additionally, two skip connections performing concatenation over channels are added between (3, 9) and (6, 12) layers as is shown in Figure 8.  (2) Our current learning strategy is optimal in the search space;

Hyper-parameters Selection
(3) The sliding window strategy outperforms the fixed window one under the optimal experimental setting.

B.3. Additional Experimental Results
Per-frame metrics for KTH action dataset. The per-frame metrics are illustrated in Figure 10. . 20 frames prediction on KTH given 10 input frames. Every 2 frames are shown. The first frames (t = 1 and 11) are animations.
To view the animation, Adobe reader is required.