Temporal Factorization of 3D Convolutional Kernels
  3D convolutional neural networks are difficult to train because they are
parameter-expensive and data-hungry. To solve these problems we propose a
simple technique for learning 3D convolutional kernels efficiently requiring
less training data. We achieve this by factorizing the 3D kernel along the
temporal dimension, reducing the number of parameters and making training from
data more efficient. Additionally we introduce a novel dataset called
Video-MNIST to demonstrate the performance of our method. Our method
significantly outperforms the conventional 3D convolution in the low data
regime (1 to 5 videos per class). Finally, our model achieves competitive
results in the high data regime (>10 videos per class) using up to 45% fewer
parameters.
