Factorized Higher-Order CNNs with an Application to Spatio-Temporal
  Emotion Estimation
  Training deep neural networks with spatio-temporal (i.e., 3D) or
multidimensional convolutions of higher-order is computationally challenging
due to millions of unknown parameters across dozens of layers. To alleviate
this, one approach is to apply low-rank tensor decompositions to convolution
kernels in order to compress the network and reduce its number of parameters.
Alternatively, new convolutional blocks, such as MobileNet, can be directly
designed for efficiency. In this paper, we unify these two approaches by
proposing a tensor factorization framework for efficient multidimensional
(separable) convolutions of higher-order. Interestingly, the proposed framework
enables a novel higher-order transduction, allowing to train a network on a
given domain (e.g., 2D images or N-dimensional data in general) and using
transduction to generalize to higher-order data such as videos (or
(N+K)-dimensional data in general), capturing for instance temporal dynamics
while preserving the learnt spatial information.
  We apply the proposed methodology, coined CP-Higher-Order Convolution
(HO-CPConv), to spatio-temporal facial emotion analysis. Most existing facial
affect models focus on static imagery and discard all temporal information.
This is due to the above-mentioned burden of training 3D convolutional nets and
the lack of large bodies of video data annotated by experts. We address both
issues with our proposed framework. Initial training is first done on static
imagery before using transduction to generalize to the temporal domain. We
demonstrate superior performance on three challenging large scale affect
estimation datasets, AffectNet, SEWA, and AFEW-VA.
