Tensor Regression Networks with various Low-Rank Tensor Approximations
  Tensor regression networks achieve high compression rate of neural networks
while having slight impact on performances. They do so by imposing low tensor
rank structure on the weight matrices of fully connected layers. In recent
years, tensor regression networks have been investigated from the perspective
of their compressive power, however, the regularization effect of enforcing
low-rank tensor structure has not been investigated enough. We study tensor
regression networks using various low-rank tensor approximations, aiming to
compare the compressive and regularization power of different low-rank
constraints. We evaluate the compressive and regularization performances of the
proposed model with both deep and shallow convolutional neural networks. The
outcome of our experiment suggests the superiority of Global Average Pooling
Layer over Tensor Regression Layer when applied to deep convolutional neural
network with CIFAR-10 dataset. On the contrary, shallow convolutional neural
networks with tensor regression layer and dropout achieved lower test error
than both Global Average Pooling and fully-connected layer with dropout
function when trained with a small number of samples.
