Compression of Fully-Connected Layer in Neural Network by Kronecker
  Product
  In this paper we propose and study a technique to reduce the number of
parameters and computation time in fully-connected layers of neural networks
using Kronecker product, at a mild cost of the prediction quality. The
technique proceeds by replacing Fully-Connected layers with so-called Kronecker
Fully-Connected layers, where the weight matrices of the FC layers are
approximated by linear combinations of multiple Kronecker products of smaller
matrices. In particular, given a model trained on SVHN dataset, we are able to
construct a new KFC model with 73\% reduction in total number of parameters,
while the error only rises mildly. In contrast, using low-rank method can only
achieve 35\% reduction in total number of parameters given similar quality
degradation allowance. If we only compare the KFC layer with its counterpart
fully-connected layer, the reduction in the number of parameters exceeds 99\%.
The amount of computation is also reduced as we replace matrix product of the
large matrices in FC layers with matrix products of a few smaller matrices in
KFC layers. Further experiments on MNIST, SVHN and some Chinese Character
recognition models also demonstrate effectiveness of our technique.
