Compression and Interpretability of Deep Neural Networks via Tucker
  Tensor Layer: From First Principles to Tensor Valued Back-Propagation
  This work aims to help resolve the two main stumbling blocks in the
application of Deep Neural Networks (DNNs), that is, the exceedingly large
number of trainable parameters and their physical interpretability. This is
achieved through a tensor valued approach, based on the proposed Tucker Tensor
Layer (TTL), as an alternative to the dense weight-matrices of DNNs. This
allows us to treat the weight-matrices of general DNNs as a matrix unfolding of
a higher order weight-tensor. By virtue of the compression properties of tensor
decompositions, this enables us to introduce a novel and efficient framework
for exploiting the multi-way nature of the weight-tensor in order to
dramatically reduce the number of DNN parameters. We also derive the tensor
valued back-propagation algorithm within the TTL framework, by extending the
notion of matrix derivatives to tensors. In this way, the physical
interpretability of the Tucker decomposition is exploited to gain physical
insights into the NN training, through the process of computing gradients with
respect to each factor matrix. The proposed framework is validated on both
synthetic data, and the benchmark datasets MNIST, Fashion-MNIST, and CIFAR-10.
Overall, through the ability to provide the relative importance of each data
feature in training, the TTL back-propagation is shown to help mitigate the
"black-box" nature inherent to NNs. Experiments also illustrate that the TTL
achieves a 66.63-fold compression on MNIST and Fashion-MNIST, while, by
simplifying the VGG-16 network, it achieves a 10\% speed up in training time,
at a comparable performance.
