Understanding and Training Deep Diagonal Circulant Neural Networks
  In this paper, we study deep diagonal circulant neural networks, that is deep
neural networks in which weight matrices are the product of diagonal and
circulant ones. Besides making a theoretical analysis of their expressivity, we
introduced principled techniques for training these models: we devise an
initialization scheme and proposed a smart use of non-linearity functions in
order to train deep diagonal circulant networks. Furthermore, we show that
these networks outperform recently introduced deep networks with other types of
structured layers. We conduct a thorough experimental study to compare the
performance of deep diagonal circulant networks with state of the art models
based on structured matrices and with dense models. We show that our models
achieve better accuracy than other structured approaches while required 2x
fewer weights as the next best approach. Finally we train deep diagonal
circulant networks to build a compact and accurate models on a real world video
classification dataset with over 3.8 million training examples.
