Recursive Least Squares for Training and Pruning Convolutional Neural
  Networks
  Convolutional neural networks (CNNs) have succeeded in many practical
applications. However, their high computation and storage requirements often
make them difficult to deploy on resource-constrained devices. In order to
tackle this issue, many pruning algorithms have been proposed for CNNs, but
most of them can't prune CNNs to a reasonable level. In this paper, we propose
a novel algorithm for training and pruning CNNs based on the recursive least
squares (RLS) optimization. After training a CNN for some epochs, our algorithm
combines inverse input autocorrelation matrices and weight matrices to evaluate
and prune unimportant input channels or nodes layer by layer. Then, our
algorithm will continue to train the pruned network, and won't do the next
pruning until the pruned network recovers the full performance of the old
network. Besides for CNNs, the proposed algorithm can be used for feedforward
neural networks (FNNs). Three experiments on MNIST, CIFAR-10 and SVHN datasets
show that our algorithm can achieve the more reasonable pruning and have higher
learning efficiency than other four popular pruning algorithms.
