Feature Flow Regularization: Improving Structured Sparsity in Deep
  Neural Networks
  Pruning is a model compression method that removes redundant parameters in
deep neural networks (DNNs) while maintaining accuracy. Most available filter
pruning methods require complex treatments such as iterative pruning, features
statistics/ranking, or additional optimization designs in the training process.
In this paper, we propose a simple and effective regularization strategy from a
new perspective of evolution of features, which we call feature flow
regularization (FFR), for improving structured sparsity and filter pruning in
DNNs. Specifically, FFR imposes controls on the gradient and curvature of
feature flow along the neural network, which implicitly increases the sparsity
of the parameters. The principle behind FFR is that coherent and smooth
evolution of features will lead to an efficient network that avoids redundant
parameters. The high structured sparsity obtained from FFR enables us to prune
filters effectively. Experiments with VGGNets, ResNets on CIFAR-10/100, and
Tiny ImageNet datasets demonstrate that FFR can significantly improve both
unstructured and structured sparsity. Our pruning results in terms of reduction
of parameters and FLOPs are comparable to or even better than those of
state-of-the-art pruning methods.
