On the Compression of Neural Networks Using $\ell_0$-Norm Regularization
  and Weight Pruning
  Despite the growing availability of high-capacity computational platforms,
implementation complexity still has been a great concern for the real-world
deployment of neural networks. This concern is not exclusively due to the huge
costs of state-of-the-art network architectures, but also due to the recent
push towards edge intelligence and the use of neural networks in embedded
applications. In this context, network compression techniques have been gaining
interest due to their ability for reducing deployment costs while keeping
inference accuracy at satisfactory levels. The present paper is dedicated to
the development of a novel compression scheme for neural networks. To this end,
a new $\ell_0$-norm-based regularization approach is firstly developed, which
is capable of inducing strong sparseness in the network during training. Then,
targeting the smaller weights of the trained network with pruning techniques,
smaller yet highly effective networks can be obtained. The proposed compression
scheme also involves the use of $\ell_2$-norm regularization to avoid
overfitting as well as fine tuning to improve the performance of the pruned
network. Experimental results are presented aiming to show the effectiveness of
the proposed scheme as well as to make comparisons with competing approaches.
