A Partial Regularization Method for Network Compression
  Deep Neural Networks have achieved remarkable success relying on the
developing availability of GPUs and large-scale datasets with increasing
network depth and width. However, due to the expensive computation and
intensive memory, researchers have concentrated on designing compression
methods in order to make them practical for constrained platforms. In this
paper, we propose an approach of partial regularization rather than the
original form of penalizing all parameters, which is said to be full
regularization, to conduct model compression at a higher speed. It is
reasonable and feasible according to the existence of the permutation invariant
property of neural networks. Experimental results show that as we expected, the
computational complexity is reduced by observing less running time in almost
all situations. It should be owing to the fact that partial regularization
method invovles a lower number of elements for calculation. Surprisingly, it
helps to improve some important metrics such as regression fitting results and
classification accuracy in both training and test phases on multiple datasets,
telling us that the pruned models have better performance and generalization
ability. What's more, we analyze the results and draw a conclusion that an
optimal network structure must exist and depend on the input data.
