DHP: Differentiable Meta Pruning via HyperNetworks
  Network pruning has been the driving force for the acceleration of neural
networks and the alleviation of model storage/transmission burden. With the
advent of AutoML and neural architecture search (NAS), pruning has become
topical with automatic mechanism and searching based architecture optimization.
Yet, current automatic designs rely on either reinforcement learning or
evolutionary algorithm. Due to the non-differentiability of those algorithms,
the pruning algorithm needs a long searching stage before reaching the
convergence.
  To circumvent this problem, this paper introduces a differentiable pruning
method via hypernetworks for automatic network pruning. The specifically
designed hypernetworks take latent vectors as input and generate the weight
parameters of the backbone network. The latent vectors control the output
channels of the convolutional layers in the backbone network and act as a
handle for the pruning of the layers. By enforcing $\ell_1$ sparsity
regularization to the latent vectors and utilizing proximal gradient solver,
sparse latent vectors can be obtained. Passing the sparsified latent vectors
through the hypernetworks, the corresponding slices of the generated weight
parameters can be removed, achieving the effect of network pruning. The latent
vectors of all the layers are pruned together, resulting in an automatic layer
configuration. Extensive experiments are conducted on various networks for
image classification, single image super-resolution, and denoising. And the
experimental results validate the proposed method.
