Make $\ell_1$ Regularization Effective in Training Sparse CNN
  Compressed Sensing using $\ell_1$ regularization is among the most powerful
and popular sparsification technique in many applications, but why has it not
been used to obtain sparse deep learning model such as convolutional neural
network (CNN)? This paper is aimed to provide an answer to this question and to
show how to make it work. We first demonstrate that the commonly used
stochastic gradient decent (SGD) and variants training algorithm is not an
appropriate match with $\ell_1$ regularization and then replace it with a
different training algorithm based on a regularized dual averaging (RDA)
method. RDA was originally designed specifically for convex problem, but with
new theoretical insight and algorithmic modifications (using proper
initialization and adaptivity), we have made it an effective match with
$\ell_1$ regularization to achieve a state-of-the-art sparsity for CNN compared
to other weight pruning methods without compromising accuracy (achieving 95\%
sparsity for ResNet18 on CIFAR-10, for example).
