Exploring Structural Sparsity of Deep Networks via Inverse Scale Spaces
  The great success of deep neural networks is built upon their
over-parameterization, which smooths the optimization landscape without
degrading the generalization ability. Despite the benefits of
over-parameterization, a huge amount of parameters makes deep networks
cumbersome in daily life applications. Though techniques such as pruning and
distillation are developed, they are expensive in fully training a dense
network as backward selection methods, and there is still a void on
systematically exploring forward selection methods for learning structural
sparsity in deep networks. To fill in this gap, this paper proposes a new
approach based on differential inclusions of inverse scale spaces, which
generate a family of models from simple to complex ones along the dynamics via
coupling a pair of parameters, such that over-parameterized deep models and
their structural sparsity can be explored simultaneously. This kind of
differential inclusion scheme has a simple discretization, dubbed Deep
structure splitting Linearized Bregman Iteration (DessiLBI), whose global
convergence in learning deep networks could be established under the
Kurdyka-Lojasiewicz framework. Experimental evidence shows that our method
achieves comparable and even better performance than the competitive optimizers
in exploring the sparse structure of several widely used backbones on the
benchmark datasets. Remarkably, with early stopping, our method unveils
`winning tickets' in early epochs: the effective sparse network structures with
comparable test accuracy to fully trained over-parameterized models, that are
further transferable to similar alternative tasks. Furthermore, our method is
able to grow networks efficiently with adaptive filter configurations,
demonstrating a good performance with much less computational cost. Codes and
models can be downloaded at {https://github.com/DessiLBI2020/DessiLBI}.
