Growing Efficient Deep Networks by Structured Continuous Sparsification
  We develop an approach to training deep networks while dynamically adjusting
their architecture, driven by a principled combination of accuracy and sparsity
objectives. Unlike conventional pruning approaches, our method adopts a gradual
continuous relaxation of discrete network structure optimization and then
samples sparse subnetworks, enabling efficient deep networks to be trained in a
growing and pruning manner. Extensive experiments across CIFAR-10, ImageNet,
PASCAL VOC, and Penn Treebank, with convolutional models for image
classification and semantic segmentation, and recurrent models for language
modeling, show that our training scheme yields efficient networks that are
smaller and more accurate than those produced by competing pruning methods.
