A Feature-map Discriminant Perspective for Pruning Deep Neural Networks
  Network pruning has become the de facto tool to accelerate deep neural
networks for mobile and edge applications. Recently, feature-map discriminant
based channel pruning has shown promising results, as it aligns well with the
CNN objective of differentiating multiple classes and offers better
interpretability of the pruning decision. However, existing discriminant-based
methods are challenged by computation inefficiency, as there is a lack of
theoretical guidance on quantifying the feature-map discriminant power. In this
paper, we present a new mathematical formulation to accurately and efficiently
quantify the feature-map discriminativeness, which gives rise to a novel
criterion,Discriminant Information(DI). We analyze the theoretical property of
DI, specifically the non-decreasing property, that makes DI a valid selection
criterion. DI-based pruning removes channels with minimum influence to DI
value, as they contain little information regarding to the discriminant power.
The versatility of DI criterion also enables an intra-layer mixed precision
quantization to further compress the network. Moreover, we propose a DI-based
greedy pruning algorithm and structure distillation technique to automatically
decide the pruned structure that satisfies certain resource budget, which is a
common requirement in reality. Extensive experiments demonstratethe
effectiveness of our method: our pruned ResNet50 on ImageNet achieves 44% FLOPs
reduction without any Top-1 accuracy loss compared to unpruned model
