{'fraction_non_alphanumeric': 0.044597509218673906, 'fraction_numerical': 0.02292492868573019, 'mean_word_length': 4.630043086564826, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rethinking Class-Discrimination Based CNN Channel Pruning

Channel pruning has received ever-increasing focus on network compression. In particular, class-discrimination based channel pruning has made major headway, as it fits seamlessly with the classification objective of CNNs and provides good explainability. Prior works singly propose and evaluate their discriminant functions, while further study on the effectiveness of the adopted metrics is absent. To this end, we initiate the first study on the effectiveness of a broad range of discriminant functions on channel pruning. Conventional single-variate binary-class statistics like Student's T-Test are also included in our study via an intuitive generalization. The winning metric of our study has a greater ability to select informative channels over other state-of-the-art methods, which is substantiated by our qualitative and quantitative analysis. Moreover, we develop a FLOP-normalized sensitivity analysis scheme to automate the structural pruning procedure. On CIFAR-10, CIFAR-100, and ILSVRC-2012 datasets, our pruned models achieve higher accuracy with less inference cost compared to state-of-the-art results. For example, on ILSVRC-2012, our 44.3% FLOPs-pruned ResNet-50 has only a 0.3% top-1 accuracy drop, which significantly outperforms the state of the art.


Introduction
Convolutional neural networks (CNNs) have become a mainstream machine learning model for various computer vision tasks, such as image classification [17,46], image super resolution [31,49], and semantic segmentation [38,45]. To gain better prediction performance, a popular approach is to grow deeper and wider models. However, such CNNs require larger storage space and higher computational cost, making them unsuitable for edge devices like phones and embedded sensors.
Channel pruning has recently been fostered by the notion of evaluating the class-discriminative information with state-of-the-art works [55,27]. Such methodology has two unique advantages: (1) It aims to find channels that have great linear separability with respect to the labels, which fits seamlessly with the goal of classification CNNs, to transform original distribution into a linearly separable space [16]. (2) It makes the pruning process more explainable, as it directly assesses what the channels have learned and quantitates the information of the channels in rigorous math. As illustrated in Figure 1, the channel with a low class-discriminative score has overlapped 2D projection and shows little class discrepancy visually. Such redundant channel is pruned away while the one with rich classification information is kept. Figure 1: An illustration of discrimination-based channel pruning. Feature maps of two channels are picked from the second layer of LeNet-5 [30] fed with MNIST [29] input. The pruning decision is visually explainable, as shown in feature maps' 2D projection. The channel with a low discrimination score (in dashed) is discarded as it has little class discrepancy. In contrast, the channel with a higher score displays clearer class separability and is kept during the pruning process. (Best viewed in color) However, the effectiveness of the class-discriminative metrics in the prior works is not well studied. While [55] inserts a cross-entropy type loss and [27] uses a closed-form function, the metrics are proposed and evaluated singly without comparing to other existing well-defined discriminant functions, e.g., Maximum Mean Discrepancy (MMD) [11] on the channel pruning task. Moreover, well known single-variate binary-class statistics, like Student's T-Test [32], are also mathematically defined to test the significance of two class's difference, whilst no attempt has been made to generalize them for high-dimensional multi-label channel scoring. Moreover, current practices require iterative optimization steps or heavy matrix operation like matrix inversion to obtain the discrimination score, which can hinder the methods to be applied to larger networks and datasets.
We here have the first attempt to study the effectiveness over a variety of class-discriminative functions on channel pruning. The group of the interested functions includes not only high-dimensional metrics like the one proposed in [27] and MMD, but also several well-known single-variate binary-class statistics like Student's T-Test [32], for which we initiate an intuitive and lightweight approach to generalize them for the high-dimensional multi-class channel scoring. Surprisingly, we find that one of our generalized single-variate metric, generalized Symmetric Divergence (G-SD), outperforms other functions notably, while having a relatively low computation complexity. Qualitative and quantitative analyses show that G-SD largely improves the selected channels' quality over prior methods, including non-discriminative type metrics. In addition, we propose a FLOP-normalized sensitivity analysis scheme to evaluate the layer's pruning sensitivity under the same FLOPs reduction. This scheme allows us to fairly compare layers' redundancy across the model and automatically determine the pruning layers and number of pruning channels, which further enhances the pruning performance. We integrate G-SD and the FLOP-normalized sensitivity analysis into a unified pruning algorithm, via which we achieve state-of-the-art channel pruning results.
Our contributions are summarized as follows: (1) We conduct the first study on the effectiveness of a group of discriminant functions for channel pruning. The effectiveness of the winning metric, G-SD, is well explained by our qualitative and quantitative channel selection analysis, where it outperforms state-of-the-art methods with a clear margin. (2) We propose a FLOP-normalized sensitivity analysis for cross-layer pruning decision. We develop a pruning algorithm with the scheme and G-SD, which allows us to remove redundant channels without performance loss. (3) We demonstrate the advantages of our approach on three benchmark datasets (CIFAR-10, CIFAR-100, and ILSVRC-2012) with two widely used networks (VGGNet and ResNet). On ILSVRC-2012, our pruned ResNet-50 achieves 75.85% top-1 accuracy with 44.3% FLOPs reduction, surpassing the state of the art.

Related Work
Non-pruning Approaches. To reduce the computation complexity of CNNs, early works propose to factorize convolution kernels [23,28,53,47]. Jaderberg et al. [23] explore a low-rank basis of filters to represent the full convolutional kernels, while Lebedev et al. [28] decompose a wide convolutional layer into a sequence of thinner layers. In addition, weight quantization methods are studied for CNN reduction [4,5,13,44,54]. Chen et al. [4] hash weights to different groups for storage saving.
Courbariaux et al. [5] and Rastegari et al. [44] binarize the network's weights for inference speedup. Our methods can be applied directly upon these approaches to further accelerate the CNNs without specialized hardware/software.
Weight-level Pruning. Weight-level pruning aims to remove redundant weights, resulting in sparsely compressed networks [15,14,12,48,3,52]. Han et al. [14] trim weights whose magnitudes are lower than a predefined threshold. Zhang et al. [52] adopt the alternating direction method of multipliers [2] for weight pruning. However, the produced sparse networks require specialized software libraries and hardware architectures to achieve the nominal acceleration.
Channel-level Pruning. In contrast, channel-level pruning can reduce network size and accelerate inference speed without specialized tools [33,37,39,21,40,51,22,55,20,27,50]. The core idea of channel pruning is to score the importance of channels and then drop the worthless ones. The channels' importance can be assessed via their associated filters [33,37,39,20,18]. Li et al. [33] adopt 1-norm of the filters as the pruning criterion. He et al. [20] calculate the filters' geometric median and discard filters closest to it. Scoring channels' saliency based on their activations is another popular direction [21,40,22,51,55,27,36]. He et al. [21] and Luo et al. [40] remove channels that have the least impact on the reconstructed mean square error. Yu et al. [51] propose neural importance score propagation (NISP) to score the channel importance in a more general manner.
In line with our work, several discrimination-based channel scoring methods have improved the efficacy of pruning [55,27]. Zhuang et al. [55] insert discriminant losses to intermediate layers and prune channels that are less correlated to the losses after iterative steps. Kung et al. [27] achieve state-of-the-art pruning results by using a closed-form class-discriminative function, Discriminant Information (DI), for channel selection. While these studies solely focus on a single discriminant loss/metric, we provide more analysis on the effectiveness across a wide range of metrics, which is absent in the prior study. Moreover, we also pioneer to generalize conventional univariate binary-class statistics to serve for our channel importance evaluation. The winning metric in our study has much less time complexity comparing to the time-consuming loss optimization and DI's heavy matrix inversion, while achieves a noticeable improvement upon them.

Discriminant Function for Channel Pruning
Function Effectiveness Study. Although discriminant functions are all theoretically convincing, their empirical effectiveness on channel pruning remains understudied. To obtain more insights into their practical usefulness, we conduct a study over a group of discriminant functions. Our study includes high dimensional discrepancy metrics: Discriminant Information (DI) [27] and Maximum Mean Discrepancy (MMD) [11]. We also observe a group of conventional single-variate binary-class metrics which are well-defined for two-class discrimination measurement: Student's T-Test (Ttest) [32], Absolute SNR (AbsSNR) [10], Symmetric Divergence (SD) [41], and Fisher Discriminant Ratio (FDR) [43]. We provide an intuitive approach to generalize them to highdimensional multi-class channel scoring and name the generalized ones with a prefix 'G-' (G-Ttest, G-SD, .etc). Our implementation of these functions can be found in the Supplementary Material.
We study the empirical effectiveness of these functions by using them to conduct one-shot pruning for VGG-16 on CIFAR-10 and ResNet-38 on CIFAR-100, whose results are shown in Fig. 2. For each metric, we uniformly prune 10%, 20%, 30%, and 40% of channels with the lowest scores for all layers in both networks. These pruned models are then fine-tuned for 150 epochs by the SGD optimizer with Nesterov Momentum. All training hyper-parameters are the same for all metrics to allow a fair comparison. We evaluate the accuracies of the pruned models before and after the fine-tuning process, and include a random scoring method as the baseline. We find that the group of discriminant functions outperform the random baseline in nearly all pruning ratios, with and without fine-tuning. Moreover, we find a subset of leading metrics: G-SD, DI, and MMD, where G-SD has a clear improvement over the others. Without retraining, G-SD outperforms DI by 5.5% accuracy on CIFAR-10 with 40% channels removed and has an 8% accuracy gain on CIFAR-100 with 30% channels removed over MMD. With retraining, G-SD also performs the best among all other metrics on both tasks. For instance, it gains around 0.5% accuracy over MMD on CIFAR-100 when 30% channels removed. Based on such consistent winning results, we adopt G-SD as our channel selection criterion to conduct more class-discriminative pruning experiments.
denote two partitions of D based on the labels, we can then formally define Symmetric Divergence (SD) [26] of D as: where µ P and σ 2 P are the sample mean and sample variance of P, and µ Q and σ 2 Q are the corresponding statistics of Q.
SD is initially proposed to select discriminative individual features in bioinformatics feature vectors for dimension reduction and efficient classification [41]. The discriminant power of a single scalar feature is quantified by the first two terms of SD. The first term measures the divergence of two classes symmetrically. It achieves a high score for the feature whose σ 2 P or σ 2 Q is much smaller than the other, which indicates that one class of the feature has a more concentrated distribution, and thus the feature is more class distinguishable. The second term 1 awards features that have large "Signal-to-Noise Ratio (SNR)" [24] where the centroids distance (µ P − µ Q ) 2 can be viewed as "class discrimination signal" and the class variance sum σ 2 P + σ 2 Q is regarded as "class variational noise". Therefore, SD's ability to find discriminant features is mathematically supported.
To generalize such function for the channel's importance evaluation, the key is to find the corresponding statistics, µ P , σ 2 P , µ Q , and σ 2 Q for the setting of channel scoring. To achieve this, we partition the feature maps of a channel in a one-versus-rest manner. We then apply a similar statistics operation on the partitioned high-dimensional tensor to obtain their corresponding µ P , σ 2 P , µ Q , and σ 2 Q . These statistics are then plugged back to Eqn. 1 to obtain the SD score for such partition. We finally aggregate the SD score for each partition to obtain a single scalar, G-SD score, which summarizes the channel's importance. The details of such generalization can be found in Supplementary Material. The effectiveness of G-SD over state-of-the-art methods are further evidenced in our channel selection analyses shown in Fig. 3 and Fig. 4. G-SD incurs no expensive operations (e.g., matrix inversion, SVD), which makes it scalable to large networks and datasets.

FLOP-Normalized Sensitivity Analysis
Pruning Algorithm. For deep CNNs, layers are constructed heterogeneously as the number of FLOPs, the number of parameters, and the spatial size all vary across layers. Due to this, selecting pruning layers and determining number of pruning channels are crucial in the pruning procedure. A method to address the issue is to conduct the layer sensitivity analysis [33], where a two-step approach is proposed to quantitate a layer's sensitivity: (1) remove a certain ratio of redundant channels from the layer; (2) evaluate the validation accuracy of the single-layer-pruned model without retraining. Such evaluation is repeated with different pruning ratios, and a pruning ratio vs. accuracy curve can be drawn for each layer to visualize the layer's sensitivity. This inevitably makes the method labor-intensive as the pruning layers and the number of pruned channels are decided via human analysis on each layer's curve. Moreover, in step (1), removing the same ratio of channels in different layers could result in drastically different FLOPs reduction, which could make the analysis unfair.
To remedy these issues, we propose FLOP-normalized sensitivity analysis in Algorithm 1 to automate the structural pruning process with attention to the normalization of FLOP reduction. The effectiveness of our approach is shown in Fig. 5. Let us denote an L-layer model by M, we aim to automatically determine the structure of the pruned model M . We denote a single-layer-pruned model by M n , which indicates that n channels are removed from layer l in M. For each layer, we remove one single channel and calculate the resultant FLOPs reduction of the network by Eqn. 2. Such single-channel FLOPs loss at the l-th layer is denoted as FLOSS .
The number of pruning channels from layer l, n l , can thus be determined in Eqn. 3, where FLOSS max = max ∈[1:L] FLOSS and α is a scaling factor to control the pruning rate. Via the normalization of FLOSS max /FLOSS , removing n l channels from layer l would have the same overall network's FLOPs reduction. The derived n l will be further rounded to integer.
We then follow the two-step anlaysis scheme: (1) individually prune n l channels with the least G-SD from layer l to obtain M n ; (2) test the validation accuracy of M n , Acc l . Acc l effectively quantifies layer l's sensitivity at the same FLOP's reduction level. Finally, the pruned network M is obtained by choosing k layers with the highest Acc l and pruned associated n l channels from these layers. After structural pruning, M will be fine-tuned to recover the accuracy loss. By iterating this pruning-retraining process, we can obtain a compact model with high performance. Test validation accuracy of M n , Acc l 6 end 7 Select layers with top-k Acc, {l1, · · · , l k }, and their number of pruning channels{n1, · · · , n k } 8 Obtain Pruned Structure: for i ∈ [1 : k] do 9 Permanently prune ni channels in layer li from M 10 end 11 return M Handling Skip Connection. For CNNs like ResNet, the input of a residual block is added to its output, which requires the number of channels to be the same for the block's input and output. We adopt the idea of using a channel selection layer at the beginning of a residual block [21,37]. Such layer samples a subset of input channels for the first convolution layer of the block based on their discriminative scores, which further reduces the block's FLOPs and parameters without overheads.
Training. Experiments are implemented in TensorFlow [1] and are carried out with NVIDIA Tesla P100 GPUs. We use the SGD optimizer with Nesterov Momentum [42] with a momentum of 0.9. The weight decay factor is set to 0.0001. We use the standard data augmentation scheme proposed in [17] for all datasets. On CIFAR, we fine-tune 200 epochs with a minibatch size of 128. The learning rate is initialized at 0.01 and divided by 5 at epoch 80 and 160. On ILSVRC-2012, we use a batch size of 256 to fine-tune VGG-16/ResNet-50 with 30/120 epochs. The learning rate is started at 0.0003 and multiplied by 0.4 at 40% and 80% of the total number of epochs. During the evaluation, the images are resized and center-cropped to the size of 224×224.  Pruning. On CIFAR, all training data are feed-forwarded for G-SD channel scoring. On ILSVRC-2012, we randomly sample 10,000 training images for G-SD calculation in each pruning iteration. We choose k to be around one-third of the total number of layers and α ∈ [2, 4] for Algorithm 1.

CIFAR Experiments
We examine the performance of our scheme on CIFAR with five popular networks, VGG-16,19 and ResNet-56,110,164. We use the same VGGNet structures as [33,37] with slim fully connected layers. As shown in Table 1, we achieve state-of-the-art results on both datasets for all networks.
On CIFAR-10, our VGG-16 has 5× storage saving and 3× inference speedup, while having 0.26% higher accuracy compared to GAL [36]. Our G-SD-A of ResNet-56 outperforms DCP [55] with 0.46% accuracy gain at the same FLOPs reduction ratio. Compared to FPGM [20] on ResNet-110, G-SD-B achieves 94.04% accuracy, which is 0.3% higher while having 8% less FLOPs. On CIFAR-100, our pruned VGG-19 has an accuracy gain of around 0.2% comparing to SLIM [37] with over 20% less FLOPs and around 10% fewer parameters. On ResNet-164, our pruned model outperforms DI [27] by 1.29% in accuracy with 5% less FLOPs and 16% fewer parameters. These results demonstrate that G-SD can produce more compact models with better performance compared to prior methods.

ILSVRC-2012 Experiments
We further conduct experiments on VGG-16 and ResNet-50 for large-scale ImageNet classification tasks. The results are summarized in Table 2 and we achieve state-of-the-art performance, again.
On VGG-16, our G-SD-A increases the performance in both top-1 and top-5 accuracy by 0.58% and 0.56% with 2.3× acceleration from baseline. This result shows that the proposed metric has a strong regularization effect on deep CNNs, which helps remove deleterious nodes and improves generalization performance. Compared to FBS [9], our 3.3× accelerated G-SD-B owns a higher speedup and compression ratio, while exceeds by 0.50% in top-5 accuracy. On ResNet-50, G-SD-A  increases top-1 and top-5 accuracy both by around 0.05% at 30% of FLOPs reduction, which is rarely reported on ILSVRC-2012. As ResNet-50 has much less redundancy than VGG-16, such accuracy gain is even more promising. Moreover, G-SD-B achieves a 44.3% of FLOPs reduction with just 0.30% top-1 accuracy loss, which surpasses all previous methods. Compared to GAL [36], G-SD-B has a higher speedup ratio while achieves a nearly 4% top-1 accuracy gain from the same baseline.

Ablation Study
Our noticeable pruning performance is directly related to the use of G-SD and FLOP-normalized sensitivity analysis. We conduct the following ablation study to further illustrate their effectiveness in comparison with state-of-the-art methods.

Qualitative Channel Selection
Single Channel Visualization. We visualize the high G-SD and the low G-SD channels at the second layer of the first block in ResNet-50 with ILSVRC-2012 inputs. As shown in Col. 1-3 of Fig. 3, we observe that the channel with low G-SD value (Col. 2) tends to generate indistinguishable responses for different classes, while the high G-SD channel (Col. 3) well preserves the image patterns, which is more informative for classification. These suggest that G-SD is promising for channel selection.
Multiple Channels Visualization. To further visually compare the effectiveness of our G-SD with other channel selection criteria [33,37,20,27], we score the channel activations and compute an average response over the top-10 channels with highest scores for each metric as shown in Col. 4-8 in Fig. 3. We observe that the average responses picked by G-SD (Col. 8) tends to display more class information than the ones from the other metrics (Col. 4-7). For example, in the first row, G-SD clearly separates the ostrich from the background grass, while others tend to generate mixed responses. Moreover, G-SD is the only one that preserves both the vertical nail and its long diagonal shadow in the fourth row. Such visual comparison speaks well on behalf of G-SD effectiveness.

Quantitative Channel Selection
We demonstrate the advantage of G-SD over state-of-the-art methods [33,37,27] quantitatively in Fig. 4, with VGG-16 on ILSVRC-2012 and ResNet-110 on CIFAR-10. Similar to the effectiveness study in 3.1, we use each metric to uniformly remove r% of the lowest-scored channels in all layers and include random scoring as the baseline. The accuracy of the pruned networks is evaluated without retraining to serve as the indicator of channels' quality and the delegate of fine-tuned accuracy [14,19]. r is set to be: 5,10,15,20,25,30,35,40. We observe that G-SD wins against other methods in nearly all FLOPs reduction ratios on both networks and datasets. This analysis well explains G-SD's effectiveness by comparing the selected channel's quality directly and eliminating the effect of different retraining settings (optimizer, number of epochs, etc.) across literature.  [33], batch-norm scaling factor [37], filter's geometric median [20], DI [27], and G-SD.

FLOP-Normalized Sensitivity Analysis
We compare our FLOP-normalized sensitivity analysis with the conventional sensitivity analysis proposed in [33] with ResNet-56 on CIFAR-10. Starting with a 31% FLOPs-pruned ResNet-56, we conduct five pruning-retraining iterations with G-SD by two methods. As shown in Fig. 5, we find a clear advantage for our methods over Li's in every iteration. Moreover, the final network obtained by Li's method suffers an accuracy drop of 0.3% while our model does not have accuracy loss.

Conclusion
In this paper, we conduct the first study on the empirical effectiveness of a broad range of classdiscriminative functions in channel pruning. We provide an intuitive generalization to enable singlevariate binary-class functions for channel scoring. We find that the winning metric in our study, generalized Symmetric Divergence (G-SD), selects channels with more information over prior arts via qualitative and quantitative analysis. In addition, we develop a FLOP-normalized sensitivity analysis scheme to enhance our pruning performance. Experimental results on three benchmark datasets demonstrate the advantage of our mechanism over state-of-the-art methods. In the future, we will explore the relationship between the channel's class-discrimination and its redundancy in a more theoretical manner to help design better metrics.

Supplementary Materials
We organize our supplementary material as follows. In Section 7, we present the definitions of the studied discriminant functions (except SD) as well as the details of our generalization for singlevariate binary-class functions to high-dimensional multi-class channel scoring. In Section 8, we discuss the detailed structures of the pruned VGG-16 and ResNet-50 models on ILSVRC-2012 by generalized Symmetric Divergence (G-SD). We discuss an empirical study on the time complexity of two discriminant functions, G-SD and Discriminant Information (DI) [27], for channel scoring in Section 9. In Section 10, we provide more visualizations of our qualitative channel selection analysis.

Single-Variate Binary-Class Metrics
We adopt the same notation of the n-sample binary-class single-variate dataset as in the main paper. Our implementation of Absolute SNR (AbsSNR) [10], Fisher Discriminant Ratio (FDR) [43] and Student's T-Test (Ttest) [32] are: where |P| and |Q| denote the number of samples in P and Q.

Generalization
We observe that the computation of these three metrics along with SD mainly requires four statistics: µ P , σ 2 P , µ Q , and σ 2 Q . We here use SD as an example to demonstrate our way to find these statistics for CNN's channel scoring, while the other three metrics can follow the similar way.
Notation. For an N -sample C-class dataset, we denote the feature maps of a CNN channel as Method. We first partition F as F c and By this partition, we can find the corresponding statistics of F in a two-class manner. Note each f i in F c is a 2D feature map with W × H activations, and thus there are |F c | × W × H activations in F c in total. We then define two statistics operators g mean and g var on F c , which return the mean and variance over these |F c | × W × H activations. We can thus obtain the statistics µ c = g mean (F c ) and σ 2 c = g var (F c ) for F c . The counterparts of F −c , µ −c and σ 2 −c , can be computed in the same way. The SD score for the two-class partition, (F c , F −c ), can thus be derived as: SD(F c , F −c ) captures the discriminativenesss of class c relative to the other classes in F. In general, we want to select channels that distinguish all classes well on average. Thus, under the C-class setting, the generalized Symmetric Divergence (G-SD) of F is formally defined as:

High-Dimensional Metrics
We present the implementation of Discriminant Information (DI) [27] and Maximum Mean Discrepancy (MMD) [11] here. We use the same notation as in Section 7.2 and regard f i as a flattened 1D vector.
DI. We first define two scatter matrices,S and S B : Nc Nc i=1 f i represents the centroids of the all data and class c (c ∈ [1 : C]), respectively. DI is then defined as: where ρ is a small ridge factor to ensure invertibility. In practice, we set ρ = 0.0001.

MMD.
We define the rbf kernel as: The two-class MMD on a (F c , F −c ) partition can thus be defined as: following the similar one-versus rest setting, the MMD score of a channel can be defined as: In practice, we set σ = 1 for the rbf kernel.

Pruned Models Details
We show the detailed structures of our pruned VGG-16 and ResNet-50 on ILSVRC-2012 in Table 3 and Table 4. Both pruned models are named as G-SD-B in the experimental results section. On VGG-16, we find that our algorithm prunes more channels in shallow layers, which indicates the shallow layers are less sensitive to pruning compared to the deep ones under the same overall FLOPs reduction. On ResNet-50, our model preserves more channels at the starting blocks of each stage. These blocks are the ones where the stride-2 downsamplings happen, which means that preserving information for the process of resolution reduction would be crucial to maintain the network's performance.

More Visualizations
We repeat the analysis in Section 5.1 with additional images from different classes (not shown in the main paper). Visually speaking, we again observe that our G-SD selects more class-discriminative channels compared to state-of-the-art metrics.   Table 4: Detailed structure of the reported ResNet-50 pruned model, G-SD-B (Top1-Accuracy: 75.85%). We choose not to prune Conv1 and fc1 for the ease of implementation.  1-norm [33], batch-norm scaling factor [37], filter's geometric median [20], DI [27], and G-SD.