{'fraction_non_alphanumeric': 0.039435190160606606, 'fraction_numerical': 0.01680697355065974, 'mean_word_length': 4.66723058704111, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Network Pruning using Adaptive Exemplar Filters

Popular network pruning algorithms reduce redundant information by optimizing hand-crafted models, and may cause suboptimal performance and long time in selecting filters. We innovatively introduce adaptive exemplar filters to simplify the algorithm design, resulting in an automatic and efficient pruning approach called EPruner. Inspired by the face recognition community, we use a message passing algorithm Affinity Propagation on the weight matrices to obtain an adaptive number of exemplars, which then act as the preserved filters. EPruner breaks the dependency on the training data in determining the"important"filters and allows the CPU implementation in seconds, an order of magnitude faster than GPU based SOTAs. Moreover, we show that the weights of exemplars provide a better initialization for the fine-tuning. On VGGNet-16, EPruner achieves a 76.34%-FLOPs reduction by removing 88.80% parameters, with 0.06% accuracy improvement on CIFAR-10. In ResNet-152, EPruner achieves a 65.12%-FLOPs reduction by removing 64.18% parameters, with only 0.71% top-5 accuracy loss on ILSVRC-2012. Our code can be available at https://github.com/lmbxmu/EPruner.

Based on the filter preserving policy, we empirically classify the filter pruning approaches into three categories: (1) Rule-based pruning decides the pruned network architecture by hand-crafted designation and the pruned network usually inherits the most important filter weights either measured by an intrinsic property of the pre-trained model, e.g., the 1 -norm [26], or through iterative refinement on training data [19], [27]. Typically, the designated architecture is sub-optimal. Moreover, it usually performs layer-wise fine-tuning/optimization to recover the accuracy, which is computationally intensive.
(2) Regularization-driven pruning retrains networks with handcrafted constraints, such as sparsity [20], [23], [28] and budget awareness [29]. The trained filters below a given threshold are removed and the weights of preserved filters are inherited by the pruned network. However, retraining the model is expensive, and the introduced hyper-parameters also require manual analysis. (3) Architecture-search-driven pruning focuses on searching for a better architecture, typically through heuristic-based policies, such as evolutionary algorithm [24] or the artificial bee colony algorithm [30]. The filter weights of these methods are randomly sampled for the follow-up finetuning. However, the architecture search is data-dependent thus also computationally intensive, as validated in Tab. VI of the experiments. Besides, the search results are not deterministic. Overall, existing methods still heavily depend on hand-crafted rules from humans, and/or involve time-consuming retraining/search. How to achieve automatic and efficient model compression still remains an open problem.
Thus, existing methods suffer either great cost on human labor or large time complexity in pruning. In this paper, we propose a novel filter pruning method, termed EPruner, which selects the most important filters (exemplars), to solve the above problems. We argue that the removal of informational redundancy is an underlying theme behind all these three types of approaches. As stressed in network interpretation [31]- [33], the filters, even within the same layer, have different impacts. This indicates that different filters have different properties and there exist some exemplars which contribute more in the network inference. Thus, the challenge lies in figuring out the exemplar number without human involvement and identifying which filter could be the exemplar such that the efficient end-to-end fine-tuning can be conducted. Inspired by the face recognition community, we use a graph message-passing algorithm Affinity Propagation [34] to make our approach adaptive to the pre-trained CNN in both finding the number of exemplars and identifying more informative exemplars.
Specifically, as shown in Fig. 1, we treat each filter as a data point, and connect them with weighted edges. Via message passing among the filters, Affinity Propagation could select the representative exemplars from all the filters. From a network interpretation perspective [31]- [33], this can be treated as selecting the most impacting filters from a filter pool. While the weight matrices in different layers may have different numbers and variances, Affinity Propagation is able to obtain an adaptive number of exemplars, with a nonlearning hyper-parameter controlling the compression strength.
To model the similarity between each pair of filters, it is critical to choose a proper distance metric. However, we found the proposed approach works surprisingly well even with the naive negative Euclidean distance, over-performing state-ofthe-art pruning algorithms [19], [23]- [25], [27], [30], [35] with less time consumption in determining the important filters. We argue that the innovative introduction of adaptive exemplar filters greatly simplifies the problem structure. Another counter-intuitive finding is related with the observation [36] that randomly initialized weights can perform better than inheriting the most important filters. In this paper, we find that proper filter weight initialization based on the exemplars could significantly outperform randomly initialized filter weights, which thus breaks the observation in [36]. As a summary, our contributions are three-fold: • Our EPruner is able to figure out an adaptive number of exemplars and identify more informative exemplars in a unified framework. Thus, the network pruning can be efficiently implemented via an end-to-end fine-tuning. • With a higher reduction of model complexity, EPruner lessens the involvement of human labor in the pruning. Besides, it also leads to less time consumption in important filter selection and deterministic pruning results. • We also provide extensive experiments on VGGNet [1], GoogLeNet [37] and ResNets [38] to demonstrate the efficacy of EPruner in reducing the model complexity and better performance in comparison with several state of the arts. In the following, we will first discuss the related work in Sec. II. Then, we elaborate our EPruner for network pruning in Sec. III. In Sec. IV, the experimental results are provided and analyzed. Lastly, we conclude this paper in Sec. V.

II. RELATED WORK
Unstructured weight pruning removes the individual neurons in the filter or the connection between fully-connected layers. The pioneering work [15] utilizes the second derivatives to balance the training loss and model complexity. Han et al. [39] proposed to recursively prune the low-weight connections and retrain the remaining subnetwork with the 2 normalization. Dynamic network surgery [16] performs weight pruning and splicing. The pruning is made to compress the network model and the slicing is used to enable connection recovery. Aghasi et al. [40] removed connections by solving a convex optimization program. In [13], the compression is formulated as the constrained Bayesian optimization solved by the annealing strategy. The lottery ticket hypothesis [14] randomly initializes a dense network and trains it from scratch. The subnet with high-weight values are extracted, and retrained with the initial weight values of the dense model. Unstructured weight pruning results in an irregular sparsity, which requires customized hardware and software to support the practical speedup. In contrast, structured filter pruning is especially advantageous by removing the entire filters and the corresponding channels in the next layer directly. It has no extra requirements for the inference engine and thus can be easily deployed.
To this end, rule-based pruning requires human experts to specify the pruned architecture. Hu et al. [18] observed the large output of zero activations in the network and the filter weights with a lower percentage of zero outputs are inherited. The 1 -norm pruning [26] assumes filters with large norms are more informative, and thus the corresponding weights are inherited. Lin et al. [25] preserved the filter weights with highrank feature map outputs. Another direction in filter pruning formulates the pruning as an optimization problem using the statistics information from the next layer [19] or a linear least square to reconstruct the outputs [27], and the optimized weights are forwarded for fine-tuning.
Regularization-driven pruning performs model complexity reduction by retraining the network with hand-crafted con-straints. Liu et al. [20] and Zhao et al. [22] imposed a sparsity constraint on the scaling factor of the batch normalization layer and the weights with higher scaling factors are preserved for the fine-tuning. Huang et al. [35] and Lin et al. [23] introduced a sparse soft mask on the feature map outputs or the filters. The filters with non-zero masks are preserved and the corresponding trained weights are preserved. Lemaire et al. [29] proposed a knowledge distillation loss function combined with a budget-constrained sparsity loss to train the network and guarantee the neuron budget at the same time.
Architecture-search-driven pruning tries to search for the optimal pruned architecture. In [24], a large auxiliary Prun-ingNet has to be trained in advance to evaluate the performance of each potential pruned architecture derived from the evolutionary procedure. In [30], artificial bee colony algorithm is applied to search for the pruned architecture and the accuracy is regarded as the fitness of each architecture. In light of the recent work [36] which claims that the essence of network pruning lies in finding the optimal pruned architecture rather than selecting the most important filter weights, the initializations of these methods are randomly sampled from the Gaussian distribution [24] or the pre-trained model [30] for fine-tuning.
Compared with the rule-based pruning, the novelty of our EPruner is two-fold: First, the pruned network architecture and filter selection are automatic and adaptive, which lessens the human involvement. Second, the fine-tuning is implemented in an efficient end-to-end manner rather than the inefficient layerwise manner. Compared with regularization-driven pruning, the novelty of our EPruner lies in no hand-crafted constraints and training from scratch. Compared with the architecturesearch-driven pruning, the novelty of our Epruner falls into its adaptivity to finding optimal pruned architecture without computationally intensive search progress and the pruning result is deterministic.

A. Problem Definition
Given a pre-trained CNN model F with a total of L convolutional layers, its filter weights can be represented as a set of 4-D tensors W = {w k } L k=1 . The k-th layer parameters of w k have the shape of c k × c k−1 × h k × w k , where c k represents the number of filters, c k−1 represents the number of input channels of each filter, and h k and w k represent the height and width of each filter. As can be seen, the number of channels, i.e., c k−1 , is equal to the number of filters in the (k − 1)-th layer.
For the ease of the following context, we rewrite the parameter tensor of w k as a 2-D data matrice with the shape of c k ×(c k−1 ·h k ·w k ). And then, we append the biases of the filters to w k , thus its dimension becomes c k ×(c k−1 ·h k ·w k +1). Without loss of generality, we denote the i-th filter as w ki .
The goal of filter pruning is to obtain a compressed representation of the parametersW = {w k } L k=1 and remove the filters of informational redundancy. Eachw k has a smaller shape

Notation
Meaning w k Filters in the k-th layer w ki The i-th filter in the k-th layer w k The exemplar filters in the k-th layer w ki The i-th exemplar filter in the k-th layer How well the filter w kj is suited to be the exemplar of the filter w ki r k (i, j) The "responsibility" message in the Affinity Propagation a k (i, j) The "availability" message in the Affinity Propagation works follow: (1) Human designated pruned architecture and inheriting the most important filter weights via rules [19], [25].
(2) Training from scratch with hand-crafted constraints [23], [35]. (3) Searching optimal pruned architecture via heuristicbased search algorithm [24], [30]. However, these methods suffer either human effort, or heavy computation as discussed in Sec. I. We propose EPruner towards solving the above problems as stated in Sec. I. The goal of our EPruner is to selectc k high-quality exemplar filters for the k-th layer, denoted asw k from w k by the affinity propagation [34]. The exemplarsw k (k = 1, 2, ..., L) has the shape ofc k × (c k−1 · h k · w k + 1) andc k makes up the pruned network architecture which will be initialized for the end-to-end fine-tuning. Our method is directly applied on top of the CNN weights, thus the specific value ofc k is adaptive to the input CNNs. Besides, it requires neither human designation, nor search progress in our EPruner.
We summarize the main notations used in this paper in Tab. I, more of which will be detailed in the following contents.

B. The Proposed EPruner
Affinity propagation [34] was originally proposed to select exemplars for the data points with different properties. Early works on the network interpretation [31]- [33] reveal that the filters, even within the same layer, have different impacts to the network. This indicates that different filters have different properties and there exist some exemplars which contribute more to the network.
As illustrated in Fig. 1, we propose to regard each filter w ki as a high-dimensional data point by reformatting it in a vector form, i.e., w ki ∈ R c k−1 ·h k ·w k . For any two filters w ki and w kj , affinity propagation takes as input their similarity graph s k (i, j) which reflects how well the filter w kj is suited to be the exemplar of the filter w ki . We found our method can well perform with the naive negative Euclidean distance as: When i = j, it indicates the suitedness of filter w i to be the exemplar of itself (self-similarity). Following [34], it can be defined as: where median(·) returns the median value of the input. Larger s k (i, i) leads to more exemplar filters, which however returns less complexity reduction. Using the median value of the whole weight in the k-th layer (w k ) in Eq. 2 would result in a moderate number of exemplars. To solve, we reformulate Eq. 2 as follows: where β is a pre-given hyper-parameter. Eq. 3 differs from Eq. 2 in two-fold: First, the median value is obtained upon the i-th filter w ki rather than the whole weight w k . Thus, the similarity s k (i, i) can be more adaptive to the filter w ki . Second, the introduced β provides an adjustable reduction of model complexity. As shown in Sec. IV-F, large β leads to a high complexity reduction, and vice versa.
Besides the similarity, there are also two kinds of messages passing between filters, i.e., "responsibility" and "availability", to decide which filters are exemplars, and for every other filter, which exemplar it belongs to.
The "responsibility" r k (i, j) indicates how well the filter w kj is suited to serve as the exemplar of the filter w ki by considering other potential exemplars for filter w ki . The updating of r(i, j) follows: where a(i, j) is the "availability" below and initialized to zero. Initially, r(i, j) is set to s(i, i) minus the largest of the similarities between filter w ki and other filters. Later, if one filter is assigned to other exemplars, its availability is smaller than zero per Eq. (6) below, which further decreases the effectiveness of s(i, j ) in Eq. (4), thus w kj is removed from the exemplar candidates.
For i = j, the "self-responsibility" is given as: which is set to s(i, i) minus the largest of the similarities between filter w ki and other filters. It reflects possibility that filter w ki can be an exemplar by considering how ill-suited it is to be assigned to another exemplar. As for the "availability", we first give its updating rule as: The availability a(i, j) is set as r(j, j) plus the sum of other responsibilities that filter w kj receives from others. The max() excludes the negative responsibilities since we only need to focus on the good filters (positive responsibilities). The r(j, j) < 0 denotes that filter w kj is more appropriate to belong to another exemplar rather than being an exemplar itself. It can be seen that the availability of filter w kj as an exemplar increases if some other filters have positive responsibilities for w kj . Thus, the "availability" a k (i, j) reflects how appropriate filter w ki chooses filter w kj as its exemplar by considering the support from other filters that filter w kj should be an exemplar. Lastly, the min() limits the influence of strong positive responsibilities, such that the total sum cannot go above zero.
For i = j, the "self-availability" is defined as: which reflects suitedness that filter w ki is an exemplar, based on the positive responsibilities from other filters. The updating of "responsibility" and "availability" is iterative. To avoid the numerical oscillations, we consider the weighted sum for each message at the t-th updating stage : where 0 ≤ λ ≤ 1 is a weighted factor and is set to 0.5 in our experiments.
After a fixed number of iterations (200 in our experiments), filter w ki selects as its exemplar another filter w kj that satisfies: When i = j, filter w ki selects itself as the exemplar. All selected filters make up the exemplarsw k . Thus, the number of exemplars, i.e.,c k , is self-adaptive without the human designation. In Sec. IV-F, we demonstrate that the exemplar selection can be efficiently implemented on a single CPU, leading to a magnitude-order reduction of time consumption.

C. Weight Initialization
The pruned network architecture consists of the number of exemplar filters in each layer, i.e.,c i . The fine-tuning is required to recover the accuracy of pruned network such that it would keep a better or at least comparable performance against the pre-trained model. Thus, the key now falls in how to feed a good weight initialization to the pruned network architecture for the follow-up fine-tuning. To that effect, we consider four different weight initialization scenarios: (1) Exemplar Weights. The weights of exemplar filters are regarded as the initial weights. To ensure the dimension consistency between the exemplar filtersw k with the shape of c k × (c k−1 · h k · w k + 1) and pruned network parametersw k with the shape ofc k × (c k−1 · h k · w k + 1), the corresponding channels in the weights of exemplars are removed directly.
(2) Random Projection. Instead of directly removing the corresponding channels in the weights of exemplars, we apply the sparse random projection [41] to reduce the dimension of exemplars, which is able to preserve the similarity between data points as demonstrated in [42].
(3) 1 -norm weights. Filters with large norms are believed to have the ability to retain more information, thus the corresponding weights are inherited as a warm-up for the follow-up fine-tuning [26]. Such a scenario has been widely-known in the network pruning.
(4) Random Initialization. Recent progress [36] shows that the essence of filter pruning lies in finding the optimal pruned Fig. 2. Pruning strategy for VGGNet, ResNet and GoogLeNet in our EPruner. For VGGNet, all convolutional layers are pruned. For ResNet, we only prune the first two convolutional layers in each residual block. For GoogLeNet, we only prune the branches with more than one convolutional layer. For all pruned filters, the channels in the next layer are removed correspondingly. (Best viewed with zooming in) architecture and random initialization with Gaussian distribution can perform better than inheriting the most important filters. Thus, we also consider random Gaussian distribution.
We use the exemplar weights by default. In Sec. V, we conduct ablation study regarding these different ways of initialization and justify the correctness of inheriting the most important filters in the literature.

IV. EXPERIMENTS
We apply the proposed EPruner on the CIFAR-10 benchmark [43] using three classic deep networks including VG-GNet [1], GoogLeNet [37] and ResNets [38], and the ILSVRC-2012 benchmark [44] using ResNets [38] with different depths. VGGNet contains sequential convolutions, GoogLeNet has the convolutions with multiple branches, and ResNet is especially designed with residual blocks. Fig. 2 outlines our pruning strategy for the three kinds of networks. We carry out our experiments on NVIDIA Tesla V100 GPUs. All models are implemented and trained using Pytorch [45].

A. Implementation Details
Training Settings. We fine-tune all models by using Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9 and the batch size is set to 256. Also, on CIFAR-10, we use a weight decay of 5×10 -3 and 150 epochs are given for fine-tuning. The learning rate starts from 0.01 and is reduced by a factor of 10 after 50 and 100 epochs. On ILSVRC-2012, the weight decay is set to 1×10 -4 and we fine-tune the networks for 90 epochs. The learning rate is initially set to 0.1 and divided by 10 every 30 epochs. Without specifications, for all methods, we apply the random crop and horizontal flip to the input images, which are also official operations in Pytorch. To stress, other techniques for image augmentation, such as lightening and color jitter, can be applied to further improve the accuracy performance as done in the source codes of [24], [46], [47]. Besides, the cosine scheduler for learning rate can also be used to replace the step learning, which also has been demonstrated to be able to boost the performance in [12], [48], [49]. We do not consider these in this paper since we aim to show the performance of pruning algorithms themselves.
Performance Metrics. We report channels, FLOPs (floating-point operations) and parameters to measure the effect of pruning. Channels reflect the memory footprint. FLOPs and parameters reflect the computation cost and storage space, respectively. Also, their corresponding pruning rates are reported. Besides, for CIFAR-10, top-1 accuracy of pruned models is provided. For ILSVRC-2012, both top-1 and top-5 accuracies are reported.

B. Testing Results on CIFAR-10
Tab. II displays our pruning results for VGGNet, GoogLeNet and ResNets on CIFAR-10. More detailed analyses are provided as below: VGGNet. We choose to prune the 16-layer VGGNet model on CIFAR-10. As can be seen from Tab. II, EPruner greatly simplifies the model complexity by reducing 67.73% channels, 76.34% FLOPs and 88.80% parameters, meanwhile it still obtains an accuracy improvement of 0.06% (93.08% for EPruner vs. 93.02% for the baseline). This greatly facilitates VGGNet model, a popular backbone for object detection and semantic segmentation, to be deployed on mobile devices. Hence, EPruner demonstrates its ability to compress and accelerate the neural network with the sequential structure.
GoogLeNet. For GoogLeNet, as shown in Tab. II, EPruner removes 22.70% channels, 67.36% FLOPs and 64.20% parameters with a negligible accuracy drop (94.99% for EPruner vs. 95.05% for the baseline). Besides, we observe that for GoogLeNet, the channel reduction is not so high as that in VGGNet. To analyze, as shown in Fig. 2, we do not prune the branch with only one convolutional layer and the last convolutional layer in the branch with more than one layer is also not pruned, which explains the above observation. Nevertheless, as can be seen, the reductions of FLOPs and parameters are still significant. Hence, EPruner can be well applied in the neural network with a multi-branch structure.
ResNets. To evaluate the power of EPruner in complexity reduction for network with residual blocks. We choose to prune ResNets with different depths of 56 and 110. As shown in Tab. II, EPruner boosts the computation by decreasing about 61.33% FLOPs for ResNet-56 and 65.91% FLOPs for ResNet-110, and it also saves more than half storage space by reducing 54.20% parameters for ResNet-56 and 76.30% parameters for ResNet-110. Comparing ResNet-56 with ResNet-110, we observe that more FLOPs and parameters are reduced in ResNet-110. The potential reason might be that the deeper ResNet-110 suffers more over-parameterized burden. Besides, it can also be observed that, similar to GoogLeNet, the channel reduction for ResNets is also limited. As explained in Fig. 2, we do not prune the convolution in the skip connection and the last convolutional layer in the residual block. Lastly, we can see that EPruner can well maintain the accuracy performance of the baseline model. For ResNet-56, EPruner leads to little accuracy loss (93.18% for EPruner vs. 93.26% for the baseline), while for ResNet-110, EPruner gains an obvious accuracy improvement of 0.12%. The above observations verify the effectiveness of EPruner in compressing and accelerating the residual-designed networks.
Besides, Tab. II also displays the results of uniform pruning. The digit after the network denotes the percentage of preserved filters in each layer of the network. As can be seen, our EPruner outperforms uniform pruning regarding the accuracy performance even with more reductions of the model complexity, which well demonstrates that EPruner can return an adaptive pruned network architecture with a better performance.
Also, we observe that the performance of pruned VGGNet-16 and ResNet-110 is better than that of the models before pruning. The rationale behind this is that pruning a big neural network to a smaller one also has the advantage of avoiding overfitting and improving generalization [50], [51]. The VGGNet-16 and ResNet-110 are two over-parameterized networks, which often result in an over-fitting problem when trained on the small CIFAR-10. Our network pruning helps to relieve overfitting and improve generalization as discussed above, and thus better performance can be observed.
As can be observed from Tab. III, the accuracy performance for compressed models on ILSVRC-2012 generates more compared with that on CIFAR-10. For an in-depth analysis, ILSVRC-2012 is a large-scale benchmark with 1,000 categories which poses a greater challenge, compared with smallscale CIFAR-10 containing only 10 categories, to recognize the images using the compressed models. Nevertheless, as can be seen, the drops of accuracies are still tolerable. Besides, we observe that the efficacy of EPruner is more advantageous in compressing deeper networks. For example, in the deepest ResNet-152, EPruner achieves a 65.12%-FLOPs reduction by removing 64.18% parameters and 18.52% channels, with only 1.48% top-1 and 0.71% top-5 accuracy loss. While in ResNet-18, 43.88% FLOPs 48.25% parameters and 10.00% channels are removed with more accuracy drops of 2.35% in the top-1 and 1.38% in the top-5. To explain, the shallow ResNet-18 has less redundancies thus more accuracy drops occur even with less reduction of model complexity. Nevertheless, this observation is meaningful since deeper networks are more constrained in the resource-limited environment.
Moreover, similar to the observation in Tab. II, EPruner also advantages in its supreme performance in comparison with the uniform pruning even on the large-scale ILSVRC-2012, which again well demonstrates the effectiveness of our EPruner in reducing the model complexity while retains a better performance.

E. In-depth Analysis
The experiments in Tab. II, Tab. III and Tab. IV show that EPruner can be well applied to reduce the complexity of CNNs while keeping a better or at least comparable accuracy performance against the pre-trained models or state-of-the-art methods. The efficacy of our EPruner is mainly from two points: self-adapting to the optimal pruned architecture and inheriting the most important filter weights.
Adaptive Pruned Architecture. Rule-based pruning and regularization-driven require the involvement of human labor to designate the pruned architecture or hyper-parameter analysis, results of which are usually sub-optimal, while architecture-search-driven pruning has to be re-run for some times to pick up the best one. To analyze EPruner, we show the layer-wise pruning when β = 0.73 using VGGNet-16 in Fig. 3. As can be seen, the pruning rate varies across different layers. More filters are preserved in the middle layers (3 2 to 4 1) while the other layers tend to remove more filters. By selecting the exemplars via message passing among filters, EPruner self-adapts to the filter property and derives the deterministic optimal pruned architecture without human involvement.
Important Filter Weights. Recent work [36] shows that the essence of filter pruning lies in finding the optimal pruned architecture rather than selecting the most important filter weights as in [18], [25], [26]. In Tab. V, we show a different  15(92.97) observation by feeding the pruned architectures with different initial weights as stated in Sec. III-C. From Tab. V, using exemplar weights obtains the best accuracy performance while random projection of exemplar weights and weights with larger 1 -norm feed back the second best, three of which show consistently higher performance than the random initialization. It indicates that the pre-trained exemplar weights are already a "distilled" piece of information from the large-scale training data of CNNs, thus they can provide a better warm-up for fine-tuning, which re-justifies the correctness of inheriting the most important filter weights. We assume that the criteria for measuring the most important filter weights in previous rulebased pruning methods are not robust and supportive, thus random initialization shows better results [36].

F. Ablation Study
In this section, we show the influence of β and the comparison of practical time consumption in finding out the optimal pruned architecture.
Influence of β. The β is used to make the reduction of model complexity more adjustable. As stressed in Sec. III-B, smaller β leads to more exemplars, thus fewer reductions of model complexity, but better accuracy performance. To demonstrate this, Fig. 4 shows the influence of β by using VGGNet-16 on CIFAR-10. Fig. 4(a) verifies our analysis. The accuracy performance is relatively stable when β falls within 0.0 to 0.7. It starts to drop as β increases, which makes sense since the pruning rate drastically goes up at the point of β = 0.7. Then we show a more subtle interval with β ranging from 0.7 to 0.8 in Fig. 4(b). As can be seen, the suitable value of β would be between 0.72 and 0.73. Note that, β is used to reach the expected reduction of model complexity, which however does not involve in the process of finding the optimal architecture, which lessens human labors.
Practical Time Consumption. We choose ABCPruner, which shows the best efficiency and effectiveness among compared methods, for comparison. Tab. VI shows the practical time consumption on finding out the pruned architecture between EPruner and ABCPruner. As can be seen, ABCPruner requires a large search time on NVIDIA Tesla V100 GPUs, even multiple GPUs are necessary in deeper networks, e.g., ResNet-152. The consumption would be significantly heavy in other platforms, such as GTX-1080TI GPUs and CPUs, (b) β ranges from 0.7 to 0.8. Fig. 4. The influence of β for VGGNet-16 on CIFAR-10. Generally, large β obtains higher pruning rates but more accuracy drops. which is unacceptable. In comparison, EPruner can find out the pruned architecture in seconds simply on CPUs, a magnitudeorder reduction of time consumption. To analyze, ABCPruner is a data-driven approach where the training data has to be used to train each potential architecture for some epochs and evaluate its fitness by observing the accuracy, while EPruner selects the exemplars among filters, which is data-independent and thus more efficient.
V. CONCLUSIONS In this paper, we present a novel filter pruning method, called EPruner, which aims to select exemplars among filters. To that effect, we regard the filters as a set of high-dimensional data points and the affinity propagation is applied to generate high-quality exemplars. The lack of human involvement and parameter analysis makes the implementation simpler and more effective. The optimal architecture with EPruner can be efficiently implemented within a few seconds simply on the CPUs, leading to a magnitude-order reduction of time consumption. We show that the weights of exemplars can serve as a better warm-up for fine-tuning the network, which justifies the correctness of inheriting the most important filter weights. We demonstrate the applicability of EPruner in compressing various networks and its superiorities over competing SOTAs. Yan Wang works as a software engineer in Search at Pinterest. With a Ph.D degree on Electrical Engineering from Columbia University, Yan published over 20 papers on top international conferences and journals, and holds 10 US or international patents. He has broad interests on deep learning's applications on multimedia retrieval.
Yongjian Wu received the master degree in computer science from Wuhan University, China, in 2008. He is currently the Expert Researcher and the Director of the Youtu Lab, Tencent Co., Ltd. His research interests include face recognition, image understanding, and large scale data processing.