{'fraction_non_alphanumeric': 0.029208371246587806, 'fraction_numerical': 0.013254473764027904, 'mean_word_length': 4.686616074508451, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

This work presents deep asymmetric networks with a set of node-wise variant activation functions. The nodes' sensitivities are affected by activation function selections such that the nodes with smaller indices become increasingly more sensitive. As a result, features learned by the nodes are sorted by the node indices in the order of their importance. Asymmetric networks not only learn input features but also the importance of those features. Nodes of lesser importance in asymmetric networks can be pruned to reduce the complexity of the networks, and the pruned networks can be retrained without incurring performance losses. We validate the feature-sorting property using both shallow and deep asymmetric networks as well as deep asymmetric networks transferred from famous networks.


I. INTRODUCTION
N EURAL networks usually consist of neurons that have equal learning capabilities because the mathematical models of neurons are identical for all the neurons in a network. Neurons are trained by capturing the relation between their inputs and outputs. Thus, all the neurons in a network have an equal chance of learning input features. Consequently, without further inspections, one cannot tell whether a feature learned by one neuron is more or less important than features learned by other neurons. In this work, we provide network neurons with unequal feature-learning abilities; thus, some neurons learn more important features than others.
Neural networks are often trained using backpropagation [1], [2]. Errors between network outputs and target outputs are propagated backward to update the weights of nodes in previous layers of the network. The updates are proportional to both the inputs and the so-called sensitivities of the nodes. By assigning different activation functions to nodes in a layer, we allow the nodes to have different sensitivities to the same inputs. We use a set of functions parameterized by a single parameter as activation functions to assign node-wise variant sensitivities. Specifically, the slopes of the activation functions are controllable via this parameter. Thus, nodes J. Jang, J. Kim, and J. Lee are with Electronics and Telecommunications Research Institute (ETRI), Daejeon, Korea (e-mail:jangjh6297@gmail.com, jhkim504@etri.re.kr, leejy@etri.re.kr).
H. Cho and S. Yang  with smaller node indices are assigned higher sensitivities. Features learned by a network with a set of node-wise variant activation functions are sorted by importance. When nodes in a trained network are removed individually, from last to first, the network accuracy gradually deteriorates at increasingly larger increments. A network with a set of node-wise variant activation function has the ability to learn not only the features that represent the inputs but also the importance of the learned features. We call deep networks containing nodes with unequal and asymmetric learning abilities "deep asymmetric networks." Prior studies exist concerning the use of node-wise variant activation functions. Linear combinations of multiple activation functions and piece-wise linear activation functions were used as activation functions in [3], [4], [5]. In this approach, the parameters that define the shapes of the activation functions assigned to the nodes are learned during training. The purpose of varying the activation functions in these studies was to improve network performance. In contrast, in our work, the purpose of varying the activation functions is to build networks that learn and sort features by their importance.
Deep networks have achieved great successes across a wide range of fields; however, they are usually computationally expensive and memory intensive. Many studies have investigated designing more efficient deep networks [6]. We can use the asymmetric network capacity to learn the importance of features to design more efficient deep networks. Because the nodes in a deep asymmetric networks sort features by importance, it is possible to prune the nodes from least to most important to meet network computational complexity and memory requirements.
The ability to learn the importance of features is validated with both a simple network in an auto-associative setting using the MNIST dataset [7] and deep convolutional neural networks (CNN) for object recognition using the CIFAR-10 dataset [8] and for action recognition using NTU RGB+D action recognition dataset [9]. After the asymmetric networks are trained, we can analyze the individual contribution of each node to the reconstruction error and recognition accuracy. The experimental results show that the reconstruction error and the accuracy are increasingly influenced by the nodes with smaller indices than by those with larger indices, which indicates that the features learned by the asymmetric networks are sorted in the order of their importance. We applied the featuresorting property to prune the deep asymmetric CNN without a loss of recognition accuracy. Using the asymmetric technique, we were also able to prune deep networks transferred from famous complex networks. To do this, we prepared VGG [10] and ResNet [11] for a facial expression recognition task.
Then, we transferred the weights from the famous networks to the asymmetric networks and trained them on the CK+ dataset [12]. The asymmetric networks pruned by the proposed procedure result in smaller (and thus more efficient) networks but exhibit no loss of accuracy. We also compared our pruning to results reported in studies using the MNIST and CIFAR-10 datasets.
The rest of this paper is organized as follows. Section II proposes the use of node-wise variant activation functions in asymmetric deep networks. The network architecture and training are introduced in Section II-A. In section II-B, the feature-sorting property of asymmetric networks is analyzed using a simple shallow network with linear activation functions. Section III presents the pruning procedure used to design more efficient deep asymmetric networks. A review of pruning methods is given in Section III-A, and the pruning algorithm is given in Section III-B. Experimental validations are presented in Section IV for shallow, deep, and transferred deep asymmetric networks in Sections IV-A, IV-B, and IV-C, respectively. Finally, conclusions are given in Section V.

A. Network Architecture
We consider a network with L layers trained with a training set of input and output pairs (x, y). The number of nodes in the lth layer is n l . The relationship between the output x l i and the input x l−1 j of the lth layer are given by for a fully connected layer, or for a convolutional layer, which is activated by for i = 1, 2, · · · , n l , where W l ij or W l are the weights. Instead of using a single activation function, we use a set of activation functions in each layer. The activation function of the lth layer f l (·; s l i ) takes the parameter s l i , which assigns a different activation function for each node. The selected activation functions are which use the parameter set where f 0 (·) is an activation function such as a rectified linear unit (ReLU), hypertangent, or sigmoid function. The set of activation functions in each layer satisfy for all i and Layer Input Fully-Connected Activation Layer Output for i = 1, 2, · · · , n l − 1 for all u. As a result, the nodes with smaller indices are assigned activation functions that have increasingly steep slopes. Fig. 1 shows a schematic of the proposed network with a set of node-wise variant activation functions (ReLU activation functions in this case). A network can be trained to minimize a cost function E that penalizes the difference between the network outputs and the desired outputs. When the network is trained using backpropagation [1], [2], the weight matrix W l , whose element is the weight W l ij of the lth layer, is updated by where η is the step size and x l−1 is a vector whose element is x l−1 i . The element of the sensitivity vector δ l is for the last layer and for the other layers. Because we adopt a set of activation functions whose first derivatives vary with the node indices, the sensitivities of the nodes in the last layer in (9) to the same weighted input u L i and error E differ from node to node. In turn, these different sensitivities affect the sensitivities of the nodes in all the layers through (10). As a result, some nodes in a layer respond more sensitively than others when the weights are updated. In particular, the nodes with smaller node indices become increasingly more sensitive based on our choice of activation functions. We call such networks consisting of nodes with unequal and asymmetric sensitivities "deep asymmetric networks." For activation functions given in the form of (4), the weight matrix update in (8) can be rewritten as follows: where s l is a vector whose element is the parameter s l i , δ 0 l is the sensitivity vector when a fixed activation function f 0 is used for all the nodes, and • is the Hadamard product. The updated (11) in vector notation is where S l is a diagonal matrix whose diagonal elements are repetitions of the vector s l . Note that (12) is the update for the diagonally scaled steepest descent algorithm resulting from the variable transforms for a quadratic penalty function [13]. The diagonal scaling is originally used to transform the variables to have equal dynamic ranges for faster convergence. We assume that the inputs to the network are already properly scaled for fast convergence. By using the set of parameterized activation functions, we intentionally make the cost function steeper in some directions than others. While the overall convergence may be delayed by this variable transform, the convergence in the steeper directions become much faster compared to that in the other, less steep directions.

B. Analysis of Asymmetric Networks
For analysis, consider a network that consists of one hidden layer with linear activation functions. The asymmetric network can be modeled byŷ where x ∈ R n andŷ ∈ R m are the input and output of the network, respectively. The matrices W 1 ∈ R p×n and D ∈ R p×p represent the operation of the first layer, and the matrix W 2 ∈ R m×p represents the operation of the second layer. The rows of W 2 and W 1 contain the weights for the output nodes of the layers. The activation function in the output layer, the second layer in this example, is usually selected based on the problem at hand. Hence, we consider the use of a set of node-wise variant activation functions for the first layer. The elements of the diagonal matrix D are the slopes s i of the linear activation functions.
The network weights are found by minimizing the following cost function: where the summation is over the pairs (x k , y k ) in a training set. We denote the covariance matrices of the sample pairs by Σ xx , Σ xy , and Σ yx . The cost function is uniquely minimized by and the minimum value is not changed by the introduction of the diagonal matrix D [14]. Let the matrices W 1 and W 2 be respectively, where C is an invertible matrix and the columns of U are the eigenvectors of Then, the cost function in (15) can be written as follows: where the λ i values are the eigenvalues of the matrix Σ.
Hence, the p largest eigenvectors arranged in the matrix U minimize the cost function. The weights of the network are related to the eigenvectors of Σ through the matrices C and D. The introduction of the diagonal matrix D does not change the fact that the weight matrices are related to the eigenvectors [14]. Using a node-wise variant activation function affects both the sensitivities of the nodes and the convergence speeds of the weights in particular directions. Consider the following extreme case, where ≈ 0. Because the insensitive nodes contribute little, the problem can be approximated to the case where p = 1, i.e. where The network can minimize the cost function in (15) by relating the w 1 1 and w 2 1 to the eigenvector corresponding to the largest eigenvalues. Hence, the network learns the most important feature for the first node. After the weight of the first node is found, we can set the slopes as follows: Then, the problem is to approximate the network operation by The weights of the first and second nodes will reflect the eigenvectors that correspond to the first and second largest eigenvalues. This process continues to find the weights of the p nodes. Based on the choice of the slope parameters s i in (5), convergence in the directions of the weights will be increasingly slow for nodes with larger indices; however, the weights of the nodes with smaller indices can be expected to converge faster than those with larger indices. Based on the above observation in the extreme case-where the node weights are converged individually-we can assume that the features learned by the network are sorted in order of their importance: from the first hidden node to the last hidden node. Deep networks with nonlinear activation functions trained by the backpropagation are not quite as straightforward to analyze as is the shallow network with linear activation functions. Nevertheless, they can be expected to behave similarly. There will be some difficulties associated with the nonlinearity of the activation functions, random initialization, and the herd effect of the backpropagation, and some nodes may end up having similar weights. We want to prevent nodes from having similar weights while allowing weights with negative correlations. To avoid such problems, a regularization term is often added to the cost function. We add the following regularization term during training that penalizes positive correlations between the normalized weights [15] The second summation is for i = 1, · · · , n l and j = 1, · · · , n l , where i = j, and the normalized weight θ i is given by

A. Review of Pruning Methods
Deep networks are typically computationally expensive and have high memory requirements. To deploy deep networks on devices with restricted computational capabilities, the complicated models of deep networks must be compressed. Previous studies have also investigated designing compact and efficient networks by pruning unimportant nodes.
In [16], [17], the sensitivity of a cost function to small changes in weights of a network is measured based on the Hessian of the cost function. The weights are sorted by sensitivity; then, the nonessential small-sensitivity weights are pruned from the network. The sensitivity to small changes in node outputs [18] and the sensitivity to the removal of weights [19] have also been considered for pruning. A regularization term that penalizes large weights can be added to a cost function during network training to favor sparse weights. After training, the weights are sorted by some measure, and small weights are removed from the trained network. The l 2 , l 1 , and l 0 norms, among other, have been used as regularization terms to cause weights to be sparse to enable the pruning of small weights from networks [20], [21], [22], [23], [24]. The weight decay term was used as a regularization factor to favor small weights in [25]. The energy term was used as a regularization term to penalize nodes with small outputs in [26]. After the nonessential nodes with small sensitivity or small weights have been pruned, the networks are usually retrained to regain their performance. The pruning and retraining procedures can be applied iteratively. The performance of pruned networks depends on how effectively the sensitivity or the regularization identify the nonessential weights or nodes. However, because the nodes in deep asymmetric networks are already sorted in order of their importance, the nonessential nodes can be removed from the network without requiring further inspection of their importance.
The computational complexity and memory requirements of deep networks can also be reduced by introducing more structured architectures. In [27], fully connected layers of networks were replaced by the Hadamard transform, diagonal matrices, and random permutation matrix. These altered networks can perform the same tasks as the unaltered versions but at less computational expense. In [28], sparsity patterns were introduced that reduced the computations of convolutional networks by zeroing out filter coefficients. In [29], the filter coefficients were quantized and compressed to reduce the memory requirements. A survey of efficient deep network design methods can be found in [6], [30].

B. Pruning Deep Asymmetric Networks
The nodes in a layer of an asymmetric network are sorted in order of their importance. Thus, given an asymmetric network with p hidden nodes, when asked to remove one hidden node, we can know exactly which node to discard. The last hidden node, which corresponds to the least important feature, can be removed from the network with the least accuracy loss. Hence, the pruning strategy for designing efficient deep asymmetric networks is both simple and effective.
First, the accuracy of a trained asymmetric network is measured using a validation set. Then, the network is pruned by removing the nodes individually to ensure that the network accuracy does not fall below a certain level. The pruning process can starts from the last hidden layer and continues up to the first layer. The nodes in each layer are discarded one-by-one from last to first. The accuracy of the network is measured after removing each node is measured. The pruning of a current layer stops just before the accuracy drops below a predefined threshold; then, the pruning continues with the previous layer, again discarding nodes from last to first. Our empirical data indicates that pruning works better when it begins with a layer that has the largest number of nodes and continues with layers with smaller and smaller number of nodes. Thus, we prepare a set of layer indices L sorted by the number of nodes, and prune a network following the layer index in the set. The pseudocode of this pruning strategy is listed in Algorithm 1.

IV. EXPERIMENTS A. Asymmetric Networks
The use of a set of node-wise variant activation functions in an asymmetric network is validated with a simple fully connected network with one hidden layer using the MNIST dataset [7]. A network with n 1 hidden nodes is prepared in an auto-associative setting to reconstruct the input. A set of activation functions with the ReLU function f 0 is used. The parameters s i are set as shown in Fig. 2(b) (this parameter choice is discussed in Section IV-B). The network is implemented using the Keras Python deep learning library. The learning rate is set to the same learning rate that would be used for a symmetric network with a single activation function Remove node x l i from network. 10: end if 11: end for 12: end for for all the nodes. Because the slope parameters are set as described by (5), the convergence of the asymmetric network is guaranteed when the symmetric network converges.   Fig. 3(a) shows the 256 principal components, while Fig. 3(b) shows the average mean square error (MSE) values between the inputs and their reconstructions using the p principal components. Fig. 4 shows the features learned by the asymmetric network and the reconstruction errors when only the first p nodes are used to reconstruct the inputs. Because of the matrix C in (17), the features learned by the network do not exactly match the principal components. Moreover, the workload seems to be shared by the nodes as stated in [14]. The MSEs do not drop as sharply as in Fig. 3(b). However, the average MSE of the images reconstructed by the first p nodes shows that the features are sorted in order of importance. The average MSE drops by larger amounts when nodes with smaller indices are removed and does not improve much when adding nodes with larger indices. This learned-feature sorting occurs consistently for asymmetric networks with different numbers of hidden node n 1 = 64, 128, and 256 as shown in (a), (b), and (c), respectively. By using a set of node-wise variant activation functions, the asymmetric network not only learns the input features but also their importance.
The feature sorting exhibited by asymmetric networks may be affected by random initialization of weights during the training. To check whether the feature-sorting is repeatable, we trained the asymmetric network five times using the same training set. To check whether it is reproducible, we also trained the asymmetric network five times, but used a different training set each time. The average MSE values of images reconstructed with the first p nodes are shown in Fig. 5.

B. Deep Asymmetric Networks
In this section, we apply a set of node-wise variant activation functions to deep networks, whose performance is then analyized in object recognition using the CIFAR10 dataset [8] and action recognition using the NTU RGB+D action recognition dataset [9]. Object detection is relatively simple and a deep asymmtric network is trained from a scratch. Action recognition is more complicated and a deep asymmetric network is trained with weights initialized with those of a deep symmetric network.
The network for object recognition consists of four convolutional layers and two fully connected dense layers. All the layers utilize max pooling and activation functions. The convolution layers utilize 3×3 filters. There are 128, 128, 256, and 256 nodes in the convolutional layers and 512 and 10 in the fully connected dense layers. The final layer provides the classification. The activation functions in each layer consist of a set of parameterized ReLU functions. We used the parameters shown by the s i values in Fig. 2 and applied the same learning rate that would be used for a symmetric network. Specifically, the learning rate was set to 1e-3 with a decay rate of 1e-6. Adam [31] is used as an optimizer. The network was trained using 50,000 images and 10,000 images were used for testing. Fig 6 illustrates the training process of deep networks with the CIFAR-10 dataset. The losses and validation accuracies shown are for the asymmetric network, which uses the set of node-wise variant activation functions, and for a symmetric network that shares the same architecture but uses the same ReLU activation function for all nodes. We trained both the asymmetric and the symmetric network. With the variable transform in (13), convergence in some directions is intentionally delayed compared to convergence in other directions. The result is that the asymmetric network converges more slowly than the symmetric network as shown in Fig  6(a) and (b). The training delay can be compensated for by considering the average slopes of the activation functions. The average node slopes with the parameters set shown as in Fig.  2 (b) is approximately 1/3. Therefore, we set the learning rate of the asymmetric network to three times that of the symmetric network; then, the asymmetric network converged successfully in approximately the same number of iterations as the symmetric network as shown in (c) and (d). After training, the test accuracy was 0.8328 for the symmetric network and 0.8299 for the asymmetric network.  Fig. 7 (d) shows how the performance of the asymmetric network deteriorates as nodes are removed from the network. The test accuracy is measured when a given percentage of nodes are removed from last to first in a layer while leaving the remaining layers intact. The test accuracy is averaged over five trainings of the network. It can be seen that the network performance gradually falls as the nodes in each layer are removed from last to first. The gradual decline of performance indicates that the nodes with larger indices are of lesser importance. For comparison, symmetric networks with the same layers and one activation function assigned to all the nodes were trained with either the l 2 or l 1 norms of the weights or the correlation between the weights as regularization terms. We evaluated the performance of the symmetric network while removing nodes based on sorting by the l 2 or l 1 norms of the weights [20], [21] or based on sorting by the weight correlations [15]. The results are reported in Fig. 7(a), (b), and (c). The asymmetric network retains its performance better than does the symmetric network when the same number of nodes are removed from each network. For example, when we removed 90% of the nodes in the first layer, the accuracy of the networks becomes 16.85%, 18.21%, 19.91%, and 53.25%, for the l 2 , l 1 , correlation based methods, and the proposed method, respectively. This suggests that the asymmetric network learns the importance of features better than is possible by evaluating the weights in the symmetric network using various measures. The asymmetric network is trained with the correlation as regularization. The comparison to the symmetric network trained with only the correlation regularization validates that the performance improvement of the asymmetric network is from the use of the asymmetric activation functions. The trained asymmetric deep network is pruned by the proposed pruning algorithm. The target accuracy used in Algorithm 1 is set to 90% of the training accuracy of the unpruned network. The pruned asymmetric network is retrained using the training set. The ratio of the number of weights is ratio = # of weights after pruning # of weights before pruning (27) and the accuracy of the networks before pruning, after pruning, and after retraining are reported in Table II. We were able to retrain the pruned asymmetric network without a loss of accuracy. Table II also compares the accuracy when the slope parameters s i of the activation functions are chosen differently from the ones shown in Fig. 2 (a), (b), and (c). The parameters in (a) set the node sensitivities to vary linearly. The parameters in (b) set the node sensitivities to vary close to linearly at smaller indices and limits node to small sensitivities at larger indices. The parameters in (c) set the node sensitivities to one and zero for the small and large indices, respectively, and to vary linearly in the middle. The parameters in (b) yielded the highest accuracy with the smallest number of weights. Thus, we used the parameters in (b) in the rest of our experiments.  Table III shows the number of nodes in each layer of the asymmetric network before and after the pruning. Overall, all but 18.13% of nodes can be removed from the network without losing accuracy. The ratio of nodes that can be removed from the layers is higher in the deeper layers. Because the computational complexity of the deeper layers is higher, pruning the deeper layers contributes more toward reducing the overall computational complexity. We also compared the complexity and performance of pruned asymmetric networks to the pruning results reported in [20], [24], [27], [28], [32]. An asymmetric network with the same architecture as LeNet-5 [33] was prepared, and the weights are initialized randomly. The network was trained with MNIST dataset, pruned, and then retrained. The ratios of the weights after pruning and the classification errors are reported in Table IV. Our pruning algorithm works with all the layers in LeNet-5 while achieving comparable accuracy and yielding smaller networks.  [20] LeNet-5 0.80% 8.24% 0.77% [24] LeNet-5 0.73% (modified) 10.25% 0.76% [27] LeNet-5 0.87% 9.01% 0.71% [28] LeNet-5 N/A conv layers only 8.33% 1.70% [32] LeNet-5 0.94% dense layer only 16.00% 1.65% dense layer only 12.00% 2.01% Proposed LeNet-5 0.81% 6.73% 0.71% The network for action recognition is from [9]. The network uses two stream CNN, with three convolutional layers in each stream, followed by four dense layers. We initialized the asymmetric network with the same weights provided with the code. The activation functions in each layer are assigned with a set of parameterized ReLU functions. To assign the parameter s i to a node, node sorting may be required to determine which nodes in the symmetric network should be assigned with smaller indices in the asymmetric network. We can sort the nodes of a symmetric network using various measure, for example, by the l 2 norm of the weights, and then assign the node indices based on the sorted results. However, such node pre-sorting complicates the performance analysis because any performance variations may be due to either the pre-sorting or to the feature sorting of the asymmetric network. Hence, we assign the parameters without any pre-sorting of the node indices. After the parameter assignment, the asymmetric network is trained with NTU RGB+D action recognition dataset. Since the initial weights are already trained for good performance, we trained the asymmetric network without the correlation regularization.
The trained asymmetric deep network is pruned by the proposed pruning algorithm. The target accuracy used in Algorithm 1 is set to 90% of the training accuracy of the unpruned network. The pruned asymmetric network is retrained using the training set. Table V shows the number of nodes in each layer of the asymmetric network before and after the pruning. Overall, all but 38.63% of nodes can be removed from the network without losing accuracy. Accuracy of the network at each step of pruning is given in Table VI.

C. Transferred Deep Asymmetric Networks
A network can share its architecture with that of a famous network previously proven to be excellent for some other purpose. The network can be initialized using the weights transferred from the famous network and then fine-tuned with a training set prepared for a given application. Famous networks are usually trained for tasks that are more complicated than the complexity of a specific task. Hence, although weight transfer is a well-established way to achieve good network performance, the end result can be an excessively large and heavy network architecture that is computationally too expensive for a given task. In this section, we transfer famous networks to asymmetric networks and then prune the asymmetric network to design efficient and compact networks. Since weights of famous networks are already trained for good performance, we trained the asymmetric network without the correlation regularization.
We transferred VGG [10] and ResNet [11] to asymmetric networks for the purpose of facial expression recognition. The asymmetric networks consist of the same layers as VGG-16 or ResNet-50 followed by an output layer that classifies input facial images into seven emotions: {anger, contempt, disgust, fear, surprise, happiness, sadness}.
The network is trained using the CK+ dataset [12]. We selected 325 sequences of 118 subjects that are classified as displaying one of the seven emotions. The so-called "apex frames" that occur at the peak of the expression were collected as labeled facial images. The network was trained using ten-fold cross validation. The labeled images are divided into ten folds, nine of which were used for training, and the remaining fold was used for evaluation.
To transfer weights from a symmetric network to an asymmetric network, node sorting may be required to determine which nodes in the symmetric network should be assigned with smaller indices in the asymmetric network. However, node pre-sorting complicates the performance analysis because any performance variations may be due to either the pre-sorting or to the feature sorting of the asymmetric network. Hence, we transferred the weights without any pre-sorting of the node indices. After the weight transfer, the asymmetric network is trained with a training set. Then, the trained asymmetric network is pruned using Algorithm 1. After the pruning, we retrain the pruned networks with the training set. Table VII shows the pruning results of the asymmetric network transferred from VGG-16 and ResNet-50. The target accuracy was set to 90% of the training accuracy before pruning. We were able to prune the VGG and ResNet transferred asymmetric networks down to 24.48% and 16.82%, respectively, without loss of accuracy. The layer-wise pruning ratios are reported in Table VIII and IX for the VGG-16 and ResNet-50 transferred asymmetric networks, respectively. It is interesting to note that all the nodes in the 44, 45, 48, and 49th layers of ResNet-50 were removed during pruning. ResNet has bypass paths that enables the network to learn the error instead of the input itself. The removal of all the nodes in these convolutional layers changes the residual block operations to simple bypasses. Hence, our pruning procedure changes the network architecture from residual blocks to convolutional layers with 1 × 1 filters.  VGG-16 6.75% 36.0% 6.60% Proposed VGG-16 6.67% 29.6% 6.17% [23] ResNet-56 6.96% 1st layer of residual blocks 86.3% 6.94% Proposed ResNet-56 6.82% 26.2% 6.75%