{'fraction_non_alphanumeric': 0.02368709432689876, 'fraction_numerical': 0.0059006996543875916, 'mean_word_length': 4.844334975369458, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks

Sparse neural networks are important for achieving better generalization and enhancing computation efficiency. This paper proposes a novel learning approach to obtain sparse fully connected layers in neural networks (NNs) automatically. We design a switcher neural network (SNN) to optimize the structure of the task neural network (TNN). The SNN takes the weights of the TNN as the inputs and its outputs are used to switch the connections of TNN. In this way, the knowledge contained in the weights of TNN is explored to determine the importance of each connection and the structure of TNN consequently. The SNN and TNN are learned alternately with stochastic gradient descent (SGD) optimization, targeting at a common objective. After learning, we achieve the optimal structure and the optimal parameters of the TNN simultaneously. In order to evaluate the proposed approach, we conduct image classification experiments on various network structures and datasets. The network structures include LeNet, ResNet18, ResNet34, VggNet16 and MobileNet. The datasets include MNIST, CIFAR10 and CIFAR100. The experimental results show that our approach can stably lead to sparse and well-performing fully connected layers in NNs.


Introduction
Deep neural network (DNN) as a flexible function approximator has been very successful in a broad range of tasks [1,2]. However, previous researches have shown that they are over-parameterized because some weights can be pruned without reducing accuracy [3,4,5]. Furthermore, they can easily overfit and even memorize random patterns in the data, if not properly regularized [6].
A way to address above issues is to reduce the complexity of DNNs. There have been studies on this topic, including network pruning [7,8], knowledge distillation [9,10], network architecture search [11], introduce sparse regularization [12,13], and adaptive model compression [14]. The existing methods usually sacrifice the network performance for sparser structures. Furthermore, additional fine-tuning process is required in some existing methods. In the mammalian brain, synapses are created in the first few months of a child's development, followed by gradual pruning of little-used connections, falling to typical adult values [15,16]. It means that the cognitive process of the mammalian brain is to dynamically pruning synaptic connections and renew the remaining ones. This inspires our work in this paper. We imagine there is an additional schema of simplifying the structure of the network, along with the optimization of its weights. We introduce a switcher neural network (SNN) to perform this task, which utilizes the knowledge contained in the weights of DNN to switch each connection in DNN and simplify its structure automatically. Based on SNN, we propose an efficient and novel learning framework, which is shown in Figure 1. We can see that our framework is composed of two parts, task neural network (TNN) and SNN. TNN is the usual network, for finding a winning-ticket [8]. The SNN explores the knowledge contained in the weights of TNN to optimize its structure. Then, the TNN renews it weights to adapt to this new structure. This leads to an alternating optimization of SNN and TNN. It can be seen as gradually pruning unimportant connections during the training and optimizing the remaining connections. To achieve the good results of sparse learning, the SNN structure is designed to be U-net like, composed of an encoder and a decoder. The proposed SNN-TNN framework let us obtain the optimal structure and the optimal parameters of TNN simultaneously and directly, without retraining or fine-tuning. Furthermore, the loss functions used in the two procedures of alternate learning are both the performance evaluation of TNN, such as commonly used Cross Entropy (CE) in image classification applications in this paper. So, we avoid imposing unreasonable sparsity on over-parameterized models.
Some existing methods for simplifying networks also rely on the knowledge contained in network weights, but usually simple and setting-by-experience, such as low weights indicating unimportant connections [7]. Differently, we try to represent and mine more sophisticated knowledge in network weights with a DNN, i.e., SNN. To our best knowledge, this is the first time of such efforts.
We apply the proposed approach to obtain sparse fully connected layers (FCL) in DNNs. The FCLs, even if they are in the minority in number, are responsible for the majority of the parameters in mainly used DNNs [17]. Experiments on various types of neural network models and datasets are conducted to confirm the effectiveness of our method.
The main contributions of this paper are summarized as follows: 1. We propose a novel network structure learning approach by introducing the idea of switcher neural network (SNN) to represent and explore the knowledge contained in the network weights for switching connections in the network.
2. The structure of SNN is designed based on the thinking of fully exploring the relationship among weights to determine the importance of each connection. It is divided into an encoder and a decoder, like U-net shape, But the feature maps generated by the encoder need to be transformed before they are inserted into the decoder path.
3. We develop a learning algorithm to optimize the SNN and the TNN alternately under the same learning objective. It can learn structure and parameters at the same time, without additional regular items, re-training and fine-tuning.
4. The proposed approach is applied to optimize the FCLs of DDNs for classification. It leads to not only sparser but also more accurate classifiers in the experiments on various types of networks and various datasets.

Related Work
The main methods on the topic of sparse DNNs are introduced briefly as follows.

The Proposed Approach
The core idea of our proposed approach is to explore the knowledge contained in the network weights for determining the importance of each connection of the network. The SNN is introduced to fulfill this idea, which takes the weight matrix between the i-th layer and the i+1-th layer of the TNN as the inputs and outputs the values for switching the connections in the i+1-th layer of TNN.

The Network Architecture
The overall framework of the proposed approach is illustrated in Figure 2, which consists of two components: the TNN is used for performing actual tasks, such as classification in the experiments of this paper, and the SNN represents and explores the knowledge contained in the weights of TNN for optimizing TNN structure. Actually, the outputs of SNN are multiplied with the weights of corresponding connections in TNN, thus it plays a role of weighting connections and the zero outputs of SNN means the pruning of corresponding connections. In this paper, we explore the weight matrix between the i-th layer and the i+1-th layer to switch connections in the i+1-th layer. We consider two strategies. The first strategy is to take all the output connections of a neuron as a whole. The processing effect is equal with switching each neuron in i-th layer. Thus the SNN outputs a different value for each neuron and let it multiplied with the output of the corresponding neuron. The second strategy is to single out each connection. Thus the SNN outputs a different value for each connection and let it multiplied with the weight of the corresponding connection.  Figure 3, where Process1 and Process2 represent the process of connection switching and that of neuron switching, respectively. Let m and n be the number of neurons in the i-th and i+1-th layer, respectively, then in both process1 and process 2, the SNN will accept m*n weights. But it will output m*n values in Process1 to indicating the importance degree of each connection. Then these the matrix constructed by these m*n values are multiplied with the weight matrix, such as shown as g1*w2 in Figure 3. The SNN output m values in Process2 to indicating the importance degree of each neuron. These the vector constructed by these m values are multiplied with the input data of this layer, such as shown as neuron*g2 in Figure 3.
The following formulas help to better express our method. The forward computation of three-layer fully connected networks as TNN is Where Wk (k=1-3) denotes the weight matrix of the k-th layer, Φ denotes the activation function, X denotes the input data, and ŷ denotes the predicted value of the network.
We set SNNs for each FCL of the TNN. Let gk be a SNN for the k-th layer.
The output of TNN for our first strategy of switching neurons is Then the output of TNN for our second strategy of switching connections is where  denotes the element wise product.

Alternating Learning Algorithm
The satisfactory TNN is determined by the optimal parameters of SNN and the optimal parameters of TNN. In order to optimize the two networks, we present a learning algorithm to train these two parts together, which is composed of two phases that are performed alternatively and iteratively. In the first phase, the TNN is remained and the SNN parameters are optimized to choose a reasonable structure of TNN. In the second phase, the SNN is remained and only TNN parameters are optimized to adapt to this new structure. We use the same loss function in these two phases, such as the CE loss in the applications of this paper. By using this strategy, the network sparsity is not required explicitly, but is achieved indirectly by requiring the optimized performance in applications. It is similar to the working way of our mammalian brain that continuously and dynamically optimizes the synaptic connections in the process of learning new things [15,16]. We believe that it is more natural for the network to find the optimal structure through targeting the direct application goals and exploring the relationship among its own neuron connection than the way of exerting human-experienced regularization terms.
The regularization items on sparsity could impose unreasonable sparsity to cause accuracy drop. Our experimental results confirm that our learning strategy can effectively avoid this problem and leads to even better performance with sparser structures.
The above alternative learning algorithm is summarized in Algorithm.

SNN Structure
The purpose of SNN is to determine the importance of the connections based on the knowledge contained in TNN's weights. The corresponding structure of SNN is designed as that illustrated in Figure 4, which is explained as follows.
The purpose of SNN is to mine TNN weights, thus we need to firstly extract the features in the weight matrix for making decisions for switching connections. Considering that the convolutional neural network (CNN) is a useful tool for extracting the features, we apply it to our problem. The CNN as the encoder subnetwork of our SNN is shown in Figure 4, which is composed of multiple blocks of convolution, ReLU activation, and max pooling. Based on the extracted features, we add a decoder subnetwork to obtain the values reflecting the importance of connections. In this part, we only upsample the widths of decoded feature maps, and the corresponding channel numbers do not change. The feature map obtained by downsampling is spliced with the corresponding upsampling feature map through row convolution. Noted that in the first strategy of switching TNN neurons, because the size of SNN output is not same as its input, the encoded features cannot be directly copied to the corresponding layer of the decoder. We introduce a row convolution to transform the encoded features to the same shape of decoded feature map. Furthermore, the widths of decoded feature maps are always 1 to determine the weighting degree of each neuron.