Hierarchical Group Sparse Regularization for Deep Convolutional Neural
  Networks
  In a deep neural network (DNN), the number of the parameters is usually huge
to get high learning performances. For that reason, it costs a lot of memory
and substantial computational resources, and also causes overfitting. It is
known that some parameters are redundant and can be removed from the network
without decreasing performance. Many sparse regularization criteria have been
proposed to solve this problem. In a convolutional neural network (CNN), group
sparse regularizations are often used to remove unnecessary subsets of the
weights, such as filters or channels. When we apply a group sparse
regularization for the weights connected to a neuron as a group, each
convolution filter is not treated as a target group in the regularization. In
this paper, we introduce the concept of hierarchical grouping to solve this
problem, and we propose several hierarchical group sparse regularization
criteria for CNNs. Our proposed the hierarchical group sparse regularization
can treat the weight for the input-neuron or the output-neuron as a group and
convolutional filter as a group in the same group to prune the unnecessary
subsets of weights. As a result, we can prune the weights more adequately
depending on the structure of the network and the number of channels keeping
high performance. In the experiment, we investigate the effectiveness of the
proposed sparse regularizations through intensive comparison experiments on
public datasets with several network architectures. Code is available on
GitHub: "https://github.com/K-Mitsuno/hierarchical-group-sparse-regularization"
