OrthoReg: Robust Network Pruning Using Orthonormality Regularization
  Network pruning in Convolutional Neural Networks (CNNs) has been extensively
investigated in recent years. To determine the impact of pruning a group of
filters on a network's accuracy, state-of-the-art pruning methods consistently
assume filters of a CNN are independent. This allows the importance of a group
of filters to be estimated as the sum of importances of individual filters.
However, overparameterization in modern networks results in highly correlated
filters that invalidate this assumption, thereby resulting in incorrect
importance estimates. To address this issue, we propose OrthoReg, a principled
regularization strategy that enforces orthonormality on a network's filters to
reduce inter-filter correlation, thereby allowing reliable, efficient
determination of group importance estimates, improved trainability of pruned
networks, and efficient, simultaneous pruning of large groups of filters. When
used for iterative pruning on VGG-13, MobileNet-V1, and ResNet-34, OrthoReg
consistently outperforms five baseline techniques, including the
state-of-the-art, on CIFAR-100 and Tiny-ImageNet. For the recently proposed
Early-Bird Ticket hypothesis, which claims networks become amenable to pruning
early-on in training and can be pruned after a few epochs to minimize training
expenditure, we find OrthoReg significantly outperforms prior work. Code
available at https://github.com/EkdeepSLubana/OrthoReg.
