High-dimensional Bayesian Optimization for CNN Auto Pruning with
  Clustering and Rollback
  Pruning has been widely used to slim convolutional neural network (CNN)
models to achieve a good trade-off between accuracy and model size so that the
pruned models become feasible for power-constrained devices such as mobile
phones. This process can be automated to avoid the expensive hand-crafted
efforts and to explore a large pruning space automatically so that the
high-performance pruning policy can be achieved efficiently. Nowadays,
reinforcement learning (RL) and Bayesian optimization (BO)-based auto pruners
are widely used due to their solid theoretical foundation, universality, and
high compressing quality. However, the RL agent suffers from long training
times and high variance of results, while the BO agent is time-consuming for
high-dimensional design spaces. In this work, we propose an enhanced BO agent
to obtain significant acceleration for auto pruning in high-dimensional design
spaces. To achieve this, a novel clustering algorithm is proposed to reduce the
dimension of the design space to speedup the searching process. Then, a
roll-back algorithm is proposed to recover the high-dimensional design space so
that higher pruning accuracy can be obtained. We validate our proposed method
on ResNet, MobileNet, and VGG models, and our experiments show that the
proposed method significantly improves the accuracy of BO when pruning very
deep CNN models. Moreover, our method achieves lower variance and shorter time
than the RL-based counterpart.
