Hyperparameter Optimization with Neural Network Pruning
  Since the deep learning model is highly dependent on hyperparameters,
hyperparameter optimization is essential in developing deep learning
model-based applications, even if it takes a long time. As service development
using deep learning models has gradually become competitive, many developers
highly demand rapid hyperparameter optimization algorithms. In order to keep
pace with the needs of faster hyperparameter optimization algorithms,
researchers are focusing on improving the speed of hyperparameter optimization
algorithm. However, the huge time consumption of hyperparameter optimization
due to the high computational cost of the deep learning model itself has not
been dealt with in-depth. Like using surrogate model in Bayesian optimization,
to solve this problem, it is necessary to consider proxy model for a neural
network (N_B) to be used for hyperparameter optimization. Inspired by the main
goal of neural network pruning, i.e., high computational cost reduction and
performance preservation, we presumed that the neural network (N_P) obtained
through neural network pruning would be a good proxy model of N_B. In order to
verify our idea, we performed extensive experiments by using CIFAR10, CFIAR100,
and TinyImageNet datasets and three generally-used neural networks and three
representative hyperparameter optmization methods. Through these experiments,
we verified that N_P can be a good proxy model of N_B for rapid hyperparameter
optimization. The proposed hyperparameter optimization framework can reduce the
amount of time up to 37%.
