Multi-level Training and Bayesian Optimization for Economical
  Hyperparameter Optimization
  Hyperparameters play a critical role in the performances of many machine
learning methods. Determining their best settings or Hyperparameter
Optimization (HPO) faces difficulties presented by the large number of
hyperparameters as well as the excessive training time. In this paper, we
develop an effective approach to reducing the total amount of required training
time for HPO. In the initialization, the nested Latin hypercube design is used
to select hyperparameter configurations for two types of training, which are,
respectively, heavy training and light training. We propose a truncated
additive Gaussian process model to calibrate approximate performance
measurements generated by light training, using accurate performance
measurements generated by heavy training. Based on the model, a sequential
model-based algorithm is developed to generate the performance profile of the
configuration space as well as find optimal ones. Our proposed approach
demonstrates competitive performance when applied to optimize synthetic
examples, support vector machines, fully connected networks and convolutional
neural networks.
