Combination of Hyperband and Bayesian Optimization for Hyperparameter
  Optimization in Deep Learning
  Deep learning has achieved impressive results on many problems. However, it
requires high degree of expertise or a lot of experience to tune well the
hyperparameters, and such manual tuning process is likely to be biased.
Moreover, it is not practical to try out as many different hyperparameter
configurations in deep learning as in other machine learning scenarios, because
evaluating each single hyperparameter configuration in deep learning would mean
training a deep neural network, which usually takes quite long time. Hyperband
algorithm achieves state-of-the-art performance on various hyperparameter
optimization problems in the field of deep learning. However, Hyperband
algorithm does not utilize history information of previous explored
hyperparameter configurations, thus the solution found is suboptimal. We
propose to combine Hyperband algorithm with Bayesian optimization (which does
not ignore history when sampling next trial configuration). Experimental
results show that our combination approach is superior to other hyperparameter
optimization approaches including Hyperband algorithm.
