Towards Assessing the Impact of Bayesian Optimization's Own
  Hyperparameters
  Bayesian Optimization (BO) is a common approach for hyperparameter
optimization (HPO) in automated machine learning. Although it is well-accepted
that HPO is crucial to obtain well-performing machine learning models, tuning
BO's own hyperparameters is often neglected. In this paper, we empirically
study the impact of optimizing BO's own hyperparameters and the transferability
of the found settings using a wide range of benchmarks, including artificial
functions, HPO and HPO combined with neural architecture search. In particular,
we show (i) that tuning can improve the any-time performance of different BO
approaches, that optimized BO settings also perform well (ii) on similar
problems and (iii) partially even on problems from other problem families, and
(iv) which BO hyperparameters are most important.
