Bayesian Optimization is Superior to Random Search for Machine Learning
  Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
  This paper presents the results and insights from the black-box optimization
(BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The
challenge emphasized the importance of evaluating derivative-free optimizers
for tuning the hyperparameters of machine learning models. This was the first
black-box optimization challenge with a machine learning emphasis. It was based
on tuning (validation set) performance of standard machine learning models on
real datasets. This competition has widespread impact as black-box optimization
(e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost
every machine learning project as well as many applications outside of machine
learning. The final leaderboard was determined using the optimization
performance on held-out (hidden) objective functions, where the optimizers ran
without human intervention. Baselines were set using the default settings of
several open-source black-box optimization packages as well as random search.
