EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on
  Expensive Black-box Functions
  Surrogate algorithms such as Bayesian optimisation are especially designed
for black-box optimisation problems with expensive objectives, such as
hyperparameter tuning or simulation-based optimisation. In the literature,
these algorithms are usually evaluated with synthetic benchmarks which are well
established but have no expensive objective, and only on one or two real-life
applications which vary wildly between papers. There is a clear lack of
standardisation when it comes to benchmarking surrogate algorithms on
real-life, expensive, black-box objective functions. This makes it very
difficult to draw conclusions on the effect of algorithmic contributions. A new
benchmark library, EXPObench, provides first steps towards such a
standardisation. The library is used to provide an extensive comparison of six
different surrogate algorithms on four expensive optimisation problems from
different real-life applications. This has led to new insights regarding the
relative importance of exploration, the evaluation time of the objective, and
the used model. A further contribution is that we make the algorithms and
benchmark problem instances publicly available, contributing to more uniform
analysis of surrogate algorithms. Most importantly, we include the performance
of the six algorithms on all evaluated problem instances. This results in a
unique new dataset that lowers the bar for researching new methods as the
number of expensive evaluations required for comparison is significantly
reduced.
