Olympus: a benchmarking framework for noisy optimization and experiment
  planning
  Research challenges encountered across science, engineering, and economics
can frequently be formulated as optimization tasks. In chemistry and materials
science, recent growth in laboratory digitization and automation has sparked
interest in optimization-guided autonomous discovery and closed-loop
experimentation. Experiment planning strategies based on off-the-shelf
optimization algorithms can be employed in fully autonomous research platforms
to achieve desired experimentation goals with the minimum number of trials.
However, the experiment planning strategy that is most suitable to a scientific
discovery task is a priori unknown while rigorous comparisons of different
strategies are highly time and resource demanding. As optimization algorithms
are typically benchmarked on low-dimensional synthetic functions, it is unclear
how their performance would translate to noisy, higher-dimensional experimental
tasks encountered in chemistry and materials science. We introduce Olympus, a
software package that provides a consistent and easy-to-use framework for
benchmarking optimization algorithms against realistic experiments emulated via
probabilistic deep-learning models. Olympus includes a collection of
experimentally derived benchmark sets from chemistry and materials science and
a suite of experiment planning strategies that can be easily accessed via a
user-friendly python interface. Furthermore, Olympus facilitates the
integration, testing, and sharing of custom algorithms and user-defined
datasets. In brief, Olympus mitigates the barriers associated with benchmarking
optimization algorithms on realistic experimental scenarios, promoting data
sharing and the creation of a standard framework for evaluating the performance
of experiment planning strategies
