Warm Starting CMA-ES for Hyperparameter Optimization
  Hyperparameter optimization (HPO), formulated as black-box optimization
(BBO), is recognized as essential for automation and high performance of
machine learning approaches. The CMA-ES is a promising BBO approach with a high
degree of parallelism, and has been applied to HPO tasks, often under parallel
implementation, and shown superior performance to other approaches including
Bayesian optimization (BO). However, if the budget of hyperparameter
evaluations is severely limited, which is often the case for end users who do
not deserve parallel computing, the CMA-ES exhausts the budget without
improving the performance due to its long adaptation phase, resulting in being
outperformed by BO approaches. To address this issue, we propose to transfer
prior knowledge on similar HPO tasks through the initialization of the CMA-ES,
leading to significantly shortening the adaptation time. The knowledge transfer
is designed based on the novel definition of task similarity, with which the
correlation of the performance of the proposed approach is confirmed on
synthetic problems. The proposed warm starting CMA-ES, called WS-CMA-ES, is
applied to different HPO tasks where some prior knowledge is available, showing
its superior performance over the original CMA-ES as well as BO approaches with
or without using the prior knowledge.
