Efficient Automatic CASH via Rising Bandits
  The Combined Algorithm Selection and Hyperparameter optimization (CASH) is
one of the most fundamental problems in Automatic Machine Learning (AutoML).
The existing Bayesian optimization (BO) based solutions turn the CASH problem
into a Hyperparameter Optimization (HPO) problem by combining the
hyperparameters of all machine learning (ML) algorithms, and use BO methods to
solve it. As a result, these methods suffer from the low-efficiency problem due
to the huge hyperparameter space in CASH. To alleviate this issue, we propose
the alternating optimization framework, where the HPO problem for each ML
algorithm and the algorithm selection problem are optimized alternately. In
this framework, the BO methods are used to solve the HPO problem for each ML
algorithm separately, incorporating a much smaller hyperparameter space for BO
methods. Furthermore, we introduce Rising Bandits, a CASH-oriented Multi-Armed
Bandits (MAB) variant, to model the algorithm selection in CASH. This framework
can take the advantages of both BO in solving the HPO problem with a relatively
small hyperparameter space and the MABs in accelerating the algorithm
selection. Moreover, we further develop an efficient online algorithm to solve
the Rising Bandits with provably theoretical guarantees. The extensive
experiments on 30 OpenML datasets demonstrate the superiority of the proposed
approach over the competitive baselines.
