A multi-objective perspective on jointly tuning hardware and
  hyperparameters
  In addition to the best model architecture and hyperparameters, a full AutoML
solution requires selecting appropriate hardware automatically. This can be
framed as a multi-objective optimization problem: there is not a single best
hardware configuration but a set of optimal ones achieving different trade-offs
between cost and runtime. In practice, some choices may be overly costly or
take days to train. To lift this burden, we adopt a multi-objective approach
that selects and adapts the hardware configuration automatically alongside
neural architectures and their hyperparameters. Our method builds on Hyperband
and extends it in two ways. First, we replace the stopping rule used in
Hyperband by a non-dominated sorting rule to preemptively stop unpromising
configurations. Second, we leverage hyperparameter evaluations from related
tasks via transfer learning by building a probabilistic estimate of the Pareto
front that finds promising configurations more efficiently than random search.
We show in extensive NAS and HPO experiments that both ingredients bring
significant speed-ups and cost savings, with little to no impact on accuracy.
In three benchmarks where hardware is selected in addition to hyperparameters,
we obtain runtime and cost reductions of at least 5.8x and 8.8x, respectively.
Furthermore, when applying our multi-objective method to the tuning of
hyperparameters only, we obtain a 10\% improvement in runtime while maintaining
the same accuracy on two popular NAS benchmarks.
