FlexiBO: Cost-Aware Multi-Objective Optimization of Deep Neural Networks
  One of the key challenges in designing machine learning systems is to
determine the right balance amongst several objectives, which also oftentimes
are incommensurable and conflicting. For example, when designing deep neural
networks (DNNs), one often has to trade-off between multiple objectives, such
as accuracy, energy consumption, and inference time. Typically, there is no
single configuration that performs equally well for all objectives.
Consequently, one is interested in identifying Pareto-optimal designs. Although
different multi-objective optimization algorithms have been developed to
identify Pareto-optimal configurations, state-of-the-art multi-objective
optimization methods do not consider the different evaluation costs attending
the objectives under consideration. This is particularly important for
optimizing DNNs: the cost arising on account of assessing the accuracy of DNNs
is orders of magnitude higher than that of measuring the energy consumption of
pre-trained DNNs. We propose FlexiBO, a flexible Bayesian optimization method,
to address this issue. We formulate a new acquisition function based on the
improvement of the Pareto hyper-volume weighted by the measurement cost of each
objective. Our acquisition function selects the next sample and objective that
provides maximum information gain per unit of cost. We evaluated FlexiBO on 7
state-of-the-art DNNs for object detection, natural language processing, and
speech recognition. Our results indicate that, when compared to other
state-of-the-art methods across the 7 architectures we tested, the Pareto front
obtained using FlexiBO has, on average, a 28.44% higher contribution to the
true Pareto front and achieves 25.64% better diversity.
