AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with
  Autotuned Data-Parallel Training for Tabular Data
  Developing high-performing predictive models for large tabular data sets is a
challenging task. The state-of-the-art methods are based on expert-developed
model ensembles from different supervised learning methods. Recently, automated
machine learning (AutoML) is emerging as a promising approach to automate
predictive model development. Neural architecture search (NAS) is an AutoML
approach that generates and evaluates multiple neural network architectures
concurrently and improves the accuracy of the generated models iteratively. A
key issue in NAS, particularly for large data sets, is the large computation
time required to evaluate each generated architecture. While data-parallel
training is a promising approach that can address this issue, its use within
NAS is difficult. For different data sets, the data-parallel training settings
such as the number of parallel processes, learning rate, and batch size need to
be adapted to achieve high accuracy and reduction in training time. To that
end, we have developed AgEBO-Tabular, an approach to combine aging evolution
(AgE), a parallel NAS method that searches over neural architecture space, and
an asynchronous Bayesian optimization method for tuning the hyperparameters of
the data-parallel training simultaneously. We demonstrate the efficacy of the
proposed method to generate high-performing neural network models for large
tabular benchmark data sets. Furthermore, we demonstrate that the automatically
discovered neural network models using our method outperform the
state-of-the-art AutoML ensemble models in inference speed by two orders of
magnitude while reaching similar accuracy values.
