Automatic Configuration of Deep Neural Networks with EGO
  Designing the architecture for an artificial neural network is a cumbersome
task because of the numerous parameters to configure, including activation
functions, layer types, and hyper-parameters. With the large number of
parameters for most networks nowadays, it is intractable to find a good
configuration for a given task by hand. In this paper an Efficient Global
Optimization (EGO) algorithm is adapted to automatically optimize and configure
convolutional neural network architectures. A configurable neural network
architecture based solely on convolutional layers is proposed for the
optimization. Without using any knowledge on the target problem and not using
any data augmentation techniques, it is shown that on several image
classification tasks this approach is able to find competitive network
architectures in terms of prediction accuracy, compared to the best
hand-crafted ones in literature. In addition, a very small training budget (200
evaluations and 10 epochs in training) is spent on each optimized architectures
in contrast to the usual long training time of hand-crafted networks. Moreover,
instead of the standard sequential evaluation in EGO, several candidate
architectures are proposed and evaluated in parallel, which saves the execution
overheads significantly and leads to an efficient automation for deep neural
network design.
