HyperNOMAD: Hyperparameter optimization of deep neural networks using
  mesh adaptive direct search
  The performance of deep neural networks is highly sensitive to the choice of
the hyperparameters that define the structure of the network and the learning
process. When facing a new application, tuning a deep neural network is a
tedious and time consuming process that is often described as a "dark art".
This explains the necessity of automating the calibration of these
hyperparameters. Derivative-free optimization is a field that develops methods
designed to optimize time consuming functions without relying on derivatives.
This work introduces the HyperNOMAD package, an extension of the NOMAD software
that applies the MADS algorithm [7] to simultaneously tune the hyperparameters
responsible for both the architecture and the learning process of a deep neural
network (DNN), and that allows for an important flexibility in the exploration
of the search space by taking advantage of categorical variables. This new
approach is tested on the MNIST and CIFAR-10 data sets and achieves results
comparable to the current state of the art.
