CMA-ES for Hyperparameter Optimization of Deep Neural Networks
  Hyperparameters of deep neural networks are often optimized by grid search,
random search or Bayesian optimization. As an alternative, we propose to use
the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known
for its state-of-the-art performance in derivative-free optimization. CMA-ES
has some useful invariance properties and is friendly to parallel evaluations
of solutions. We provide a toy example comparing CMA-ES and state-of-the-art
Bayesian optimization algorithms for tuning the hyperparameters of a
convolutional neural network for the MNIST dataset on 30 GPUs in parallel.
