Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep
  Learning
  Gray-box hyperparameter optimization techniques have recently emerged as a
promising direction for tuning Deep Learning methods. In this work, we
introduce DyHPO, a method that learns to dynamically decide which configuration
to try next, and for what budget. Our technique is a modification to the
classical Bayesian optimization for a gray-box setup. Concretely, we propose a
new surrogate for Gaussian Processes that embeds the learning curve dynamics
and a new acquisition function that incorporates multi-budget information. We
demonstrate the significant superiority of DyHPO against state-of-the-art
hyperparameter optimization baselines through large-scale experiments
comprising 50 datasets (Tabular, Image, NLP) and diverse neural networks (MLP,
CNN/NAS, RNN).
