HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter
  Optimization
  We present a new software, HYPPO, that enables the automatic tuning of
hyperparameters of various deep learning (DL) models. Unlike other
hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models
and directly accounts for uncertainty in model predictions to find accurate and
reliable models that make robust predictions. Using asynchronous nested
parallelism, we are able to significantly alleviate the computational burden of
training complex architectures and quantifying the uncertainty. HYPPO is
implemented in Python and can be used with both TensorFlow and PyTorch
libraries. We demonstrate various software features on time-series prediction
and image classification problems as well as a scientific application in
computed tomography image reconstruction. Finally, we show that (1) we can
reduce by an order of magnitude the number of evaluations necessary to find the
most optimal region in the hyperparameter space and (2) we can reduce by two
orders of magnitude the throughput for such HPO process to complete.
