Scalable Gradient-Based Tuning of Continuous Regularization
  Hyperparameters
  Hyperparameter selection generally relies on running multiple full training
trials, with selection based on validation set performance. We propose a
gradient-based approach for locally adjusting hyperparameters during training
of the model. Hyperparameters are adjusted so as to make the model parameter
gradients, and hence updates, more advantageous for the validation cost. We
explore the approach for tuning regularization hyperparameters and find that in
experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels
are within the optimal regions. The additional computational cost depends on
how frequently the hyperparameters are trained, but the tested scheme adds only
30% computational overhead regardless of the model size. Since the method is
significantly less computationally demanding compared to similar gradient-based
approaches to hyperparameter optimization, and consistently finds good
hyperparameter values, it can be a useful tool for training neural network
models.
