Training Deep Neural Networks by optimizing over nonlocal paths in
  hyperparameter space
  Hyperparameter optimization is both a practical issue and an interesting
theoretical problem in training of deep architectures. Despite many recent
advances the most commonly used methods almost universally involve training
multiple and decoupled copies of the model, in effect sampling the
hyperparameter space. We show that at a negligible additional computational
cost, results can be improved by sampling nonlocal paths instead of points in
hyperparameter space. To this end we interpret hyperparameters as controlling
the level of correlated noise in training, which can be mapped to an effective
temperature. The usually independent instances of the model are coupled and
allowed to exchange their hyperparameters throughout the training using the
well established parallel tempering technique of statistical physics. Each
simulation corresponds then to a unique path, or history, in the joint
hyperparameter/model-parameter space. We provide empirical tests of our method,
in particular for dropout and learning rate optimization. We observed faster
training and improved resistance to overfitting and showed a systematic
decrease in the absolute validation error, improving over benchmark results.
