How much progress have we made in neural network training? A New
  Evaluation Protocol for Benchmarking Optimizers
  Many optimizers have been proposed for training deep neural networks, and
they often have multiple hyperparameters, which make it tricky to benchmark
their performance. In this work, we propose a new benchmarking protocol to
evaluate both end-to-end efficiency (training a model from scratch without
knowing the best hyperparameter) and data-addition training efficiency (the
previously selected hyperparameters are used for periodically re-training the
model with newly collected data). For end-to-end efficiency, unlike previous
work that assumes random hyperparameter tuning, which over-emphasizes the
tuning time, we propose to evaluate with a bandit hyperparameter tuning
strategy. A human study is conducted to show that our evaluation protocol
matches human tuning behavior better than the random search. For data-addition
training, we propose a new protocol for assessing the hyperparameter
sensitivity to data shift. We then apply the proposed benchmarking framework to
7 optimizers and various tasks, including computer vision, natural language
processing, reinforcement learning, and graph mining. Our results show that
there is no clear winner across all the tasks.
