Using a thousand optimization tasks to learn hyperparameter search
  strategies
  We present TaskSet, a dataset of tasks for use in training and evaluating
optimizers. TaskSet is unique in its size and diversity, containing over a
thousand tasks ranging from image classification with fully connected or
convolutional neural networks, to variational autoencoders, to non-volume
preserving flows on a variety of datasets. As an example application of such a
dataset we explore meta-learning an ordered list of hyperparameters to try
sequentially. By learning this hyperparameter list from data generated using
TaskSet we achieve large speedups in sample efficiency over random search. Next
we use the diversity of the TaskSet and our method for learning hyperparameter
lists to empirically explore the generalization of these lists to new
optimization tasks in a variety of settings including ImageNet classification
with Resnet50 and LM1B language modeling with transformers. As part of this
work we have opensourced code for all tasks, as well as ~29 million training
curves for these problems and the corresponding hyperparameters.
