Hyperparameter Optimization with Differentiable Metafeatures
  Metafeatures, or dataset characteristics, have been shown to improve the
performance of hyperparameter optimization (HPO). Conventionally, metafeatures
are precomputed and used to measure the similarity between datasets, leading to
a better initialization of HPO models. In this paper, we propose a cross
dataset surrogate model called Differentiable Metafeature-based Surrogate
(DMFBS), that predicts the hyperparameter response, i.e. validation loss, of a
model trained on the dataset at hand. In contrast to existing models, DMFBS i)
integrates a differentiable metafeature extractor and ii) is optimized using a
novel multi-task loss, linking manifold regularization with a dataset
similarity measure learned via an auxiliary dataset identification meta-task,
effectively enforcing the response approximation for similar datasets to be
similar. We compare DMFBS against several recent models for HPO on three large
meta-datasets and show that it consistently outperforms all of them with an
average 10% improvement. Finally, we provide an extensive ablation study that
examines the different components of our approach.
