AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient
  Hyper-parameter Tuning
  Deep neural networks have seen great success in recent years; however,
training a deep model is often challenging as its performance heavily depends
on the hyper-parameters used. In addition, finding the optimal hyper-parameter
configuration, even with state-of-the-art (SOTA) hyper-parameter optimization
(HPO) algorithms, can be time-consuming, requiring multiple training runs over
the entire dataset for different possible sets of hyper-parameters. Our central
insight is that using an informative subset of the dataset for model training
runs involved in hyper-parameter optimization, allows us to find the optimal
hyper-parameter configuration significantly faster. In this work, we propose
AUTOMATA, a gradient-based subset selection framework for hyper-parameter
tuning. We empirically evaluate the effectiveness of AUTOMATA in
hyper-parameter tuning through several experiments on real-world datasets in
the text, vision, and tabular domains. Our experiments show that using
gradient-based data subsets for hyper-parameter tuning achieves significantly
faster turnaround times and speedups of 3$\times$-30$\times$ while achieving
comparable performance to the hyper-parameters found using the entire dataset.
